HBM is the DRAM industry’s attempt to short-circuit the slowing pace of Moore’s Law by using 3D chip-packaging technology. Each HBM chip is made up of as many as 12 thinned-down DRAM chips called dies. Each die contains a number of vertical connections called through-silicon vias. The dies are piled atop each other and connected by arrays of microscopic solder balls aligned to the TSVs. This DRAM tower—well, at about 750 micrometers thick, it’s more of a brutalist office block than a tower—is then stacked atop what’s called the base die, which shuttles bits between the memory dies and the proc
How Will the Tech Industry Handle the DRAM Shortage?… The recently released B300 uses eight HBM chips, each of which is a stack of 12 DRAM dies. Competitors’ use of HBM has largely mirrored Nvidia’s. AMD’s MI350 GPU, for example, also uses eight 12-die chips. With so much demand, an increasing fraction of the revenue for DRAM makers comes from HBM. …
… HBM is a particular type of memory product tailor-made to serve AI processors . …
… This is paired with custom memory aggregation chips that sit physically next to memory modules and coordinate memory across the server. “It’s an endpoint for that high-speed interface and fans out to many, many commodity DRAM chips,” explains Rabii. …
… High‑bandwidth memory HBM marks a major inflection along this path. By stacking DRAM dies — scaling to 16 layers and beyond — and placing memory much closer to the processor, HBM enables rapid access to ever‑larger working datasets. …
… Instead of relying on high-bandwidth memory HBM situated next to GPUs , it leans on SRAM memory integrated within the processor itself. …
… Tensordyne’s Napier AI chip includes 144 gigabytes of HBM, but the real power comes from its unusual math. …