… I also detailed how Meta Platforms had created what amounts to a memory hypervisor, called “Neo,” that stages parameters and embeddings not only across the CPU and GPU memory hierarchies within a node, but also across a cluster of machines. …
… Eventually, China will come up with a new memory scheme that it can manufacture – stacked LPDDR memory or Z-Angle memory or a large number of banks of HBM3E memory attached to relatively modest AI compute engines and just scale it all out. …
… This I/O may link out over DPUs to Ethernet switches, with both running the ESUN memory coherency protocol that Meta Platforms is pushing. …
… We have gone vertical with DRAM memory with HBM stacks, which is relatively simple given the low power draw of memory chips compared to the ASICs that shuttle data around and compute upon it. …
… Here is how the Hopper NVL8, Grace-Blackwell NVL72, and Vera-Rubin NVL72 systems stack up in terms of throughput tokens per second per megawatt and interactivity tokens per second per user : It is immediately obvious that the larger shared GPU memory domain enabled by NVSwitch helps stretch the cur… …
… With the “Vera” VC100 CPUs and “Rubin” R200 GPUs in the Vera-Rubin platforms coming later this year , Nvidia will introduce a new G3.5 tier that is called inference context memory storage, which is basically use BlueField-4 DPUs used as a storage controller and put inside of the node/rack to be eve… …
… Equally importantly, SambaNova has kept the HBM memory bandwidth roughly the same at 1.84 TB/sec and the capacity of the HBM memory the same at 64 GB even as it doubles up the cores and more than doubles the performance – and even backstepped to HBM2E memory in the SN50 from HBM3 memory used in the… …
… There’s a bulked-up Agent Runtime for supporting agents that can run for days at a time and keep their context with persistent memory via Memory Bank. …
… But that is part of the exotic memory story at Micron, too. Neither MCDRAM nor 3D XPoint worked out, but Micron learned a thing or two from the experience with Intel, which sought to keep both technologies tied closely to its own Xeon server CPU platforms. …
… The DRAM market is expanding on so many fronts, and Micron plays in all of them including datacenter-class LPDDR5 memory for AI servers like Nvidia’s “Grace” and “Vera” Arm server CPUs, DDR5 memory for high-end servers generally X86 machines but also including IBM Power and z CPUs and various Arm d… …