In addition to Muon, NVIDIA also supports many other optimizers for the research community to explore, including: The ultimate form of orthogonalized optimizer MOP (Momentum Orthogonalized by Polar decomposition) An advanced SOAP variant that updates eigen basis per step with eigen decomposition plus KL correction in REKLS
Advancing Emerging Optimizers for Accelerated LLM Training with NVIDIA Megatron | NVIDIA Technical Blog…supporting sequences of 128K tokens, 256K tokens, and beyond. However, training these models with extended context lengths presents significant computational and communication challenges. As context lengths grow, the memory and communication overhead…
…Slurm is configured with SwitchType = switch/nvidia_imex to use NVIDIA IMEX for cross-node GPU communication. Topology-aware scheduling: Slurm 25.11 supports TopologyParam=BlockAsNodeRank with TopologyPlugin=topology/block , ensuring allocations…
…The Save As dialog now supports the .ncu-repz file extension for compressed reports. NVIDIA Nsight Compute CLI Mandatory concurrent kernels (e.g. NCCL communication kernels) can now be profiled across processes…
…How the SGLang community is contributing Mooncake: Initial SGLang support in AIConfigurator AIConfigurator initially supported only TensorRT LLM, reserving interfaces for SGLang and vLLM without full implementation. Contributors from Mooncake (an open…
…Individual agents can be powerful on their own, but coordinated groups of agents can accomplish far more, much like human societies scale their capability through collective intelligence and coordination. Supporting these emerging…
…expanding their context windows, with recent models supporting sequences of 128K tokens, 256K tokens, and beyond.... 9 MIN READ Feb 02, 2026 Optimizing Communication for Mixture-of-Experts Training with Hybrid Expert…
…This collaboration will boost GPU efficiency, eliminate context-switching for smoother gameplay, and provide a unified workflow for the broader development community. Building on last summer’s preview of Cooperative Vector support…
Networking / Communications Design, Simulate, and Scale AI Factory Infrastructure with NVIDIA DSX Air Mar 16, 2026 By Ranga Maddipudi , Avi Alkobi and Taylor Allison Discuss (0) Discuss (0) L T F R…
…Even though these sequences fit on a single GPU, they’re partitioned due to a longer sequence in the same batch, resulting in unnecessary CP communication overhead. Usually, computation hides CP communication…
…unified communications and collaboration (UCC) and Unified Communication X (UCX), RDMA verbs, GPUDirect® Network acceleration SDK: NVIDIA Accelerated Switching and Packet Processing (ASAP2)™ software-defined networking (SDN), emulated VirtIO, Firefly time synchronization…