In addition to Muon, NVIDIA also supports many other optimizers for the research community to explore, including: The ultimate form of orthogonalized optimizer MOP (Momentum Orthogonalized by Polar decomposition) An advanced SOAP variant that updates eigen basis per step with eigen decomposition plus KL correction in REKLS
Advancing Emerging Optimizers for Accelerated LLM Training with NVIDIA Megatron | NVIDIA Technical Blog…expanding their context windows, with recent models supporting sequences of 128K tokens, 256K tokens, and beyond.... 9 MIN READ Feb 02, 2026 Optimizing Communication for Mixture-of-Experts Training with Hybrid Expert…
…Students only need a web browser and internet connection to get started. * Subject to limits and approval by program administration Teaching Kits in Action Resources Community Forum and Support Please read our…
…For questions and community support, visit the NVIDIA Developer Forum . Join NVIDIA founder and CEO Jensen Huang for the NVIDIA GTC Taipei 2026 Keynote and learn more with related sessions . Discuss (0…
…Multi-Teacher On-Policy Distillation enables continuous improvement and domain specialization by training Nemotron 3 Ultra with dense feedback from over ten domain-specific teacher models, supported by a massive and transparent…
…support for local models, coming soon. NVIDIA has helped H Company optimize their new models and harness to deliver over 2x performance on NVIDIA GPUs. How are NVIDIA and the OSS community…
…Omniverse libraries support agentic orchestration via Model Context Protocol (MCP) servers, facilitating LLM-based agent workflows, and are being piloted by industry leaders including ABB Robotics, PTC, Siemens, and Synopsys to enable…
…The project supports community contributions, enables organization-specific extensions, and updates recipes as new validated configurations are available, with ongoing development for broader platform and workload support. AI-generated content may summarize…
…Tensor parallelism is extended to support sharded quantized weights, with scaling factors handled per shard to preserve numerical correctness. Megatron fused kernels and communication patterns are retained, ensuring that ternary quantization delivers…
…MLOPart devices have the same compute capability, PCI IDs, and support for peer-to-peer communication as their underlying Blackwell GPU, but offer reduced streaming multiprocessors, partitioned memory (with shared virtual address…
…Accelerate Your Startup NVIDIA Inception—an acceleration platform for AI, data science, and HPC startups—supports over 7,000 startups worldwide with go-to-market support, expertise, and technology. Startups get access…