Optimizing Communication for Mixture-of-Experts Training with Hybrid Expert Parallel | NVIDIA Technical Blog
… The design goals and core optimization directions of Hybrid-EP include leveraging the latest communication technologies on the NVIDIA platform, such as TMA commands for data communication on NVLink scale-up networks, and low-level IBGDA network technology for RDMA networks. …
