Every new LLM architecture comes with its own inference challenges, from transformer models to hybrid vision language models (VLMs) to state space models (SSMs). Turning a reference implementation into a high-performance inference engine typically requires adding KV cache management, sharding weights across GPUs, fusing operations, and tuning the execution graph for specific hardware. AutoDeploy shifts this workflow toward a compiler-driven approach. Instead of requiring model authors to manually reimplement inference logic, AutoDeploy automatically extracts a computation graph from an off-the
Automating Inference Optimizations with NVIDIA TensorRT LLM AutoDeploy | NVIDIA Technical Blog…The team ran a large LLM pre-training job to show this strategy. Timeline A: Normal workflow Figure 5 shows the AllGather bus bandwidth for the mixed network + NVLink collectives in one…
…Learn more In today’s data-driven world, organizations increasingly rely on video to capture critical information, yet extracting meaningful, real-time insights from massive amounts of footage remains a challenge. NVIDIA…
…Learn more The next generation of AI-driven robots like humanoids and autonomous vehicles depends on high-fidelity, physics-aware training data. Without diverse and representative datasets, these systems don’t get…
…View all posts by Alex Watson View all posts by Alex Watson About John Myers John Myers is a senior director of software engineering at NVIDIA who is leading OpenShell engineering efforts…
…About the Authors About Andreas Kieslinger Andreas Kieslinger is a senior development technology engineer for Generative AI and LLMs at NVIDIA. His current focus is to accelerate AI inference in projects like…
…Omniverse libraries support agentic orchestration via Model Context Protocol (MCP) servers, facilitating LLM-based agent workflows, and are being piloted by industry leaders including ABB Robotics, PTC, Siemens, and Synopsys to enable…
…View all posts by Allen Zhao View all posts by Allen Zhao About Ivan Yin Ivan Yin (Wenzhi Yin) is a senior computer architect engineer specializing in GPU compiler engineering and high…