Nemotron 3 Ultra: Advancing Agentic Reasoning via Open MoE Hybrid Mamba-Transformer Architecture

NVIDIA introduces Nemotron 3 Ultra, a novel large language model utilizing a Mixture-of-Experts (MoE) hybrid architecture that combines Mamba and Transformer layers to optimize agentic reasoning and computational efficiency.

Architectural Innovation: The Mamba-Transformer Hybrid

Nemotron 3 Ultra represents a significant shift in model architecture by integrating the linear-time scaling properties of Mamba with the robust attention mechanisms of the Transformer. This hybrid approach aims to overcome the quadratic complexity associated with standard Transformers, allowing for more efficient processing of long sequences while maintaining the high-fidelity contextual understanding required for complex tasks.

Mixture-of-Experts (MoE) for Enhanced Reasoning

To support "agentic reasoning"—the ability of a model to plan, execute, and refine multi-step tasks autonomously—Nemotron 3 Ultra employs a Mixture-of-Experts (MoE) framework. By activating only a subset of parameters for each token, the model achieves a massive increase in total parameter capacity without a proportional increase in inference latency, enabling more specialized knowledge retrieval and sharper logical deduction.

Targeting Agentic Workflows

The technical report emphasizes the model's optimization for agentic workflows. Unlike standard chat-based LLMs, Nemotron 3 Ultra is engineered to function as a core engine for AI agents, focusing on improved tool-use capabilities, long-term memory management, and the ability to maintain coherence across extended reasoning chains.

Note: As the provided source is a technical report PDF without a detailed summary of benchmark results, specific performance metrics and training dataset compositions are not detailed in this overview.