NVIDIA Model Optimizer (ModelOpt) is an open-source library that integrates a suite of model optimization techniques—including quantization, pruning, Neural Architecture Search (NAS), distillation, speculative decoding, and sparsity—to accelerate large language and vision-language models. It accepts Hugging Face, PyTorch, and ONNX model inputs, and provides Python APIs for composing these techniques into optimized quantized checkpoints. The library is tightly integrated with NVIDIA Megatron-Bridge, Megatron-LM, and Hugging Face Accelerate, and exports seamlessly to downstream inference frameworks such as TensorRT-LLM, vLLM, SGLang, and TensorRT. Recent benchmarks demonstrate significant performance gains: a W4A4 NVFP4 plus quantization-aware distillation pipeline for Qwen3.6-35B-A3B achieved 1.30× vLLM throughput over BF16 with 3.1× smaller checkpoints; quantizing Nemotron 3 Ultra (550B) to NVFP4 delivered up to 5.9× higher decode-heavy throughput compared to GLM-5.1 754B FP4 while matching BF16 accuracy; and a pruning-plus-two-phase-distillation-plus-FP8 pipeline for Nemotron-3-Nano-30B-A3B yielded 2.6× vLLM throughput and 2.6× memory reduction. Customer deployments include Domyn compressing Colosseum-355B to 260B and Bielik.AI producing a Minitron 7B model that is 33% smaller, 50% faster, and retains 90% quality. Installation is available via pip (`pip install -U nvidia-modelopt[all]`) or through NVIDIA container images on NGC, with quantized checkpoints publicly accessible on Hugging Face.
Read original
github-trending/python