NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB

NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is an open-weight, multimodal Mixture-of-Experts (MoE) model engineered specifically for edge execution and local agentic autonomy. Released under the OpenMDW-1.1 license, the architecture compresses the capabilities of the larger Nemotron 3.5 Super VL 120B-A12B down to a 74-billion parameter network where only 9 billion parameters activate per token. The primary engineering goal behind this build is singular: squeezing high-tier frontier intelligence into exactly 24GB of physical memory, allowing single-GPU workstations to host production-grade agents without offloading or cloud dependencies.

Rather than training a small model from scratch, NVIDIA used Star Elastic compression followed by post-training and Quantization-Aware Distillation (QAD). By quantizing the MoE feed-forward experts down to roughly 2-bit precision (specifically IQ2_XXS at 2.0625 bits per weight), the entire 74B parameter graph fits directly within standard consumer and workstation VRAM budgets. It targets local developers running system-level agents, terminal automations, and visual analysis pipelines alongside other active workstation software.

As a local AI model 74B parameters 2026 release, Nemotron 3.5 Bolt sits in a unique performance tier. It challenges typical 30B to 70B dense checkpoints by relying on sparse computation for speed, paired with a hybrid state-space architecture. It competes directly against models like Qwen2.5-VL-72B and deep MoE architectures like Mixtral 8x22B, offering a dramatically smaller deployment footprint without sacrificing the reasoning required for automated web and terminal workloads.

The core architecture of Nemotron 3.5 Bolt is a hybrid Mamba-2 and Transformer LatentMoE network paired with a C-RADIO v4-H vision encoder. This hybrid formulation solves the traditional memory and compute bottlenecks that plague dense 70B-class Transformers during long-context agent loops.

Standard Transformers suffer from quadratic computational scaling and massive KV cache memory footprints as context windows expand. By alternating Transformer attention layers with Mamba-2 state-space layers, the model maintains linear compute scaling and minimal state memory overhead across long input sequences. The MoE routing mechanism isolates computation so that each forward pass routes through just 9B active parameters out of 74B total parameters.

This routing approach maximizes NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB MoE efficiency:

• Routing Sparsity: The active compute cost mirrors that of a sub-10B parameter model, minimizing execution latency on modern tensor cores.
• Latent MoE Representation: Routing occurs within a compressed latent space, reducing cross-expert communication bottlenecks and routing instability.
• Context Capacity: The architecture natively supports context lengths up to 256K tokens, making multi-document ingestion and extended terminal history feasible without immediate out-of-memory errors.

The model ships natively packed for 24GB deployment. Typical post-training quantization at 2 bits introduces severe perplexity loss and degrades reasoning logic. NVIDIA mitigated this by pairing structural pruning (Star Elastic) with Quantization-Aware Distillation (QAD) directly against the 120B parent model. The active attention layers and vision backbone retain precision, while the expert matrices operate at an ultra-compact ~2.0625 bits per element.

To mitigate memory bandwidth bottlenecks common in local inference, the release integrates the LiLiCorr speculative decoding drafter via DFlash. LiLiCorr generates 7 speculative draft tokens per pass, validating them through parallel matrix operations in a single network step. This architectural feature allows the model to sustain high generation throughput despite running on consumer-tier memory buses.

Nemotron 3.5 Bolt handles text, multi-image, and video inputs, outputting structured text, function payloads, and reasoning traces. Dynamic thinking controls (enable_thinking=True/False) allow developers to toggle between fast, direct inference for deterministic tools and deep step-by-step reasoning traces for complex problem sets.

The model is specifically tuned for agentic benchmarks, demonstrating strong results on Terminal-Bench 2.1 and BrowseComp. It parses messy terminal outputs, interprets stdout/stderr errors, and self-corrects bash commands. When connected to headless browsers, its C-RADIO vision encoder analyzes DOM trees and screenshots simultaneously, allowing the agent to locate UI elements, parse web tables, and execute multi-step form submissions without external vision APIs.

Using NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB for coding extends beyond standard code completion. The model natively outputs clean JSON schemas and validates function-calling arguments against structured APIs. It can inspect legacy repositories, read stack traces, write bug patches, and emit unit tests within local IDE extensions.

The multimodal engine processes dense technical diagrams, architectural blueprints, UI mockups, and mixed-format PDF scans. For local enterprise document pipelines, it extracts tabular data and converts complex diagrams into machine-readable Markdown or structured JSON while maintaining spatial reasoning across high-resolution image inputs.

Running a 74B model on a single GPU was historically impossible without heavy offloading to system RAM, which dropped speeds to unusable levels. Because this model is natively compressed to a 24GB memory footprint, sizing your hardware correctly is critical to prevent out-of-memory (OOM) allocation errors when initializing the context cache.

Understanding NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB VRAM requirements depends on target context length:

• Base Weights Footprint: The 2-bit packed model occupies roughly 19.5GB to 20.5GB of VRAM.
• Runtime and Context Overhead: At 8K to 16K context, VRAM usage sits around 22.5GB to 23.5GB.
• Maximum Context (256K tokens): Running the full 256K window requires either FP8/INT4 KV cache quantization or scaling across multiple GPUs (e.g., dual 24GB cards or a single 48GB workstation GPU like the RTX 6000 Ada / A6000).

The best GPU for NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB in standard desktop setups is the NVIDIA GeForce RTX 4090 (24GB) or the enterprise RTX Spark module. The Ada Lovelace architecture provides the memory bandwidth and fourth-generation tensor cores necessary to hit high generation speeds.

For developers asking how to run 74B model on RTX 4090 hardware: the native 2-bit weights load directly into device memory. When using vLLM or specialized backends with the LiLiCorr speculative drafter, NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB tokens per second figures regularly hit 80 to 84 TPS on dedicated Spark configurations, and 45 to 65 TPS on standard desktop RTX 4090 GPUs.

To run NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB locally, you can deploy via vLLM or Ollama/llama.cpp engines.

While community users often reach for Q4_K_M as the best quantization for standard open weights, this specific release is already deeply optimized around its native IQ2_XXS structure. Deploying the official 24GB packed checkpoint preserves the quantization-aware distillation work done by NVIDIA.

Deploying with vLLM (Recommended for Maximum Throughput)

Using vLLM enables full tensor parallelism and hardware acceleration for the speculative decoding drafter:

1vllm serve nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB \
2    --speculative-model nvidia/Nemotron-3.5-Bolt-LiLiCorr-Drafter \
3    --num-speculative-tokens 7 \
4    --gpu-memory-utilization 0.95 \
5    --max-model-len 32768 \
6    --trust-remote-code

Running via Ollama (Easiest Local Setup)

For fast local integration into tools like Continue.dev, Open WebUI, or custom scripts:

1. Create a Modelfile:

1FROM nemotron-3.5-bolt-74b-24gb.gguf
2PARAMETER temperature 1.0
3PARAMETER top_p 0.95
4PARAMETER stop "<|im_end|>"

1. Build and run:

1ollama create nemotron-bolt -f ./Modelfile
2ollama run nemotron-bolt

Evaluating NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB vs competitor models reveals clear trade-offs between dense precision and sparse architectural speed.

Qwen2.5-VL-72B is an industry standard for dense multimodal performance. However, deploying Qwen2.5-VL-72B locally requires significant compromises:

• Memory Footprint: Qwen2.5-VL-72B needs at least a 4-bit quant (Q4_K_M), taking roughly 44GB to 48GB of VRAM. It cannot run on a single 24GB card without massive CPU offloading.
• Inference Speed: Even on dual RTX 4090 setups, Qwen2.5-VL-72B generates at roughly 20 to 30 TPS because it must compute all 72 billion parameters every token.
• Accuracy Trade-off: Qwen retains higher raw scores on fine-grained visual OCR and general knowledge, but Nemotron 3.5 Bolt runs on a single consumer GPU while outperforming Qwen on continuous agentic loops, terminal debugging, and autonomous web browsing.

Mixtral 8x22B uses a classic Transformer MoE architecture (141B total parameters, roughly 39B active):

• Active Compute: Mixtral requires 39B active parameters per token, demanding substantial memory bandwidth and multi-GPU tensor parallelism. Nemotron uses only 9B active parameters, cutting memory traffic dramatically.
• Multimodal Support: Mixtral 8x22B is fundamentally a text-only model. Nemotron natively ingests video and image feeds via its C-RADIO v4-H encoder.
• Hardware Barrier: Running Mixtral 8x22B even at aggressive 2-bit quantizations typically overflows a single 24GB card once the KV cache scales, whereas Nemotron 3.5 Bolt was explicitly pre-calibrated to fit within that 24GB envelope.

Select Nemotron 3.5 Bolt 74B-A9B if you operate within a strict single-GPU budget (24GB VRAM) and need an autonomous agent capable of processing visual desktop feeds, executing bash commands, and sustaining rapid decoding rates without third-party API dependencies.