Nemotron 3.5 Bolt 74B-A9B 24GB

NVIDIA's Nemotron 3.5 Bolt 74B-A9B 24GB is an open hybrid vision-language mixture-of-experts model built specifically to solve the local deployment problem for autonomous agent workloads. Released on October 15, 2026, the model features 74 billion total parameters with only 9 billion active parameters per token. Instead of training a lightweight model from scratch, NVIDIA derived Bolt from its larger Nemotron 3.5 Super VL 120B-A12B foundation model using Star Elastic compression alongside 2-bit quantization-aware distillation (QAD). The result is an agent-focused multimodal model that fits squarely within a 24GB memory ceiling.

This architecture targets practitioners who want to run Nemotron 3.5 Bolt 74B-A9B 24GB locally on workstation-class desktop hardware without offloading layers to system RAM or relying on remote API calls. By combining state-space sequence modeling with sparse routing, the model occupies a unique niche: it maintains the high contextual understanding and tool-use precision of a 70B-class system while delivering the inference throughput and hardware footprint traditionally limited to sub-14B models.

The primary objective behind the 24GB footprint is co-location. A local agent cannot monopolize the entire system; it must run alongside an operating system, browser instances, development environments, and automation runtimes. Nemotron 3.5 Bolt 74B-A9B is tuned directly for screen parsing, bash execution, and multi-step tool invocation, making it a standout local AI model 74B parameters 2026 option for practical, local agentic loops.

The core architecture combines a Mamba-2 state space model backbone with Transformer attention layers and a sparse Mixture of Experts (MoE) routing mechanism. It integrates a C-RADIO v4-H vision encoder to ingest image frames alongside text tokens.

1[Vision Input (C-RADIO v4-H)] \
2                               ==> [Hybrid Mamba-2 / Transformer Backbone] ==> [Latent MoE: 74B Total / 9B Active] ==> Output Tokens
3[Text / Tool Call Prompt]     /

• Total vs. Active Parameters: The model maintains 74 billion total weights in memory, providing a vast reservoir of stored world knowledge, specialized domain reasoning, and syntactical patterns. During inference, the router directs each token through only 9 billion parameters. This sparse activation cuts computational complexity dramatically, allowing modern GPUs to process tokens without the compute bottlenecks typical of dense 70B models.
• Mamba-Transformer Hybridization: Pure Transformers suffer from quadratic scaling during the prefill phase when reading long traces of shell outputs or dense web pages. By interleaving Mamba-2 layers with sparse self-attention layers, the model maintains linear compute scaling and constant-time state updates on long sequential contexts, while retaining standard attention heads for cross-token relational reasoning.
• Native 2-Bit Quantization-Aware Distillation (QAD): Fitting 74 billion parameters into 24GB requires radical compression. Standard post-training quantization to 2 bits typically destroys language model coherence. NVIDIA avoided this by utilizing Star Elastic pruning followed by 2.0625-bit (IQ2_XXS) quantization-aware distillation against the uncompressed 120B parent. The model weights are structurally native to this precision rather than crudely truncated post-hoc.
• Speculative Drafting with LiLiCorr: To maximize decoding speed, NVIDIA bundles a dedicated LiLiCorr speculative decoding drafter via DFlash. The drafter generates candidate tokens ahead of the primary model, allowing the system to verify multiple tokens in parallel.

Nemotron 3.5 Bolt 74B-A9B 24GB is not an open-ended creative writer. It is an operations-focused engine built for agentic execution across terminals, browsers, and software development stacks.

Equipped with native vision and structural text reasoning, the model parses raw HTML, accessibility trees, and visual screenshots to interact with web interfaces. It reads web pages, extracts structured payloads, identifies clickable UI coordinates, and plans follow-up actions without human intervention. On the BrowseComp benchmark, its parent distillation allows it to track dynamic elements and handle authentication states across long task sequences.

Nemotron 3.5 Bolt 74B-A9B 24GB excels at command-line workflows. Tested on environments like Terminal-Bench 2.1, it generates accurate bash commands, debugs POSIX exit codes, traverses directory structures, and edits configuration files on the fly. It is directly applicable for local site reliability automation, repo refactoring, and CI/CD diagnostic tasks.

The model natively supports structured JSON generation and strict schema compliance. In workflows requiring parallel tool calling (querying SQLite, polling external APIs, reading local Git logs, executing Python scripts), it determines tool selection without drifting into invalid syntax or hallucinatory parameters.

When evaluating Nemotron 3.5 Bolt 74B-A9B for coding, the model shows deep competence in systems programming, Python automation, and unit test generation. It features an integrated reasoning mode via chat-template controls (enable_thinking=True/False), allowing practitioners to toggle step-by-step reasoning tokens on or off depending on whether a task requires deep algorithmic planning or low-latency responses.

The main appeal of this model is running a 74B-class architecture within consumer hardware constraints. Below is what you need to know about Nemotron 3.5 Bolt 74B-A9B 24GB hardware requirements and real-world deployment.

Because the base release uses an aggressive native 2-bit quantization (IQ2_XXS) via quantization-aware training, the packed model weights consume roughly 19GB to 20GB of memory. This leaves 3GB to 4GB of remaining space for the KV cache and activation memory on a standard 24GB GPU.

• Native IQ2 / 2-bit (Recommended): Requires ~21GB-23GB VRAM total (including base context). This is the best quantization for Nemotron 3.5 Bolt 74B-A9B 24GB because the model was specifically distilled at this precision to preserve benchmark accuracy.
• Higher Quantizations (Q4_K_M / Q8): Decompressing or running uncompressed variants requires 48GB to 80GB of VRAM (dual RTX 3090/4090 setups or an A100/H100), eliminating the specific 24GB design advantage of this release.

How to run 74B model on consumer GPU setups comes down strictly to memory bandwidth and unified allocations:

• NVIDIA GeForce RTX 4090 / RTX 3090 (24GB): The primary consumer target. The model fits entirely within the 24GB VRAM frame buffer. However, deep contexts will quickly exhaust the remaining 2GB-3GB overhead, so keep context allocations bounded or offload parts of the KV cache.
• NVIDIA RTX Spark & DGX Spark: The specialized hardware targets referenced in the model's design, delivering optimized inference loops for everyday local agent workflows.
• Apple Silicon (M3/M4 Max with 36GB+ Unified Memory): Metal-accelerated inference platforms provide ample memory headroom for both the 24GB footprint and larger context allocations, though token throughput will depend directly on memory bandwidth (e.g., 300-400 GB/s on Max configurations).
• Enterprise GPUs (A100 80GB, H100, Blackwell): Excellent for batch deployments using vLLM, allowing deep context expansion up to its limit.

The Nemotron 3.5 Bolt 74B-A9B 24GB MoE efficiency shows its strength in generation speed. Because only 9B parameters are active during decoding:

• RTX Spark / RTX 4090: With the LiLiCorr speculative drafter active, Nemotron 3.5 Bolt 74B-A9B 24GB tokens per second reach up to 84 TPS on standard agent execution benchmarks. Without speculative decoding, raw decoding ranges between 45 and 60 TPS depending on prompt length and memory bus saturation.
• Apple Silicon (M4 Max): Typically yields 30 to 45 TPS using standard GGUF runtimes.

To run the model locally using Ollama (once the model weights or GGUF ports are pulled):

1# Pull and initialize the 24GB optimized run
2ollama run nemotron-3.5-bolt:74b-a9b-24gb

For production agent runtimes leveraging speculative decoding on NVIDIA hardware, vLLM is the recommended engine:

1python3 -m vllm.entrypoints.openai.api_server \
2   --model nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB \
3   --speculative-model lilicorr-drafter \
4   --num-speculative-tokens 7 \
5   --gpu-memory-utilization 0.95 \
6   --max-model-len 16384

Evaluating Nemotron 3.5 Bolt 74B-A9B 24GB vs competitor architectures highlights where NVIDIA's compression strategy succeeds and where standard tradeoffs apply.

1+------------------------------------+------------------+------------------+---------------------+
2| Model                              | Total Footprint  | Active Params    | Best Hardware Target|
3+------------------------------------+------------------+------------------+---------------------+
4| Nemotron 3.5 Bolt 74B-A9B (IQ2)    | ~20GB (24GB min) | 9B               | Single RTX 4090/3090|
5| Qwen 2.5 72B Instruct (Q4_K_M)     | ~43GB (48GB min) | 72B (Dense)      | 2x RTX 4090 / A100  |
6| Mixtral 8x7B Instruct (Q4_K_M)     | ~28GB (32GB min) | ~13B             | 2x RTX 3060/4060 Ti |
7| Llama 3.3 70B Instruct (Q2_K)      | ~26GB (32GB min) | 70B (Dense)      | Single 24GB (tight) |
8+------------------------------------+------------------+------------------+---------------------+

Qwen 2.5 72B is an exceptionally strong dense model for reasoning and programming. However, running Qwen 2.5 72B at usable quantizations (such as Q4_K_M) requires at least 45GB to 48GB of VRAM, mandating a dual-GPU setup. Nemotron 3.5 Bolt achieves competitive tool execution and system reasoning within a single 24GB card. While Qwen holds an edge in nuanced multilingual prose and deep mathematical synthesis, Nemotron Bolt delivers significantly faster generation speeds (84 TPS vs. 15-20 TPS on consumer hardware) due to its 9B active MoE layout.

Running a standard 70B dense model (like Llama 3.3 70B) in a 24GB envelope requires reducing it to 2-bit or 3-bit weights via post-training quantization algorithms like AWQ, GPTQ, or standard GGUF IQ2. In almost all cases, post-training 2-bit quantization causes catastrophic degradation in syntax structure, function-calling syntax, and complex reasoning benchmarks.

NVIDIA's use of Quantization-Aware Distillation during training preserves reasoning accuracy far better than applying 2-bit quantization to a pre-trained dense model. For practitioners seeking the best GPU for Nemotron 3.5 Bolt 74B-A9B 24GB, a single 24GB card running Bolt natively will consistently outperform an aggressively crushed dense 70B model in both speed and agent task completion.