Nemotron 3.5 Lightning

Nvidia released Nemotron 3.5 Lightning to solve a clear operational bottleneck in agentic workflows: frontier reasoning models are too slow and resource-heavy to handle repetitive execution loops. In multi-turn systems, agents spend the vast majority of their runtime validating tool outputs, parsing structured data, and delegating sub-tasks. Nemotron 3.5 Lightning occupies the execution layer in this pipeline, operating as an open-weights Mixture-of-Experts (MoE) model built to handle high-frequency agent actions without burning unnecessary compute.

With 30B total parameters and only 3B active parameters per token, this local AI model 30B parameters 2026 release brings datacenter-grade agent throughput to single-GPU workstations and local development setups. Rather than relying on cloud endpoints that introduce latency spikes and network overhead, developers can run Nemotron 3.5 Lightning locally to maintain deterministic, high-speed control loops. The model weights are openly available to download, modify, and run under Nvidia's OpenMDW license, targeting developers who want complete ownership of their local agent infrastructure.

Unlike generalist chat models that treat tool-calling as an afterthought, Nemotron 3.5 Lightning was trained specifically for structured execution. It combines rapid token generation with solid reasoning benchmarks, outperforming comparable dense alternatives on agentic benchmark suites while requiring a fraction of the compute per token generated.

Nemotron 3.5 Lightning relies on a hybrid architecture that departs from standard dense Transformer stacks. Nvidia engineered the model by interleaving Mamba-2 state space layers, sparse Mixture-of-Experts feedforward layers, and selective multi-head attention. This hybrid design allows the model to capture long-range dependencies efficiently while keeping the memory footprint and FLOP overhead during token generation remarkably low.

The core of the model's throughput comes from its MoE routing. While all 30B parameters remain resident in memory, the gating network routes each token through only 3B active parameters. For local practitioners, this Nemotron 3.5 Lightning MoE efficiency creates a distinct hardware dynamic:

• Compute footprint: During generation, the GPU only executes matrix multiplications for 3B parameters. This makes the compute load comparable to running a small 3B model, yielding exceptionally fast decoding.
• Memory footprint: Because the entire 30B-parameter weight matrix must reside in VRAM to avoid PCIe offloading penalties, the memory capacity requirement matches that of a 30B model.

Nvidia also baked speculative decoding and multi-token prediction directly into the model during pretraining, offering companion draft models (DSpark and DFlash). Context length extends up to 1M tokens, enabling deep agent history tracking, multi-document retrieval evaluation, and continuous execution loops without aggressive context trimming. The model ships in native BF16 format alongside an official NVFP4 checkpoint that utilizes specialized Tensor Core kernels optimized across Ampere, Hopper, and Blackwell GPU architectures.

Nemotron 3.5 Lightning is tuned for the mechanical, deterministic tasks that dominate agent runtimes. While it handles open-ended conversational chat, its strengths lie in code generation, multi-step reasoning, and strict schema compliance.

• Function Calling and Tool Execution: The model parses complex function schemas, constructs structured JSON calls, and validates returned payloads with high reliability. It serves as an ideal worker engine inside orchestration frameworks where an orchestrator assigns sub-tasks to local subagents.
• Code Generation and Debugging: Nemotron 3.5 Lightning for coding excels at generating boilerplate, writing glue code, inspecting error traces, and refactoring scripts. It is well-suited for automated test runners and code interpretation loops.
• Multi-Step Tool Reasoning: On the GPQA Diamond benchmark (text-only, zero-shot without external tools), the model hits roughly 75.4% in BF16 and 75.5% in NVFP4. On MMLU Pro, it scores over 81.6%, showing strong foundational reasoning when deciding which tool to call next.
• Result Validation and Delegation: In continuous execution pipelines, the model checks outputs from external APIs, verifies regex constraints, and delegates sub-tasks to downstream workers without lagging the main agent thread.

In addition to English and programming languages, the model supports German, French, Spanish, Italian, and Japanese for localized agent workflows.

Running this model effectively requires balancing VRAM capacity against memory bandwidth. Because the compute requirement is only 3B parameters, memory bandwidth is the primary bottleneck for generation speed.

The full 30B parameter set must fit into GPU memory for smooth inference:

• BF16 / FP16: Requires approximately 60 GB to 64 GB of VRAM. You will need dual RTX 3090/4090 GPUs (48 GB pooled via pipeline parallelism with partial offloading) or a workstation card like an RTX 6000 Ada (48 GB) or A100/H100 (80 GB).
• 8-bit Quantization (FP8 / INT8): Requires around 32 GB to 34 GB of VRAM. Achievable on dual consumer cards or a single professional GPU.
• 4-bit Quantization (NVFP4 / Q4_K_M): Requires approximately 18 GB to 20 GB of VRAM. This is the sweet spot that fits entirely onto a single 24 GB consumer card.

For single-card local execution, the best GPU for Nemotron 3.5 Lightning is an Nvidia GeForce RTX 4090 24 GB or an RTX 3090 24 GB using 4-bit weights. The best quantization for Nemotron 3.5 Lightning for most developers is Q4_K_M (or native NVFP4 on Blackwell and supported architectures), which preserves functional accuracy while fitting comfortably within 24 GB of VRAM with room left over for a modest context buffer.

Apple Silicon users with unified memory can run the model via llama.cpp or MLX. A Mac equipped with an M3 Max, M4 Max, or M2 Ultra and at least 36 GB to 48 GB of unified memory can run 4-bit and 8-bit quantizations locally without swap overhead.

Because active compute is limited to 3B parameters, Nemotron 3.5 Lightning tokens per second figures significantly outpace standard 32B dense models:

• Single RTX 4090 (4-bit quantization): Expect generation speeds between 65 and 95 tokens per second under standard sampling settings (Temperature 1.0, Top_P 0.95). When speculative decoding is enabled with compatible inference engines, generation bursts can exceed 110 tokens per second.
• Apple Silicon (M3/M4 Max, 4-bit): Expect consistent generation speeds around 35 to 50 tokens per second.

The quickest way to get started is via Ollama or vLLM:

1# Pull and run the model via Ollama once published to the library
2ollama run nemotron-3.5-lightning
3
4# Alternatively, serve the checkpoint using vLLM for high-throughput API endpoints
5vllm serve nvidia/nemotron-3.5-lightning-30b-a3b --tensor-parallel-size 1 --max-model-len 32768

For teams looking to customize the model, Nvidia supports parameter-efficient fine-tuning via LoRA, full supervised fine-tuning using NeMo Automodel or the NeMo Megatron Bridge, and reinforcement learning alignment through NeMo RL.

Evaluating Nemotron 3.5 Lightning vs Qwen3.6 35B and Qwen 2.5 32B highlights the difference between general-purpose language models and execution-oriented architectures.

• Nemotron 3.5 Lightning vs Qwen3.6 35B: On benchmarked agent environments, Nvidia reports that Nemotron 3.5 Lightning completes 10,000 tasks on PinchBench roughly 30 percent faster than Qwen3.6 35B while maintaining equivalent accuracy. Qwen models offer broader multilingual support and strong creative writing, but Nemotron's lower active parameter count gives it a decisive advantage in token latency and tool-calling execution loops.
• Nemotron 3.5 Lightning vs Qwen 2.5 32B: Qwen 2.5 32B is a dense model where all 32B parameters activate on every token. While Qwen 2.5 32B provides exceptional general knowledge and coding depth, its generation speed on a single RTX 4090 hovers around 25 to 35 tokens per second at 4-bit. Nemotron 3.5 Lightning produces more than double the throughput on identical hardware due to its 3B active parameter routing.

If your local workload requires a standalone model for creative prose, open-ended document drafting, or broad world knowledge, a dense 32B model remains a strong option. However, if you are building an autonomous agent stack where your model spends its time invoking terminal commands, evaluating code executions, and interacting with sub-agents, Nemotron 3.5 Lightning delivers a superior throughput-to-accuracy ratio on consumer hardware.