NVIDIA-Nemotron-Labs-Dory1.0-4B-MXFP8

NVIDIA-Nemotron-Labs-Dory1.0-4B-MXFP8 is a 3.98-billion-parameter text-only dense model developed by NVIDIA that introduces elastic latent-recurrent inference to edge-scale deployments. Released under the commercially permissive OpenMDW-1.1 license, the model targets practitioners who need strong agentic coding and reasoning performance without loading a massive parameter footprint into local memory. Rather than relying on standard fixed-depth feed-forward passes, Dory dynamically trades inference compute for accuracy through a looped architecture that reuses internal layers at runtime.

The checkpoint ships pre-quantized in microscopic-scaling 8-bit floating point (MXFP8), alongside BF16, NVFP4, and GGUF variants. By decoupling the static parameter footprint from computational depth, NVIDIA allows local developers to serve a sub-4B weight footprint that scales its reasoning capabilities on demand. When evaluated at higher recurrent depth, it achieves task accuracy competitive with models more than twice its physical size, positioning it directly against modern lightweight options like Qwen3.5-9B and Phi-4 mini.

For practitioners looking to run NVIDIA-Nemotron-Labs-Dory1.0-4B-MXFP8 locally, the primary appeal is its massive 262,144-token (256K) context window paired with a small physical VRAM footprint. It serves as a drop-in reasoning engine for localized agent frameworks, large code repository parsing, and complex instruction-following workflows where host memory constraints typically rule out larger foundation models.

Dory1.0-4B uses a normalized-GPT (nGPT) dense architecture structured into three distinct stages: an 8-layer input block, a 20-layer shared recurrent block, and an 8-layer output block. The total parameter count sits at 3.98B. The model was initially pre-trained natively in NVFP4 before continuing in BF16, producing weights that retain high numerical fidelity even in compressed representations like MXFP8.

The core differentiator is its latent recurrence. During inference, the shared 20-layer recurrent block can execute 1, 2, or 4 loops before passing hidden states to the output block. The input representation is reinjected at every loop step, forcing the model to spend extra FLOPs per token refining its internal state in latent space rather than generating bloated reasoning traces. Because of this:

• Loop Count 1: Operates as a fast, low-latency 36-layer equivalent pass for basic extraction and instruction tasks.
• Loop Count 2 (Recommended default): Strikes an optimal balance between execution speed and deep semantic reasoning.
• Loop Count 4: Maximizes reasoning depth, pushing compute to 96 effective layer passes while holding identical VRAM for weights.

Handling attention over its 262,144-token context length is managed via an interleaved design combining full attention layers with sliding-window attention (SWA). Standard looped models incur severe memory penalties because each pass creates a separate Key-Value (KV) cache. Dory circumvents this by supporting last-loop KV caching, which preserves only a single cache per recurrent attention layer. Under this configuration, runtime KV cache memory requirements match those of an unlooped model regardless of whether you run 1, 2, or 4 loops.

Dory1.0-4B-MXFP8 focuses strictly on English text processing, excelling across coding, multi-step reasoning, structured function-calling, and rigorous instruction-following.

NVIDIA-Nemotron-Labs-Dory1.0-4B-MXFP8 for coding represents a significant architectural leap for local workstations. On the SWE-bench Verified benchmark, vendor evaluations report scores scaling directly with recurrence: 50.3 at 1 loop, 59.5 at 2 loops, and 61.8 at 4 loops. A score of 61.8 places this sub-4B footprint neck-and-neck with standard 9B models. Paired with its 256K window, it can ingest entire software modules, generate multi-file diffs, and self-correct via loop recurrence without exceeding consumer GPU limits.

On reasoning benchmarks, running 4 loops allows the model to process complex logic internally. Instead of generating thousands of reasoning tokens to solve math or system problems, it resolves ambiguities within the latent layers. This yields dense, precise answers faster than chain-of-thought methods on standard small parameter models.

The model adheres cleanly to rigid JSON schemas and API calling contracts. In localized agent pipelines, it functions reliably as a task-router or parameter extractor, operating within terminal environments and agentic tool harnesses without drifting from prompt constraints.

To run NVIDIA-Nemotron-Labs-Dory1.0-4B-MXFP8 locally, developers must note current software ecosystem support. As of late 2026, baseline support for Dory's recurrent layer looping in mainstream engines like vLLM (PR #60149) and llama.cpp (PR #30306) requires compiling directly from active pull requests rather than stable release builds.

The MXFP8 format uses 8-bit precision with microscopic scaling blocks, meaning the static weights consume roughly 4.2 GB to 4.5 GB of VRAM. Total memory overhead depends primarily on how much of the 262,144-token context window you allocate:

• Short Context (up to 8K tokens): ~6 GB VRAM minimum.
• Mid Context (up to 64K tokens): ~10 GB to 12 GB VRAM.
• Full Context (256K tokens with FP8 KV cache): ~18 GB to 22 GB VRAM.

The best GPU for NVIDIA-Nemotron-Labs-Dory1.0-4B-MXFP8 depends on your context target:

• NVIDIA GeForce RTX 4060 Ti (16GB) / RTX 4070 (12GB): Ideal for local workstations running code-completion tasks with context windows up to 32K tokens.
• NVIDIA GeForce RTX 4090 / RTX 3090 (24GB): The baseline standard if you intend to exploit large context windows (128K to 256K) alongside 4-loop inference execution.
• Apple Silicon (M3/M4 Max with 36GB+ Unified Memory): Excellent platform for llama.cpp PR builds, offering sufficient unified memory bandwidth to hold maximum context lengths comfortably.

While the MXFP8 release offers high efficiency on native Ada Lovelace and Blackwell architectures, practitioners using older hardware or CPU-based setups will find GGUF formats (specifically Q4_K_M or Q5_K_M) more practical. The best quantization for general local execution remains Q4_K_M, bringing base weight consumption down to approximately 2.6 GB with minimal loss in reasoning accuracy.

In terms of tokens per second, performance scales inversely with loop counts:

• 1 Loop: 85 to 110 tokens/sec on an RTX 4090.
• 2 Loops: 55 to 70 tokens/sec on an RTX 4090.
• 4 Loops: 28 to 35 tokens/sec on an RTX 4090.

For beginners wondering how to run 3.98B model on consumer GPU setups, building llama.cpp from the Dory PR and converting to GGUF provides the simplest path until unified runtime tools like Ollama package native looped execution defaults into stable drops.

Evaluating NVIDIA-Nemotron-Labs-Dory1.0-4B-MXFP8 performance requires looking at standard alternatives in the 3B to 9B parameter bracket.

• Dory1.0-4B vs Qwen3.5-9B: Qwen3.5-9B provides higher baseline terminal task scores and broader native multilingual support out of the box. However, Qwen requires at least 10 GB VRAM just to seat an 8-bit quantization before context allocation. Dory fits inside half that parameter footprint while matching Qwen's SWE-bench Verified performance (61.8 vs ~61) when configured to 4 loops.
• Dory1.0-4B vs Phi-4 mini (3.8B): Microsoft's Phi-4 mini executes static feed-forward passes quickly, offering higher raw tokens-per-second throughput out of the box. However, Dory offers dynamic inference flexibility: you can run it light for low-latency tasks or dial up to 4 loops for complex reasoning without swapping model checkpoints in VRAM. Additionally, Dory's native 256K context ceiling significantly outperforms the standard windows found on most edge-class models.

Choose Dory1.0-4B-MXFP8 if you operate on constrained local VRAM, need massive context window capability, and require high-level code engineering reasoning that can be toggled dynamically at runtime. Choose standard dense architectures if your pipeline requires high tokens-per-second streaming without custom runtime engine compilations.