DiffusionGemma 26B-A4B

Google DeepMind’s DiffusionGemma 26B-A4B is the first open-weight text-diffusion model built for production inference. Rather than generating tokens one at a time in a left-to-right autoregressive chain, it denoises entire blocks of 256 tokens in parallel. This shifts the primary bottleneck from memory bandwidth to compute, delivering over 1,000 tokens per second on a single H100 and up to 4× faster generation than its Gemma 4 sibling at the same parameter count.

The model is multimodal—accepting text, image, and video input—and features a 256K token context window. It ships under Apache 2.0, making it freely usable for research, commercial products, and local deployment. DiffusionGemma is not a drop-in replacement for every LLM workload; it trades a measured quality regression for a massive improvement in generation speed. If your application needs real-time text output, code infilling, or interactive editing, this model is purpose-built for that use case.

DiffusionGemma 26B-A4B uses a Mixture-of-Experts (MoE) encoder-decoder architecture with 25.2B total parameters and only 3.8B active per forward pass. The model activates 8 out of 128 total experts (plus one shared expert) across 30 layers. This sparsity keeps memory footprint low while maintaining strong reasoning capacity—a key design choice for local execution.

The generation process works in three stages:

1. Encoder prefill – The autoregressive encoder processes the prompt context and caches it as a KV cache.
2. Canvas denoising – The decoder starts with a block of 256 random tokens and iteratively refines them over multiple steps using discrete diffusion, with bidirectional attention over the canvas. Every token can attend to every other token in the block, enabling self-correction and global coherence.
3. Canvas commit – Once denoised, the block is appended to the KV cache, and the encoder prepares for the next canvas.

This block-autoregressive approach bypasses the sequential decode bottleneck that limits traditional LLMs. Instead of moving model weights from memory to compute for each token, DiffusionGemma feeds the GPU a large parallel workload, keeping tensor cores saturated.

The model supports a native system role for structured prompts, a sliding window of 1024 tokens, and a vocabulary size of 262K tokens. The vision encoder adds roughly 550M parameters for image/video processing.

DiffusionGemma excels in scenarios where generation speed matters more than maximum benchmark accuracy. Its capabilities span:

• Chat & instruction-following – Natural dialogue and task completion with configurable thinking modes (reasoning enabled/disabled).
• Code generation & infilling – Bidirectional attention during generation is a major advantage for code completion and inline editing. The model can evaluate the entire code block simultaneously, making it ideal for IDEs and real-time coding assistants.
• Vision & multimodal reasoning – Accepts images and videos interleaved with text. Supports variable aspect ratios and resolutions. Scores 70.5 on MATH-Vision, indicating strong performance on visual math problems.
• Multilingual text – Supports multiple languages (exact list not published, but built on Gemma 4’s multilingual backbone).
• Function calling – Can integrate with external tools and APIs when prompted appropriately.
• Math & reasoning – Scores 77.6 on MMLU Pro, 73.2 on GPQA Diamond, 69.1 on AIME 2026 (no tools), and 69.1 on LiveCodeBench v6. These numbers are 5–19 points below Gemma 4 26B on the same benchmarks, so choose this model only if speed is the priority.

Best-fit use cases:

• Real-time chat applications with low latency requirements
• Code editing tools (e.g., IDE autocomplete, code review comments)
• Interactive document drafting with inline corrections
• Any pipeline where you need fast streaming output and can tolerate moderate accuracy tradeoffs

This model is designed to run on a single capable GPU, not a cluster. Here is what you need to know to run DiffusionGemma 26B-A4B locally.

• Full FP16 – Requires roughly 48 GB VRAM for the full 25.2B model (active + inactive experts). Not feasible on consumer GPUs.
• Q4_K_M quantization – Fits comfortably in 18–20 GB VRAM. This is the recommended default for most users. With 3.8B active parameters, the model is highly efficient even when quantized.
• Q4 (4-bit) – Can run on an RTX 4090 (24 GB) or RTX 5090 (32 GB) with room for KV cache at 256K context.
• Q3 (3-bit) – Possible on 16 GB GPUs (e.g., RTX 4070 Ti) but expect quality degradation.

• Best performance – Single NVIDIA H100 (80 GB). Achieves 1,000+ tokens per second.
• Consumer flagship – RTX 5090 (32 GB) or RTX 4090 (24 GB) with Q4_K_M quantization. Expect 700+ tokens per second on the 5090, ~500+ on the 4090.
• Apple Silicon – M4 Max with 36 GB+ unified memory can run Q4_K_M via MLX, but token throughput will be lower (~200–300 t/s). M4 Ultra (64+ GB) improves that.
• Minimum viable – Any GPU with 16 GB VRAM and support for FP8/FP4 (RTX 40-series, AMD RX 7900 XTX with ROCm). Expect 3–5 t/s with aggressive quantization.

The quickest way to run DiffusionGemma 26B-A4B locally is via Ollama. After installing Ollama, pull the model:

1ollama pull diffusiongemma:26b

Then run:

1ollama run diffusiongemma:26b

For more control, use the Hugging Face transformers library with the official Google repository. The model supports NVFP4 (NVIDIA’s 4-bit floating-point) on Blackwell GPUs for further acceleration.

• Tokens per second – Varies by hardware and quantization. On an RTX 4090 with Q4_K_M, expect 400–600 t/s. On an H100, 1,000+ t/s. On a mid-range GPU (16 GB), expect 100–200 t/s.
• Generation latency – First token latency is higher than autoregressive models because the entire canvas must be denoised initially. After the first block, subsequent tokens stream out at canvas rate (~256 tokens per denoising step).

Choose DiffusionGemma when speed and bidirectional generation are critical. Choose Gemma 4 if you need maximum accuracy and can tolerate slower inference.

Both are MoE, but DeepSeek V2 has a 236B total parameter count and ~21B active. DeepSeek V2 scores higher on complex math and coding benchmarks but requires multiple GPUs for local deployment (80 GB+ VRAM). DiffusionGemma is more practical for single-GPU setups and is much faster on consumer hardware.

Llama 3.1 70B requires 70 GB even for quantized versions, making it infeasible for most local hardware. DiffusionGemma’s MoE efficiency (3.8B active) offers a dramatic VRAM advantage while providing competitive speed for its parameter class.

Bottom line: DiffusionGemma 26B-A4B is the fastest open-weight model under 25B total parameters for local deployment. If your workload is latency-sensitive and you can tolerate benchmark gaps of 5–15 points vs. top dense models, this is the clear choice. For maximum accuracy on math, reasoning, or complex multilingual tasks, stick with standard autoregressive models at the same active parameter count.