Best GPUs to Run DiffusionGemma 26B-A4B Locally

The AMD Radeon RX 7800 XT with 16 GB of VRAM is the smallest device in our directory that runs DiffusionGemma 26B-A4B at 4-bit, at roughly 48 tok/s. Any card with at least 10 GB will fit it.

We size the model at 4-bit (Q4_K_M), add the KV cache for its context length and a fixed runtime overhead, then compare that to each device VRAM. A device is graded on how comfortably the model fits and how fast it should run, based on the card memory bandwidth.

It is an estimate of decode speed in tokens per second at 4-bit, derived from the device memory bandwidth and the model size. Real speed varies with the runtime, batch size, and context length, so treat it as a ballpark for comparing devices, not a guarantee.

4-bit (Q4_K_M) is what most people actually run locally. It roughly quarters the memory a model needs versus 16-bit, with a small quality trade-off, so far more hardware can run the model. Each model page has a full quantization breakdown if you want other levels.