NVIDIA GeForce RTX 3090

The NVIDIA GeForce RTX 3090 is a prosumer GPU that, since its launch, has become a de facto standard for running large language models (LLMs) locally. Built on the Ampere architecture (GA102 die) and fabricated on Samsung’s 8nm process, it packs 28.3 billion transistors into a 628 mm² die. Its 24 GB of GDDR6X VRAM gives it the memory capacity to load models that most consumer GPUs cannot touch, which is why it remains a top choice for AI engineers, ML researchers, and hobbyists running local inference in 2026.

Priced at an MSRP of $1,499, the RTX 3090 occupies a unique niche: it delivers datacenter-like VRAM capacity at a fraction of the cost of an A100 or H100. It’s the spiritual successor to NVIDIA’s Titan line, and while it was marketed for 8K gaming and creative workflows, its real value for the AI community lies in its 24 GB unified memory, 936 GB/s memory bandwidth, and 3rd-gen Tensor Cores. For practitioners who need to run 13B parameter models or quantized 30B–70B models on a single machine, the RTX 3090 remains a compelling, in-stock option.

This page covers the RTX 3090’s AI-specific specs, real-world model compatibility, expected tokens-per-second, and how it stacks up against alternatives like the RTX 4090 and used datacenter cards.

For AI inference, especially transformer-based models, three metrics matter most: VRAM capacity, memory bandwidth, and compute throughput.

VRAM: 24 GB GDDR6X — This is the RTX 3090’s killer feature. 24 GB lets you load 13B models in FP16 without offloading (e.g., Llama 3.1 8B Q8 fits with headroom, Mistral 7B Q4_K_M fits easily). With 4-bit quantization, you can squeeze in 30B–34B models like Qwen 2.5 32B, DeepSeek-R1 33B, or YI-34B. Two RTX 3090s connected via NVLink give you 48 GB combined, enabling models like Llama 3.1 70B in 4-bit with tensor parallelism.

Memory Bandwidth: 936 GB/s — On a 384-bit bus with 19.5 Gbps GDDR6X, the RTX 3090 delivers high bandwidth that directly impacts token generation speed. For a 13B model in 4-bit, expect 35–70 tokens/sec on a single card — enough for interactive chatbot use. Bandwidth is the primary bottleneck for autoregressive generation; the RTX 3090’s 936 GB/s outperforms the RTX 4090’s 1,008 GB/s by only a small margin, and far exceeds the 600–700 GB/s of cards like the A4000.

Compute Performance: 35.58 TFLOPS FP32, 142 TFLOPS FP16 Tensor — The 328 3rd-gen Tensor Cores support mixed-precision operations (TF32, BF16, FP16, INT8). In practice, inference uses FP16 or quantized INT8, so the 142 TFLOPS (dense) for FP16 tensor math is the relevant figure. That’s enough to keep the memory pipeline fed for most LLM inference workloads. For training, the RTX 3090 falls short of the RTX 4090 (330 TFLOPS FP16) or RTX 6000 Ada, but for fine-tuning (QLoRA, LoRA) it works well for models up to 13B.

Power and Cooling: 350W TDP — The RTX 3090 is power-hungry. A recommended 750W PSU with 2x 8-pin PCIe power connectors is a minimum. The founder’s edition is a 3-slot card, 313 x 138 mm, weighing 2.1 kg. For multi-GPU setups, ensure adequate airflow — the card can hit 93°C under sustained load. Still, for a card that can run a 13B model at 50+ tokens/sec, the power draw is justifiable vs. a $15,000 A100.

NVLink (3rd Gen, 2-way) — A rare feature at this price point. Two RTX 3090s can be linked with NVLink (not SLI) for 112.5 GB/s bidirectional bandwidth, enabling efficient model parallelism. In practice, 2x 3090 with NVLink yields 40–60% throughput improvement over 2x PCIe-only for LLM inference, particularly for models that don’t fit in a single card.

This is where the RTX 3090 earns its keep. Here’s a realistic breakdown by model size and quantization.

Models that fit comfortably in 24 GB (single card):

• 7B–8B models (Llama 3.1 8B, Mistral 7B, Qwen 2.5 7B) in FP16 or Q8: ~90–130 tokens/sec with Q4_K_M quantization. These run with plenty of VRAM to spare for context (128K+ tokens with sliding window or Ring Attention).
• 13B models (Llama 3 13B, Mistral 13B, CodeLlama 13B) in 4-bit (Q4_K_M or GPTQ): ~35–70 tokens/sec. FP16 versions also fit (21–24 GB), leaving less headroom for long contexts. Use vLLM or llama.cpp with flash attention for best throughput.
• 20B–34B models (Qwen 2.5 32B, DeepSeek-R1 33B, YI-34B) in 4-bit or AWQ: They fit with careful quantization. Expect ~15–30 tokens/sec. Some models like Mixtral 8x7B (MoE, ~47B total parameters) can fit in 24 GB with 4-bit and sparse inference, though only 12B are active per token — 35–40 tokens/sec reported.
• Multimodal models (LLaVA-NeXT 13B, Qwen-VL 7B): Fit easily with the vision encoder consuming 2–4 GB. Full 24 GB allows high-resolution image processing.

Models requiring 48 GB (two RTX 3090s with NVLink):

• 70B–72B models (Llama 3.1 70B, Qwen 2.5 72B, DeepSeek-V2 67B) in 4-bit: ~10–20 tokens/sec with tensor parallelism. Without NVLink, PCIe-only setups see ~30% lower throughput.
• Large MoE models (Mixtral 8x22B, DeepSeek-V2 236B) – only possible with heavy quantization and model sharding; not recommended for real-time inference.

Models that are challenging or impossible:

• Unquantized 70B models (140 GB+ in FP16) are out of reach even with two cards.
• 120B+ dense models like Qwen 2.5 110B or Llama 3 405B require cloud GPUs.
• Long-context fine-tuning (>32K tokens) may exceed VRAM for 13B+ models in FP16.

Real-world tokens per second benchmarks (from community and web research):

• Llama 3.1 8B Q4_K_M: ~120 tokens/sec (1x 3090)
• Mistral 7B Q4_K_M: ~130 tokens/sec
• Qwen 2.5 7B Q4_K_M: ~110 tokens/sec
• Llama 3.1 13B Q4_K_M: ~55 tokens/sec
• Qwen 2.5 32B Q4_K_M: ~20 tokens/sec
• Mixtral 8x7B Q4_K_M: ~38 tokens/sec
• Llama 3.1 70B Q4_K_M (2x 3090 NVLink): ~12 tokens/sec

These numbers depend on backend (llama.cpp, vLLM, ExLlamaV2), context length, batch size, and GPU clock/power limits. The RTX 3090’s max supported model parameters is listed as 13B for FP16, but with quantization you can push well beyond that.

The RTX 3090 serves a broad range of AI practitioners:

• Local LLM hobbyists who want a private, uncensored assistant running at home (e.g., Llama 3.1 13B, Qwen 2.5 32B). 24 GB VRAM allows them to run large context windows without cloud dependencies. The 3090 is the most cost-effective entry into “big model” territory.
• Developers building AI-powered applications that need offline inference — agentic workflows, coding assistants, RAG pipelines. A single 3090 can serve multiple concurrent requests for 7B models at 30+ tokens/sec, or handle batch processing of 13B models with vLLM.
• Teams running inference servers on a budget. Two 3090s with NVLink provide equivalent VRAM to an A6000 (48 GB) at roughly half the cost. For serving models like Llama 3.1 70B in 4-bit to a small team, this is a practical, self-hosted solution.
• Edge deployment – While the 350W TDP limits battery-powered scenarios, the RTX 3090 is used in desktop-edge environments (e.g., field workstations, automated driving labs). Its 8K HDR output also suits visualization and VR applications.
• Fine-tuning with QLoRA – For models up to 13B, you can fine-tune in FP16 with LoRA adapters. 24 GB allows a batch size of 1–2 for a 13B model. Training larger models requires multi-GPU or cloud instances.

Training vs. inference: The RTX 3090 is a strong inference card but only a decent training card. Its FP16 tensor performance (142 TFLOPS) is roughly half that of the RTX 4090 (330 TFLOPS) and far behind the RTX 6000 Ada (91 TFLOPS FP32, but higher VRAM). If your primary workload is training 7B+ models from scratch, look at the RTX 4090 or datacenter cards. For inference and fine-tuning, the 3090 remains a better value per GB of VRAM.

The RTX 3090’s closest competitors are the RTX 4090 and used RTX A6000 (48 GB). Here’s a factual breakdown:

| Feature | RTX 3090 (24 GB) | RTX 4090 (24 GB) | RTX A6000 (48 GB) |

|---------|------------------|------------------|------------------|

| VRAM | 24 GB | 24 GB | 48 GB |

| Memory Bandwidth | 936 GB/s | 1,008 GB/s | 768 GB/s |

| FP16 Tensor | 142 TFLOPS | 330 TFLOPS | 75 TFLOPS (approx) |

| TDP | 350 W | 450 W | 300 W |

| NVLink | Yes (2-way) | No | No (NVLink on A6000 is rare) |

| Used Price (2026) | $800–$1,200 | $1,600–$2,000 | $4,000–$6,000 |

RTX 4090 – Faster compute, slightly higher memory bandwidth, but same VRAM. For inference on models that fit in 24 GB (7B–13B), the RTX 4090 is 30–50% faster. However, it lacks NVLink, so you cannot combine two cards into a 48 GB memory pool. The RTX 3090 is the better choice if you need to scale beyond 24 GB or if you are on a tighter budget.

RTX A6000 – 48 GB VRAM is a clear advantage for 70B models in 4-bit on a single card. But the A6000’s memory bandwidth (768 GB/s) is lower, leading to 10–20% slower token generation than the RTX 3090. The A6000 is also significantly more expensive new, though used datacenter cards can be found. For multi-GPU setups, the RTX 3090’s NVLink makes it more flexible.

VS AMD (7900 XTX) – For AI inference, NVIDIA’s CUDA ecosystem and tensor core support make the RTX 3090 the clear choice. AMD’s ROCm software compatibility is improving, but for LLM inference, PyTorch, vLLM, and ollama all favor NVIDIA. The RTX 3090’s 24 GB VRAM also matches the 7900 XTX’s 24 GB while offering better performance on transformer models.

When to pick the RTX 3090:

• You need 24 GB VRAM at the lowest cost per gigabyte.
• You plan to run 13B–34B models locally with quantization.
• You want to pair two cards via NVLink for 48 GB on a sub-$2,500 budget.
• You are fine with 350W power draw and can manage airflow.

When to skip the RTX 3090:

• You need >48 GB VRAM for 70B FP16 models.
• Pure inference speed is your only concern and your models fit on a single RTX 4090.
• You are constrained by power or size (e.g., SFF builds).

For developers and teams evaluating hardware for local AI agents and LLM inference in 2026, the NVIDIA GeForce RTX 3090 remains one of the most practical, well-supported, and cost-effective GPUs available. Its 24 GB VRAM, NVLink support, and strong community software support make it a reliable workhorse for running a wide range of open-source models locally.