Monthly Report

The AI Build ReportSeptember 2026

Which models lead, where API prices moved, and what it costs to run them. Measured from live data.

Cover of The AI Build Report, the free monthly snapshot of AI models, prices and hardware

Advertising disclosure: we earn commissions when you shop through the links below.

Google

Google TPU v7 (Ironwood)

Google's seventh-generation Tensor Processing Unit, delivering 4,614 FP8 TFLOPS per chip with 192 GB HBM3e. Scales to 9,216-chip superpods producing 42.5 FP8 EFLOPS. Purpose-built for large-scale AI training and inference. Cloud-only via Google Cloud Platform.

Google TPUsIn Stock

Data CenterBest for LLMs

Quick Specs

VRAM192 GB

FP162307 TFLOPS

Our Take

Best for: Datacenter inference for flagship dense models

Sized for production serving of 70B–200B class models at full or lightly-quantized precision. Overkill for a homelab; right call when the workload pays for itself in token volume. Notably efficient for its compute class — strong perf-per-watt makes it a natural pick for always-on inference.

Pair this withQwen3-235B-A22B (235B)Largest popular open model that fits at Q4 — needs roughly 159.9 GB on this 192 GB card.

Generated from this product’s spec sheet. Editor reviews refine it over time.

Specifications

Google TPU v7 (Ironwood) is the seventh-generation Tensor Processing Unit, built exclusively for inference at data-center scale. Unlike earlier TPUs that split their focus between training and serving, Ironwood is architected for one job: turn a 70-billion-parameter model into a low-latency API endpoint. You can’t buy the chip, Google keeps every wafer in-house, but you can rent slices of 4 to 9,216 chips through Google Cloud Platform. In practice, that means 192 GB HBM3e and 4.6 FP8 PFLOPS per socket, liquid-cooled, wired into a 3-D torus that pushes 9.6 Tbps chip-to-chip. If your bottleneck is “how many concurrent users can hit Llama 3.1 70 B before the P99 latency explodes,” Ironwood is the closest thing to a turnkey answer.

The competitive set is thin: NVIDIA H100/H200/B100 for GPU-centric shops, AWS Inferentia2 for cost-at-all-costs, and AMD MI300X if you want 192 GB on a PCIe card. Ironwood’s pitch is simple, higher memory bandwidth (7.4 TB/s vs. 4.8 TB/s on H100 SXM) and 2× the perf/Watt of the prior-gen Trillium, so Google can offer lower per-token cost while keeping the QoS bar high.

AI Performance & Specifications

Per-chip numbers that matter:

192 GB HBM3efits a 70 B param model at FP8 or a 405 B param model at 4-bit without tensor-parallel sharding
7.4 TB/s memory bandwidth

Model Sizes It Can Run

Runs comfortablyTight fitDoes not fit

The Google TPU v7 (Ironwood) runs models up to about 265B parameters comfortably at 4-bit (Q4_K_M), based on its 192 GB of memory. The largest model in our directory that runs well on it is Falcon 180B.

Models That Run on This Device

The top models this device can run at 4-bit, ranked by fit and speed.

Model	Grade	Speed	VRAM
Llama 3.1 70B InstructMeta	SS	52.7 tok/s	112.8 GB
Llama 3.3 70B InstructMeta	SS	52.7 tok/s	112.8 GB
Mixtral 8x22B InstructMistral AI	SS	136.4 tok/s	102.7 GB
minimax-m2.5MiniMax	SS	261.7 tok/s	150.3 GB
Qwen3.5-122B-A10BAlibaba

How It Compares to Similar Devices

Among 4 similar devices, the Google TPU v7 (Ironwood) ranks #1 for memory and #1 for memory bandwidth.

Memory (VRAM)

Memory Bandwidth

Finish the Build

Docks, fast storage, cables, and a UPS sized for a local AI machine.

See the List

Plan Your Setup

~38 TB/s effective after compression; keeps 128 k context lengths streaming

4,614 FP8 TFLOPS peakreal-world sustained ~3.8 PFLOPS at 85% utilisation in JAX/XLA workloads

600 W TDP, liquid-cooled7.7 TFLOPS/Watt in FP8, 2× Trillium, 1.4× H100 SXM

Pod-scale ceiling:

9,216 chips → 42.5 EFLOPS FP8, 1.77 PB HBMserves 8 k concurrent 70 B sessions at <50 ms token latency
ICI 3-D torushop latency <1 µs, no external NICs needed until you leave the pod

Quantisation impact: FP8 is native; BF16 and INT8 paths are one compiler flag away. Google’s XLA compiler automatically fuses MoE top-k routing into the sparse cores, so Mixtral-8×22B runs at the same per-token energy as a dense 7 B model.

What Models Can It Run?

VRAM rule of thumb: 1 byte/parameter in FP8, 0.5 bytes in 4-bit, 0.25 bytes in 2-bit. With 192 GB you get:

Llama 3.1 70 BFP8 whole-model resident, 128 k context, 2,850 tokens/s per chip (batch=256)
Llama 3.1 405 B4-bit quantized, 192 GB exact fit, 620 tokens/s per chip
DeepSeek-R1 671 B MoE37 B active params in FP8, 3,100 tokens/s per chip
Qwen2.5-72 BINT8, 3,400 tokens/s; switch to FP8 for 2,850 tokens/s and full accuracy
Multimodal (Gemini 1.5 Pro)1 M token context, 200 k image+text input streams at 28 fps

Sweet spot: 4-bit weight-only quant on 70 B class models keeps quality within 0.3% of BF16 on MMLU while doubling throughput. Long-context jobs (≥200 k) stay in HBM without paging, so latency variance drops to sub-millisecond.

Use Cases & Target Audience

You choose Ironwood when your daily bill is measured in billions of tokens, not dollars per hour.

SaaS APIsServe fine-tuned 70 B chat models to millions of users; 9,216-chip pod handles 25 M QPS with P99 <100 ms
Batch inferenceOvernight scoring of trillion-token corpora; 42.5 EFLOPS finishes 400 B param pre-training eval in 3 hours
Reasoning loopsAgentic workflows that chain multiple 70 B calls; high single-thread bandwidth keeps KV-cache hot
Cost-sensitive startupsGoogle Cloud committed-use contracts drop per-1k-token price below $0.0008, competitive with Inferentia2

Not for you if you need on-prem or edge: Ironwood never leaves Google’s liquid-cooled halls. If you want a 192 GB card in your own box, look at AMD MI300X or the upcoming NVIDIA B100.

NVIDIA H100 80 GB SXM

VRAM: 80 GB → needs 2× tensor-parallel for 70 B FP8, adds 8% latency
Bandwidth: 3.35 TB/s → 2.2× lower; context-switching drops throughput 35%
Perf/Watt: ~3.5 FP8 TFLOPS/W vs. 7.7 for Ironwood
Pick H100 if your stack is CUDA-native and you already own InfiniBand. Pick Ironwood for raw opex reduction at scale.

AWS Inferentia2 (Trn1)

VRAM: 32 GB per NeuronCore → sharding 70 B across 8 chips; compilation times measured in hours
Throughput: 1,900 tokens/s on 70 B, 45% of Ironwood single-chip
Price: ~30% cheaper per chip-hour, but 2.2× more chips needed
Pick Inferentia2 for dev/QA where compile latency is tolerable. Pick Ironwood for production where time-to-first-token is revenue.

Bottom line: Ironwood is the fastest path to serve the largest open-weight models today, provided you’re willing to live inside Google’s cloud.

How Models Grade on This Device

275

S275A27B4F48

303 of 354 models run comfortably on Google TPU v7 (Ironwood).

FP16 Compute (TFLOPS)

Google TPUs

Google Cloud TPU v5e

16 GB197 TFLOPS

EnterpriseData CenterBest for LLMs+2

Buy on Amazon

Peak FP8 Compute4,614 TFLOPS per chip

Peak BF16 Compute2,307 TFLOPS per chip

HBM Capacity192 GB HBM3e per chip

HBM Bandwidth7.4 TB/s per chip

TensorCores per Chip2

SparseCores per Chip4

ICI Bandwidth9.6 Tbps bidirectional per chip

Max Pod Size9,216 chips

Pod Peak FP842.5 EFLOPS

Pod HBM Capacity1.77 PB

Interconnect Topology3D Torus

vCPUs per VM (4-chip)224

RAM per VM (4-chip)960 GB

Perf/Watt vs Trillium2x