Distil-Whisper Large v3.5

Distil-Whisper Large v3.5 is an English-only automatic speech recognition (ASR) model developed by Hugging Face’s Distil-Whisper team. It is a knowledge-distilled version of OpenAI’s Whisper Large v3, optimized for local inference on consumer hardware. At 0.8B parameters (756M actual, rounded to 0.8B), it occupies a sweet spot between accuracy and speed: it runs approximately 1.5× faster than the official Whisper Large v3 Turbo (809M parameters) while maintaining comparable word error rates (WER).

What makes this model stand out is its training recipe. Unlike earlier Distil-Whisper variants, v3.5 was trained on over 98,000 hours of diverse public data (more than 4× the data used for v3), using a “patient” teacher strategy and SpecAugment for aggressive data augmentation. This produces a model that is more robust to noise, accents, and varied recording conditions than previous distilled checkpoints. It is released under the MIT license, making it suitable for commercial and research applications.

For practitioners who want to run ASR locally without sacrificing accuracy for speed, Distil-Whisper Large v3.5 is a drop-in replacement for Whisper Large v3 and its Turbo variant — delivering near-identical accuracy on short-form transcription while being significantly faster on long-form audio.

Distil-Whisper Large v3.5 uses a standard dense encoder-decoder transformer architecture, the same as the original Whisper. It has 756 million parameters (0.8B), all active during inference — no Mixture-of-Experts or sparse activation. This means VRAM usage is proportional to the full parameter count.

The model retains the 30-second chunk processing design of Whisper: it transcribes audio in fixed 30-second segments, making it well-suited for both short clips and long-form audio (via sliding window). The context length is effectively the 30-second window; the model does not have an extended context beyond that, unlike text-based LLMs.

Key architectural details from training:

• Patient teacher distillation: The teacher model (Whisper Large v3) generates soft labels over multiple intermediate layers, not just the final output. This improves knowledge transfer.
• SpecAugment: Frequency and time masking are applied during training to improve robustness.
• English-only: The model’s vocabulary and training data are limited to English. Multilingual distillation is not yet supported in this release.

Because the encoder is frozen during distillation, Distil-Whisper Large v3.5 can also be used as a draft model for speculative decoding with the full Whisper Large v3. This allows you to load only two extra decoder layers and share the encoder, achieving ~2× faster inference than Whisper Large v3 while producing identical outputs. This technique is especially useful for latency-sensitive applications where you need the accuracy of the full model but the speed of the distilled one.

This model is a text-only ASR system: it takes audio waveforms as input and outputs transcribed text. It is not a speech-to-text + translation model (English only).

What it excels at:

• Real-time transcription on consumer GPUs — low latency for live captioning, voice assistants, and meeting transcription.
• On-device speech recognition where Whisper Large v3 is too slow or memory-heavy.
• Batch transcription of long audio files (podcasts, lectures, interviews) with high throughput.
• Speculative decoding with Whisper Large v3 for applications that need the full model’s accuracy but want faster inference (e.g., server-side ASR).

Where it falls short:

• Non-English languages: not supported. Use Whisper Large v3 Turbo or multilingual distillations if needed.
• Extremely noisy environments: while robust, the original Whisper Large v3 still leads on the hardest benchmarks.

Concrete use cases:

• Running a local voice assistant on an RTX 4060 or M4 Mac.
• Transcribing customer support calls on a dedicated GPU server.
• Integrating into a Raspberry Pi-based kiosk with a small language model for voice commands (using the small.en variant if memory is tighter, but v3.5 can work with quantization).

This model is designed to run on consumer hardware with moderate VRAM. Here’s what you need:

VRAM requirements (approx.)

• Full‑precision (float32): ~3.0 GB VRAM (overkill; not recommended).
• FP16 / BF16 inference: ~1.5 GB VRAM — runs on any GPU with 4+ GB VRAM.
• 8‑bit quantization (INT8): ~0.9 GB VRAM.
• 4‑bit quantization (e.g., Q4_K_M): ~0.6 GB VRAM — barely fits on 4 GB cards but comfortable on 6+ GB.

Recommended quantization for most users: Q4_K_M

This preserves near‑lossless accuracy while halving memory. Benchmarks from the Distil-Whisper team show <0.5% WER degradation at Q4_K_M relative to FP16. For maximum accuracy with moderate VRAM, use FP16.

Consumer hardware that works:

• NVIDIA RTX 3060 (12GB) — can run FP16 batch sizes of 8+ sequences.
• RTX 4090 (24GB) — batch size 32+ for high-throughput transcription.
• M4 Max (64GB unified memory) — runs FP16 easily; M2/M3 also fine.
• GTX 1650 (4GB) — possible with 4‑bit quantization and small batch size.
• Apple Silicon M1 (8GB) — quantized to Q4_K_M works for single‑stream real‑time.

Expected performance (tokens per second)

• On RTX 4090 (FP16, batch=1): ~800–1000 tokens/sec (roughly 10–15× real‑time for 30‑second clips).
• On M4 Max (FP16): ~500–700 tokens/sec.
• Quantized to Q4_K_M: roughly 1.1–1.3× faster, so ~900–1300 tokens/sec on RTX 4090.

Quickest way to get started:

Use [Ollama](https://ollama.com) — it supports Distil-Whisper models via the distil-whisper tag. Run ollama run distil-whisper and feed audio files. For custom integration, use Hugging Face Transformers with from_pretrained('distil-whisper/distil-large-v3.5').

Hardware tip: If you plan to use speculative decoding with Whisper Large v3, you need enough VRAM to hold both the full model and the draft decoder (about 1.5 GB extra). An RTX 3090/4090 is ideal.

When to choose Distil-Whisper Large v3.5:

• You need the fastest inference among these three models.
• Your workload is dominated by short‑form transcription (e.g., voice commands, short clips).
• You want a drop‑in improvement over distil‑large‑v3 without changing libraries.

When to choose Whisper Large v3 Turbo:

• Long‑form accuracy is critical (e.g., transcribing hour‑long podcasts without segmentation artifacts).
• You are okay with slightly slower speed for better WER on OOD long‑form audio.

When to choose an even smaller model:

• For resource‑constrained devices (mobile, Raspberry Pi) look at distil‑small.en (166M) or distil‑medium.en (394M). They are faster but have higher WER.

Distil-Whisper Large v3.5 also supports speculative decoding — if you need the full model’s accuracy but can accept variable latency, combine it with Whisper Large v3 for nearly identical output at 2× the throughput. No other distilled Whisper variant offers this pairing natively.