Granite Speech 3.3 8B

Our Take

A workable 9B-parameter dense audio model from IBM. Treat the modality benchmarks above as the leading indicator of fit — composite scoring across modalities is still maturing.

Generated from this model’s benchmarks and ranking signals. Editor reviews refine it over time.

Granite Speech 3.3 8B is a dense model with 9B total parameters—there are no mixture‑of‑experts gimmicks. For local inference, this means VRAM consumption scales linearly with the full parameter count rather than with a smaller active subset, but the architecture is straightforward to quantize and run on consumer hardware.

The speech‑specific components are:

• Conformer acoustic encoder – uses block attention and self‑conditioning with connectionist temporal classification (CTC) loss to produce audio frame embeddings.
• Windowed query‑transformer speech adapter – downsamples acoustic embeddings temporally and maps them into the text embedding space of the base language model.
• LoRA adapters – additional low‑rank adapters tuned during training to further align the language model for speech tasks.

In speech mode, the encoder, projector, and LoRA adapters are active. In text mode, those components are bypassed and the core Granite 3.3 8B Instruct runs directly (without LoRA), preserving its original text‑based reasoning and safety.

Context length is not officially specified, but the underlying Granite 3.3 8B supports at least 8K tokens. For ASR, output length is roughly proportional to audio duration; for AST, the translated text typically adds minimal overhead. Practitioners planning long‑form transcription should test with their own audio lengths using available quantized versions.

Granite Speech 3.3 8B excels at two tasks:

1. English automatic speech recognition – The model’s primary design goal. Benchmarks (LibriSpeech, Common Voice, etc.) show it outperforms several other open and proprietary ASR systems of similar size. Revision 3.3.2 uses a deeper acoustic encoder and additional data, further reducing word‑error rate.

1. Speech translation – Input can be in English, French, German, Spanish, or Portuguese. Output translation is to English (X‑En) or from English to those same languages (En‑X). Translation quality is competitive with dedicated AST models trained on far larger datasets.

Concrete use cases:

• Privacy‑first transcription – run entirely offline for medical, legal, or enterprise meetings.
• Local voice assistants – transcribe user speech, then pass the text to a local LLM for intent parsing or action generation.
• Multilingual content pipelines – transcribe podcasts or webinars in French or German, then translate the transcript to English for further analysis.
• Low‑latency translation – real‑time subtitle generation for video conferencing or live events, provided you have enough GPU headroom.

The two‑pass design means you must explicitly call the language model for translation after the ASR call. This is not a limitation for most batch workflows but is something to account for in streaming or real‑time applications.

At full FP16 precision, a 9B parameter model requires roughly 18 GB of VRAM (9B × 2 bytes). Add overhead for the encoder and adapter weights, and you should plan for 20–24 GB of GPU memory for unaudited inference.

Quantization brings this into reach of consumer hardware:

Recommended quantization for most users: Q4_K_M. It fits comfortably on an 8 GB RTX 3070/4060 Ti or an Apple M1/M2 with unified memory above 16 GB. For optimal speed on an RTX 4090 (24 GB), try Q8_0 or Q5_K_M to maximize token generation rate while staying under VRAM limits.

Consumer GPUs that can run it:

• NVIDIA RTX 3090/4090 (24 GB) – can run Q8_0 or even FP16.
• RTX 3080 / 4070 Ti (12–16 GB) – Q4_K_M works, though batch size may need to be 1.
• Apple Mac with M2 Ultra / M3 Max (at least 32 GB unified memory) – runs Q4_K_M natively via llama.cpp.
• Lower‑end cards (8 GB) – Q4_K_M is possible but may max out VRAM, especially with long audio inputs.

Speed depends on audio length and quantization. ASR token output is typically short (a few hundred tokens per minute of speech), so throughput is less critical than in text generation. On an RTX 4090 with Q5_K_M, expect 80–120 tokens/second for the language model pass. The acoustic encoder (conformer) adds about 0.1–0.3 seconds per file for typical lengths (30 seconds of speech). Total end‑to‑end time for a 5‑minute audio file is usually under 2 seconds on fast GPUs.

The easiest way to run locally is via Ollama (if the model community ports it) or directly using llama.cpp with quantization support. Example command using llama-server:

Then send audio as base64 or via a file endpoint. See the model’s GitHub repository for inference scripts in Jupyter notebooks.

Choose Granite Speech 3.3 8B when:

• You need the best English ASR accuracy for a given parameter count.
• You want full Apache 2.0 license without restrictions on commercial use.
• You plan to run on a single consumer GPU (e.g., 12–24 GB) and can tolerate a slightly larger model size than Whisper.

Choose Whisper large‑v3 when:

• You require speech recognition for many languages (beyond the five that Granite supports).
• You need a smaller model footprint (1.5B vs 9B) and don’t need translation.

Choose SeamlessM4T‑v2 when:

• You need built‑in text‑to‑speech or many‑to‑many translation (e.g., Japanese to French).
• You’re willing to accept a more complex pipeline with larger total memory requirements.

Granite Speech’s two‑pass design means you cannot run it as a single end‑to‑end model for translated speech output in one call; you must chain the ASR and then the LLM call. This is a minor concession for the benefit of a clearly separated, debuggable pipeline.