Granite 4.0 1B Speech

Granite 4.0 1B Speech is IBM’s latest compact speech-language model for multilingual automatic speech recognition (ASR) and bidirectional automatic speech translation (AST). Despite the “1B” in its name, this is a 2B-parameter dense model — half the size of its predecessor, granite-speech-3.3-2b, while delivering higher English transcription accuracy and faster inference. It recently claimed the #1 spot on the Hugging Face OpenASR leaderboard with an average Word Error Rate (WER) of 5.52, beating larger open models and even proprietary systems.

This model is built for practitioners who need on‑device speech processing — no cloud dependency, no data leaving the machine. It occupies a sweet spot between dedicated ASR engines (like Whisper variants) and larger speech‑language models that require datacenter GPUs. With its Apache 2.0 license and native support in transformers and vLLM, Granite 4.0 1B Speech is immediately usable in local pipelines for transcription, translation, and keyword‑driven recognition.

Granite 4.0 1B Speech uses a dense encoder‑decoder design with 2 billion total parameters. Unlike mixture‑of‑experts (MoE) models where only a subset of parameters activates per token, this dense architecture keeps the full parameter set active at all times. For a 2B model, that means predictable memory usage and consistent throughput — no token‑dependent VRAM spikes.

The model is built by modality‑aligning the granite-4.0-1b-base language model to speech using public and synthetic audio‑text corpora. A two‑pass pipeline is used: first, audio is processed by the acoustic encoder to produce a transcription (or translation), then the underlying Granite language model can be invoked separately for downstream text tasks. This separation gives developers explicit control over the workflow.

Key architectural details:

• Context length: Not specified by IBM. In practice, the acoustic encoder handles audio chunks of variable duration. For transcription, context is effectively the length of the input audio (no fixed token limit is documented).
• Speculative decoding: Supported out of the box, significantly improving tokens‑per‑second during inference without accuracy loss.
• Keyword biasing: A new capability that lets you supply a list of keywords (names, acronyms) to improve recognition of rare or domain‑specific terms. This is implemented via the encoder’s attention mechanism — no separate post‑processing required.
• Quantization: Works with standard methods (GGUF, GPTQ, AWQ). The model is compatible with llama.cpp and Ollama via conversion.

Granite 4.0 1B Speech is optimized for two core tasks:

• Multilingual ASR in six languages: English, French, German, Spanish, Portuguese, and Japanese. The Japanese support is new in this release, trained on synthetic datasets to handle the writing system and phonetic structure.
• Bidirectional speech translation (AST): Transcribe speech in one language and output text in another — for example, English audio → French text (En‑X), or French audio → English text (X‑En). Translation quality is competitive with dedicated AST systems at the same parameter scale.

Concrete use cases:

• Edge transcription devices – On a Raspberry Pi 5 with a Coral TPU or an NVIDIA Jetson Orin, the model can transcribe real‑time meeting audio without internet.
• Local speech‑to‑text for sensitive data – Medical dictation, legal depositions, or financial calls where cloud upload is prohibited. Run on a laptop with 8 GB VRAM.
• Keyword‑boosted transcription – For call‑center analytics where product names, acronyms, or customer names must be captured accurately. Supply a keyword list and WER on those terms drops significantly.
• Offline translation – Travel or field‑work scenarios where a translator app must work without connectivity. Transcribe a French interview into English text on a tablet.

This model is designed for consumer and edge hardware. Here’s what you need to know to run it on‑prem.

The model’s dense architecture means VRAM scales linearly with quantization level. For most users, Q4_K_M is the sweet spot — it preserves >98% of FP16 accuracy while cutting memory in half. The total VRAM includes the model weights, audio input buffer, and key‑value cache for the language decoder.

• Consumer GPUs: RTX 3060 12GB (Q4_K_M with headroom for audio batches), RTX 4090 24GB (can run FP16 and batch multiple audio streams).
• Mac: M1/M2/M3/M4 with at least 8 GB unified memory; M4 Max 16 GB can run Q4_K_M comfortably at 15–20 tokens/sec.
• Edge devices: Jetson Orin NX 8 GB (Q4_K_M), Raspberry Pi 5 with 8 GB + external NPU (Q3_K_S).
• CPU‑only: Any modern x86 with 8 GB RAM and AVX2 support. Expect 2–4 tokens/sec — enough for batch transcription of offline files.

With speculative decoding enabled and Q4_K_M quantization on an RTX 4090, you can expect:

• English ASR: 40–60 tokens/second (real‑time factor ~0.3×).
• Multilingual ASR (Japanese): 30–45 tokens/second.
• AST (En‑Fr): 25–40 tokens/second, depending on phrase length.

On a laptop RTX 4060 8 GB, tokens‑per‑second drop to 15–25. Still comfortably above real‑time for most applications.

The quickest way to run Granite 4.0 1B Speech locally is via Ollama. Once the model is converted to GGUF format (IBM provides a conversion script), you can pull it with:

1ollama run granite-4.0-1b-speech:q4_K_M

Alternatively, use the transformers library directly with speculative decoding flag:

1from transformers import AutoModelForCTC, AutoProcessor
2model = AutoModelForCTC.from_pretrained("ibm-granite/granite-4.0-1b-speech")
3# Enable speculative decoding
4model.generation_config.speculative_decoding = True

At the 2B parameter scale, Granite 4.0 1B Speech competes primarily with Whisper Small (1.5B parameters) and OpenAI Whisper Medium (0.8B). Here’s how they stack up:

When to choose Granite 4.0 1B Speech: You need the best possible WER in the 2B class, especially for English and the supported European languages plus Japanese. The keyword biasing is a genuine differentiator for enterprise use cases — Whisper has no equivalent. You also benefit from speculative decoding for faster inference.

When to choose Whisper Small/Medium: If your workflow requires support for languages outside Granite’s six (e.g., Arabic, Mandarin, Hindi) or you need a model that is already packaged in many open‑source tools with minimal conversion. Whisper also has more established community documentation for edge deployment.

Tradeoffs: Granite’s dense 2B architecture uses slightly more VRAM than Whisper Small (which is also dense) — about 0.5 GB extra at the same quantization. The real penalty is language coverage: if you need 96 languages, Whisper is the only viable choice in this weight class. For the languages it does support, Granite’s accuracy is clearly superior.