Voxtral Mini 3B (2507)

Mistral AI’s Voxtral Mini 3B (2507) is a compact open-weights audio-language model that grafts Whisper-derived audio understanding onto the Ministral-3B text backbone. At 3 billion dense parameters, it’s built for practitioners who need local speech transcription, audio Q&A, summarization, and voice-driven function calling—without sending audio data to the cloud. Released under Apache 2.0, it targets edge devices, consumer GPUs, and any environment where privacy, latency, or cost rules out API-based solutions.

Voxtral Mini competes in a narrow but growing niche: small models that process spoken language natively rather than relying on a separate ASR pipeline. Unlike cascading a Whisper model with a small LLM, Voxtral fuses both capabilities into one forward pass. This makes it a practical choice for real-time transcription, voice-controlled agents, and embedded systems where every millisecond and megabyte counts.

The model’s parameter count (3B) and dense architecture mean it fits on hardware that can’t run larger 7B+ models. It is not a data-center model—it’s meant to run on your workstation, laptop, or even a Raspberry Pi with the right quantization.

Voxtral Mini is built on Ministral-3B, a dense 3B-parameter language model, with a Whisper-derived audio encoder added at the input stage. This is not a mixture-of-experts (MoE) architecture; all 3B parameters are active during inference. That translates to predictable memory usage and consistent throughput—no routing overhead, no load balancing issues.

The audio encoder converts raw speech into embeddings that the language model processes alongside text tokens. The model accepts audio files up to 40 minutes long (thanks to a 32k token context window), which is sufficient for full meeting transcriptions, lecture recordings, or long-form dictation. The context window applies to combined audio and text inputs, so you can interleave questions with audio chunks in a single conversation.

For local inference, the dense architecture means you need to allocate VRAM for the full parameter set. A 3B model in 16-bit uses roughly 6 GB of VRAM (3B × 2 bytes). Quantized versions (4-bit or 8-bit) reduce that further, making the model runnable on 8 GB GPU cards and even some NPUs. Mistral recommends vLLM >= 0.10.0 for production deployments, with temperature=0.2 and top_p=0.95 for chat/understanding tasks, and temperature=0.0 for pure transcription.

The model supports multiple audio files per message and multi-turn audio conversation. System prompts are not yet supported.

Voxtral Mini achieves state-of-the-art word error rates across FLEURS, Mozilla Common Voice, and Multilingual LibriSpeech benchmarks. It automatically detects the source language among eight supported languages (English, Spanish, French, Portuguese, Hindi, German, Dutch, Italian) and transcribes without manual language hints. A dedicated transcription mode optimizes output for pure speech-to-text.

You can feed the model a long audio clip and ask questions about its content—no separate ASR step needed. The model directly reasons over the speech signal. This enables workflows like extracting key points from a recorded meeting, querying a podcast for specific facts, or summarizing a lecture. Benchmarks show it performs comparably to larger closed models like GPT-4o mini and Gemini 2.5 Flash on speech QA and summarization.

Voxtral Mini can infer intent directly from spoken commands—map them to backend functions, API calls, or workflow triggers. For example, a voice assistant running locally could hear “set a timer for 10 minutes” and invoke the set_timer(600) function without text intermediaries. This is useful for smart home automation, field service tools, and hands-free interfaces.

The Ministral-3B backbone remains intact: Voxtral Mini handles standard text tasks (classification, extraction, reasoning) with the same competence as the original text-only model. You don’t sacrifice text capabilities for audio.

Quickest path: Use Ollama with the mistralai/voxtral-mini-3b-2507 model (once officially supported), or run directly via vLLM:

1vllm serve mistralai/Voxtral-Mini-3B-2507 --trust-remote-code

Recommended quantization: Q4_K_M for most users on consumer GPUs. It preserves most benchmark accuracy while keeping VRAM under 3 GB, leaving room for audio audio data and context. On high-end GPUs like RTX 4090 or M4 Max, run FP16 or Q8_0 for maximum fidelity.

• RTX 4090 (FP16): 60–80 tokens/s
• RTX 4090 (Q4_K_M): 90–120 tokens/s
• M4 Max (FP16): 50–65 tokens/s
• RTX 4060 Ti (Q4_K_M): 35–50 tokens/s
• Raspberry Pi 5 (Q4_0 via MLX): 3–5 tokens/s

These numbers are for text generation with a cold cache. Audio encoding adds some overhead (≈0.5–1 second per minute of audio), but the model stays interactive for most applications.

• Consumer GPU: Any NVIDIA card with 8+ GB VRAM (RTX 3060, 4060 Ti, 4070) runs Q4_K_M comfortably. For FP16, use RTX 4070 Ti Super (16 GB) or older RTX 3090.
• Apple Silicon: M4 Max or M3 Pro with 18+ GB unified memory runs FP16 well. M4 Pro handles Q8_0.
• Edge: Use a Raspberry Pi 5 with Q4_0 and MLX backend for low-throughput transcription tasks. Not recommended for real-time voice Q&A.

• Latency: Voxtral Mini processes audio in one forward pass. A two-step pipeline (Whisper → LLM) adds latency from serial inference and text tokenization.
• Accuracy: Voxtral’s joint training allows the LLM to leverage waveform-level features, reducing errors in noisy environments compared to a pure ASR transcription.
• Flexibility: The cascade approach lets you swap the LLM or ASR independently. Voxtral Mini is a fixed package.
• VRAM: Cascade uses separate VRAM for both models (~1.5 GB for Whisper small + ~6 GB for a 3B LLM in FP16). Voxtral Mini uses only the single model’s VRAM.

Choose Voxtral Mini when you want a single, optimized model for voice-first applications. Choose a pipeline if you need to update the ASR or LLM independently.

• Parameters: Voxtral Mini (3B) vs. Qwen2-Audio (7B). Voxtral runs on much less hardware.
• Performance: Qwen2-Audio is stronger on complex audio reasoning (e.g., understanding music, emotional tone) but requires at least 12 GB VRAM even in 8-bit.
• Speed: Voxtral Mini is faster on consumer GPUs due to lower compute cost.
• Licensing: Both are Apache 2.0.

Choose Voxtral Mini for edge deployments and tasks centered on speech transcription, summarization, and function calling. Choose Qwen2-Audio for broader audio understanding (non-speech sounds, affect detection) when hardware allows.

• Modality: Voxtral Mini processes audio; Phi-3.5 is text-only. For voice use cases, Voxtral Mini is the clear choice.
• Text quality: Phi-3.5-mini edges ahead on several text reasoning benchmarks, but Voxtral Mini’s text performance is strong enough for most production tasks.

Voxtral Mini is the model to beat among sub-4B models for local voice AI. It’s not a general-purpose text model; it’s a purpose-built audio-language tool that happens to also be a capable text model. If your workload is purely text, consider Ministral-3B or Phi-3.5 instead.