Phi-4-multimodal-instruct

Phi-4-multimodal-instruct is a lightweight open multimodal model from Microsoft that unifies text, image, and audio inputs through a single transformer backbone with mixture-of-LoRAs modality adapters (dedicated speech-LoRA and vision-LoRA). It builds on research and data from Phi-3.5 and Phi-4.0 and supports a 128K token context length.

• Decoder-only transformer (Phi4MMForCausalLM) with 5.6B parameters.
• Speech encoder + speech-LoRA adapter and vision encoder + vision-LoRA adapter feed into the shared LLM.
• Post-trained with supervised fine-tuning, DPO, and RLHF for instruction following and safety.

• ASR and speech translation for 8 languages: English, Chinese, German, French, Italian, Japanese, Spanish, Portuguese.
• Surpasses Whisper-v3 on ASR and SeamlessM4T-v2-Large on ST; ranked #1 on the HF Open ASR Leaderboard at release (March 2025).
• First open-sourced model capable of speech summarization; also supports speech QA.

Trained on ~512 A100-80G GPUs for 28 days over ~5T text tokens, ~2.3M hours of speech, and ~1.1T image-text tokens. Data cutoff: June 2024.

On-device multimodal assistants, transcription + reasoning pipelines, voice-controlled apps, document + audio understanding, accessibility, and edge/IoT deployments.