Nemotron 3.5 Transcribe

Nvidia released Nemotron 3.5 Transcribe on October 28, 2026, targeting engineers who need high-throughput, instruction-steered speech processing on dedicated hardware. Built as a speech-augmented language model (SALM), it bridges the gap between raw acoustic recognition and intelligent text formatting. Instead of forcing developers into fixed, hardcoded output formats, Nemotron 3.5 Transcribe takes standard audio along with natural language prompts to output verbatim transcripts, polished written copy, domain-specific terminology, or translated text.

The model is structured as a 34B-parameter hybrid Mamba-Transformer mixture-of-experts (MoE) network that activates approximately 4.3B parameters per token during inference. By coupling a dedicated Transformer audio encoder and a frozen diarization model with an efficient MoE language backbone, Nvidia delivers a system capable of handling complex acoustic environments, including overlapping speakers and long-form recordings. Released under the open OpenMDW-1.1 license, it provides practitioners with an open-weight alternative to closed enterprise speech APIs.

For local infrastructure operators, Nemotron 3.5 Transcribe changes the economics of production speech pipelines. Traditional enterprise transcription requires cascading multiple disparate models: an automatic speech recognition (ASR) engine, a standalone diarization pipeline, a punctuation restorer, and an LLM for formatting or entity correction. Nemotron 3.5 Transcribe consolidates these stages into a single forward pass, delivering state-of-the-art accuracy on public benchmarks while operating entirely within local hardware environments.

Nemotron 3.5 Transcribe relies on a Parallel Expert Encoder framework integrated into a 34B parameter MoE language model backbone (Nemotron 3.5 Lightning). The acoustic pipeline feeds audio into a Transformer speech encoder while simultaneously routing signals through a frozen Nemotron 3 Sortformer diarization model. Diarization representations are injected directly into the speech encoder embeddings, enabling the system to track who spoke what without requiring architectural bloat inside the downstream language model.

The core language model uses a hybrid Mamba-2 and Transformer MoE architecture. Out of the 34B total parameters, only 4.3B are active during decoding. This distinction defines Nemotron 3.5 Transcribe MoE efficiency:

• Compute footprint: The compute overhead per generated token matches that of a small ~4B dense model, allowing rapid output generation and low latency.
• Memory footprint: The full 34B parameter weight set must remain resident in VRAM, which dictates system memory capacity even though floating-point operations remain low.

The architecture includes a 65,536-token context window, providing enough sequence length to ingest up to one hour of continuous audio alongside complex instruction prompts and entity reference lists.

For inference acceleration, the model incorporates native Multi-Token Prediction (MTP). During local serving with engines like vLLM or SGLang, the internal MTP layer acts as an integrated draft model for speculative decoding. Nvidia also distributes an optional external DFlash2 draft model for high-concurrency setups. To extract precise alignment data without polluting the generative text decoder, the model uses a lightweight post-decode Connectionist Temporal Classification (CTC) adapter to output speaker-attributed, word-level timestamps.

Nemotron 3.5 Transcribe supports 40 languages for transcription and handles speech-to-text translation across 31 languages to and from English. By taking instruction prompts alongside raw audio, it addresses workloads that typically require multi-model orchestration:

• Speaker-Attributed Meeting Transcription: The model separates and labels up to eight distinct speakers in overlapping, noisy audio across sessions up to 60 minutes.
• Contextual and Domain-Specific ASR: Users can pass structured lists of proper nouns, technical acronyms, or product catalogs directly in the prompt context. On specialized benchmarks like ContextASR-Bench, this contextual steering reduces named-entity transcription errors up to six-fold.
• Verbatim vs. Normalized Text Formatting: Natural language system prompts dictate whether the model preserves disfluencies ("um", "uh", false starts) for legal and clinical records, or normalizes text with standardized capitalization, numerical punctuation, and clean formatting for executive summaries.
• Audio Question Answering and Code-Switching: Because the generative core is an instruction-tuned LLM, it reliably parses multilingual code-switched speech and answers contextual questions about the spoken content in real time.

Running this local AI model 34B parameters 2026 release requires balancing active compute demands with total parameter memory storage. While the model computes like a lightweight 4.3B network, your GPU configuration must hold the full 34B structure.

Memory allocation depends on precision and whether you run speculative draft models alongside the main weights:

• BF16 (Unquantized): Requires ~72GB to 80GB VRAM. This configuration requires an enterprise card such as an Nvidia A100 (80GB), H100 (80GB), or a dual-GPU workstation.
• FP8 Precision: Requires ~38GB to 42GB VRAM. This runs comfortably across two 24GB GPUs (such as dual RTX 4090s or RTX 3090s) or a single workstation GPU like the RTX 6000 Ada (48GB).
• NVFP4 / Q4 Quantization: Requires ~20GB to 24GB VRAM. This profile fits inside a single 24GB GPU frame, though context allocation for maximum one-hour recordings will push memory near capacity.

Engineers exploring how to run 34B model on consumer GPU setups can look at 24GB cards like the Nvidia RTX 4090 or RTX 3090 using 4-bit quantizations. For Apple Silicon environments, unified memory configurations make local execution straightforward: an M3 Max or M4 Max with 64GB or 128GB of unified RAM easily holds the entire parameter space in memory using GGUF builds via NeMo-Speech.cpp or llama.cpp.

The best GPU for Nemotron 3.5 Transcribe in production depends on deployment goals. For single-card desktop inference, an RTX 4090 running a 4-bit quantization provides a balanced footprint. For enterprise data processing, the RTX PRO 6000 Blackwell or H100 allows serving the native FP8 or NVFP4 checkpoints. When running NVFP4 with the DFlash2 speculative decoding draft model on Blackwell architecture, the model reaches processing speeds up to 2,238 seconds of audio transcribed per real-time second.

For most local setups, the best quantization for Nemotron 3.5 Transcribe is FP8 if multi-GPU hardware is available, or Q4_K_M (or NVFP4 on supported architectures) for single-card consumer rigs. Because the acoustic encoder and diarization module feed into an MoE language backbone, 4-bit weight quantization introduces minimal degradation to Word Error Rate (WER) while cutting physical memory consumption from over 70GB down to roughly 22GB.

The fastest path to run Nemotron 3.5 Transcribe locally is through NeMo Speech, vLLM, or SGLang. For simple desktop deployments, Ollama and standard GGUF runtimes provide straightforward command-line serving:

1# Example serving via NeMo Speech CLI
2nemo-speech-serve \
3  --model-name nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-FP8 \
4  --tensor-parallel-size 2 \
5  --port 8000

On a dual-RTX 4090 setup using FP8, expect sustained decoding rates between 45 and 75 Nemotron 3.5 Transcribe tokens per second during the text generation phase, with prefill latency largely governed by audio duration.

Evaluating Nemotron 3.5 Transcribe against existing local speech tools highlights significant trade-offs in capability and infrastructure overhead.

OpenAI's Whisper Large v3 is an encoder-decoder model with roughly 1.5B dense parameters.

• VRAM and Compute: Whisper runs in under 6GB of VRAM on entry-level GPUs. Nemotron 3.5 Transcribe requires a minimum of 20GB to 24GB even when quantized to 4 bits, demanding significantly more baseline memory.
• Task Flexibility: Whisper outputs fixed transcription based on acoustic prediction. It cannot follow complex negative constraints, insert contextual glossaries, or dynamically change style from verbatim to summarized output without external post-processing pipelines.
• Diarization: Whisper does not identify distinct speakers natively, requiring a separate framework like PyAnnote. Nemotron 3.5 Transcribe includes integrated diarization for up to eight speakers within a unified model.

Qwen2-Audio integrates audio understanding into a standard 7B dense transformer.

• Inference Speed: Although Qwen2-Audio is smaller in total storage size (7B vs 34B total weights), Nemotron 3.5 Transcribe activates only 4.3B parameters per token. During active token generation, Nemotron delivers faster per-token throughput than dense 7B audio LLMs when memory bandwidth is sufficient.
• Long-Form Stability: Qwen2-Audio focuses heavily on short audio reasoning and conversational QA. Nemotron 3.5 Transcribe is specifically optimized for sustained, long-form transcription up to one hour, backed by CTC-aligned timestamping and targeted diarization tracking.