Kandinsky S2V 1.0 Pro

Kandinsky S2V 1.0 Pro is an open-weights, 21-billion-parameter speech-to-video diffusion transformer developed by Sber's Kandinsky Lab. Designed for controllable, audio-driven human animation, the model synthesizes 25 FPS video from three inputs: a single reference portrait or character image, an input speech audio track, and a descriptive text prompt. Unlike head-only talking-face models, Kandinsky S2V 1.0 Pro animates the entire subject, synchronizing lip movements, facial expressions, head pose, and body gestures to the cadence of the speech waveform.

Released under the permissive MIT license, the Pro model represents the top tier of the Kandinsky S2V family alongside its smaller 2B Lite counterpart. It operates in the latent space of a frozen 3D video autoencoder, chaining sequential 5-second video chunks through a multi-scale memory mechanism to generate coherent continuous speech clips up to 60 seconds or longer. For engineering teams evaluating local speech-to-video pipelines, it stands as one of the largest and most expressive open-source avatar animation backbones available.

Deploying the model on local infrastructure requires significant compute. The dense 21B parameter diffusion backbone, combined with its auxiliary conditioning models, demands high-end hardware with substantial VRAM. It targets practitioners running workstations with server-class NVIDIA GPUs or multi-GPU nodes who need sovereign, uncensored video generation free from external API costs or data-retention risks.

Kandinsky S2V 1.0 Pro is built as a 3D Diffusion Transformer (DiT) operating under a flow-matching objective, initialized from Kandinsky 5.0 Video text-to-video weights. The generation pipeline distributes computation across a multimodal encoder stack, a dense generative backbone, and a spatio-temporal decoder:

• Multimodal Conditioning Stack: Instead of relying on standalone speech models like Wav2Vec 2.0 or Whisper alongside separate text encoders, the model draws conditioning features from a frozen Qwen2.5-Omni-7B model. Its audio tower extracts 3584-dimensional acoustic embeddings at 25 tokens per second directly from the speech waveform, while its Thinker module outputs hidden states for text prompts. A frozen CLIP ViT-L/14 model provides a pooled global text embedding.
• Generative DiT Backbone: The core 21.4B parameter transformer contains 60 visual blocks. Each block executes video self-attention (V2V), cross-attention to text (V2T), and feed-forward operations. To inject acoustic dynamics without destabilizing visual quality, audio cross-attention (V2A) adapters are applied across 18 periodically spaced blocks. These adapters enforce strict temporal alignment, ensuring each latent frame attends only to its corresponding audio slice.
• Latent Video VAE: Video encoding and decoding are handled by a frozen HunyuanVideo 3D VAE. The reference image and noisy latent targets are packed into this shared latent space.
• FramePack Memory: Long-form synthesis relies on a reference-aware, three-scale visual memory system. As the model synthesizes sequential 5-second chunks (125 frames per chunk), past latents are compressed at progressively coarser spatial patch sizes. This preserves visual identity across multi-chunk rollouts while suppressing pose stagnation along the temporal RoPE axis.
• Inference Distillation: Alongside the standard flow-matching base model, Sber provides a π-Flow distilled variant. The distilled version generates output in 16 sampling steps without classifier-free guidance, yielding a 4.5x to 5.4x speedup over the base model.

Kandinsky S2V 1.0 Pro is built for naturalistic avatar production where rigid, synthetic head rotation is unacceptable. It handles:

• Full-Figure Speech Animation: Generates coordinated motion across shoulders, hands, torso, and head based on speech intensity and pacing, rather than just warping mouth coordinates.
• Complex Visual Stylizations: Because training incorporated broad visual data, the model animates oil paintings, 2D/3D cartoon figures, and non-human characters alongside photorealistic humans.
• Long-Form Speech Rollouts: The three-scale FramePack memory mechanism allows for generating continuous video up to 60 seconds without visual degradation, facial drift, or identity collapse between 5-second chunks.
• Multilingual Audio Support: Although trained predominantly on a curated corpus of 1.17 million single-speaker English and Russian clips, the underlying Qwen2.5-Omni acoustic conditioning generalizes across various languages, dialects, and non-verbal vocalizations.
• Variable Resolutions: The native weights generate standard definition (512x768) and high definition (1280x768) video at the input image's native aspect ratio without architectural modifications.

Running Kandinsky S2V 1.0 Pro locally requires understanding that this is not an LLM that can be dropped into an inference framework like Ollama or llama.cpp. It is a multi-stage diffusion pipeline written in PyTorch requiring CUDA 13 capabilities and driver version 580 or newer.

The primary checkpoint (kandinsky_s2v_pro.safetensors) is 39.8 GiB in bfloat16. However, inference requires holding the 21B DiT, Qwen2.5-Omni-7B, CLIP ViT-L/14, and the HunyuanVideo 3D VAE in memory simultaneously. Total cached model weights on disk equal roughly 65 GB.

• Minimum VRAM (Single GPU, SD 512x768): 46 GiB of VRAM (48 GB physical capacity, such as an RTX 6000 Ada or A40).
• Recommended VRAM (HD 1280x768): 55 GiB to 80 GiB of VRAM (NVIDIA A100 80GB or H100 80GB).
• Reduced Memory Mode: Enabling offload.vae_tiling: true lowers peak VRAM during the decoding stage to roughly 28 GiB, though it can introduce faint spatial seams between video tiles.

Practitioners asking how to run a 21B model on consumer GPUs should note that a single 24 GB card (such as an RTX 3090 or RTX 4090) cannot run the unquantized Pro pipeline without aggressive CPU offloading or layer-by-layer disk swapping. This slows inference down significantly.

For local consumer rigs, the best GPU for Kandinsky S2V 1.0 Pro is either a dual RTX 3090/4090 setup using sequence parallel sharding, or a professional workstation card like the RTX 6000 Ada (48 GB). Apple Silicon hardware (such as an M3 Max or M4 Max with 64 GB to 128 GB unified memory) can physically fit the weights in RAM, but official repository support currently targets Linux with CUDA and FlashAttention-3.

Throughput depends heavily on whether you run the base model or the distilled configuration, and whether torch.compile is enabled. Performance is measured in seconds of computation per one second of synthesized 25 FPS video:

• Pro 21B Base (1x H100, SD): ~189 seconds per 1 second of video (~152s with torch.compile).
• Pro 21B Distilled (1x H100, SD): ~41.5 seconds per 1 second of video (~33.6s with torch.compile).
• Pro 21B Distilled (8x H100 parallel, SD): ~11.4 seconds per 1 second of video.
• Pro 21B Distilled (1x H100, HD): ~137 seconds per 1 second of video (~29s on an 8x H100 node).

For production batch runs, using the distilled model (s2v_pro_distill.yaml) is mandatory. It cuts generation latency by more than 75% while maintaining parity in lip synchronization and image fidelity metrics.

Evaluating Kandinsky S2V 1.0 Pro against other open-weights avatar generation pipelines highlights distinct tradeoffs in parameter scale, compute footprint, and output naturalism:

• Kandinsky S2V 1.0 Pro vs Kandinsky S2V 1.0 Lite: The Lite model shrinks the DiT backbone down to 2B parameters (32 blocks, 13 audio cross-attention layers). Lite runs on an individual 24 GB GPU using VAE tiling and synthesizes a second of video in roughly 8.7 seconds on an H100. However, the 21B Pro model yields noticeably better lip-sync precision on complex phonemes and produces more diverse upper-body gestures without losing fine facial identity.
• Kandinsky S2V 1.0 Pro vs Hallo2 / LivePortrait: LivePortrait relies on landmark-based implicit keypoint retargeting. It runs at near real-time speeds on consumer GPUs, but it is limited to head crops and often looks robotic. Hallo2 generates diffusion-based video, but its reliance on smaller audio encoders (like Wav2Vec) often leads to identity drift over longer clips. Kandinsky S2V 1.0 Pro trades lightweight deployment for full-body coherence and stable long-sequence generation backed by its FramePack architecture.

For developers with access to data-center hardware or multi-GPU clusters, Kandinsky S2V 1.0 Pro delivers state-of-the-art visual expressiveness and audio synchronization under a permissive MIT license. Teams constrained to single consumer GPUs should look toward the 2B Lite variant instead.