PAN 2

PAN 2 is an open foundation world model developed by the Institute of Foundation Models (IFM) at MBZUAI. Unlike conventional generative video models that simply interpolate visual aesthetics from broad prompts, PAN 2 is built specifically for physical intelligence and simulative reasoning. The model simulates how environments mutate in direct response to text-described actions, rendering video sequences up to 20 seconds at 704x1248 resolution and 24 frames per second. All weights, inference pipelines, and training implementations are published under an Apache 2.0 license.

The system addresses a core bottleneck in autonomous agent deployment: enabling an embodied agent or planner to evaluate the outcomes of candidate physical decisions before executing them in the physical world. By coupling conceptual language planning with a dense 29B parameter perceptual diffusion backbone, PAN 2 provides a fully open simulation environment. Practitioners can deploy, fine-tune, and inspect the model without proprietary API barriers.

Within the landscape of generative simulators, PAN 2 occupies a distinct position alongside systems like Sora, Gen-3, and Cosmos, but with complete architectural transparency and local reproducibility. For engineers looking to run PAN 2 locally, the 29-billion-parameter dense architecture presents demanding yet approachable compute targets on modern workstation hardware.

PAN 2 implements the Generative Latent Prediction (GLP) framework, dividing world simulation into two interconnected stages: conceptual reasoning and perceptual generation.

1[Input: Action Text + Optional State Image]
2                 │
3                 ▼
4┌─────────────────────────────────────────┐
5│   Conceptual Reasoner (Multimodal LLM)  │
6│ - Actor-action binding                  │
7│ - Precise timestamps & camera trajectory │
8└─────────────────────────────────────────┘
9                 │
10                 ▼ Time-grounded structured plan
11┌─────────────────────────────────────────┐
12│ Perceptual Backbone: 29B Dense MMDiT    │
13│ - Predicts future video latents         │
14│ - Cross-attention conditioning          │
15└─────────────────────────────────────────┘
16                 │
17                 ▼ Latent representation
18┌─────────────────────────────────────────┐
19│           Wan 2.2 VAE Decoder           │
20│ - Uncompresses latents to 704x1248 @ 24fps
21└─────────────────────────────────────────┘

The first phase relies on a multimodal conceptual reasoning model. This stage ingests the current observation (an initial condition image plus prompt) and generates a structured, time-grounded plan. This plan binds specific actions to agents, specifies exact action start and end timestamps, and dictates continuous camera trajectories.

The second phase is the perceptual backbone, which uses a 29B parameter dense Multimodal Diffusion Transformer (MMDiT). Operating entirely in latent space, the 29B MMDiT receives the time-grounded conditioning vectors and iteratively denoises video latents representing the future world state. The training regime relies on FSDP2 hybrid sharding, context parallelism, and the Muon optimizer, trained on packed token budgets over precomputed latents. Crucially, the perceptual backbone bypasses pixel decoding during intermediate steps. Final video frames are reconstructed through a high-efficiency Wan 2.2 VAE decoder, maintaining spatial consistency across long 20-second temporal horizons.

Because this is a dense 29B parameter model rather than a Mixture of Experts (MoE), all 29 billion parameters participate in every inference step. This architecture demands consistent memory bandwidth and compute across diffusion steps, but it eliminates routing overhead and keeps memory access patterns uniform during generation.

PAN 2 focuses on deterministic, physically coherent state transitions rather than cinematic stylization. Its primary capabilities include:

• Timed Action Intervention: Actions take effect at precise timestamps. Changing an action timestamp from 00:00:03.000 to 00:00:06.000 fundamentally alters downstream physics, enabling fine-grained sensitivity testing for robotic trajectories.
• Simulative Reasoning for Robotics: Autonomous agents can roll out multiple candidate motion paths (such as grasping, navigating dynamic obstacles, or fluid manipulation) and evaluate the simulated visual states before sending control commands to physical hardware.
• Counterfactual Physical Simulation: Simulating multi-body interactions where timing determines collision or clearance, evaluating contact mechanics, liquid flow, and object deformation.
• Synthetic Data Generation for Vision-Language-Action (VLA) Models: Generating verified visual outcome sequences to train end-to-end policy networks on rare failure modes without damaging physical rigs.

Running a dense 29B diffusion model locally requires careful planning around VRAM capacity, precision formats, and memory bandwidth. Unlike pure text LLMs, PAN 2 requires memory for the 29B perceptual transformer, the text conditioning encoder, and the VAE decoding pipeline.

The perceptual backbone alone requires roughly 58 GB of VRAM in unquantized FP16/BF16 format. When accounting for KV cache, diffusion activation buffers, and the VAE decoder, full-precision inference requires multi-GPU infrastructure. Quantization significantly broadens hardware accessibility:

For production simulation pipelines, the best GPU for PAN 2 is an enterprise card with high memory bandwidth, such as the NVIDIA H200 (141GB HBM3e) or RTX 6000 Ada (48GB GDDR6). High bandwidth is essential: diffusion models require tens of iterative passes through the entire parameter set per video generation.

If you are evaluating how to run 29B model on consumer GPU setups:

• NVIDIA RTX 4090 (24GB): Can run PAN 2 using 4-bit quantization (such as GGUF or AWQ/NF4 kernels) combined with sequential CPU offloading for the text encoder and Wan 2.2 VAE decoder. Expect tight headroom.
• Dual RTX 3090 / 4090 (48GB combined): An ideal prosumer configuration. Using pipeline parallelism or Diffusers device mapping, you can run FP8 weights cleanly across two cards without paging activations to system RAM.
• Apple Silicon (M3 Max / M4 Max, 64GB-128GB Unified Memory): Metal Performance Shaders (MPS) can host the full FP16 or FP8 model in unified memory, though step latency will be compute-bound compared to CUDA execution.

For the majority of local practitioners, FP8 represents the sweet spot for PAN 2 performance, preserving subtle physical trajectories and fine visual textures without introducing diffusion artifacts. If hardware constraints force 4-bit execution, Q4_K_M or AWQ maintains structural coherence, though edge stability and long-range temporal consistency in 20-second clips degrade slightly compared to FP8.

Because PAN 2 generates temporal latents rather than discrete text tokens, throughput is measured in sampling latency per video second rather than standard PAN 2 tokens per second.

• On an NVIDIA H100 (80GB, FP8), generating a 5-second clip (120 frames at 24 fps) over 30 diffusion steps averages roughly 45 to 75 seconds.
• On a Dual RTX 4090 setup (FP8), expect generation times between 2.5 to 4 minutes per 5-second sequence depending on context parallelism configurations and kernel optimizations (such as FlashAttention-4 and Quack kernels).

PAN 2 inference is integrated with both Hugging Face diffusers and vLLM for conditioning tasks. The official repository provides launch scripts utilizing PyTorch FSDP2 and custom Triton kernels. Ensure your environment uses Python 3.10+, PyTorch 2.4+, and current CUDA drivers before pulling weights from Hugging Face.

Evaluating PAN 2 requires contrasting it against both conventional video generators and domain-specific physical world models.

Open-source video foundations like HunyuanVideo and Wan 2.1 focus heavily on visual prompt fidelity, artistic motion, and dynamic rendering.

• Parameter Scale: At 29B parameters, PAN 2 allocates significantly more capacity to world state modeling and spatial-temporal grounding than 13B-14B models.
• Conditioning: Standard open video models take natural language prompts and render a single interpretation. PAN 2 accepts explicit actor-action timestamps, allowing users to modify timing parameters independently to evaluate cause-and-effect branches.
• VRAM Demands: HunyuanVideo and Wan 2.1 are easier to fit on single 24GB GPUs at FP8. PAN 2 demands at least 48GB for uncompromised multi-step physical simulation.

NVIDIA Cosmos models are purpose-built physical world simulators designed for robotics and autonomous driving.

• Architecture: Cosmos models rely on specialized post-training recipes tuned heavily toward autonomous driving and robot manipulation domains. PAN 2 uses the GLP architecture to balance broader semantic action reasoning with physics.
• Licensing & Extensibility: PAN 2 is released under the permissive Apache 2.0 license with open training scripts (using Muon and FSDP2) and inference pipelines. This makes PAN 2 exceptionally attractive for research labs and commercial teams building local AI model 29B parameters 2026 workflows without restrictive commercial terms.