Cola2

Cola2 is an open-weight base language model developed by ByteDance-Seed that departs from standard token-by-token autoregression. Built with an 8.77B parameter dense architecture, the system plans text hierarchically: it compresses source text into continuous semantic spans, proposes latent representations using a block-causal Diffusion Transformer (DiT), and realizes those latents into discrete tokens through a specialized decoder. Published on Hugging Face with an Apache-2.0 codebase, Cola2 represents an architectural departure from conventional causal language models, exploring continuous latent planning to tackle sequence modeling.

Trained on 346.03B source tokens, Cola2 is strictly a foundational base model rather than an instruction-tuned assistant. Despite processing a fraction of the data seen by mainstream open models, it achieves a reported 60.10 average across eight standard language tasks. The checkpoint is not designed as a drop-in replacement for everyday conversational agents. Instead, it serves as an empirical platform for engineers and researchers evaluating diffusion-based text generation, hierarchical planning, and local inference mechanics.

For practitioners looking to run a local AI model with 8.77B parameters in 2026, Cola2 presents a distinct workload profile. Because it alternates between latent diffusion steps and discrete span decoding, its compute profile diverges from standard transformer key-value (KV) caching patterns, creating unique considerations for local GPU deployment.

Cola2 uses an 8.77B dense parameter network divided between continuous semantic compression and discrete token realization. Rather than predicting the next discrete token given a prior sequence, Cola2 operates across two decoupled levels of abstraction:

• Dynamic-Compression VAE: Text is segmented into dynamic, content-aware semantic spans. A continuous encoder maps each span into a compact latent vector, controlling compression without requiring unstable auxiliary losses.
• Block-Causal Latent DiT: An approximately 4B-parameter Diffusion Transformer models the sequence of continuous latents using Flow Matching. It processes sequences block-by-block, maintaining causal consistency across span blocks while diffusing representations within each block.
• Span-Causal Token Decoder: A discrete token decoder consumes the continuous latents alongside prior context to reconstruct readable text. Classifier-free guidance (CFG) dynamically balances fidelity between strict semantic reconstruction and open-ended generation.

During inference, Cola2 repeatedly proposes a continuous latent block, decodes the corresponding tokens, and re-encodes the verified text under a fixed span layout to keep continuous and discrete states aligned. Because all 8.77B parameters are active on every pass, memory bandwidth and raw compute remain predictably tied to dense matrix multiplication, without the sparse routing overhead seen in mixture-of-experts (MoE) designs.

The official release does not specify a maximum sequence context length. Because the model relies on block-causal diffusion rather than an arbitrary positional context window, practical memory usage scales with block batching and generation budgets rather than standard quadratic attention limits.

Cola2 is a raw base model trained on general text distributions. It exhibits general text-completion and reasoning traits rather than chat-style conversational behavior. Its performance reflects its limited training exposure (346.03B tokens, with its latent DiT reported as still unconverged), yet it demonstrates strong structural comprehension for its data budget.

On standard reasoning and language evaluations, the model records the following reported scores:

• StoryCloze: 80.42
• ARC-Easy: 87.62
• RACE: 69.91
• LAMBADA: 63.00
• TruthfulQA: 54.10
• MMLU: 51.32
• HellaSwag: 41.75
• NarrativeQA: 32.70

On narrative-level tracking tests like StoryCloze (80.42) and HellaSwag (41.75), Cola2 scores slightly higher than Llama-3-8B's zero-shot base scores despite seeing over 43 times fewer pretraining tokens. This demonstrates that continuous semantic latent planning provides strong inductive bias for narrative coherence.

Concrete local use cases include:

• Non-Autoregressive Generation Research: Testing whether hierarchical latent planning resolves token-level repetition or local-optima traps common to greedy next-token prediction.
• Semantic Span Analysis: Using the model's dynamic-compression encoder to extract continuous semantic representations from raw corpora for downstream retrieval and clustering.
• Domain Adaptation and Continued Pretraining: Serving as a base checkpoint for organizations researching diffusion decoders on domain-specific corpora (such as medical records or technical manuals) where global structure matters more than conversational formatting.

To run Cola2 locally, your setup must account for the model's dense parameter footprint and its multi-stage forward pass (the VAE encoder, latent DiT, and token decoder).

At 8.77B parameters, memory consumption closely mirrors standard 8B-9B class networks for weight storage:

• 16-bit (BF16 / FP16): 18 GB to 20 GB VRAM. Requires a 24 GB GPU for safe execution and workspace memory.
• 8-bit Quantization (INT8): 10 GB to 12 GB VRAM. Fits on 16 GB GPUs.
• 4-bit Quantization (INT4 / Q4_K_M): 6 GB to 7.5 GB VRAM. Fits on 8 GB to 12 GB consumer GPUs.

• Best GPU for Cola2 (Single GPU): An NVIDIA GeForce RTX 3090 (24 GB) or RTX 4090 (24 GB) provides optimal headroom to run the model at full 16-bit precision without offloading.
• Mid-Range Consumer GPUs: An NVIDIA RTX 4070 Ti Super (16 GB) or RTX 4080 (16 GB) handles 8-bit quantization comfortably. An RTX 3060 (12 GB) or RTX 4060 Ti (16 GB) represents the entry point for how to run an 8.77B model on consumer GPU setups using 4-bit precision.
• Apple Silicon: Macs equipped with unified memory (M2, M3, or M4 Pro/Max with 24 GB or more RAM) can host the unquantized weights directly through PyTorch MPS.

For general local exploration, the best quantization for Cola2 is 4-bit weight-only quantization (such as AWQ or bitsandbytes 4-bit NF4) if memory is constrained, or 8-bit precision if you have at least 16 GB of VRAM. Because diffusion-based generation involves stochastic latent steps and classifier-free guidance, aggressive sub-4-bit quantizations can introduce noise into the continuous latent space and degrade token reconstruction.

In terms of Cola2 tokens per second, generation throughput differs noticeably from standard autoregressive models. A standard 8B dense model on an RTX 4090 generates roughly 40 to 60 tokens per second. Cola2, however, spends extra compute solving the continuous flow matching trajectory for each latent block before running token decoding. As a result, Cola2 performance exhibits higher computational latency per block than pure next-token generators, trading pure token throughput for parallel span realization.

Standard local runners like Ollama or vanilla llama.cpp rely on standard transformer decoding loops and do not natively parse Cola2's custom diffusion pipeline out of the box. While practitioners frequently look to Ollama as the quickest way to get started with common models, running Cola2 requires cloning its official repository and loading the weights using Hugging Face transformers with trust_remote_code=True in Python. Ensure PyTorch and CUDA dependencies are configured prior to running the provided CLI inference scripts.

Evaluating Cola2 requires looking at competing base models in the 8B parameter bracket, particularly standard causal transformers and alternative masked diffusion architectures.

Meta's Llama-3-8B is the de facto benchmark for dense open weights around this scale.

• Training Scale: Llama-3-8B was trained on over 15 trillion tokens, whereas Cola2 consumed 346.03B tokens. Consequently, Llama-3-8B outperforms Cola2 across broader knowledge benchmarks such as MMLU (63.19 vs 51.32) and ARC-Easy (90.86 vs 87.62).
• Generation Paradigm: Llama-3-8B uses next-token autoregression supported by deep KV-cache tooling (vLLM, Ollama, llama.cpp). Cola2 uses continuous latent diffusion with block decoding, which requires its native PyTorch repository.
• When to choose which: Choose Llama-3-8B for general reasoning, zero-shot application development, or turnkey deployment. Choose Cola2 if you are researching non-autoregressive language systems or evaluating hierarchical semantic planning.

LLaDA is another contemporary non-autoregressive diffusion model operating near the 8B boundary.

• Architecture: LLaDA applies discrete diffusion directly onto masked token spaces. Cola2 instead applies continuous diffusion inside a compressed latent space produced by a dynamic VAE, then decodes latents into tokens.
• Scaling Behavior: LLaDA (trained on 2.3T tokens) achieves higher aggregate benchmark numbers across multi-task evaluations (61.27 MMLU). However, controlled scaling studies show that Cola2 continues closing performance gaps as training tokens scale, demonstrating stronger compression efficiency per latent step.
• When to choose which: LLaDA provides a purely discrete diffusion reference, while Cola2 is the primary choice for experimenting with continuous-to-discrete hierarchical translation.