GritLM-8x7B

GritLM-8x7B is a Mixture-of-Experts (MoE) model developed by Contextual AI that pulls double duty: it’s both a state-of-the-art text embedder and a strong open-source language generator. The model uses the Generative Representational Instruction Tuning (GRIT) method, which trains a single set of weights to handle embedding and generation through instruction‑based routing. This means you don’t need separate models for retrieval and response — one model does both.

With 57.9B total parameters but only 13B active per forward pass, GritLM-8x7B targets users who want best-in-class open generation without paying the full compute cost of a dense 57B model. On the Massive Text Embedding Benchmark (MTEB) it is competitive with the best dedicated embedding models, while on generative tasks it outperforms every open model up to its size that we tested. It’s released under Apache 2.0, making it free to use, modify, and deploy in production.

The model is a text‑only system. It excels at tasks requiring both retrieval (via embeddings) and generation — most notably retrieval‑augmented generation (RAG) — because the unified architecture can cut pipeline latency by over 60% for long documents, compared to using separate retrieval and generation models.

GritLM-8x7B is based on an 8‑expert MoE transformer with 57.9B total parameters. Only 13B parameters are active per token, which is the effective computational cost for inference. This design balances the quality of a large model with the throughput of a much smaller one:

• Total parameters: 57.9B
• Active parameters: 13B
• Architecture: MoE (Mixture of Experts) with 8 experts
• Routing: Learned gating routes each token to the top‑k experts

The active‑parameters count is the key number for hardware planning. A dense 57B model would require around 110 GB of VRAM in FP16, while GritLM-8x7B’s MoE sparsity brings the memory footprint closer to that of a 13B–15B dense model (plus overhead for expert weights). In practice, you can run the full FP16 model on a single high‑VRAM GPU — but only at aggressive quantization.

Context length is not officially specified, but the model inherits the 32K context length typical of its Mixtral‑8x7B base (confirm with your own testing for long‑context tasks). The tokenizer is the same as Mistral‑7B’s (v1).

GritLM-8x7B is trained for two modes: generate and embed. You choose the mode via a special instruction prefix. This makes it uniquely suited for:

• Retrieval‑Augmented Generation (RAG): Use the same model to embed your document corpus and to generate answers over those embeddings. No separate text-embedding-ada-002 or BGE model required. The pipeline becomes simpler, and for long documents the end‑to‑end latency drops by more than 60%.
• Text generation: The model achieves top‑tier results on open‑ended chat, instruction following, and reasoning benchmarks. It competes directly with dense 70B models while using far fewer FLOPs per token.
• Embedding tasks: MTEB scores place it among the best open‑source embedders. Use it for semantic search, clustering, classification, and sentence‑level tasks. The embedding mode avoids loading the LM head, saving memory when you only need vectors.
• Unified API: One model, one inference server, one pipeline. If your application alternates between embedding and generation (e.g., semantic cache + chatbot), you avoid juggling two model instances.

Concrete use cases include: building a local RAG system for internal documentation, creating a semantic search engine over your codebase, or powering a chatbot that also indexes chat history for retrieval.

Running a 57.9B MoE model on consumer hardware is feasible if you quantize well. The MoE architecture means you must load all 57.9B parameters into memory (even though only 13B are computed per token). Here’s the realistic breakdown:

Recommended for most practitioners: Q4_K_M on a dual‑GPU setup or Q3_K_M on a single 24 GB card with aggressive offloading. If you have an Apple M4 Max with 128 GB unified memory, you can run Q8_0 comfortably.

On a single RTX 4090 with Q3_K_M and CPU offloading, expect 2–4 tokens/second for generation. On two RTX 4090s with Q4_K_M, you’ll see 8–12 tokens/second. For embedding (batch size 1), throughput is higher because the forward pass is lighter (no autoregressive generation). Use llama.cpp or Ollama to get started quickly — Ollama supports GritLM-8x7B via gritlm/gritlm-8x7b and handles quantization automatically.

1ollama run gritlm/gritlm-8x7b

This downloads the Q4_K_M quantized model (about 18 GB download) and runs it with sensible defaults. For embedding, call the API with the appropriate instruction prefix.

• Minimum: RTX 4090 (24 GB) with Q3_K_M and CPU offloading – usable for small‑batch RAG.
• Recommended: Dual RTX 4090 or RTX 6000 Ada (48 GB) at Q4_K_M for reasonable speed and quality.
• Good alternative: Apple M4 Max (128 GB) – runs Q8_0 without offloading.
• Server‑grade: A100 80 GB × 1 – runs FP16 if you offload some layers to CPU, or Q8_0 comfortably.

GritLM-8x7B lives in the awkward zone between a 13B dense model and a 70B dense model. Here’s how it stacks up against two realistic alternatives:

• Generation: GritLM-8x7B outperforms Mixtral‑8x7B on open‑ended generation benchmarks (e.g., MT‑Bench, AlpacaEval) thanks to GRIT training and alignment on tulu2 data.
• Embedding: Mixtral has no native embedding capability. You’d need to switch to a separate embedder. GritLM embeds and generates in one.
• Inference speed: Both have the same active parameter count (13B), so throughput is comparable at the same quantization. GritLM’s unified pipeline can be faster overall for RAG.

• RAG latency: Two models = two inference passes, plus latency for network calls if they’re on separate servers. GritLM’s single‑model approach eliminates that overhead.
• Generation quality: GritLM-8x7B clearly beats Llama 3 8B on complex reasoning and instruction following. It’s closer to Llama 3 70B while using far fewer active parameters.
• Memory footprint: Llama 3 8B (FP16) takes ~16 GB, plus an embedder (e.g., BGE‑Small ~1 GB). Total ~17 GB. GritLM at Q4_K_M uses ~33 GB. You trade more VRAM for higher quality and unified operation.

When to choose GritLM-8x7B: You need both retrieval and generation in a single OSS model, and you have the hardware to run a 33+ GB quantized model. The primary advantage is pipeline simplification and latency reduction for local RAG.

When to avoid it: If your use case is purely generative and you can’t spare 33 GB of VRAM, a 7B or 13B dense model will be faster and easier to deploy. If you only need embeddings, a dedicated 384‑parameter embedder is cheaper and faster.