GritLM-7B

GritLM-7B is a 7.2 billion parameter dense language model from Contextual AI that does something most open models can’t: it handles both text generation and text embedding in a single unified architecture. Trained with Generative Representational Instruction Tuning (GRIT), this model doesn’t force you to choose between a chat model and an embedding model. You get both, controlled by simple instructions.

At 7.2B parameters, GritLM-7B competes directly with other 7B-class models like Mistral 7B, Llama 2 7B, and Zephyr 7B on generative tasks, but it also sets a new state of the art on the Massive Text Embedding Benchmark (MTEB) among open models of its size. That means if you need a model that can answer questions, write code, and produce high-quality semantic embeddings for retrieval or clustering, this is the one dataset to train them all.

Licensed under Apache 2.0, with no usage restrictions, GritLM-7B is a practical choice for developers who need a single local model for both RAG pipelines and conversational agents.

GritLM-7B is a dense transformer with 7.2B parameters — no mixture-of-experts gating here. Dense means all parameters are active for every forward pass, so the full model capacity is always engaged. For local hardware, this translates to predictable memory consumption: the full-precision model (float32) requires ~29 GB of VRAM, but with quantization (e.g., 4-bit) it drops to around 5-6 GB. Context length is not officially specified, but the model supports the same underlying architecture as Llama 2 (likely 4096 tokens based on available checkpoints).

The key design innovation is GRIT: the model is trained to distinguish between generation and embedding tasks based on instruction prefixes. During inference, you pass a system-level instruction (e.g., "Generate:" vs "Embed:") and the model switches behavior accordingly. This is implemented as a simple chat template; no separate adapter models or pipelines. The embedding mode uses the last hidden state, and you can specify pooling methods (mean, weighted mean, etc.) via the GritLM Python library.

For developers familiar with transformers, the model uses the same loading pipeline as any Hugging Face model. The official gritlm Python package handles both modes seamlessly and supports torch_dtype="auto" for automatic precision.

GritLM-7B excels at two distinct categories of tasks, often in the same pipeline:

• Text Generation: Instruction-following, chat, code completion, summarization, translation. It outperforms Mistral 7B and Llama 2 7B on generative benchmarks (e.g., MMLU, ARC, HellaSwag) and matches models twice its size when using the GRIT tuning. You can use it with any standard chat template or the built-in <|user|> / <|assistant|> format.

• Text Embedding: State-of-the-art performance on MTEB for a 7B model. Tasks include semantic similarity, clustering, retrieval (top 89% recall on ArguAna), and classification (96% accuracy on Amazon polarity). For RAG systems, this means you can generate embeddings and then generate answers with the same model — no need to run a separate embedding model like gte-small or e5.

Concrete use cases:

• Build a local RAG chatbot where the same model indexes documents and answers questions.
• Perform zero-shot text classification by generating label embeddings and comparing to inputs.
• Run a semantic search engine on private documents without sending data to external APIs.
• Fine-tune further on domain-specific data using the GRIT training scripts (also open-source in the repo).

Running GritLM-7B on consumer hardware is feasible with proper quantization. Here’s what you need:

• RTX 4090 (24 GB): Run Q4_K_M with headroom for context caching. Expect 50-70 tokens/second in generation mode.
• RTX 3090 (24 GB): Similar performance, slightly slower due to memory bandwidth (~40-55 tokens/second).
• RTX 4080 (16 GB): Can run Q4_K_M with batch size 1. Generation speed ~35-50 tokens/second.
• M4 Max (64 GB unified): Run FP16 or Q8_0 at high throughput (60-80 tokens/second) thanks to large unified memory bandwidth.
• Apple M1/M2 with 16 GB: Use Q4_K_M — expect 10-15 tokens/second via MLX or llama.cpp.

Install Ollama and pull the model:

1ollama pull gritlm/gritlm-7b

Then run:

1ollama run gritlm/gritlm-7b

For custom quantization, use llama.cpp directly:

1git clone https://github.com/ggerganov/llama.cpp
2cd llama.cpp
3make -j
4./quantize /path/to/gritlm-7b.gguf Q4_K_M
5./main -m gritlm-7b-Q4_K_M.gguf -p "Generate: Explain GRIT in one sentence."

For embedding, the gritlm Python package is recommended:

1from gritlm import GritLM
2model = GritLM("GritLM/GritLM-7B", torch_dtype="auto", mode="embedding")
3embedding = model.encode(["This is a test sentence."])

• Generation (Q4_K_M on RTX 4090): 50-70 tokens/second with 2048 context.
• Embedding (FP16 on RTX 4090): ~5000 sentences/second for short texts (batch size 64).
• Quantization tradeoff: Q4_K_M preserves >98% of MTEB performance. Q3 reduces embedding quality noticeably (2-4% drop on retrieval tasks).

vs Mistral 7B

Mistral 7B is a pure generative model. It slightly edges out GritLM-7B on code generation and has a larger community, but lacks any native embedding capability. If you need a single model for both tasks, GritLM-7B wins. If you only need generation and want a lighter ecosystem, Mistral remains a strong choice.

vs e5-mistral-7b-instruct

This is a dedicated embedding model fine-tuned from Mistral 7B. On MTEB, GritLM-7B performs comparably or better (SOTA at 7B when released). The difference is that e5 requires a separate generative model for chat; GritLM-7B does both. If your workflow is purely retrieval, e5 may be marginally faster on inference, but GritLM-7B simplifies your stack.

vs Llama 2 7B / Zephyr 7B

GritLM-7B outperforms both on generative benchmarks (especially instruction following) and massively surpasses them on embedding tasks. The only downside is slightly larger community support for alternatives (e.g., more pre-quantized GGUF variants for Llama 2). But for practitioners who value a unified model, GritLM-7B is the clear winner in its class.