Octen-Embedding-8B

Octen-Embedding-8B is a domain-tuned text embedding model developed by Octen AI, currently ranking #1 on the RTEB Leaderboard (as of January 2026) with a Mean (Task) score of 0.8045. It is a dense 7.6B parameter model built as a LoRA fine-tune of Qwen3-Embedding-8B, targeting high-precision retrieval for industry workloads. This model competes directly with closed-source embedding APIs like Voyage-3-large and Cohere Embed v4, but with the advantage of being fully open-source under Apache 2.0 — meaning you can run it on your own hardware without per-query costs or data leakage.

The embedding space is 4096 dimensions, and the model supports a 32,768 token context length, making it suitable for long-document retrieval in legal, medical, and financial settings. If you need a local alternative to API-based embedders that matches or beats top commercial offerings on public and private benchmarks, this is the model to evaluate.

Octen-Embedding-8B is a dense transformer with 7.6B parameters, not MoE. Every inference call uses the full parameter count, so VRAM consumption is predictable: approximately 15–16 GB in FP16, 8–9 GB in INT8, and around 5–6 GB with 4-bit quantization (e.g., Q4_K_M). The architecture inherits Qwen3’s dual-encoder design for symmetric (query-document) and asymmetric retrieval, with a LoRA adapter trained on domain-specific data.

• Embedding dimension: 4096
• Max tokens: 32,768
• Base model: Qwen3-Embedding-8B
• Fine-tuning method: LoRA (low-rank adaptation)
• Supported quantization: INT8 (available via HuggingFace Octen/Octen-Embedding-8B-INT8), plus community Q4_K_M and Q5_K_M through GGUF.

The dense architecture means you get full performance on every request — no routing overhead or active-parameter variance. For local deployment, the trade-off is higher VRAM than a comparable MoE model, but the retrieval quality is state-of-the-art.

Octen-Embedding-8B excels in domain-specific retrieval where generic embeddings fall short. Octen AI tuned it on vertical datasets spanning four key areas:

• Legal: Contract clauses, case law, regulatory documents
• Finance: Earnings reports, Q&A datasets, personal finance content
• Healthcare: Medical Q&A, clinical dialogues, health consultation logs
• Code: Programming problems, code search, SQL queries

The model supports 100+ natural languages and several programming languages, with strong cross-lingual and multilingual retrieval performance. It achieves a Public dataset score of 0.7953 and Private dataset score of 0.8157 on RTEB, indicating minimal overfitting. Use it for:

• Semantic search over long internal documents (32K token context)
• RAG pipelines where retrieval quality directly impacts LLM output
• Code search in multi-repo environments
• Multilingual document matching without language-specific pipelines

This is where Octen-Embedding-8B differentiates itself from API-only models. You can deploy it on consumer GPUs, and the open license means no rate limits or data privacy risks.

For most users, Q4_K_M is the sweet spot: it preserves retrieval quality (within 1–2% of FP16 on MTEB benchmarks) while fitting on widely available cards like the RTX 3060 12GB. If you have a 24GB card like the RTX 4090, INT8 or even FP16 is feasible and maximizes precision for edge-case queries.

Embedding models are measured in tokens per second during inference, not generation speed. On a single RTX 4090:

• FP16: ~800–1000 tokens/second
• INT8: ~1100–1400 tokens/second
• Q4_K_M: ~1300–1600 tokens/second

Batch size affects throughput substantially. With a batch size of 16 and Q4_K_M, you can embed ~25,000 tokens per second. On an M4 Max (64GB unified), expect ~600–800 tokens/second in FP16 — sufficient for real-time retrieval in most RAG setups.

The fastest way to run Octen-Embedding-8B locally is via Ollama (once a GGUF variant is available). Alternatively, use the official INT8 model from HuggingFace with sentence-transformers:

1from sentence_transformers import SentenceTransformer
2
3model = SentenceTransformer("Octen/Octen-Embedding-8B-INT8")
4embeddings = model.encode(["Your query text here"])

For production deployments, consider using vLLM with the Nomic embedding backend or a custom Triton server.

• Best GPU for Octen-Embedding-8B: RTX 4090 (24GB) — fits FP16 with headroom for batch processing.
• Budget option: RTX 3060 12GB + Q4_K_M quantization.
• Apple Silicon: M4 Max 64GB unified memory can run FP16 comfortably. M2 Ultra also works.

If you need to run a 7.6B model on a consumer GPU like the RTX 3060, use Q4_K_M and keep the batch size under 8. For latency-sensitive applications, the INT8 variant on an RTX 4080 Super is a solid middle ground.

Octen-Embedding-8B competes primarily with two models at similar scale:

vs Voyage-3-large (API-only, 1024 dim)

Voyage-3-large scores 0.7812 Mean (Task) on RTEB vs Octen’s 0.8045. Octen also offers 4× the embedding dimensions (4096 vs 1024), which can improve recall on fine-grained retrieval. The trade-off: Octen requires local hardware and is larger (7.6B vs Voyage’s ~1.5B). If you need zero-maintenance cloud retrieval, Voyage is simpler; if you want control and better scores, Octen wins.

vs Qwen3-Embedding-8B (base model)

Octen-Embedding-8B is a refined version of Qwen3-Embedding-8B, which scores 0.7547. Octen’s domain tuning adds ~5 points on Mean (Task) and significantly improves performance on legal, finance, and medical tasks. If you’re already using Qwen3-Embedding-8B, upgrading to Octen costs nothing (license compatibility) and gives you a measurable quality lift without changing your inference stack.

When to choose Octen-Embedding-8B:

• You need top-tier retrieval quality on proprietary or domain-specific documents.
• You want to avoid API costs and data egress.
• You have at least 6 GB VRAM available for quantization.

When to choose an alternative:

• You have extremely limited hardware (under 4 GB VRAM) — consider Octen-Embedding-0.6B.
• You need 128K+ context (Cohere Embed v4) — Octen caps at 32K.