F2LLM-v2-8B

F2LLM-v2-8B is a general-purpose, multilingual embedding model developed by CodeFuse-AI (Ant Group). At 7.6B parameters (dense architecture), it is one of the largest publicly available text embedding models designed for local inference. Its primary use case is generating high-quality dense vector representations for retrieval-augmented generation (RAG), semantic search, clustering, and classification across more than 200 languages.

The model is the second generation of the F2LLM family, trained on a curated composite of 60 million publicly available high-quality examples. Unlike many embedding models that cap out at a few hundred million parameters, F2LLM-v2-8B pushes into a scale that previously required proprietary APIs. It delivers near-flagship embedding quality at an inference cost that fits on a single consumer GPU—making it a practical choice for developers who need state-of-the-art multilingual retrieval without paying per-query API bills.

F2LLM-v2-8B is released under the Apache 2.0 license. The family includes base (Preview) and instruct variants; for production use, the instruct version (codefuse-ai/F2LLM-v2-8B) is recommended.

F2LLM-v2-8B is a dense transformer encoder, not a decoder-only generative model. It uses a bidirectional attention mechanism optimized for producing fixed-size embeddings from variable-length text inputs. The architecture is built on top of Qwen3 backbone (as indicated in the training code) and supports Matryoshka Representation Learning (MRL) and knowledge distillation—both included as new features in V2.

• Parameters: 7.6B (dense, all active at inference)
• Context length: Not officially specified, but practical testing suggests at least 8,192 tokens (consistent with typical Qwen3 limits)
• Embedding dimension: 4096 (default), though MRL enables truncation to lower dimensions (e.g., 1024, 2048) for storage or speed trade-offs without retraining
• Supported languages: 200+, with particular emphasis on mid- and low-resource languages (e.g., Swahili, Catalan, Burmese, Tajik)
• Pipeline tag: feature-extraction – designed for use with libraries like HuggingFace Transformers and Sentence Transformers

Because it is a dense model at 7.6B, peak VRAM at FP16 precision is approximately 15.2 GB. This is a key consideration for local deployment. However, quantization reduces the footprint significantly—see the Running Locally section for concrete numbers.

F2LLM-v2-8B is an embedding model, not a chat or completion model. Its strength lies in converting text into high-quality vectors that capture semantic, multilingual, and cross-lingual relationships.

Primary use cases:

• Multilingual semantic search – Index documents in 200+ languages and query in any of them. The model’s training data deliberately included low-resource languages, making it a strong choice for global products.
• Cross-lingual retrieval (CLIR) – Retrieve English documents from a Spanish query, or French documents from a Chinese query. The model learns language-agnostic representations.
• Dense retrieval for RAG – Generate embeddings for a local knowledge base and retrieve relevant chunks with high recall. Can compete with API-based embedders like OpenAI text-embedding-3-large at a fraction of the latency.
• Clustering and classification – Use the embeddings as features for downstream tasks (e.g., topic modeling, intent classification) without fine-tuning.

Concrete example: A developer building a support bot that serves users in Spanish, Arabic, and Vietnamese can use F2LLM-v2-8B to index their FAQ database once and serve queries in any language. The same pipeline works for documentation retrieval, product catalogs, or internal knowledge bases.

This model is sized to run on consumer hardware with careful quantization choices. Here are the real-world specs.

• RTX 4090 (24GB) – Run FP16 or Q8_0 comfortably; multiple parallel encoding streams possible.
• RTX 4080 / 4070 Ti Super (16GB) – FP16 is tight but works; Q8_0 is safe.
• RTX 4060 / 3060 (8–12GB) – Use Q4_K_M or Q5_K_M. Expect good throughput for batch encoding.
• Apple Silicon (M4 Max 24–48GB) – FP16 runs in unified memory; performance is limited by memory bandwidth (around 200–400 embeddings per second depending on sequence length).
• MacBook Pro M1/M2 (8–16GB) – Only feasible with Q4_K_M. Not ideal for high-throughput production, but fine for prototyping.

Performance depends heavily on sequence length and batch size. For a typical batch of 1 query (512 tokens) with a single embedding:

• RTX 4090 (FP16): ~500–800 embeddings per second
• RTX 4090 (Q4_K_M): ~800–1200 embeddings per second
• M4 Max (FP16): ~200–400 embeddings per second
• RTX 3060 12GB (Q4_K_M): ~200–300 embeddings per second

For indexing large corpora, batching (e.g., batch size 32) multiplies throughput linearly until memory is exhausted.

The fastest way to run F2LLM-v2-8B locally is via Ollama (if a GGUF conversion is available) or directly with the HuggingFace transformers library using the Sentence Transformers integration. The model card on HuggingFace provides a ready-to-run snippet:

1from sentence_transformers import SentenceTransformer
2
3model = SentenceTransformer("codefuse-ai/F2LLM-v2-8B")
4embeddings = model.encode(["Your text here"])

For quantized GGUF versions, use llama.cpp or Ollama after converting the model with convert.py.

F2LLM-v2-8B competes in the league of large, open-source multilingual embedding models. The most direct alternatives are:

• intfloat/multilingual-e5-large (335M parameters) – Much smaller, far less VRAM (under 1GB at FP16). Good for low-resource setups, but quality on mid/low-resource languages is noticeably lower. F2LLM-v2-8B outperforms it on cross-lingual retrieval benchmarks (per the F2LLM paper).
• BAAI/bge-m3 (568M parameters) – Another strong multilingual embedder with dense retrieval and ColBERT support. BGE-M3 is more memory-efficient (2–3GB FP16) and supports dense+sparse hybrid search. F2LLM-v2-8B wins on raw representation quality and language coverage.
• Cohere Embed V3 (proprietary API) – F2LLM-v2-8B offers comparable quality offline without per-query costs. The trade-off is self-hosting overhead.

When to choose F2LLM-v2-8B: You need high multilingual quality, especially for low-resource languages, and have a GPU with at least 8GB VRAM. You want full control over the embedding pipeline and no API dependency.

When to choose a smaller model: If you are constrained to a CPU or an 8GB GPU and must prioritize throughput over maximum accuracy, models like BGE-M3 or multilingual-e5-large are more practical.

Trade-off summary: F2LLM-v2-8B delivers state-of-the-art multilingual embeddings at a size that is demanding but feasible for a single consumer GPU. It is the best open choice for developers who need an on-premises alternative to large API-based embedders and are willing to allocate 5–8GB VRAM for quantization.