Falcon3-3B-Base

Falcon3-3B-Base is a 3-billion-parameter causal decoder-only language model developed by the Technology Innovation Institute (TII) in Abu Dhabi. Released in December 2024 as part of the Falcon3 open foundation model lineup, it occupies the lightweight, high-efficiency tier designed for low-latency local deployment and edge inference. Unlike instruction-tuned conversational checkpoints, Falcon3-3B-Base is a raw, pretrained foundation model. It provides an unaligned base for developers looking to build task-specific completions, run continuous pretraining on domain data, or perform custom supervised fine-tuning (SFT) and direct preference optimization (DPO).

TII engineered this model through structured pruning across both depth and width from Falcon3-7B-Base, followed by knowledge distillation on 100 gigatokens of curated web, code, STEM, and multilingual data. Operating within the compact sub-4B parameter category, Falcon3-3B-Base directly challenges established alternatives such as Meta's Llama 3.2 3B and Alibaba's Qwen 2.5 3B. It delivers surprisingly competitive reasoning, math, and code capabilities relative to its training budget, making it an attractive target for developers evaluating a local AI model with 3B parameters in 2026.

Distributed under the TII Falcon-LLM License 2.0 with weights hosted on Hugging Face, Falcon3-3B-Base supports text generation across English, French, Spanish, and Portuguese. Its compact footprint makes it especially relevant for practitioners who want to avoid API lock-in and run models locally on edge devices, developer workstations, and entry-level discrete GPUs.

Falcon3-3B-Base employs a standard dense causal transformer architecture configured to optimize inference throughput and memory access efficiency:

• Parameter Count: 3 billion dense parameters
• Decoder Layers: 22 transformer blocks
• Attention Configuration: Grouped Query Attention (GQA) utilizing 12 query heads and 4 key-value heads
• Head Dimension: 256, providing wider representation capacity per attention head
• Context Length: 8,192 tokens natively supported
• Position Embeddings: Rotary Position Embeddings (RoPE) with a high base frequency (rope_theta = 1000042) for robust long-context processing
• Activation and Normalization: SwiGLU activation functions paired with RMSNorm
• Vocabulary Size: 131,072 tokens

The adoption of Grouped Query Attention with a 3:1 query-to-key-value ratio significantly curbs the memory overhead of the KV cache during sustained inference sessions. Across the native 8,192-token context window, this GQA implementation allows the KV cache to remain compact (under 500 MB in FP16), enabling high batch sizes or multiple concurrent streams on hardware with restricted memory bandwidth.

The distillation process from Falcon3-7B-Base involved depth and width pruning using 1,024 NVIDIA H100 GPUs. Rather than training a 3B model from scratch across tens of trillions of tokens, TII transferred the representations learned by the 7B teacher model into a shallower, 22-layer network. This architectural compression allows the model to retain a high degree of the linguistic and logical coherence found in larger systems while slashing the per-token computational cost.

Because Falcon3-3B-Base is an unaligned base model, it does not respond to zero-shot conversational prompts like a chat assistant. Instead, it generates completions based on prompt structure. In raw form, it functions as a text completer, few-shot in-context learner, and an ideal substrate for domain adaptation.

• Domain-Specific Fine-Tuning: The model serves as a clean starting point for LoRA or QLoRA fine-tuning on proprietary enterprise documentation, technical manuals, or structured formatting tasks (such as extracting entities into JSON).
• Falcon3-3B-Base for Coding: Distilled with a significant proportion of STEM and code repositories, the base model understands syntax, logic flow, and documentation structure in languages like Python, C++, and JavaScript. It functions well for local code completion engines embedded inside IDEs without introducing cloud latency.
• Multilingual Applications: Falcon3-3B-Base natively processes English, French, Spanish, and Portuguese, enabling translation pipelines and multilingual document classification on lightweight hardware.
• Math and Symbolic Reasoning: In internal evaluations, the model displays high raw mathematical execution for its footprint. On the GSM8K (5-shot) benchmark, Falcon3-3B-Base achieves a raw score of 63.9, outperforming Llama 3.2 3B (26.7) and approaching Qwen 2.5 3B (69.0). On the MATH Level-5 (4-shot) test, it achieves 9.4, leading its direct weight-class competitors.
• Academic and Logical Benchmarks: The Falcon3-3B-Base reasoning benchmark results show strong zero-shot and few-shot numbers, including a 31.2 on GPQA (0-shot) and 54.9 on ARC-Challenge (25-shot).

The primary appeal of a 3B parameter model is running it on consumer hardware without complex multi-GPU setups. Falcon3-3B-Base hardware requirements are modest, making local deployment practical across a wide range of devices.

Memory requirements depend on the chosen precision. Because the parameter count is small, running full FP16 or BF16 is feasible on standard desktop graphics cards:

• BF16 / FP16 (Unquantized): ~6.0 GB to 6.5 GB VRAM for model weights. Add 1 GB to 2 GB for the 8K KV cache and runtime overhead. A GPU with 8 GB VRAM handles this at short-to-medium contexts; 12 GB VRAM is recommended for the full 8K context.
• Q8_0 (8-bit Quantization): ~3.2 GB to 3.5 GB VRAM. Runs comfortably on any modern 6 GB or 8 GB graphics card with zero discernible loss in output quality.
• Q4_K_M (4-bit Quantization): ~1.9 GB to 2.2 GB VRAM. Total memory footprint during inference remains below 3.5 GB even at full context.

The best quantization for Falcon3-3B-Base depends on your target hardware:

• For GPUs with 8 GB or more VRAM (RTX 3060, RTX 4060, Apple unified memory): Run Q8_0 or unquantized BF16. At 3B parameters, 4-bit quantizations can introduce subtle degradation in precision-sensitive tasks like code generation and math, whereas 8-bit retains baseline accuracy with negligible memory cost.
• For edge systems, low-power laptops, or integrated GPUs: Use Q4_K_M. It provides the best trade-off between footprint and perplexity, fitting inside devices with constrained shared memory.

To run Falcon3-3B-Base locally at scale, consider these hardware profiles:

• Consumer Desktop GPUs (RTX 3060 12GB, RTX 4060 8GB, RTX 4070): These are the best GPU options for Falcon3-3B-Base when balancing price and performance. On an RTX 4070 (12GB), you can expect Falcon3-3B-Base performance to exceed 120 to 160 tokens per second using quantized GGUF or vLLM backends. Even an entry-level RTX 3060 generates 80 to 110 tokens per second at Q4_K_M.
• Apple Silicon (M2/M3/M4 Series): Base M-series chips with 8 GB or 16 GB unified memory run the model entirely in RAM via Metal acceleration. An M4 Max processes generation speeds above 100 tokens per second, while base M2/M3 hardware achieves 40 to 60 tokens per second.
• High-End Desktop (RTX 4090): Yields over 200 tokens per second, making it ideal for high-throughput batch synthetic data generation or continuous fine-tuning evaluation loops.
• Integrated Graphics and Mini PCs: AMD APUs (like the Ryzen 7 7840HS or 8840U with Radeon 780M) running GGUF formats under llama.cpp comfortably achieve 25 to 40 tokens per second at Q4_K_M.

To quickly run the model locally, modern inference runtimes like Ollama, llama.cpp, or Hugging Face Transformers provide fast setup paths.

Using the official Hugging Face transformers library on an NVIDIA GPU:

1import torch
2from transformers import pipeline
3
4pipe = pipeline(
5    "text-generation",
6    model="tiiuae/Falcon3-3B-Base",
7    torch_dtype=torch.bfloat16,
8    device_map="auto"
9)
10
11# Base models require prompt completion patterns, not chat dialog
12prompt = "def calculate_fibonacci(n):\n    \"\"\"Return the nth Fibonacci number efficiently.\"\"\"\n"
13output = pipe(prompt, max_new_tokens=150, do_sample=False)
14print(output[0]['generated_text'])

For zero-setup desktop execution, practitioners commonly pull community GGUF conversions into Ollama or run them directly via llama.cpp using the command-line interface:

1# Example running with llama.cpp CLI
2./llama-cli -m falcon3-3b-base-q4_k_m.gguf -p "The primary components of a distributed database include:" -n 256

Falcon3-3B-Base operates in an active segment of the local LLM space. Understanding how it compares to alternative models helps determine the right fit for your deployment stack:

• Reasoning and Math: Falcon3-3B-Base holds a substantial advantage in internal benchmarks for multi-step logic and mathematics. On GSM8K (63.9 vs 26.7) and MATH Level-5 (9.4 vs 1.4), the Falcon3 architecture delivers higher raw accuracy.
• General Knowledge: Llama 3.2 3B demonstrates slightly stronger broad encyclopedic knowledge retrieval on standard MMLU benchmarks (56.1 vs 55.5).
• Ecosystem: Llama 3.2 has broader out-of-the-box toolchain integration and fine-tuning recipe availability across commercial providers, but Falcon3-3B-Base offers a superior starting checkpoint for math- and logic-heavy domain distillation.

• Language Support: Qwen 2.5 3B is trained on a broader set of Asian and global languages. Falcon3-3B-Base focuses specifically on English, French, Spanish, and Portuguese.
• Performance Tradeoffs: Qwen 2.5 3B leads on MMLU (65.6 vs 55.5) and marginally edges out GSM8K (69.0 vs 63.9). However, Falcon3-3B-Base achieves a higher score on the challenging GPQA zero-shot reasoning benchmark (31.2 vs 27.5) and MATH Level-5 (9.4 vs 8.4).
• Efficiency: Falcon3-3B-Base uses 22 decoder layers compared to Qwen 2.5 3B's 36 layers. The shallower architecture of Falcon3-3B-Base translates to fewer sequential operations per token, which can yield lower per-token latency on bandwidth-constrained consumer hardware.