joey-1

joey-1 is an open-weight, 2B-parameter decision model released by Base Labs at Baseten under the permissive MIT license. Built as a specialized post-trained model from a Qwen3.5-2B dense base, joey-1 addresses a fundamental inefficiency in modern AI agent pipelines: using heavy, autoregressive text generation to make simple, discrete choices. Instead of generating streamable markdown or fragile JSON strings that downstream systems must parse, joey-1 processes input state and emits calibrated probability distributions over typed choices, yes/no checks, and rubric scores in a single forward pass.

This operational design dramatically reduces compute overhead. While standard large language models require iterative decoding loops, consuming memory bandwidth for every generated token, joey-1 functions as a fast classification and routing engine. When deployed locally or on bare-metal hardware, it handles workflow orchestration, triage, content moderation, and ticket routing at minimal latency. As an open-source local AI model with 2B parameters, joey-1 gives local AI engineers a drop-in replacement for the classification tasks that typically stall compound AI architectures.

For developers building local-first agent frameworks, joey-1 represents a shift from generative bloat to purpose-built inference. By offloading routing logic and boolean gates from large 70B parameter models to a dedicated 2B parameter decision model, local pipelines achieve lower overall latency, eliminate JSON formatting failures, and free up GPU memory for generative tasks.

joey-1 utilizes a dense transformer architecture derived directly from Qwen3.5-2B. Rather than relying on a Mixture-of-Experts (MoE) configuration, every one of its 2 billion parameters is active on every forward pass. In small-parameter regimes, dense models offer predictable memory access patterns and optimal cache locality, ensuring consistent execution speed without the routing overhead associated with MoE layers.

The defining architectural deviation in joey-1 is its extraction mechanism. In standard decoder-only models, the model generates output autoregressively, predicting the next token one position at a time until an end-of-sequence token is reached. joey-1 bypasses autoregressive decoding for decision outputs. It processes the prompt state and attached questions simultaneously, using specialized decision heads to produce confidence scores, discrete category logits, and calibrated probabilities in one forward compute operation.

Because the model evaluates multiple structured questions against a single shared state prefix, KV cache reuse is maximized. Local inference engines can cache the context prefix once and evaluate multiple decision branches with near-zero latency impact. This dense, single-pass design minimizes memory bandwidth consumption, which is the primary bottleneck for inference speed on modern accelerator hardware.

joey-1 is designed for high-throughput decision-making, policy checks, and workflow control where generating verbose text is counterproductive.

• Agent Routing and Workflow Branching: Rather than prompting an 8B or 70B model with strict JSON schemas to decide which tool an agent should invoke, joey-1 classifies queries across custom intent criteria.
• Triage and Ticket Escalation: Support engineering pipelines can run joey-1 locally to evaluate incoming user reports against categorical buckets (such as technical support vs. billing disputes) and score issue severity simultaneously.
• Rubric-Based Scoring: When evaluating data against qualitative standards, joey-1 scores states along ordered scales (for example, grading conversational tone, relevance, or factual clarity) while returning calibrated probabilities.
• Guardrails and Yes/No Filtering: Security and moderation pipelines can run joey-1 as a deterministic gatekeeper to screen inputs for policy violations before dispatching prompts to heavier generative backends.

Because the outputs are typed probabilities rather than parsed text strings, the model avoids hallucinated schemas, malformed JSON syntax, and regex parsing errors completely.

Deploying joey-1 locally is straightforward due to its compact 2B parameter footprint. Because inference does not require sustained autoregressive generation loops, hardware resource demands are light, allowing it to run smoothly on budget desktop GPUs, Apple Silicon laptops, or lightweight edge nodes.

The physical memory footprint for joey-1 depends on your execution precision:

• FP16 / BF16 (Unquantized): Requires approximately 4.5 GB to 5.0 GB of VRAM. This preserves complete numerical precision for logit scoring.
• Q8_0 (8-bit Quantization): Requires approximately 2.5 GB to 2.8 GB of VRAM. There is virtually zero observable loss in decision calibration compared to native FP16.
• Q4_K_M (4-bit Quantization): Requires approximately 1.5 GB to 1.8 GB of VRAM. This is the optimal configuration for edge hardware, low-spec laptops, and environments where GPU memory is prioritized for a larger co-located model.

When calculating joey-1 hardware requirements, remember to budget roughly 500 MB of additional memory for system context and runtime overhead, especially when processing long, multi-thousand-token prompt prefixes.

Understanding how to run a 2B model on consumer GPUs comes down to memory bandwidth and compute alignment:

• Consumer Desktop GPUs: The NVIDIA GeForce RTX 3060 (12GB) or RTX 4060 (8GB) represent the sweet spot for self-hosted production setups. Both cards feature more than enough VRAM to host joey-1 unquantized alongside full application runtimes, while maintaining sub-millisecond layer compute times. For high-volume multi-agent systems, the best GPU for joey-1 is an RTX 4090 or RTX 3090, which can easily host joey-1 in VRAM alongside an open-weight generative model like Llama-3-70B.
• Apple Silicon: On Apple hardware (M1 through M4, including base, Pro, and Max tiers), joey-1 runs comfortably within unified memory. Even an entry-level MacBook Air with 8 GB or 16 GB of RAM handles joey-1 without competing with macOS background tasks.
• CPU Inference: Because the model is small and outputs are generated in a single pass, modern x86 and ARM CPUs running AVX2/AVX-512 can execute joey-1 inference with practical latencies (sub-100 ms per decision), eliminating the hard requirement for dedicated accelerator silicon.

The best quantization for joey-1 in local production environments is Q8_0 or Q4_K_M. Because joey-1 relies heavily on calibrated output logit probabilities rather than qualitative prose generation, aggressive low-bit quantization (such as 2-bit or 3-bit formats) can alter the logit distribution and degrade confidence threshold accuracy. Q8_0 maintains near-identical logit calibration to full precision while halving the memory footprint.

Regarding joey-1 performance and throughput: because the model is designed to emit typed decisions rather than stream sentences, standard "joey-1 tokens per second" metrics for generative decode phases do not tell the whole story. Instead, performance is measured by prompt ingestion throughput and single-pass evaluation latency. On an RTX 4090 or modern workstation accelerator, joey-1 processes 2,000-token prompt prefixes and returns decision outputs in under 20 milliseconds, delivering hundreds of discrete routing operations per second under batched workloads.

The easiest path to run joey-1 locally is through runtimes supporting GGUF weights, such as llama.cpp or Ollama.

Once converted to GGUF format or pulled through community registries, you can instantiate the model using Ollama:

1ollama run joey-1

For custom Python pipelines requiring direct logit and classification access, running the model through Hugging Face transformers or serving engines like vLLM provides programmatic access to intermediate probability tensors:

1from transformers import AutoModelForSequenceClassification, AutoTokenizer
2import torch
3
4model_id = "baseten/joey-1"
5tokenizer = AutoTokenizer.from_pretrained(model_id)
6model = AutoModelForSequenceClassification.from_pretrained(
7    model_id, 
8    torch_dtype=torch.bfloat16, 
9    device_map="auto"
10)

Evaluating joey-1 vs Gemma-2-2B or joey-1 vs Qwen2.5-1.5B highlights fundamental differences in system design.

While joey-1 shares the architectural backbone of the Qwen family, base and instruction-tuned Qwen models are general autoregressive generators. To extract a routing decision from Qwen2.5-1.5B, you must prompt the model, wait for autoregressive token generation (often 10 to 50 tokens depending on formatting), and parse the output text or JSON. If the output malforms, the pipeline breaks. joey-1 trades open-ended text generation for immediate, deterministic decision heads. It executes in a fraction of the time, uses significantly fewer compute cycles, and guarantees valid categorical outputs.

Google's Gemma-2-2B is a potent small generative model that excels at reasoning, extraction, and lightweight chat. However, using Gemma-2-2B for classification tasks requires structural constraint frameworks like Outlines or JSON schema grammar masks. While grammar masking forces valid output syntax, it still suffers from autoregressive decoding overhead. joey-1 is vastly superior when your application exclusively needs classification probabilities, routing gates, or rubric grading, whereas Gemma-2-2B is the appropriate choice if you need the model to generate explanatory text justifying its choices.

Select joey-1 when architecting local multi-agent systems that need fast, reliable boolean checks, intent classification, or request triage. Use general-purpose 2B to 8B models only when free-form text generation and explanatory dialogue are strictly required.