Carnice-V2-27b

Our Take

A workable 27B-parameter dense language model from kai-os. A pragmatic middle-ground choice when you need open weights without a flagship-sized footprint.

Generated from this model’s benchmarks and ranking signals. Editor reviews refine it over time.

Carnice-V2-27b is a dense transformer model with 27B active parameters. Unlike mixture-of-experts architectures where only a subset of parameters route per token, dense models use all 27B parameters on every forward pass. The tradeoff is straightforward: you get consistent, predictable inference speeds at the cost of higher VRAM usage relative to MoE models of similar total parameter count (e.g., Mixtral 8x7B has ~47B total but only ~13B active).

Key architectural specs:

• Context length: 262,144 tokens (262K)
• Architecture variant: Qwen3.5 / Qwen3.6 hybrid attention/SSM layers (converted as qwen35 in GGUF)
• Precision: BF16 merged safetensors (51GB) and GGUF quantizations from 9.4GB to 27GB
• License: Apache 2.0

The 262K context window is a significant capability for agentic workflows that require long conversation histories, multi-turn tool calls, or document-level context. In practice, most local hardware will context-length limited by VRAM, but the model supports full-length context if you have the memory (e.g., 48GB+ GPUs or CPU offload).

A note on the BF16 loading fix: the original merged weights had an extra Unsloth wrapper prefix that caused Hugging Face Transformers to treat real weights as “unexpected”. The repo now has corrected safetensors keys, so AutoModelForImageTextToText (or AutoModelForCausalLM for text-only) loads cleanly. GGUF exports were never affected.

Carnice-V2-27b is optimized for structured, tool-oriented agent interactions. Its primary capabilities — chat, code, reasoning, function-calling, instruction-following — map directly to these use cases:

• Function-calling agents – If your agent framework expects the model to return JSON or XML tool calls (e.g., Hermes-style <functioncall> tags), Carnice-V2-27b was built exactly for that. The SFT data heavily weights properly formatted tool invocations with required parameters.
• Instruction-heavy tasks – IFEval benchmark improvements (90% strict prompt accuracy vs 85% base) indicate the model reliably follows multi-step formatting instructions, even when constraints are nested.
• Code generation and reasoning – The base Qwen3.6-27B is already strong at code and math. The SFT preserved those capabilities while improving adherence to output format rules — useful for code agents that must produce functions with specific signatures or documentation.
• Long-context reasoning – 262K context allows processing entire codebases, long chat logs, or multi-document queries, though VRAM will be the practical limiter.

It is not a multimodal model — the image-text-to-text tag on Hugging Face is misleading; the weights are Qwen3.6-27B base (text-only) and the pipeline tag inherited from the base repo. Use it for text-only agent applications.

For a 16GB GPU (RTX 4060 Ti 16GB, RTX 4080 Super): start with IQ2_M. This quantization uses an imatrix calibration pass tuned for Carnice/Hermes data, so it retains more quality than generic Q2_K. If your runtime (e.g., older llama.cpp) doesn’t support IQ quants, fall back to Q2_K (10GB). You can run with -c 8192 comfortably.

For a 24GB GPU (RTX 4090, 3090): Q4_K_M is the sweet spot — 16GB file, fits entirely on VRAM with 8K context. You can push to Q5_K_M (18GB) if you reduce context to 4K-6K or enable partial CPU offload for the KV cache.

For multi-GPU or 48GB+: Q8_0 or BF16 GGUF offers near-lossless quality.

These are rough estimates on an RTX 4090 (24GB) with q4_K_M quantization, 8K context:

• Batch size 1, no prompt processing: ~30–40 t/s for generation
• Prompt processing (prefill): ~150–200 t/s (depends on prompt length)
• Lower quantizations (IQ2_M) can yield 45–55 t/s generation

Actual throughput varies by backend (llama.cpp, Ollama, ExLlamaV2), prompt length, and context size. Longer contexts will slow down due to KV cache overhead.

The quickest way: Ollama supports this model via the GGUF quantizations at kai-os/Carnice-V2-27b-GGUF.

If you need function-calling support, ensure your Ollama version is recent enough to handle the qwen35 GGUF architecture (hybrid attention/SSM). For direct Python usage, use Hugging Face Transformers with AutoModelForCausalLM and the corrected safetensors.

vs. Qwen3.6-27B (base)

The base model is a capable general-purpose agent, but Carnice-V2-27b shows clear improvements in instruction-following (IFEval +5% strict) and lower assistant-token perplexity (1.513 vs 1.835). If you’re already using Qwen3.6-27B for agent workloads, expect fewer formatting errors and better adherence to tool-calling schemas. The tradeoff: the SFT is specialized — for pure code completion or open-ended chat, the base may feel more “creative” or less constrained.

vs. NousResearch Hermes 2 34B

Hermes 2 34B (based on Yi-34B) is a 34B dense model with similar agentic focus. Carnice-V2-27b is smaller (27B vs 34B) and runs on lower VRAM (Q4_K_M 16GB vs ~20GB for Hermes 2 34B Q4_K_M). Benchmarks are not directly comparable, but the held-out loss metric suggests Carnice-V2-27b is better regularized for agent traces. If you’re constrained to 16GB GPUs, Carnice is a better fit; if you have 24GB+ and want maximum capability, Hermes 2 34B is worth trying side-by-side.

vs. Mistral Small 3.1 24B

A 24B dense model with 128K context. Carnice-V2-27b has more parameters and a larger context window (262K vs 128K). For pure instruction-following, Carnice’s Hermes-focused training likely outperforms Mistral’s general SFT. However, Mistral Small 3.1 is more multilingual and works better out-of-box for general chat. Choose Carnice if your primary use case is structured tool use; choose Mistral Small 3.1 if you need broader language coverage.