Unsloth

Unsloth is a Python library maintained by Unsloth AI that rewrites the computational bottlenecks of fine-tuning open models to deliver dramatically faster training with far lower GPU memory consumption. Originally released in 2023 under the Apache 2.0 license, it has grown to 67,474 GitHub stars and over 2.3 million monthly PyPI downloads. The library targets practitioners who need to adapt large language models (and increasingly vision, audio, and embedding models) on a single NVIDIA GPU — whether a consumer card like an RTX 3090 or a cloud instance with a single A100.

While Unsloth is best known as a fine-tuning accelerator, the project has expanded into local inference via Unsloth Studio, a web UI that runs models entirely offline on Mac, Windows, and Linux. For engineers evaluating it as an inference engine, the honest assessment is that Unsloth’s strength remains in training: you use Unsloth to fine-tune a model, then export it (to GGUF, safetensors, or LoRA adapters) and serve it with a dedicated inference engine like vLLM, Ollama, or llama.cpp. The library does include a Python API for running models locally, but it is not designed for production-scale serving with continuous batching or multi-GPU inference.

The team behind Unsloth collaborates directly with the developers of Llama, Mistral, Qwen, Gemma, and Phi, and has contributed critical bug fixes that improved model accuracy. This engineering pedigree shows in the kernel-level optimizations that make Unsloth faster than the standard Hugging Face Transformers + PEFT training stack without sacrificing model quality.

Unsloth patches the core operations of the PyTorch training loop — attention kernels, linear layers, and activation functions — with hand-optimized Triton and CUDA kernels. When you load a model through Unsloth’s FastLanguageModel class, it replaces the standard modules with faster, memory-frugal equivalents while preserving the original model’s outputs. You then attach a LoRA adapter (or use full fine-tuning, FP8, 4-bit QLoRA, etc.) and pass the model to a standard Hugging Face Trainer.

The minimal workflow:

1pip install unsloth

1from unsloth import FastLanguageModel, is_bfloat16_supported
2import torch
3
4model, tokenizer = FastLanguageModel.from_pretrained(
5    model_name = "unsloth/Meta-Llama-3.1-8B-bnb-4bit",
6    max_seq_length = 2048,
7    dtype = None,
8    load_in_4bit = True,
9)
10
11# Add LoRA adapters
12model = FastLanguageModel.get_peft_model(
13    model,
14    r = 16,
15    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"],
16    lora_alpha = 16,
17    lora_dropout = 0,
18    bias = "none",
19    use_gradient_checkpointing = "unsloth",
20)

From there, training proceeds with the standard Trainer API. Unsloth’s custom gradient checkpointing (use_gradient_checkpointing="unsloth") recomputes fewer activations than the standard setting, further reducing memory. The result: a Llama 3.1 8B model fine-tunes in 4-bit on a single 24GB GPU — something that requires a multi-GPU cluster with the vanilla stack.

For local inference, Unsloth Studio provides a graphical interface to download and run models (GGUF, safetensors, LoRA adapters) with tool calling, web search, and code execution. You can also use the Python API: after fine-tuning, call model.generate() or export to GGUF with model.save_pretrained_gguf(). The library exposes an OpenAI-compatible API endpoint when running Unsloth Studio, but this is intended for single-user local use, not production traffic.

Unsloth is built for NVIDIA GPUs. It requires a CUDA-capable GPU with at least 4GB of VRAM for the smallest quantized models; 24GB supports fine-tuning 8B models in 4-bit, and 48GB handles 70B-class models. The library does not run on CPU, AMD ROCm, or Apple Silicon for training — however, Unsloth Studio can run inference on Apple Silicon via Metal acceleration.

The core claim: fine-tuning runs approximately 2x faster and uses up to 70% less GPU memory than the standard Hugging Face Transformers + bitsandbytes stack, with no accuracy degradation. This is achieved through:

• Quantized training in 4-bit and FP8 — the compute is done in lower precision while the optimizer state remains in higher precision, keeping convergence intact.
• Custom attention kernels — fused and tiled implementations that reduce memory reads/writes.
• Selective gradient checkpointing — the “unsloth” flavor reuses computation only where necessary, cutting activation memory by 30-50% over standard checkpointing.

For inference, Unsloth’s Dynamic 2.0 quantization (GGUF format) shows a 46% speedup on GPU and Apple Silicon compared to standard k-quant methods — hitting 127 t/s on an RTX 3090 for a 4-bit Qwen3.6-35B-A3B model at 10K context. However, on CPU-only systems, the same quantizations run roughly 30% slower than non-Unsloth equivalents, so practitioners serving on CPU should benchmark before switching.

Memory use depends on model size and quantization. A 4-bit 35B model requires about 23GB total for inference. Fine-tuning an 8B model in 4-bit with LoRA fits comfortably in 24GB VRAM. Multi-GPU training is supported but the team describes the current implementation as basic, with a more robust version in development.

Faster training. Unsloth’s rewritten kernels cut fine-tuning time by roughly half compared to the default training stack. This does not require changing the model architecture or sacrificing final accuracy.

Lower memory use. 4-bit QLoRA training fits large models onto smaller GPUs. A single RTX 4090 can fine-tune a 70B model in 4-bit with LoRA — impossible with the standard stack.

Ready-made notebooks. Free Google Colab and Kaggle notebooks let you start fine-tuning in minutes without any local setup. These cover popular models (Llama, Mistral, Qwen, Gemma, Phi) and common tasks (instruction tuning, chat, classification).

Quantization support. Unsloth supports 4-bit, 8-bit, FP8, and its own Dynamic 2.0 GGUF quantization for inference. The GGUF export path integrates directly with llama.cpp and vLLM.

Export to production formats. After fine-tuning, you can save the model as 16-bit safetensors or GGUF for immediate deployment with Ollama, vLLM, or TensorRT-LLM.

500+ model support. The library covers text, vision, audio, and embedding models, with continuous updates as new open models are released.

Custom fine-tuning on a single GPU. The dominant use case: an engineer takes a base model like Llama 3.1 8B, Qwen 2.5 7B, or Gemma 4 12B, adds domain-specific data (legal documents, internal chat logs, medical transcripts), and produces a usable fine-tuned model in hours on a single RTX 3090 or A10. Without Unsloth, this would typically require a multi-GPU cluster.

Budget-constrained fine-tuning. Teams that cannot rent A100 80GB nodes use Unsloth to train on cheaper instances or even consumer hardware. This is especially common for startups and academic researchers.

Rapid iteration during R&D. Because each training run completes in hours instead of days, practitioners can test hyperparameters, data mixes, and LoRA ranks quickly.

Local inference for development and testing. Unsloth Studio provides a zero-setup way to run fine-tuned models locally, verify outputs, and compare side-by-side. This is useful for debugging before deploying to production.

Poor fit: high-throughput production serving. Unsloth is not designed for serving. It lacks continuous batching, request queuing, API authorization, and horizontal scaling. Use it to fine-tune, then serve with vLLM, SGLang, or Ollama.

1. Install.

1   pip install unsloth

For Linux/WSL, you can also run curl -fsSL https://unsloth.ai/install.sh | sh to get Unsloth Studio.

1. Run a free notebook.

Open a Google Colab notebook from the Unsloth documentation (docs.unsloth.ai) and run the “LLaMA 3.1 8B 4-bit” notebook. It downloads the model, loads a dataset, and starts fine-tuning in under 60 seconds.

1. Fine-tune on your own data.

Replace the dataset with your own JSON or CSV. The library auto-creates training datasets from PDF, DOCX, and CSV files via the Data Recipes tool.

1. Export the model.

After training, call model.save_pretrained_gguf("my_model.gguf") to get a file ready for llama.cpp or Ollama.

1. Run inference locally.

Launch Unsloth Studio (unsloth studio) and load the exported GGUF or LoRA adapter. You get an OpenAI-compatible API endpoint for testing.

The full documentation is at [docs.unsloth.ai](https://docs.unsloth.ai). The community (Discord, Reddit r/unsloth, GitHub) is active and responsive — over 230 contributors have helped shape the library.

Unsloth vs. Hugging Face Transformers + PEFT (bitsandbytes).

This is the direct comparison for fine-tuning. Unsloth is roughly 2x faster and uses 70% less memory for 4-bit LoRA training, with no accuracy loss. The trade-off: you are tied to Unsloth’s custom kernels, which only support NVIDIA GPUs and require occasional updates to track upstream model changes. For teams that already use Hugging Face and need a drop-in upgrade, Unsloth is a clear choice.

Unsloth vs. vLLM / Ollama (for inference).

These are not direct competitors. Unsloth is a fine-tuning tool that can export models for inference engines. If your goal is to serve a model at scale, stick with vLLM (for high throughput with continuous batching) or Ollama (for local dev). Unsloth Studio is a convenient local runner but not production-grade.

Unsloth vs. Axolotl.

Axolotl is another fine-tuning framework with broader hardware support (AMD, Intel) and more configuration options. Unsloth is simpler to set up and faster on NVIDIA GPUs, but less flexible for complex training schedules. Choose Unsloth if you want the fastest path to a fine-tuned model on one GPU; choose Axolotl if you need to target non-NVIDIA hardware or want fine-grained control.

When to choose Unsloth: you have an NVIDIA GPU, you want to fine-tune an open model in hours on a single card, and you prioritize speed and memory efficiency over configuration surface. When to choose something else: you need to serve at scale, you have AMD or CPU-only hardware, or you require a production-grade inference stack.