LLaMA-Factory

LLaMA-Factory is a broad fine-tuning toolkit, not a dedicated inference engine. Maintained by hiyouga, it is designed to let practitioners fine-tune over a hundred open language and vision-language models with minimal code. First released in 2023 under the Apache 2.0 license, it has quickly become one of the most popular open-source training frameworks, evidenced by 72,596 GitHub stars and 283 contributors. Its primary language is Python, and it is installed via pip install llamafactory.

The core problem LLaMA-Factory solves is fragmentation in the fine-tuning ecosystem. Instead of wiring together separate tools for data loading, LoRA, quantization, and experiment tracking, it provides a single platform that covers the full training workflow. It offers both a command-line interface and a web UI, making it accessible to engineers who want to script training pipelines and to those who prefer a no-code approach. While it is not a production inference server, it includes support for serving fine-tuned models via an OpenAI-compatible API, using backends like vLLM and SGLang. This positions LLaMA-Factory as a training-first platform that can also handle light serving for testing and evaluation.

LLaMA-Factory operates as a Python library with a modular architecture. You load a base model from Hugging Face or local files, define your training method (LoRA, QLoRA, full fine-tuning, or preference tuning), and specify a dataset in a supported format. Training runs on NVIDIA GPUs with optional multi-GPU scaling via DeepSpeed or FSDP.

The web UI (LlamaBoard) runs in a browser and lets you configure training parameters, upload data, and start runs without writing a single line of code. For scripted workflows, you use the llamafactory-cli command with YAML or JSON configuration files. After training, you can merge LoRA adapters back into the base model, quantize the result, and export it.

For inference, LLaMA-Factory can spin up an OpenAI-compatible API server using one of four backends: Hugging Face Transformers (for general inference and reward models), vLLM (for high-throughput serving with PagedAttention), SGLang (for distributed serving and advanced KV caching), or KTransformers (for CPU+GPU heterogeneous inference). You select the backend via the infer_backend parameter. The API is compatible with the OpenAI chat completions endpoint, so existing clients can point to it with minimal changes.

LLaMA-Factory is built primarily for NVIDIA GPUs. It supports single GPU, multi-GPU, and multi-node training through DeepSpeed, FSDP, and Ray. Confirmed capabilities include NVIDIA GPU, multi-GPU, and quantization.

Training performance depends on the method: LoRA and QLoRA are efficient for single GPU setups, while full fine-tuning requires multiple high-memory GPUs. The framework integrates FlashAttention-2 and Unsloth for acceleration, and supports a wide range of quantization techniques (AQLM, AWQ, GPTQ, bitsandbytes, HQQ, EETQ) to reduce memory footprint.

For inference, using the vLLM backend gives you continuous batching and PagedAttention, which are critical for high-throughput serving. The SGLang backend offers advanced KV caching for distributed deployments. However, LLaMA-Factory is not optimized for inference latency in the same way as a dedicated engine like TensorRT-LLM or TGI. Its inference capabilities are best used for testing, evaluation, and low-traffic serving.

Hardware support beyond NVIDIA is limited. The documentation mentions NPU (Neural Processing Unit) support for some Chinese hardware, but AMD GPUs and Apple Silicon are not confirmed. CPU-only inference is possible via the Hugging Face backend but will be slow.

Wide model coverage. LLaMA-Factory supports LLaMA, Mistral, Qwen, Yi, Gemma, Baichuan, ChatGLM, Phi, and many others, including vision-language models like LLaVA. This breadth means you can standardize on a single training tool across different model families.

Training web UI. The LlamaBoard interface lets you upload datasets, configure hyperparameters, monitor training with TensorBoard or WandB, and start runs from a browser. This is particularly valuable for teams where not everyone is comfortable writing training scripts.

Multiple tuning methods. You can run LoRA, QLoRA, full fine-tuning, and preference tuning (DPO, PPO, KTO, ORPO) within the same framework. This includes advanced optimizers like GaLore, BAdam, DoRA, and PiSSA. The ability to compare methods on the same model and data is a practical advantage.

OpenAI-compatible API. After training, you can serve the model with a standard chat completions endpoint. This makes it easy to integrate into existing applications built for OpenAI.

Quantization support. From 2-bit to 8-bit quantization using multiple algorithms, including GPTQ, AWQ, and bitsandbytes. This allows fine-tuning and serving large models on consumer GPUs.

No-code fine-tuning. Teams that need to adapt a model to a specific domain but lack deep ML engineering resources can use the web UI to upload data and train without writing code. This is common in business analytics, legal document processing, and customer support.

Custom model on your data. Engineers who have proprietary datasets (e.g., internal documentation, codebases, or chat logs) can fine-tune a base model to improve task-specific performance. LLaMA-Factory’s support for many model families means you can pick the best base for your data size and hardware.

Comparing tuning methods. Researchers and practitioners often need to evaluate whether LoRA, full fine-tuning, or a preference method yields better results for a given task. LLaMA-Factory makes it straightforward to run multiple experiments from the same codebase.

Light production serving. While not a dedicated inference engine, LLaMA-Factory can serve fine-tuned models for internal demos, testing, or low-traffic applications using the vLLM backend. Teams that already use LLaMA-Factory for training can reuse the same environment for deployment without adding another tool.

Poor fit for: High-throughput production serving at scale, latency-sensitive applications, or deployment on non-NVIDIA hardware. For those cases, export the model to a dedicated engine like vLLM, TensorRT-LLM, or Ollama.

Install with pip:

1pip install llamafactory

The fastest path to a running model is to use the web UI:

1llamafactory-cli webui

This opens a browser where you can select a base model (e.g., LLaMA-3-8B), choose a dataset, set LoRA parameters, and start training. After training, you can merge and export the model, then serve it:

1llamafactory-cli api --model_name_or_path ./my_model --infer_backend vllm

You need an NVIDIA GPU with at least 8 GB VRAM for small models with LoRA. For larger models or full fine-tuning, you will need multiple GPUs or higher memory. The official documentation is at [llamafactory.readthedocs.io](https://llamafactory.readthedocs.io/). The GitHub repository (hiyouga/LLaMA-Factory) has examples and community discussions on Discord and WeChat.

LLaMA-Factory vs. Hugging Face Transformers + PEFT. Using HF Transformers with the PEFT library gives you similar LoRA and QLoRA capabilities, but you must assemble the pipeline yourself: data formatting, trainer configuration, quantization, and experiment tracking. LLaMA-Factory bundles all of that into a single interface with a UI. It also supports more tuning methods (preference tuning, GaLore) out of the box. Choose LLaMA-Factory if you want a unified training platform with less boilerplate. Choose raw Transformers if you need maximum flexibility or are already deeply integrated into the HF ecosystem.

LLaMA-Factory vs. vLLM (for inference). vLLM is a dedicated inference engine optimized for throughput and latency. LLaMA-Factory can use vLLM as a backend, but it adds overhead and is not designed for high-scale deployment. If your goal is to serve a fine-tuned model in production, export it to vLLM directly. If you need to quickly test a model after training, LLaMA-Factory’s built-in API is convenient.

When to choose LLaMA-Factory: You need to fine-tune multiple models and methods, want a no-code option for team members, or value having training and light inference in one tool. When to avoid it: You need pure inference performance, non-NVIDIA hardware, or a minimal footprint.