MLX

MLX is Apple’s array framework for running and training machine learning models on Apple Silicon. It was released in 2023 by Apple’s machine learning research team and has quickly become the go-to engine for getting maximum performance out of M-series Macs. With 27,295 GitHub stars, 262 contributors, and nearly 1.6 million PyPI downloads per month, it has the traction of a serious tool that engineers actually use.

What sets MLX apart from every other inference engine in this directory is its hardware dependency. This is not a general-purpose framework. It runs exclusively on Apple Silicon — no NVIDIA, no AMD, no Intel. That sounds limiting until you see what it can do on the hardware it targets. MLX takes advantage of Apple’s unified memory architecture, where the CPU and GPU share a single pool of RAM. The result: no data copies between host and device, no PCIe bottlenecks, and the ability to load surprisingly large models on a laptop.

MLX is both a library and a serving engine. The core is a NumPy-like array framework with automatic differentiation, lazy evaluation, and composable function transformations. The mlx-lm package adds the serving layer: an OpenAI-compatible API server and a simple model runner. You use it as a Python library for fine-tuning or as a local server for inference.

You get started by installing the mlx-lm package:

pip install mlx-lm

That’s the only dependency. The mlx-lm package bundles the core MLX framework along with model loading, tokenization, and inference utilities. To run a model, you either use the Python API:

1from mlx_lm import load, generate
2model, tokenizer = load("mlx-community/Llama-3.2-3B-Instruct-4bit")
3response = generate(model, tokenizer, "What is MLX?", verbose=True)

Or you start an OpenAI-compatible server:

mlx_lm.server --model mlx-community/Llama-3.2-3B-Instruct-4bit

The server exposes a standard /v1/chat/completions endpoint. You can drop it in behind any OpenAI SDK client without changing a line of code. It supports streaming, so you can incrementally return tokens to the client.

MLX loads models from a growing repository of mlx-community checkpoints on Hugging Face. These are quantized to 4-bit or 8-bit by default, which is the main way MLX makes large models fit on consumer hardware. Under the hood, MLX uses lazy computation: arrays are materialized only when needed, and operations are compiled to Metal shaders for the GPU.

MLX runs only on Apple Silicon. That means any M-series chip: M1, M2, M3, M4, and the new M5 generation. There is no CPU-only fallback and no support for discrete GPUs.

The unified memory model is the key performance lever. On a traditional NVIDIA GPU, data must be copied over PCIe between system RAM and VRAM. On Apple Silicon, the CPU and GPU see the same memory pool. An operation that requires both the CPU (tokenization, sampling) and the GPU (matrix multiplications) does not pay a transfer cost. This is why MLX can run a 70B parameter model at 4-bit quantization on a Mac with 192 GB of unified memory — the entire model fits in shared RAM, and the GPU can access it directly.

Quantization is supported at 4-bit and 8-bit, using group-wise or per-channel schemes. The mlx-lm package includes built-in conversion scripts to take a model from Hugging Face and quantize it to MLX format. Streaming is also supported natively: the server streams tokens via Server-Sent Events, and the Python generator yields tokens one at a time.

What MLX does not do is continuous batching in the style of vLLM or TensorRT-LLM. There is no paged KV cache, no dynamic request scheduling, and no multi-GPU support across multiple Macs. A single inference request processing pipeline is what you get. That is fine for single-user or low-concurrency workloads, but if you need high-throughput production serving, MLX is not there yet. Third-party projects like vllm-mlx are attempting to add continuous batching, but they are not part of the official release.

For single-stream inference — one user generating text — MLX delivers excellent latency. Independent benchmarks show it outperforms llama.cpp by 21% to 87% on models ranging from 0.6B to 30B parameters on an M4 Max, reaching up to 525 tokens per second on smaller models. This makes sense: MLX is built specifically for Apple Silicon, while llama.cpp is a portable C++ engine that must abstract over many architectures.

• Unified memory: The defining feature. Data lives in shared memory, so the GPU can access tensors without copies. This is not a minor optimization; it is the reason MLX can load models that would require a datacenter GPU on any other platform.

• mlx-lm runner: A one-line install gets you both a Python API for interactive use and a CLI server that exposes an OpenAI-compatible endpoint. You can serve a moderately sized model to a handful of clients without writing any glue code.

• On-device fine-tuning: Because MLX supports automatic differentiation and lazy computation, you can fine-tune models entirely on a Mac. The mlx.nn and mlx.optimizers APIs closely follow PyTorch. This is a practical way to adapt a small model (e.g., a 3B or 7B) to proprietary data without renting a cloud GPU.

• Quantization: Load models at 4-bit or 8-bit precision. This reduces memory usage by 4x to 8x, making it feasible to run 70B models on high-end M3/M4 Max machines.

• Streaming: The server supports token-level streaming, crucial for chat applications.

• OpenAI-compatible API: Drop-in replacement for openai.ChatCompletion. No need to change your client code if you already target the OpenAI API.

Running models on a Mac. This is the primary use case. If you own an M-series Mac and want to run open models locally, MLX is the fastest option available. Use it for personal assistants, code completion, or document analysis that stays on your machine.

On-device experiments. Researchers and engineers who need to prototype with private or sensitive data can do so entirely offline. The unified memory lets you iterate quickly without worrying about PCIe copies or cloud costs.

Local fine-tuning. MLX supports LoRA and full fine-tuning. You can adapt a base model to your domain on a single Mac, then re-quantize and serve it with the same mlx-lm server. This is cost-effective for small-scale projects, internal tools, or teams without GPU budget.

Edge cases where MLX is a poor fit: Any deployment that requires NVIDIA GPUs, high concurrency (hundreds of simultaneous users), or CPU-only inference. Also, teams that need a broader model catalogue (GGUF has thousands of community conversions; MLX’s mlx-community is smaller, though growing).

1. Install mlx-lm:

pip install mlx-lm

1. Download a pre-quantized model from the MLX community on Hugging Face. For a quick test, use mlx-community/Llama-3.2-3B-Instruct-4bit.

1. Start the server:

mlx_lm.server --model mlx-community/Llama-3.2-3B-Instruct-4bit

1. From a separate terminal, send a chat request:

1   from openai import OpenAI
2   client = OpenAI(base_url="http://localhost:8080/v1", api_key="none")
3   response = client.chat.completions.create(model="default", messages=[{"role":"user","content":"Hello"}])
4   print(response.choices[0].message.content)

That is the full path from install to first token. No configuration files, no environment variables, no GPU driver setup. The official documentation lives at ml-explore.github.io/mlx, and the GitHub repository has extensive examples for fine-tuning, custom models, and multi-device experiments.

MLX vs llama.cpp. Both run on Apple Silicon. llama.cpp is hardware-agnostic (NVIDIA, AMD, Intel, ARM) and supports the massive GGUF ecosystem with thousands of community-quantized models. MLX is faster on Mac hardware (often 2x or more on pure generation throughput) because it uses the Metal API directly and the unified memory model. Choose MLX if you primarily run on a Mac and want maximum speed. Choose llama.cpp if you need to run on multiple platforms or want access to exotic model formats.

MLX vs Ollama. Ollama is a higher-level tool that bundles either llama.cpp or MLX as the backend depending on the model and platform. It provides a simpler CLI and a model library. If you want a single command to pull and run a model, use Ollama. If you need programmatic control, fine-tuning support, or an OpenAI-compatible server, use MLX directly.

MLX vs vLLM. vLLM is designed for production serving on NVIDIA GPUs. It offers continuous batching, paged KV cache, and multi-GPU scaling. MLX does not compete in that space. If you are deploying on NVIDIA GPUs with high request rates, use vLLM. If you are working on a Mac with modest concurrency requirements, MLX is the more natural fit.

Is MLX production ready? That depends on your definition. For single-user or small-team local inference, it is stable and fast. For serving hundreds of concurrent users, it lacks the throughput infrastructure of dedicated serving engines. Consider it a high-performance library that can function as a lightweight server, not a replacement for a full production inference stack.