LMQL

LMQL is a query language and runtime for structured generation from large language models, developed and maintained by the SRI Lab at ETH Zurich. First released in 2022 under an Apache 2.0 license, it occupies a distinct category in the inference engine landscape: it is not a serving engine like vLLM or TGI, nor a model runner like Ollama or llama.cpp. LMQL is a programming language that compiles prompts with embedded constraints into token-level operations during generation.

The core insight behind LMQL is that prompting is a form of querying. Instead of writing prompt templates in one place and validation logic in another, LMQL lets you express both in a single, unified syntax. The research team behind it published the foundational paper "Prompting Is Programming" at ACM POPL 2023, and many ideas that LMQL pioneered — particularly token-level constraint enforcement during generation — have since been adopted by newer structured generation tools.

With 4,190 GitHub stars and 41 contributors, LMQL has a modest but focused community. Its 1,502 monthly PyPI downloads suggest it is used primarily by researchers and engineers who need precise control over model output, rather than by teams running high-throughput production pipelines.

LMQL extends Python with a query syntax. You write standard Python functions decorated with @lmql.query, and top-level strings within those functions become prompt statements. Template variables in square brackets are automatically completed by the model, and a where clause specifies constraints on the generation.

A minimal LMQL program looks like this:

1@lmql.query
2def greet():
3    '''lmql
4    "Greet LMQL: [GREETING]" where len(TOKENS(GREETING)) < 25
5    '''

The runtime parses this into a generation plan. Constraints in the where clause are compiled into token masks that are applied eagerly during decoding — not as post-processing. This means the model never generates tokens that would violate the constraint. The runtime supports multiple decoding strategies including greedy, beam search, and sampling, and it can execute control flow (loops, conditionals) that interleaves LLM calls with program logic.

To serve models, LMQL supports multiple backends. You can run local models through Hugging Face Transformers or llama.cpp, or use hosted APIs including OpenAI and Anthropic. There is no standalone server daemon; you run LMQL queries from Python scripts or use the provided Playground IDE (a web-based environment). LMQL does not expose an OpenAI-compatible API endpoint, so it cannot serve as a drop-in replacement for existing serving infrastructure.

LMQL runs on NVIDIA GPUs and CPU. It supports quantized models through the Hugging Face Transformers backend, including 4-bit and 8-bit quantization via bitsandbytes. On CPU, it can use llama.cpp as a backend, which provides its own quantization formats.

The confirmed capabilities are:

• NVIDIA GPU: Full support with CUDA kernels
• CPU inference: Supported via llama.cpp backend
• Structured output: Enforced at the token level during generation
• Streaming: Token-by-token streaming output

Performance is where LMQL shows its limitations. Because constraints must be evaluated and applied at every decoding step, the runtime incurs overhead compared to unconstrained generation. A GitHub issue from October 2023 documents a user running a 13B LLaMA 2 model on an A100 80GB with 4-bit quantization, achieving roughly 50 samples in 3 hours with beam search. The LMQL maintainers acknowledged this and recommended tuning the chunksize parameter (the number of tokens generated speculatively in one LLM call), suggesting values of 1-4 for better throughput with beam-based decoders.

LMQL does not support continuous batching, paged attention, or Tensor Parallelism. It runs a single query at a time. For teams comparing it to engines like vLLM or TensorRT-LLM, the throughput difference will be substantial on identical hardware. LMQL trades raw throughput for the ability to enforce complex constraints during generation.

Prompt query language: LMQL treats prompts as programs. You can use variables, control flow, and constraints within a single query string. This eliminates the split between prompt templates and post-processing validation.

Built-in constraints: Constraints operate at the token level during generation. You can limit output length, enforce exact string matches, restrict to a set of choices, and combine multiple conditions. The where clause supports functions like len(), STOPS_AT(), and TOKENS() to express constraints on character length or token count.

Multiple backends: LMQL can run against local Hugging Face models, llama.cpp, or hosted APIs (OpenAI, Anthropic). This makes it useful for development workflows where you prototype with a hosted model and later switch to a local one.

Typed variables: LMQL supports typed output variables, so you can declare that a generated value must be an integer, a list of options, or a JSON object. The runtime enforces these types during generation, not after.

Multistep prompting: The query language supports branching logic. You can conditionally prompt based on previous model outputs, loop over generated content, and compose multiple LLM calls within a single query function.

Research and teaching: LMQL is strongest as a research tool. Its academic origins and the ability to precisely control generation make it suitable for studying how constraints shape model behavior. The Playground IDE lowers the barrier for teaching structured prompting concepts.

Constrained generation: When you need to guarantee that model output adheres to a strict format or vocabulary, LMQL's token-level enforcement is more reliable than post-processing. This matters for tasks like generating structured data, classification into fixed categories, or producing outputs within strict length bounds.

Multi-step reasoning pipelines: LMQL's ability to interleave program logic with LLM calls makes it useful for chains of reasoning where each step depends on previous outputs. You can express these as a single query function rather than orchestrating multiple API calls from external code.

Poor fit for high-throughput serving: LMQL is not designed for production serving. Teams running latency-sensitive or high-throughput workloads should look at vLLM, SGLang, or TensorRT-LLM. LMQL's per-query overhead and lack of batching make it unsuitable for serving at scale.

Install LMQL with pip:

1pip install lmql

The smallest path to running a model locally requires a Hugging Face model and a GPU. For CPU inference, install the llama.cpp backend:

1pip install lmql[llama-cpp]

A minimal working query:

1import lmql
2
3@lmql.query
4def hello():
5    '''lmql
6    "Say hello in exactly five words: [RESPONSE]" where len(TOKENS(RESPONSE)) == 5
7    '''
8
9print(hello())

You need a GPU with sufficient VRAM for the model you choose, or a quantized model file for CPU inference. The documentation at lmql.ai/docs covers installation, the Playground IDE, and the full constraint API. The Discord community (linked from the GitHub repo) is active for troubleshooting.

LMQL vs. SGLang: SGLang is the closest direct competitor. Both provide a language for structured generation with constraint enforcement. SGLang has a more active development cadence, supports continuous batching and RadixAttention for prefix caching, and exposes an OpenAI-compatible API. Choose LMQL if you want a Python-native syntax with tighter integration into research workflows. Choose SGLang if you need production throughput or API compatibility.

LMQL vs. guidance: Microsoft's guidance library (now guidance-ai) offers similar constraint-based generation but with a different syntax (using gen() and select() functions rather than LMQL's query language). Guidance has broader model backend support and more active maintenance. LMQL's academic lineage and published formalism give it an edge for research contexts where reproducibility and theoretical grounding matter.

LMQL vs. vLLM: These are not direct competitors. vLLM is a serving engine optimized for throughput with PagedAttention and continuous batching. LMQL is a query language for constrained generation. If you need to serve models at scale, use vLLM. If you need to enforce constraints during generation, use LMQL on top of a serving backend, or consider SGLang which combines both capabilities.