Guidance

Guidance is a Python library for structured generation, maintained by Guidance AI (originally developed at Microsoft). It solves a specific problem: when you need a language model to produce output that fits a precise structure, conventional prompting wastes tokens and produces unreliable results. Guidance lets you define a template that mixes fixed text, logic, and generation, then constrains the model to fill only the blanks you specify.

This is not a serving engine. Guidance does not run a server, manage request queues, or expose an API endpoint. It is a library you import into your Python code and use to build generation pipelines. It competes with tools like LMQL, Outlines, and manual constrained decoding approaches. Where vLLM or TGI optimize throughput for raw text generation, Guidance optimizes for correctness and token efficiency in structured tasks.

With 21,519 GitHub stars, 84 contributors, and roughly 27,000 monthly PyPI downloads, Guidance has significant community traction. It is MIT-licensed and written in Python, first released in 2023.

Guidance operates through a Pythonic API where you define generation templates using context managers and operators. You load a model through one of several backends, then write a program that interleaves text, control flow, and generation calls.

The core abstraction is the guidance decorator and the gen() function. You mark positions in your template where the model should generate tokens, and Guidance handles the rest. The library manages the conversation state, applies constraints during decoding, and returns the completed output.

1from guidance import system, user, assistant, gen
2from guidance.models import Transformers
3
4lm = Transformers("microsoft/Phi-4-mini-instruct")
5
6with system():
7    lm += "You are a helpful assistant"
8
9with user():
10    lm += "What is the capital of France?"
11
12with assistant():
13    lm += gen(max_tokens=20)

Guidance supports multiple backends including Hugging Face Transformers, llama.cpp, and OpenAI API models. You choose the backend when you instantiate the model object. There is no server component. You run Guidance programs directly in your Python process, which means you manage model loading and memory yourself.

The library does not provide an OpenAI-compatible API. It is a programmatic interface, not a service.

Guidance runs on NVIDIA GPUs through the Transformers backend and on CPU through llama.cpp. Confirmed capabilities include NVIDIA GPU inference, CPU inference, structured output, and streaming.

Performance depends entirely on the backend you choose. Guidance itself adds minimal overhead. The primary performance benefit comes from constrained decoding: by restricting the model's output space, you generate fewer tokens and reduce the chance of retries. This can lower both latency and cost compared to unconstrained generation followed by validation.

The library supports "guidance acceleration," which maintains KV cache state across multiple generation calls within a single program. This avoids recomputing attention for the fixed parts of your template when you make sequential generation calls. For local models using the Transformers or llama.cpp backends, this can produce meaningful speedups.

Memory usage follows the backend's behavior. With Transformers, you load the full model into GPU memory. With llama.cpp, you can use quantized models to reduce memory footprint. Guidance does not implement paged attention, continuous batching, or multi-GPU sharding. It is not designed for high-throughput serving of many concurrent requests.

For throughput-sensitive workloads, Guidance is not the right tool. It is a single-process library. If you need to serve hundreds of requests per second, pair Guidance with a serving layer or use a dedicated inference engine for the generation portion.

Guided templates let you define a structure and let the model fill only the parts you mark. This is the core feature. You write a template with fixed text and gen() calls, and the model generates only the tokens needed to complete each blank. The rest of the template is not regenerated.

Constrained decoding restricts output to valid choices, patterns, or grammars. You can force the model to output only from a set of options, match a regex, or follow a context-free grammar. This eliminates the need for post-processing validation and retries. For JSON output, you can define the schema and Guidance will enforce it during generation.

Token efficiency is a direct consequence of constrained generation. The model produces only the tokens you need, not the structural tokens you already defined. This reduces token count, which lowers cost for API-based backends and reduces latency for local models.

Streaming is supported. You can receive tokens as they are generated, which is useful for real-time applications.

Reliable structured output is the primary use case. Teams use Guidance to generate JSON, code, or formatted text that must match a schema. The constrained decoding guarantees the output is valid, which eliminates parsing errors downstream.

Complex prompt logic benefits from Guidance's ability to mix conditionals, loops, and generation in a single template. You can write a program that branches based on previous model output, iterates over a list, or calls tools, all within the same generation flow.

Cost control is a practical concern for teams using API-based models. By generating only the variable parts of a template, Guidance reduces token consumption. For high-volume applications, this can meaningfully reduce API costs.

Local development and prototyping is a natural fit. Guidance's Python API makes it easy to experiment with structured generation patterns before deploying to production. The library works with small local models, which is useful for testing.

Guidance is a poor fit for high-throughput serving, multi-user applications, or any scenario where you need a standalone API server. It is also not designed for training or fine-tuning.

Install Guidance with pip:

1pip install guidance

The smallest meaningful path to a running model requires a backend. For local inference with a Hugging Face model:

1from guidance import gen, models
2
3lm = models.Transformers("microsoft/Phi-4-mini-instruct")
4lm += "The capital of France is " + gen(max_tokens=5)
5print(lm)

For CPU inference with llama.cpp, install the llama.cpp backend and use models.LlamaCpp instead.

You need a GPU for anything beyond small models. For CPU inference, use quantized models through the llama.cpp backend. The official documentation is at guidance.readthedocs.io. The GitHub repository at github.com/guidance-ai/guidance contains examples, tutorials, and issue tracking.

Guidance vs. LMQL. Both are Python libraries for structured generation with constrained decoding. LMQL offers a SQL-like query language and supports more backends including vLLM. Guidance has a more Pythonic API that feels natural for developers writing imperative code. Guidance's template syntax is simpler to learn for basic use cases. LMQL's query language is more expressive for complex constraints.

Guidance vs. Outlines. Outlines is a library focused on structured generation with regex and JSON constraints. It integrates with Hugging Face models and provides a simpler API for basic constrained decoding. Guidance offers richer control flow (conditionals, loops) within generation templates. Outlines is lighter weight if you only need output constraints without program logic.

Choose Guidance when you need to interleave complex logic with generation, when you want a Python-native programming model, and when you are building single-process applications. Choose a serving engine like vLLM or TGI when you need throughput, batching, and a production API. Guidance complements those tools rather than competing with them.