Guidance

Guidance supports multiple backends including Hugging Face Transformers, llama.cpp, and OpenAI API models. You choose the backend when you instantiate the model object. There is no server component. You run Guidance programs directly in your Python process, which means you manage model loading and memory yourself.

The library does not provide an OpenAI-compatible API. It is a programmatic interface, not a service.

Guidance runs on NVIDIA GPUs through the Transformers backend and on CPU through llama.cpp. Confirmed capabilities include NVIDIA GPU inference, CPU inference, structured output, and streaming.

Performance depends entirely on the backend you choose. Guidance itself adds minimal overhead. The primary performance benefit comes from constrained decoding: by restricting the model's output space, you generate fewer tokens and reduce the chance of retries. This can lower both latency and cost compared to unconstrained generation followed by validation.

The library supports "guidance acceleration," which maintains KV cache state across multiple generation calls within a single program. This avoids recomputing attention for the fixed parts of your template when you make sequential generation calls. For local models using the Transformers or llama.cpp backends, this can produce meaningful speedups.

Memory usage follows the backend's behavior. With Transformers, you load the full model into GPU memory. With llama.cpp, you can use quantized models to reduce memory footprint. Guidance does not implement paged attention, continuous batching, or multi-GPU sharding. It is not designed for high-throughput serving of many concurrent requests.

For throughput-sensitive workloads, Guidance is not the right tool. It is a single-process library. If you need to serve hundreds of requests per second, pair Guidance with a serving layer or use a dedicated inference engine for the generation portion.

Guided templates let you define a structure and let the model fill only the parts you mark. This is the core feature. You write a template with fixed text and gen() calls, and the model generates only the tokens needed to complete each blank. The rest of the template is not regenerated.

Constrained decoding restricts output to valid choices, patterns, or grammars. You can force the model to output only from a set of options, match a regex, or follow a context-free grammar. This eliminates the need for post-processing validation and retries. For JSON output, you can define the schema and Guidance will enforce it during generation.

Token efficiency is a direct consequence of constrained generation. The model produces only the tokens you need, not the structural tokens you already defined. This reduces token count, which lowers cost for API-based backends and reduces latency for local models.

Streaming is supported. You can receive tokens as they are generated, which is useful for real-time applications.

Reliable structured output is the primary use case. Teams use Guidance to generate JSON, code, or formatted text that must match a schema. The constrained decoding guarantees the output is valid, which eliminates parsing errors downstream.

Complex prompt logic benefits from Guidance's ability to mix conditionals, loops, and generation in a single template. You can write a program that branches based on previous model output, iterates over a list, or calls tools, all within the same generation flow.

Cost control is a practical concern for teams using API-based models. By generating only the variable parts of a template, Guidance reduces token consumption. For high-volume applications, this can meaningfully reduce API costs.

Local development and prototyping is a natural fit. Guidance's Python API makes it easy to experiment with structured generation patterns before deploying to production. The library works with small local models, which is useful for testing.

Guidance is a poor fit for high-throughput serving, multi-user applications, or any scenario where you need a standalone API server. It is also not designed for training or fine-tuning.

Install Guidance with pip:

The smallest meaningful path to a running model requires a backend. For local inference with a Hugging Face model:

For CPU inference with llama.cpp, install the llama.cpp backend and use models.LlamaCpp instead.

You need a GPU for anything beyond small models. For CPU inference, use quantized models through the llama.cpp backend. The official documentation is at guidance.readthedocs.io. The GitHub repository at github.com/guidance-ai/guidance contains examples, tutorials, and issue tracking.

Guidance vs. LMQL. Both are Python libraries for structured generation with constrained decoding. LMQL offers a SQL-like query language and supports more backends including vLLM. Guidance has a more Pythonic API that feels natural for developers writing imperative code. Guidance's template syntax is simpler to learn for basic use cases. LMQL's query language is more expressive for complex constraints.

Guidance vs. Outlines. Outlines is a library focused on structured generation with regex and JSON constraints. It integrates with Hugging Face models and provides a simpler API for basic constrained decoding. Guidance offers richer control flow (conditionals, loops) within generation templates. Outlines is lighter weight if you only need output constraints without program logic.

Choose Guidance when you need to interleave complex logic with generation, when you want a Python-native programming model, and when you are building single-process applications. Choose a serving engine like vLLM or TGI when you need throughput, batching, and a production API. Guidance complements those tools rather than competing with them.

• Guided templates

  Define a structure and let the model fill only the parts you mark.

• Constrained decoding

  Restrict output to valid choices, patterns, or grammars.

• Token-efficient

  Generate only what is needed, which can lower cost and latency.