DSPy

DSPy is a programming framework for building and optimizing LLM pipelines, maintained by the Stanford NLP group. It is not an inference engine in the traditional sense. You do not serve models with DSPy. You do not load weights or manage KV caches. DSPy sits one layer up: it orchestrates calls to underlying models and automatically optimizes the prompts and few-shot examples used in those calls.

The core problem DSPy solves is the brittleness of hand-tuned prompts. Most LLM applications today are built around string templates that engineers tweak by trial and error. When the model changes or the task drifts, those templates break. DSPy replaces this with a programmatic approach: you define your task as typed signatures, compose modules into pipelines, and let the framework search for the prompts and examples that maximize a metric you care about.

With over 35,000 GitHub stars, 430+ contributors, and 6 million monthly PyPI downloads, DSPy has become the standard library for prompt optimization in the Python ecosystem. It is licensed under MIT, requires Python 3.10 or later, and is designed to work with any language model accessible through a Python API.

DSPy competes with manual prompt engineering workflows and libraries like LangChain or Haystack. But unlike those, DSPy does not just help you build pipelines. It optimizes them. That makes it useful for any team running LLMs in production who wants to move from guessing at prompts to measuring and improving them systematically.

DSPy operates through three core abstractions: Signatures, Modules, and Optimizers.

Signatures are typed declarations of a task. You define inputs and outputs as Python class fields, not prompt strings. For example, a classification task looks like this:

1class Triage(dspy.Signature):
2    """Route a support ticket."""
3    ticket: str = dspy.InputField()
4    urgency: Literal["low", "high"] = dspy.OutputField()
5    team: str = dspy.OutputField()

Modules control how a signature executes. dspy.Predict produces a direct completion. dspy.ChainOfThought adds step-by-step reasoning. dspy.ReAct adds tools and a reasoning loop. You swap strategies by changing the module, not the signature.

Optimizers (formerly called teleprompters) search for the prompts and few-shot examples that produce the best outputs against your chosen metric. You call optimizer.compile() with your program, training data, and evaluation metric, and DSPy does the search. This process can include prompt rewriting, example selection, and multi-stage pipeline optimization.

You use DSPy by writing Python. You connect it to a model through the dspy.LM object, which supports OpenAI, Anthropic, local models via Hugging Face Transformers, or any OpenAI-compatible endpoint. DSPy itself does not serve models. You point it at a model endpoint that someone else serves, or run a local model alongside it.

DSPy does not manage inference performance directly. It is not a serving engine like vLLM or TensorRT-LLM. It does not handle batching, KV cache management, quantization, or GPU memory allocation. Those concerns belong to the underlying model provider or inference engine you connect to DSPy.

What DSPy does affect is pipeline efficiency. Because DSPy can search for shorter, more effective prompts and fewer few-shot examples, it can reduce token usage per call. That translates to lower latency and cost on whatever engine you are using underneath.

DSPy itself runs on CPU. It is pure Python. The hardware requirements for DSPy are minimal: any machine that can run Python can run DSPy. The GPU requirements come from the models it calls. If you are running a local model, you need whatever hardware that model requires. DSPy works with any inference engine that exposes a Python API, including vLLM, Ollama, llama.cpp, and Hugging Face Transformers.

For practitioners running DSPy in production, the typical pattern is to run a model server like vLLM on a GPU node and point DSPy at its OpenAI-compatible endpoint. DSPy handles the orchestration and optimization; the inference engine handles the throughput and latency.

Pipelines as code. Task definitions are modules with typed inputs and outputs. This makes pipelines testable, versionable, and reviewable through normal code practices. No more prompt templates hidden in config files.

Automatic optimization. DSPy optimizers search for prompts and few-shot examples that maximize a user-defined metric. This replaces hours of manual prompt tuning with a compile step that runs automatically. The underlying research, published at Stanford and arXiv, shows that DSPy-optimized pipelines consistently outperform hand-tuned prompts across multiple benchmarks.

Model agnostic. The same pipeline definition works across OpenAI, Anthropic, local models, and custom endpoints. Switching models requires changing one line in the LM configuration, then re-optimizing if desired.

Structured output. DSPy signatures enforce structured outputs with typed fields. You get back Python objects, not raw text. This is critical for building reliable agents and multi-step pipelines where downstream steps depend on specific output formats.

Streaming support. DSPy supports streaming from models that expose it, useful for chat interfaces and real-time applications.

Improving agent quality. Teams building autonomous agents use DSPy to optimize the prompts behind each agent step. Instead of tuning prompts by hand until the agent behaves correctly, you define a metric and let DSPy search for the prompts that maximize it.

Reproducible pipelines. In regulated environments or team settings, treating prompt engineering as code that can be tested and versioned is a significant advantage. DSPy enables this by keeping all prompt logic in Python rather than in external templates.

Switching models cleanly. When a new model version ships, or when you want to migrate from GPT-4 to a local model, DSPy lets you re-optimize your pipeline against the new model without rewriting your task definitions.

Not a fit for: Raw inference serving, real-time latency-sensitive applications where the optimization overhead is prohibitive, or teams that want a turnkey model server.

Install with pip:

1pip install dspy

The smallest meaningful path to using DSPy is:

1. Configure a language model: lm = dspy.LM("openai/gpt-4o-mini")
2. Define a signature: a class with InputField and OutputField annotations
3. Create a module: classify = dspy.Predict(MySignature)
4. Call it: result = classify(input="your text")

To optimize, you need:

• A training set of example inputs and outputs
• A metric function that scores outputs
• An optimizer, e.g., dspy.teleprompt.BootstrapFewShot

Documentation is at dspy.ai. The GitHub repository at github.com/stanfordnlp/dspy contains examples and the active Discord community.

DSPy vs. LangChain. Both let you compose LLM pipelines as code. LangChain focuses on integrations and pre-built components. DSPy focuses on optimization. If you need to connect to 20 different vector stores and document loaders, LangChain wins. If you need to systematically improve the quality of your pipeline outputs against a measurable metric, DSPy wins. They are complementary; some teams use LangChain for integrations and DSPy for optimizing the prompts within those integrations.

DSPy vs. manual prompt engineering. This is not a comparison against another framework. It is a comparison against the status quo. Manual prompt engineering works for simple tasks with stable models. As pipelines grow in complexity or models change, manual tuning becomes unsustainable. DSPy replaces guesswork with search. The cost is learning the programming model and setting up evaluation metrics. The benefit is reproducible, optimizable, measurable pipelines.