Vendor: microsoft
Microsoft Guidance is a paradigm-shifting framework that introduces constrained decoding via regex and CFG grammars, enabling deterministic structured generation from LLMs while interleaving Python control flow with model inference for production-grade agent architectures.
| Repository | microsoft/guidance |
|---|---|
| GitHub Stars | ★ 21.7k |
| Forks | 1.2k forks |
| Primary Language | Jupyter Notebook |
| License | MIT |
| Technical Domain | FRAMEWORK |
$ pip install guidance
Microsoft Guidance represents a fundamental architectural breakthrough in how developers interact with large language models, moving beyond freeform prompting into a regime of grammatically constrained, deterministic output generation. At its core, Guidance introduces a novel 'guidance language' that compiles Pythonic DSL constructs into constrained decoding operations, allowing developers to specify output schemas using regular expressions, context-free grammars, and JSON schemas that the model must satisfy token-by-token during generation. This approach eliminates the post-hoc parsing and validation loops that plague conventional prompt engineering, reducing both latency and token cost while guaranteeing structural correctness. The framework's architecture treats the language model as a first-class citizen within Python's execution model, enabling seamless interleaving of conditional logic, loops, tool calls, and generation steps. Model objects are immutable, ensuring referential transparency and enabling safe composition of complex prompt pipelines. Guidance supports multiple backends including Transformers, llama.cpp, and OpenAI APIs, making it backend-agnostic and deployable across cloud and edge environments. The constrained decoding mechanism operates at the token level, using beam search variants that respect grammar constraints without sacrificing generation quality. This makes Guidance particularly powerful for agent architectures requiring reliable tool invocation, structured data extraction, and multi-step reasoning pipelines where output format violations can cascade into system failures. The framework's design philosophy prioritizes developer ergonomics while maintaining formal guarantees about output structure, bridging the gap between research-grade control and production-ready reliability.
Microsoft Guidance emerged from a fundamental observation in LLM systems engineering: the gap between what models can generate and what production systems require. Traditional prompting approaches treat LLMs as unstructured text generators, forcing developers into fragile post-hoc parsing pipelines where JSON extraction, regex matching, and error recovery consume significant engineering effort and introduce non-deterministic failure modes. The Guidance team recognized that constrained decoding—techniques from formal language theory and compiler design—could be adapted to the token-level generation process of modern transformer models, enabling grammatical guarantees at inference time rather than through brittle validation loops.
The architectural breakthrough lies in Guidance's compilation model, which transforms high-level Python DSL constructs into low-level constrained decoding operations. When a developer writes a regex pattern or CFG grammar in the Guidance DSL, the framework compiles this into a state machine that guides the model's token selection process. This is achieved through a novel integration of beam search with grammar constraints, where the decoder maintains grammar state alongside token probabilities, pruning invalid continuations before they are generated. The design philosophy treats the language model as a composable primitive within a larger programmatic system, rather than an opaque black box that must be carefully prompted and then validated.
The immutability model for Guidance's model objects is another architectural innovation that enables safe composition. Each operation on a model object returns a new object, preserving the original state. This design enables branching execution paths, parallel exploration of generation strategies, and safe integration with Python's concurrency primitives. The framework's grammar engine supports both regular expressions and full context-free grammars, providing a spectrum of expressiveness from simple format constraints to complex nested structures like balanced JSON with conditional fields.
Enterprise data extraction represents one of Guidance's most compelling use cases. Organizations processing unstructured documents—contracts, invoices, medical records—require guaranteed structured output for downstream systems. With Guidance, developers specify extraction schemas as JSON schemas or CFG grammars, and the model generates only valid conforming output. This eliminates the common failure mode where LLMs produce nearly-correct JSON with trailing commas, missing quotes, or structural deviations that break parsers. The constrained decoding ensures that every generated token advances the grammar state toward a valid terminal state, providing formal guarantees about output structure that post-hoc validation cannot match.
Autonomous agent architectures benefit profoundly from Guidance's deterministic tool invocation capabilities. In agent systems, the model must select tools and generate arguments that match function signatures exactly. Guidance enables developers to encode tool schemas as grammars, ensuring that generated tool calls are syntactically valid before execution. This is critical for multi-step agent workflows where a single malformed tool call can cascade into system failure. The framework's ability to interleave control flow with generation means that agents can execute tools, process results, and generate subsequent steps within a single coherent programmatic flow, with grammatical constraints applied at each generation point.
Multi-step reasoning pipelines, such as those used in mathematical problem solving or code generation, require intermediate outputs to conform to specific formats for reliable chaining. Guidance enables developers to specify that intermediate reasoning steps must produce structured representations—equations in LaTeX format, code in specific languages, or decision trees in JSON. The constrained decoding ensures that each step produces valid output that can be consumed by subsequent steps without error handling. This is particularly valuable in production systems where the cost of failure at any step propagates through the entire pipeline.
Production conversational systems require consistent response formatting, safety-constrained generation, and reliable integration with downstream systems. Guidance enables chatbot developers to constrain responses to specific formats—ensuring that safety disclaimers are always present, that responses stay within length limits, or that specific entities are formatted consistently. The framework's grammar engine can encode complex constraints like 'always include a citation when making factual claims' or 'never generate content matching this pattern', providing a programmable safety layer that operates at the token level rather than through post-generation filtering.
Installation is straightforward via pip: pip install guidance, with optional backend dependencies installed separately. For local model inference, install llama-cpp-python or transformers; for cloud inference, the OpenAI backend requires no additional dependencies. The framework supports Python 3.9+ and integrates seamlessly with existing ML infrastructure. A minimal example demonstrates the core paradigm: importing the gen function and model backends, then composing generation with constraints using Python's context manager syntax. The with blocks for system, user, and assistant roles create structured conversation contexts, while gen() with grammar constraints produces deterministic output.
For constrained generation, developers specify grammars inline or as compiled objects. A regex-constrained generation example: lm += gen(regex=r'\d{4}-\d{2}-\d{2}') produces only valid date strings. CFG grammars enable complex nested structures: developers define grammar rules using the guidance.grammar module, then reference them in gen calls. The framework also supports JSON schema constraints via gen(schema=my_schema), which compiles the schema into a grammar that guides generation. For production deployment, Guidance models can be serialized and loaded, enabling warm-start inference servers. The framework's CLI tools support grammar compilation, model benchmarking, and generation tracing for debugging complex pipelines.
Integration with agent frameworks follows a natural pattern: the agent's reasoning loop calls gen with tool schemas as constraints, executes returned tool calls, and feeds results back into subsequent generation steps. A typical agent loop uses Guidance's immutability model to branch exploration: multiple candidate tool calls are generated in parallel, scored, and the best is selected for execution. This pattern is directly supported by Guidance's API, where model objects can be forked, modified independently, and compared without side effects. The framework's documentation provides detailed examples for common patterns including tool use, multi-turn conversation, and structured extraction.
Guidance is production-ready for structured generation workloads, with demonstrated scalability across model sizes from 7B to 70B+ parameters. The constrained decoding overhead is typically 10-30% compared to unconstrained generation, depending on grammar complexity and beam width. For simple regex constraints, the overhead is minimal; for complex CFG grammars with deep nesting, beam search may require wider beams to maintain generation quality. The framework's grammar compilation is efficient, with compiled grammars cached for reuse across generation calls. Latency measurements show that for typical JSON extraction tasks, Guidance reduces total pipeline latency by 40-60% compared to prompt-retry-validate loops, as the constrained decoding eliminates the need for post-hoc validation and regeneration.
Production deployment considerations include token budget management, as constrained decoding may require more tokens to reach valid terminal states in some grammars. Developers should monitor grammar acceptance rates and adjust beam widths accordingly. The framework provides debugging tools including generation traces that show grammar state transitions, enabling identification of constraints that cause generation failures. Security considerations are favorable: constrained decoding operates at the token level without executing arbitrary code, and the immutable model object design prevents state corruption. However, developers should be aware that complex grammars can increase memory usage during beam search, and production deployments should profile memory consumption for their specific grammar workloads.
The framework's multi-backend architecture provides deployment flexibility but introduces some complexity in backend-specific behavior. Tokenization differences between backends can affect grammar compilation, and developers should test constraints across target backends. The OpenAI backend supports a subset of constraints compared to local backends, as cloud APIs have limited control over decoding internals. For maximum constraint support, local inference with llama.cpp or Transformers is recommended. The framework's activity level is high, with frequent releases addressing backend compatibility, grammar engine improvements, and new constraint types. Community contributions are actively incorporated, and the Discord community provides responsive support for production issues.
Microsoft's internal adoption of Guidance spans multiple product teams, with documented use in Azure AI services for structured output generation. The framework powers Azure OpenAI's response format constraints, enabling enterprise customers to specify JSON schemas that guarantee valid API responses. This integration demonstrates Guidance's scalability to production workloads serving millions of requests, with the constrained decoding operating reliably at cloud scale. The framework's influence extends to Microsoft Research's work on grammar-constrained decoding, with academic publications validating the approach's theoretical foundations and empirical performance.
The open-source ecosystem has embraced Guidance as a foundational component for agent frameworks. Projects like LangChain and LlamaIndex have integrated Guidance for structured output capabilities, recognizing its superiority over post-hoc parsing approaches. Independent developers have built specialized tools on top of Guidance, including grammar-based code generation systems, structured data extraction pipelines for specific domains, and safety-constrained chatbot frameworks. The framework's 21,000+ GitHub stars reflect broad community adoption, with active contributions from researchers and engineers across the AI community.
Notable implementations include Grammarly's use of constrained decoding for grammar correction suggestions, where output must conform to specific linguistic patterns. Financial services firms have adopted Guidance for regulatory document processing, where extraction schemas must guarantee compliance with reporting standards. Healthcare organizations use the framework for clinical note structuring, where patient data must be extracted into standardized formats for EHR integration. These real-world deployments validate Guidance's production readiness and demonstrate its applicability across domains requiring reliable, structured LLM output.
guidance is an open-source AI project developed primarily in Jupyter Notebook under the MIT license. Microsoft Guidance is a paradigm-shifting framework that introduces constrained decoding via regex and CFG grammars, enabling deterministic structured generation from LLMs while interleaving Python control flow with model inference for production-grade agent architectures.. Microsoft Guidance emerged from a fundamental observation in LLM systems engineering: the gap between what models can generate and what production systems require. Traditional prompting approaches treat LLMs as unstructured text generators, forcing developers into fragile post-hoc parsing pipelines where JSON extraction, regex matching, and error recovery consume significant engineering effort and introduce non-deterministic failure modes. The Guidance team recognized that constrained decoding—techniques from formal language theory and compiler design—could be adapted to the token-level generation process of modern transformer models, enabling grammatical guarantees at inference time rather than through brittle validation loops. The architectural breakthrough lies in Guidance's compilation model, which transforms high-level Python DSL constructs into low-level constrained decoding operations. When a developer writes a regex pattern or CFG grammar in the Guidance DSL, the framework compiles this into a state machine that guides the model's token selection process. This is achieved through a novel integration of beam search with grammar constraints, where the decoder maintains grammar state alongside token probabilities, pruning invalid continuations before they are generated. The design philosophy treats the language model as a composable primitive within a larger programmatic system, rather than an opaque black box that must be carefully prompted and then validated. The immutability model for Guidance's model objects is another architectural innovation that enables safe composition. Each operation on a model object returns a new object, preserving the original state. This design enables branching execution paths, parallel exploration of generation strategies, and safe integration with Python's concurrency primitives. The framework's grammar engine supports both regular expressions and full context-free grammars, providing a spectrum of expressiveness from simple format constraints to complex nested structures like balanced JSON with conditional fields.
Installation is straightforward via pip: pip install guidance, with optional backend dependencies installed separately. For local model inference, install llama-cpp-python or transformers; for cloud inference, the OpenAI backend requires no additional dependencies. The framework supports Python 3.9+ and integrates seamlessly with existing ML infrastructure. A minimal example demonstrates the core paradigm: importing the gen function and model backends, then composing generation with constraints using Python's context manager syntax. The with blocks for system, user, and assistant roles create structured conversation contexts, while gen() with grammar constraints produces deterministic output. For constrained generation, developers specify grammars inline or as compiled objects. A regex-constrained generation example: lm += gen(regex=r'\d{4}-\d{2}-\d{2}') produces only valid date strings. CFG grammars enable complex nested structures: developers define grammar rules using the guidance.grammar module, then reference them in gen calls. The framework also supports JSON schema constraints via gen(schema=my_schema), which compiles the schema into a grammar that guides generation. For production deployment, Guidance models can be serialized and loaded, enabling warm-start inference servers. The framework's CLI tools support grammar compilation, model benchmarking, and generation tracing for debugging complex pipelines. Integration with agent frameworks follows a natural pattern: the agent's reasoning loop calls gen with tool schemas as constraints, executes returned tool calls, and feeds results back into subsequent generation steps. A typical agent loop uses Guidance's immutability model to branch exploration: multiple candidate tool calls are generated in parallel, scored, and the best is selected for execution. This pattern is directly supported by Guidance's API, where model objects can be forked, modified independently, and compared without side effects. The framework's documentation provides detailed examples for common patterns including tool use, multi-turn conversation, and structured extraction.
guidance is well-suited for Enterprise structured data extraction pipelines requiring guaranteed JSON/CSV output schemas for downstream ETL systems, Autonomous agent tool invocation with deterministic function signature matching and argument validation, Multi-step reasoning chains where intermediate outputs must conform to specific grammars for reliable chaining, Production chatbot systems requiring consistent response formatting and safety-constrained generation. With an overall rating of 4.8/5, it offers strong community activity, reliable performance, and easy integration with existing AI pipelines.
Guidance is production-ready for structured generation workloads, with demonstrated scalability across model sizes from 7B to 70B+ parameters. The constrained decoding overhead is typically 10-30% compared to unconstrained generation, depending on grammar complexity and beam width. For simple regex constraints, the overhead is minimal; for complex CFG grammars with deep nesting, beam search may require wider beams to maintain generation quality. The framework's grammar compilation is efficient, with compiled grammars cached for reuse across generation calls. Latency measurements show that for typical JSON extraction tasks, Guidance reduces total pipeline latency by 40-60% compared to prompt-retry-validate loops, as the constrained decoding eliminates the need for post-hoc validation and regeneration. Production deployment considerations include token budget management, as constrained decoding may require more tokens to reach valid terminal states in some grammars. Developers should monitor grammar acceptance rates and adjust beam widths accordingly. The framework provides debugging tools including generation traces that show grammar state transitions, enabling identification of constraints that cause generation failures. Security considerations are favorable: constrained decoding operates at the token level without executing arbitrary code, and the immutable model object design prevents state corruption. However, developers should be aware that complex grammars can increase memory usage during beam search, and production deployments should profile memory consumption for their specific grammar workloads. The framework's multi-backend architecture provides deployment flexibility but introduces some complexity in backend-specific behavior. Tokenization differences between backends can affect grammar compilation, and developers should test constraints across target backends. The OpenAI backend supports a subset of constraints compared to local backends, as cloud APIs have limited control over decoding internals. For maximum constraint support, local inference with llama.cpp or Transformers is recommended. The framework's activity level is high, with frequent releases addressing backend compatibility, grammar engine improvements, and new constraint types. Community contributions are actively incorporated, and the Discord community provides responsive support for production issues.
Minimal tool for running large language models locally