Structured outputs / schema-constrained generation

Free text between machine components is a stringly-typed interface. The schema is imposed on generation, and the output is validated anyway (double control): the schema guarantees shape, only your own checks guarantee meaning.

There are three levels of strength. Asking in the prompt for JSON gets JSON most of the time and a trailing comment or a truncated object the rest. Function calling and JSON modes make the provider aim for valid JSON. Constrained decoding makes invalid output impossible: at each step the decoder masks every token that would leave the grammar. Willard and Louf, “Efficient Guided Generation for Large Language Models” (2023, the Outlines library), showed how to compile a regular expression or JSON Schema into a finite-state machine so the mask costs little per token; llama.cpp grammars and the providers’ “strict” structured-output modes apply the same idea. Support for JSON Schema features differs between implementations — check which keywords yours enforces.

// schema from the type, output into the type
record Invoice(String number, long totalCents, LocalDate due) {}

var invoice = mapper.readValue(llm.completeWithSchema(prompt, schemaOf(Invoice.class)),
                               Invoice.class); // this can throw too — handle it

type Extract struct {
    Number     string `json:"number"`
    TotalCents int64  `json:"total_cents"`
}

// json.Unmarshal + domain validation AFTER unmarshaling — a model can
// return syntactically valid, semantically false JSON

An invoice-extraction example. The schema guarantees that total_cents is an integer; it does not guarantee that it equals the sum of the line items, that due is after the issue date, or that the number was in the document at all rather than invented. Those are domain checks, done after parsing (2), with a rejection path: retry with the error in the prompt, route to a human, or mark the record unextracted. A refusal or an empty object must be a representable outcome — a schema with every field required forces the model to fill fields it could not find.

Trade-offs. Constraining format can cost answer quality; one 2024 study (Tam et al., “Let Me Speak Freely?”) reported lower reasoning scores under strict formats on some tasks. A common remedy is a free-text reasoning field before the structured fields, or a second call that only formats. Measure it on your own evals (51) rather than assuming either way. The schema is also an interface: version it with the prompt (52) and evolve it compatibly (18).