LLM output is untrusted input (prompt injection = trust boundary)

The most important principle in this section. Anything the model has read (web pages, documents, emails) may contain instructions; therefore the model’s output carries the trust level of its least trusted input. The classic risk triad: access to private data + untrusted content + an outbound channel — never all three at once.

Simon Willison named the attack “prompt injection” in September 2022, by analogy with SQL injection, after Riley Goodside’s demonstrations; Greshake et al., “Not what you’ve signed up for” (2023), showed the indirect form, where the instructions arrive inside content the application retrieves rather than from the user. Willison later called the combination of private data, untrusted content and an exfiltration channel the “lethal trifecta”. OWASP’s Top 10 for LLM Applications lists prompt injection first.

# LLM output is parsed as untrusted input — the same parse-don't-validate.
# Tagged union: tool and args are tied by a discriminator, otherwise
# {"tool": "search_orders", "args": <InvoiceArgs fields>} passes validation
class SearchOrders(BaseModel):
    model_config = ConfigDict(strict=True, extra="forbid")
    tool: Literal["search_orders"]
    args: SearchArgs

class GetInvoice(BaseModel):
    model_config = ConfigDict(strict=True, extra="forbid")
    tool: Literal["get_invoice"]
    args: InvoiceArgs

ToolCall = Annotated[SearchOrders | GetInvoice, Field(discriminator="tool")]

raw = llm.complete(prompt)
call = TypeAdapter(ToolCall).validate_json(raw)  # not eval, not getattr(obj, raw_name)

// Same principle as a type: LLM text enters the system only via a parser
pub struct LlmRaw(String);                          // untrusted

pub enum Action { SearchOrders(Query), GetInvoice(InvoiceId) }

impl LlmRaw { pub fn parse(self) -> Result<Action, Rejected> { ... } }

A schema is necessary, not sufficient. Valid JSON can still carry a perfectly legal and perfectly unwanted command. Parsing is followed by deterministic checks the model does not control: authorization on behalf of the user, not the agent; the object belonging to the tenant; semantic limits (amounts, volumes, date ranges); rate limits; egress policy.

Why there is no filter that fixes it. Unlike SQL, a prompt has no separate channel for data (34): instructions and the email being summarized are the same kind of tokens. Classifiers and “ignore instructions in the document” prompts lower the success rate of known attacks; none has a guarantee, and an attacker iterates faster than a blocklist. A support agent that reads tickets and can issue refunds is exploitable by a ticket that asks for one. The structural fixes remove a leg of the triad: the agent that reads untrusted text has no refund tool, or every refund waits for a human (49, 50).