Graceful degradation / fallback cascade

A model is a dependency with a non-trivial SLA and cost. The cascade: cache → cheap model → expensive model → rule/template → honest refusal. Circuit breakers and budgets (tokens/latency/money) are the same principles from section IV, applied to a new kind of dependency.

Graceful degradation is an old reliability idea: when a dependency fails, serve a reduced answer rather than an error. Michael Nygard’s Release It! (2007) gave backend engineers the circuit breaker and the bulkhead for it (24). An LLM provider adds two failure modes that most dependencies lack: rate limits and overload are routine, not exceptional, and every call has a price that varies by two orders of magnitude between models. The cascade as a cost tool was studied by Chen, Zaharia and Zou in “FrugalGPT” (2023): send a query to a cheap model first, score its answer, and escalate only when the score is low. They reported large cost savings at matched accuracy on their benchmarks — a result about those tasks, worth reproducing on your own evals (51) before adopting the routing.

async def classify(ticket: Ticket) -> Category:
    if (hit := semantic_cache.get(ticket.embedding, threshold=0.97)): return hit
    try:
        return await small_model.classify(ticket, timeout=1.5)
    except (Timeout, Overloaded):
        return rules_fallback(ticket)          # degradation shows in metrics, not silent

Each step of the cascade is a quality level the product has agreed to show. For ticket classification, a keyword rule that routes to a general queue is acceptable; for a legal answer, a template saying “we can’t answer this now” is the honest floor, and a cheaper model guessing is not. The degradation level is a metric and a label on the response, so a silent slide into fallback shows up on a dashboard (57) instead of in complaints.

The semantic-cache risk. Embedding similarity is not query equivalence, and a shared cache is a leak channel between tenants. The key must include the tenant, the user’s permissions, the model, the prompt version, the index version and the tool state; the false-hit rate is measured, not assumed. Retries across the cascade also multiply load on an already struggling provider — budget them (23).