The unit of failure is designed up front: customers are sharded into cells — complete, independent copies of the stack — and an incident hits one cell, not everyone. Shuffle sharding is a variant that reduces the overlap between customers combinatorially.
A cell has its own compute, its own database and its own queues; the only shared component is a thin router that maps a partition key (tenant, account) to a cell. A bad deploy, a poison message (81), a runaway tenant or a corrupted index stays inside the cell it reached, and deploys roll out cell by cell, so the first cell is the canary for the rest. The idea is the bulkhead (24) applied to the whole stack. AWS describes it in the Well-Architected guidance “Reducing the Scope of Impact with Cell-Based Architecture” and uses it for its own control planes.
Shuffle sharding is described by Colm MacCárthaigh in the Amazon Builders’ Library article “Workload isolation using shuffle-sharding”. Instead of fixed groups, each customer gets a random combination of workers. With 8 workers and 2 per customer there are 28 combinations; a customer whose requests crash their two workers takes down only the customers who share both — on average a small fraction, and clients that retry against the other workers of their shard see little effect. With plain sharding into 4 pairs, the same customer takes out a quarter of everyone.
The cost and the limits. Cells multiply fixed cost (N databases to patch, N dashboards), make cross-cell queries and cross-tenant features hard, and need a migration path for tenants that outgrow a cell. The router and the cell-assignment table are the new single point of failure, so they are kept minimal, cached at the edge and changed rarely. Global dependencies — DNS, the identity provider, a shared config push — bypass cells entirely; an incident there has the old blast radius. Below a few dozen tenants or one region the overhead usually exceeds the benefit. Expressed by deployment topology, not by code.