Observability ≠ monitoring

author’s position
opinionated
fragment

Scope. Wide events do not replace metrics. The working position is complementarity: metrics for cheap aggregate detection, traces for the causal chain, events for detail, profiles for CPU/memory. High cardinality demands control over cost and privacy.

Monitoring checks conditions formulated in advance; observability is the ability to make sense of a state nobody anticipated, without shipping new code to do it. Wide structured events with high cardinality are a strong mechanism for that ability, not its definition: metrics, traces and profiles answer different questions and are not replaced by events.

The word comes from control theory: Rudolf Kálmán (around 1960) called a system observable if its internal state can be determined from its outputs. Software borrowed it, loosely, in the 2010s; Majors, Fong-Jones and Miranda, Observability Engineering (O’Reilly, 2022), give the version most engineers now use. Monitoring is still necessary — Google’s Site Reliability Engineering (2016) names latency, traffic, errors and saturation as the four golden signals to alert on — but it only answers questions someone thought to ask. A dashboard of averages per endpoint cannot tell you that p99 rose only for one customer, on one gateway, since one deploy; you either kept that context on each event or you ship code and wait for it to happen again. A practical test: in the last incident, how many questions could only be answered after a code change or a new metric was deployed?

fragment

// One wide event per request instead of a dozen scattered log lines
logger.Info("request_completed",
    "trace_id", traceID, "customer_id", custID, "endpoint", "/charge",
    "duration_ms", ms, "retry_count", retries, "gateway", gw, "outcome", outcome,
    "amount_cents", amt, "deploy_sha", buildSHA)

// later: "show p99 by gateway for EU customers on version abc123" — answerable

Python · structlog

# context accumulates along the request path
log = structlog.get_logger().bind(trace_id=tid, customer_id=cid)
...
log.info("request_completed", duration_ms=ms, outcome="declined", reason=r)

Costs and limits. High cardinality is cheap in a columnar event store and ruinous as metric labels: customer_id on a Prometheus counter creates a time series per customer. Wide events cost storage, so they are sampled — keep every error and slow request, sample the boring ones, and record the sample rate so counts stay correct. Events also carry personal data; what goes into them falls under retention (36). Trace IDs across boundaries (29) are what let one event be joined to the rest of the request.