Control plane ≠ data plane: controllers reconcile to a level, and the data plane survives without them

The control plane changes the system; the data plane is the system doing its job. The control plane should be a set of controllers, each comparing desired state with observed state and acting on the difference — level-triggered, so an event is only a hint to look, never the input itself. The data plane should keep serving on its last known-good configuration when the control plane is gone: it gets no updates during the outage, but what worked keeps working.

Level, not edge. A handler that acts on “what changed” is only as correct as the event stream: a missed, duplicated or reordered event, a restart, or a resync silently desynchronises it. A reconciler reads the whole current state on every pass, so any number of events collapse into one correct action, and a value that went 2 → 5 → 3 is simply driven to 3. The price is that reconcile must be idempotent (22) and cheap enough to run often; the dividend is that crash recovery, resync and the normal path are the same code (25). “Done” is defined against a particular spec: status.observedGeneration equal to metadata.generation is the only honest “rolled out” signal, and deploy tooling should wait on it rather than on the API call having returned.

// The request carries only a name, never "what changed": every call reads the
// current desired and observed state and converges, so a missed or duplicated
// event costs nothing.
func (r *WebAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var app webv1.WebApp
    if err := r.Get(ctx, req.NamespacedName, &app); err != nil {
        // Gone: the Deployment is garbage-collected through its owner reference.
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    dep := &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: app.Name, Namespace: app.Namespace}}
    // Mutate only the fields this controller owns; overwriting whole structs would
    // drop server-side defaults and turn every pass into a spurious update.
    _, err := controllerutil.CreateOrUpdate(ctx, r.Client, dep, func() error {
        labels := map[string]string{"app": app.Name}
        dep.Spec.Replicas = ptr.To(app.Spec.Replicas)
        dep.Spec.Selector = &metav1.LabelSelector{MatchLabels: labels}
        dep.Spec.Template.Labels = labels
        if len(dep.Spec.Template.Spec.Containers) == 0 {
            dep.Spec.Template.Spec.Containers = []corev1.Container{{Name: "web"}}
        }
        dep.Spec.Template.Spec.Containers[0].Image = app.Spec.Image
        return controllerutil.SetControllerReference(&app, dep, r.Scheme)
    })
    if err != nil {
        return ctrl.Result{}, err // requeued with backoff; the retry re-reads everything
    }

    // Observed state, stamped with the generation it was computed against.
    want := webv1.WebAppStatus{ObservedGeneration: app.Generation, ReadyReplicas: dep.Status.ReadyReplicas}
    if app.Status == want {
        return ctrl.Result{}, nil
    }
    app.Status = want
    return ctrl.Result{}, r.Status().Update(ctx, &app)
}

func (r *WebAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
    // Watches are triggers only: a Deployment event enqueues its owner's name.
    return ctrl.NewControllerManagedBy(mgr).
        For(&webv1.WebApp{}).
        Owns(&appsv1.Deployment{}).
        Complete(r)
}

Failure modes of reconcilers. Two controllers owning the same field fight forever, each “correcting” the other — field ownership must be explicit. Writing status unconditionally can re-trigger the controller that wrote it: a hot loop. A permanent error retried with backoff is noise forever; mark it terminal and surface it in a condition. An external resource cannot be observed once its API object is gone, so its cleanup needs a finalizer. And GitOps (86) is the same loop one level up: a manual hotfix is drift and gets reverted, so the incident procedure must suspend sync explicitly rather than race it.

Static stability. The data plane must not call the control plane on the request path: no per-request lookups against the orchestrator API, no sidecar that fails closed when it cannot reach its controller, no startup that requires a control-plane write. Recovery actions should be data-plane actions against capacity provisioned in advance (97, 105). The flip side: a controller propagates a bad desired state everywhere at controller speed, so the control plane is where progressive rollout (72) and cell boundaries (27) matter most.