Production Kubernetes Upgrade Engineering: 50 Clusters, 1.24 → 1.29, Zero Incidents
Also in Chinese: 中文版
Why
Kubernetes 1.24 was aging out of support; every unsupported minor compounds CVE exposure and compliance risk. The fleet was 50 self-managed clusters on AWS — kubeadm control planes, ASG-backed workers — and the existing practice did not scale: a single cluster took 18–21 hours of hands-on work and required a two-person pairing, because the real risk model lived in senior engineers’ heads. Multiply that by per-minor hops (kubeadm upgrades one minor at a time, so 1.24 → 1.29 means every intermediate step, each touching control plane, workers, and addons) and manual execution was not a viable plan.
The reframing that shaped the project: the root problem was not “manual vs. automated” but missing explainability. At any step, nobody could answer “what evidence tells me it is safe to proceed?” — knowledge was implicit, evidence scattered, postmortems impossible to reconstruct.
How: the Upgrade Safety System
The system is a three-stage pipeline — check → plan (dry-run) → apply + evidence — implemented as a Python CLI orchestrating Ansible, boto3, and kubectl, with all stdout/stderr captured to an evidence store.
check makes the health baseline explicit. etcd quorum is verified three ways: member list (odd count, all online), raft index lag (a follower more than 1,000 entries behind the leader fails the gate — quorum can nominally exist while replication is unhealthy), and leader uniqueness. Node readiness, kube-system pod health, and active external alerts are gated the same way, with fail conditions written down rather than judged ad hoc. An etcd snapshot is mandatory before anything mutates.
plan is a real dry-run, not a document. Masters: kubeadm upgrade plan plus ansible --check --diff. Workers: an AMI diff (kubelet must equal the target version) and a Launch Template diff (only the AMI ID may change). Blast radius is quantified up front by quorum math: three masters tolerate exactly one unavailable, and serial: 1 guarantees only one is ever touched, capping the worst case at a single node.
apply executes in strict order with per-layer gates. Control plane upgrades in place, one master at a time, each with human confirmation. Workers are replaced immutably — new AMI, Launch Template update, ASG Instance Refresh in 20% batches with pause-on-failure. Addons follow a dependency order: AWS cloud-controller-manager first (an unhealthy CCM strands new workers on the uninitialized taint), then the CNI (maxUnavailable: 1, verified by cross-node pod connectivity), then Cluster Autoscaler (verified by a drain-triggered scale-out). Post-verify re-runs the full check and diffs against the pre-upgrade baseline.
Every step drops evidence: JSON health snapshots, plan diffs, full execution logs, and a per-step state file (completed / in-progress / pending). Interrupted runs resume from state; idempotent steps re-run.
Fleet rollout order: dev → preprod → prod canary → remaining prod → management cluster last (highest blast radius). Any failure pauses the fleet until a human approves continuation.
The efficiency path: 18–21 hours with a two-person pair became 6–8 hours with one operator plus the system; the automation roadmap (external alert gating, synthetic health checks, auto-promotion after canary) targets 3–4 hours per cluster, a projected 60–80% reduction.
Hard parts
1. API deprecations as a separate workstream. Removals ship per minor; the canonical case was PodSecurityPolicy, removed in 1.25. Logging and monitoring DaemonSets legitimately need privileged access, and a namespace carelessly set to enforce: restricted under Pod Security Admission blocks them outright. The fix was procedural: scan and migrate workloads before the hop, roll out PSA in warn/audit first, enforce only after zero violations, keep infra namespaces explicitly privileged. Version skew policy (kubelet may trail the apiserver by two minors) makes control-plane-first sequencing safe by construction.
2. Stateful workloads and eviction order. Draining nodes that host database pods (Kafka, MySQL, YugabyteDB, ClickHouse) is where upgrades break tenants. Database-hosting nodes were mapped ahead and sequenced first in each wave, one at a time with per-service verification. PDBs stay honest even on a dark cluster: violations there cannot hurt users, but they still block drain, so the procedure is wait-for-PDB or explicitly lower minAvailable — never force. Protection is layered — PDB for voluntary disruption, HPA for load, readiness probes for rollout — with the non-covered cases (hardware failure, application bugs) named and handled by multi-AZ placement.
3. Fleet heterogeneity without heroics. 50 clusters drift. Clusters were classified by cluster_type (workload / management / dev-staging), with per-cluster feature flags for residual exceptions — e.g. a drain timeout raised from the 300s default to 600s where eviction is known to be slow. Drift is not eliminated; it is made visible in the check baseline before every upgrade, and a growing flag count signals the classification itself is wrong.
4. Making “zero incidents” a verifiable claim. Zero incidents only means something against a pre-agreed standard. Every gate resolves a version × health four-quadrant (target version + healthy → proceed; old version + healthy → re-run; anything unhealthy → human), post-verify diffs against the recorded baseline rather than an engineer’s memory, and the check itself is cross-validated against external monitoring to defend against a wrong baseline. Validation covered business-layer correctness, not just infra health.
5. Rollback designed per layer, triggers pre-declared. An etcd snapshot restores data, not binaries — so control plane rollback is snapshot restore plus binary downgrade; worker rollback is stopping the Instance Refresh and reverting the Launch Template; addon rollback is kubectl rollout undo. Triggers were pre-written: master upgrade failure, over 50% of pods not Running, critical services unreachable, alert flood. Raft leader transfer mid-upgrade is expected (elections settle in 1–2 seconds); the real hazard is a member failing to rejoin — exactly what serial: 1 plus the snapshot gate bounds.
Production execution
Both production fleets — masters, control-plane components, and workers — were upgraded with zero customer-impacting downtime and zero rollbacks. The enabling pattern was paired-cluster traffic pre-shift: move all traffic to the sibling cluster, pause cross-cluster replication, upgrade the now-dark cluster, resume replication and wait for lag to drain below threshold, verify against the checklist, then shift traffic back. Under this pattern the 20% batch size becomes a pacing mechanism rather than user protection — small enough to pause fast when something looks wrong. Human sign-off remained at check, at plan, at every master, and at final post-verify.
Takeaways
- The hard part of an upgrade is not knowing what to do; it is justifying, with evidence, that you may do the next thing. An evidence chain converts implicit seniority into explicit gates anyone can operate.
- Blast radius is a design input, not an outcome: quorum math,
serial: 1, and 20% batches put a ceiling on the worst case before execution starts. - The durable asset is not the tool. The checklist and evidence patterns outlived the project and now run as routine infra health checks.