Refused is not timeout
Also in Chinese: 中文版
Symptom
External connections to a database service failed instantly with connection refused. Instantly is the clue: refused means the packet arrived and nothing was listening — fast and deterministic. Timeout means a listener exists but is not answering. The two words point at different halves of the stack, and conflating them wastes the first hour.
Walk the hops, stop at the first failure
The path had eight hops: client → DNS → load balancer → NodePort → ingress → Service → Endpoints → pod. The method is unglamorous: verify each hop actually listens or resolves, in order, and stop at the first one that fails. On this class of incident it has produced two different root causes:
-
The listener that looked configured. The ingress controller’s TCP-services ConfigMap existed with the right entries — but the controller had never loaded it. Configuration present, listener absent. Everything upstream of the ingress was innocent by inspection.
-
The label that went stale. Subtler: after an abnormal pod restart, the database operator failed to refresh the pod’s readiness label. The Service selector required that label, so the Endpoints list was empty — the Service existed, the pod was actually serving, and no traffic could ever arrive. Control-plane truth had diverged from data-plane truth, and only the Endpoints hop showed it.
Fix and verify
Reload the controller in the first case; restart the workload so the operator re-evaluates its labels in the second. Verification mirrors the diagnosis: confirm the port is actually in a listening state and the Endpoints list is non-empty — the two exact things that were broken, not a generic health check.
Symptom classification before tooling: refused vs. timeout, then hop by hop. The method is boring, which is why it works at 3 a.m.