Zombie System Tables: the Sub-Second OOM That 30-Second Monitoring Could Not See
Also in Chinese: 中文版
Symptoms: the alert fires three systems away from the cause
The visible signal was Kafka consumer lag: a downstream ingestion consumer group’s lag rising monotonically across multiple tenants at once, with no traffic spike to explain it. The consumer itself looked guilty and was not. Its inserts into ClickHouse were being rejected with Code 241 MEMORY_LIMIT_EXCEEDED; each failed batch sent the consumer into a 30-second sleep-and-rebalance retry loop, replaying the same records into the same failure. The distance between symptom and cause was the defining feature of this incident: the pager said Kafka, the root cause lived in ClickHouse’s own system tables.
Investigation: when the failure is faster than the scrape
Point-in-time inspection found nothing. By the time anyone exec’d into the pod, SHOW PROCESSES and system.merges were clean, and memory sat at its ~2 GiB baseline. The metrics platform, scraping at 30-second intervals, showed the same flat line. Even the error text misled: current RSS: 8.85 GiB reads like a slow leak. It was not. The case broke open on ClickHouse’s internal 1-second-granularity metric log, which showed the true shape: baseline ~2 GiB, a single-second spike to the 21.60 GiB server ceiling, an OvercommitTracker kill, then straight back to baseline. A sub-second failure is invisible to a 30-second scrape by construction; only an in-engine time series could prove it.
With the spike established, the question became what allocated 20 GiB in one shot. The answer was upgrade residue. When a ClickHouse upgrade changes a system log table’s schema, it renames the old table with a numeric suffix and creates a fresh one; it never drops the old data. This fleet had been upgraded repeatedly, and the corpses had accumulated: a renamed trace log at 1.21 TiB, a renamed profiling log at 308 GiB. These zombie tables receive no writes and have no readers, but their parts still participate in background merges. One such merge on a 1+ TiB zombie allocated its way from 2 GiB to the 21.60 GiB cap within a second, got killed, and took the concurrent insert queries down with it.
Fix and prevention
The fix was deletion, with the right scope. Truncating the active system logs relieved forward pressure but touched nothing: TRUNCATE only acts on the current no-suffix tables. The zombies had to be dropped explicitly, past ClickHouse’s 50 GB drop guard, via a per-query override; a 1+ TiB drop completed in 1-3 minutes with the server serving throughout. Because the tables had no writers and no readers, the operation was risk-free by construction. Consumer lag did not drain on its own; the consumer needed a restart to break the rebalance loop.
Prevention went three ways. First, configuration: TTLs on the internal log tables that ship without one, the highest-volume log disabled outright, and the server memory ceiling pinned explicitly. Second, process: the upgrade runbook now ends with an audit of system tables for new suffixed residue, because this is a latent defect every upgrade re-creates. Third, fleet scope: one confirmed diagnosis triggered an immediate cross-cluster audit, which found 1.65 TiB of zombies on the worst cluster and 250 GiB on a sibling in another region, already producing 3 OOMs per day and climbing the same curve.
Lessons
- Monitoring granularity must match failure timescale. A 30-second scrape cannot testify about a one-second spike; for engines that keep 1-second internal metrics, that log is the primary evidence source after any OOM.
- Raising the pod’s memory limit was the obvious knob and the wrong one: it delays an unbounded-growth failure, it does not stop it.
- Upgrade residue is a horizontal defect. If an upgrade left debris on one cluster, it left debris on every cluster with the same history: diagnose once, then audit the fleet.