僵尸系统表:30 秒采集看不见的亚秒级 OOM
一条 Kafka 消费积压告警,真因是 ClickHouse 里超过 1 TiB 的升级残留,被一次一秒内完成的后台 merge 引爆
一条 Kafka 消费积压告警,真因是 ClickHouse 里超过 1 TiB 的升级残留,被一次一秒内完成的后台 merge 引爆
reconcile、IaC、冗余、降级、限流,这几条现代云管理方法论背后其实是同一个问题:为了保住整体,你愿意主动放弃什么。好的 infra engineer,好在能在约束下做出这个取舍。
junior SRE 把 system metrics 和 app metrics 横着切,把 CPU 标出来就算监控。真正的转折是从上往下做:不同的主体关心不同的指标,每个主体的 SLO 不同,dashboard 也不同,dashboard 存在的意义是回答问题。
AI makes many things faster, but the professional gap is widening. Juniors get amplified. Seniors become more valuable but harder for organizations to recognize.
When everyone can use AI to write code, run analysis, and generate reports, execution efficiency stops being scarce. The market reprices: what capabilities are truly scarce? What can’t AI replace?
真正变化的是定价机制 AI 让很多事情变快了,但职业差距在拉大。junior 被 AI 放大,senior 更值钱,但 senior 反而更难被组织看见。 所有人都能用 AI 写代码、做分析、出报告,执行效率就不再稀缺了,市场也在重新定价:什么能力真正稀缺,AI 替不了。这篇想聊的就是这件事,不涉及怎么用 AI、prompt 工程或者求职技巧。 职业价值来自降低不确定性 不确定环境里,职业价值可以写成一个公式: 价值 = 正确性 x 执行效率 AI…
We keep saying “production environment” like it’s just another deployment target. It’s not. Demo systems optimize for success. Production systems optimize for survival. Completely different species.
When dashboards are all green but users are already down, you’re facing a control system failure, not an observability problem.
02:17 AM. A major customer reports massive request timeouts in the support channel.
02:19. Oncall gets paged.
02:21. Grafana opens. All green. CPU normal. Memory normal. Error rate near zero. The system looks completely healthy.
02:25. More customers report timeouts. Business leaders join the call: How many customers affected? What’s the impact scope? Do we need to rollback? How long to recover?
No one can answer. Monitoring systems exist, but they provide nothing usable for decisions.
这篇文章来自一次真实事故的复盘,记录的是生产组织在最需要可见性的时候为什么反而看不见。 那次凌晨两点的事故 现场 凌晨 02:17,客户支持群冒出第一条消息:某大客户请求大量超时。 02:19,Oncall 被拉醒。 02:21,打开 Grafana,一片绿色。CPU 正常,Memory 正常,Error rate 接近 0。系统看上去啥事没有。 02:25,更多客户报告超时,业务负责人进会议开始问:影响多少客户?范围多大?要不要回滚?多久能恢复?没人能答上来。监控系统在那里,但它没给出任何能拿来做决策的信息。那一刻我意识到,监控没坏,但已经失效了。…