Compiling a Document into a Control Loop: A Machine-Level Configuration Reconciler for 5G vRAN Bare Metal
Also in Chinese: 中文版
What it was, and why
Kubernetes runs a reconcile loop over containers: declare a desired state, observe the actual state, converge the difference, repeat. This project pushed that philosophy one layer down, onto the bare-metal hardware that carries real-time radio traffic in a 5G vRAN stack.
Intel’s cloud-ification of 5G moves the radio access network’s L1/L2 off purpose-built silicon and onto general-purpose Xeon servers with standard NICs and software (the FlexRAN reference architecture). Whether that software hits certified performance depends entirely on the hardware, firmware, and kernel being tuned to exactly the values Intel has validated — the Best Known Configuration, or BKC.
My work sat in the IaC layer: hardware-level configuration management for this fleet. The concrete deliverable was a self-authored, continuously running machine-level controller — one daemon per node, at the systemd layer.
The starting point is the whole point. Before this system, the BKC was not compiled into anything executable. It lived as prose and copy-paste shell snippets in a doc, as tribal knowledge in engineers’ heads, and as an assumption that “it was probably right when we installed it.” Provisioning a node meant opening that doc and pasting commands by hand — no enforcement, no audit, no drift detection. This was more primitive than Ansible: not “Ansible was not good enough,” but that there was no Ansible-tier layer at all. In one sentence, the project compiled a document into a control loop: desired state moved from scattered prose and human memory into a git-declared, machine-executable, continuously enforced form.
Why BKC is a desired state by nature
BKC is a validated recipe for a full stack that delivers certified performance: BIOS (C-state / P-state policy, Turbo, SR-IOV, VT-d/IOMMU, NUMA, power profile), kernel (an RT / low-latency build, boot parameters such as isolcpus, nohz_full, hugepages), NIC (the ice driver plus a DDP profile version), firmware, and OS tuning. It is, structurally, already a desired state — it just lacks a controller that continuously enforces it. That controller is what this project built. It also grounds an abstract claim in a concrete, commercially meaningful anchor: reconcile is a general idea, not a Kubernetes feature.
The constraints that drove it (the 5G side)
Three hard properties shaped every design decision:
- Deterministic latency budgets are microsecond-scale. Fronthaul timing (O-RAN 7.2x), C-state wake latency, cross-NUMA access — a single unclosed C-state or one drifted tuning value can eat the budget and violate a timing SLA. Determinism at the bottom layer is a physical requirement, not an optimization.
- Drift does not crash; it silently degrades. A node off-BKC usually does not error — it jitters, drops packets, breaks the fronthaul timing budget. The hardest failure to diagnose is the one where every health check is green and performance is simply worse.
- Provisioning complete does not equal certified performance in place. A node that installs, boots K8s, and runs pods says nothing about whether C-states are off, the NIC firmware is right, the DDP profile is correct, or hugepages and isolcpus actually landed. The question the system answers: how do we define and continuously enforce “5G-grade hardware readiness”?
Architecture
The design is pull-based, and the control plane is an observer rather than a commander. There are three data flows and none point from the center to a node: each node pulls its BKC profile from git and verifies the signature before applying; each pushes a heartbeat and compliance summary to a read-only aggregator; and before any disruptive action a node must acquire a token. The center degrades into a dashboard that can die without stopping a single node from enforcing.
Two facts force this shape. Convergence has to live at the systemd layer, not the K8s layer: you cannot assume K8s is even up (bare-metal delivery, testbed re-imaging, a node that just powered on), the drift lives in a layer K8s cannot see (BIOS, kernel cmdline, NIC firmware, sysctl), and a solution delivered to a customer must be self-contained — one systemd unit plus one agent binary, not a dependency on the customer already running an orchestrator. And two reconcile loops nest: systemd keeps the agent process alive, the agent keeps the node’s configuration from drifting — the same structure as kube-controller-manager over pods, with the target swapped for Linux plus hardware.
The engineering core: reconcile graded by risk
This is the line between running a script and designing a controller. A one-shot config push is not enough when the machines are live and constantly changed, so such a controller must classify every configuration item along two axes — can it be audited, and what is the cost and risk of remediating it — and act accordingly:
- 🟢 online and reversible (sysctl, CPU governor, IRQ affinity, tuned profile): reconcile immediately.
- 🟡 requires draining the node first (NIC ice driver and DDP reload, which needs the interface down).
- 🟠 requires a maintenance window and reboot (BIOS values, kernel version, kernel cmdline — none take effect until reboot).
- 🔴 audit-only, never auto-changed (NIC NVM firmware, FEC accelerator firmware — a flash is high-risk).
Three principles fall out. Audit always precedes and is independent of update: even for items the controller never changes, continuously producing per-node compliance plus a drift timestamp lights up the silent-degradation blind spot and becomes a delivery-compliance artifact. Remediation must be traffic-aware, because the target is a live node carrying real-time radio traffic. And every action is idempotent and forward-only: current equal to desired is always a no-op.
The audit-and-monitor half is the confirmed core of what I built and ran: the daemon continuously inspects each node — BIOS via Redfish/BMC and in-band vendor tools, kernel via /proc/cmdline, tuning via sysctl and tuned, NIC via ethtool and devlink — and reports compliance. The full risk-tiered model above, with drain, maintenance windows, and reboot coordination, is stated here as how such a controller must be built to act safely on live nodes.
Problems a controller like this must solve
Framed as design, not war stories — these are the problems any honest version of this system has to answer:
- The monitor must not perturb the monitored. On a node with isolcpus, the isolated cores run the RT RAN workload. An audit agent scheduled onto those cores, or one whose child processes escape onto them, becomes a noisy neighbor and manufactures the very jitter it exists to protect against. It must be pinned to housekeeping cores and refuse to run on isolated ones, its children constrained by the same cpuset.
- A cross-reboot state machine. Changes that only take effect after reboot (BIOS, kernel) mean the agent has to survive the reboot it triggers: a persisted state machine plus an intent marker read on boot, bounded retries to break boot loops, and a bootloader last-known-good fallback so a bad cmdline cannot brick a customer node.
- Reboot tokens rate-limited by failure domain. Under a pull model a single BKC bump in git is seen by every node on the next poll — without a distributed lease capping how many nodes in a domain may be disrupted at once, they would all reboot together, a self-inflicted correlated failure. Fail-safe default: token service unreachable means do not reboot.
- Signed profiles. The agent runs as root on every machine and can flash firmware, and desired state comes from git. The trust boundary has to be locked with signature verification (fail-closed), an INTENT audit log per mutation, reviewed protected branches, and human approval for the highest-risk class (firmware flash) — especially because this ships to customers running the agent as root on their own bare metal.
This is the same blast-radius discipline as the Kubernetes upgrade project’s serial:1, quorum math, and PodDisruptionBudget — one mind, two layers.
Scope
The system deliberately stops at the systemd layer and does not become a K8s Node Operator. The scenario is precisely “below K8s, on customer bare metal, possibly with no K8s at all,” so moving up into an Operator would contradict the reason the project exists. Exposing compliance as a Node label for the scheduler (the NFD-style neighbor) is an imaginable adjacent direction, explicitly out of scope.
Takeaways
- Reconcile is target-agnostic. A document becomes a control loop once desired state is declarative, machine-executable, and continuously enforced — the target can be a container or a NIC firmware version. This project is the argument that the idea belongs below K8s as much as above it.
- Audit is worth more than auto-remediation on day one. Making a hardware-layer blind spot visible, as per-node compliance and drift timestamps, is the value; safe remediation is the second step, gated by risk and traffic.