Developer Guardrails on Kubernetes: RBAC Says Which Doors, Admission Says Which Fields

发表于 分类于
Developers needed to change their own deployments without an infra ticket, but not the two fields that move cost and capacity for everyone else. RBAC cannot express that. A Kyverno admission policy can — rolled out Audit → verify → Enforce across five clusters, one of them PCI-scoped, with a one-line rollback.

Also in Chinese: 中文版

Why

Access to our clusters came in three flavors: admin, read-only, and a “developer” role that was read-only plus the right to delete pods and jobs. In practice that meant any change beyond restarting something — an env var, an image tag, a config tweak — went through infra. Developers wanted to self-serve. Infra did not want to hand out a role that could also quietly double a deployment’s replicas or triple its memory requests, because those two fields are how one team’s change turns into node pressure and a cost line for everybody else on the cluster.

So the requirement was specific: full write access to workloads, except spec.replicas and containers[].resources.

Three layers, one question

Before touching YAML I wrote the access path down, because a role change that only lands in one layer is a role that does not work. Every permission system answers the same expression — Allow(subject, action, resource, context) — and differs only in where each part is enforced. Ours had three enforcement points:

Layer Its job What a new role needs there
Approval platform decides who gets the key a role option, an approval policy, a mapping
Bastion holds the key and records its use an account bound to a ServiceAccount token
Kubernetes decides which doors the key opens ClusterRole + ServiceAccount + binding

The one-line version I used to explain it: the approval platform decides whether you get a key, the bastion keeps the key and logs every use, Kubernetes decides which doors it opens. This project is the third layer, plus a finer lock inside it.

RBAC stops at the door

RBAC’s vocabulary is resource × verb. patch deployments is one grant, and it covers every field of a Deployment — env, image, replicas, resources alike. There is no RBAC rule that says “patch, but not that field.” Splitting the role further does not help; the granularity simply is not there.

Field-level control belongs one step later in the request path, at admission. The API server authenticates, authorizes against RBAC, then hands the object to admission webhooks before it is persisted. So the design became two layers with separate jobs:

  • RBAC grants a developer-edit role full CRUD on pods, deployments, statefulsets, daemonsets, services, configmaps, jobs, ingresses and PVCs — and nothing cluster-scoped beyond reading namespaces.
  • A Kyverno ClusterPolicy (a validating admission webhook) denies an UPDATE from the developer ServiceAccounts when spec.replicas differs from the old object, or when any container’s resources differs from the same-named container in the old object.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
- name: deny-replicas-change
  match:
    any:
    - resources:
        kinds: [Deployment, Deployment/scale]
  preconditions:
    all:
    - key: "{{ request.operation }}"
      operator: Equals
      value: UPDATE
    - key: "{{ request.userInfo.username }}"
      operator: AnyIn
      value: ["system:serviceaccount:<ns>:developer", "system:serviceaccount:<ns>:developer-edit"]
  validate:
    message: "developers cannot change spec.replicas or scale"
    deny:
      conditions:
        any:
        - key: "{{ request.object.spec.replicas || `null` }}"
          operator: NotEquals
          value: "{{ request.oldObject.spec.replicas || `null` }}"

Three details decided whether the policy actually held:

  1. kubectl scale is a different resource. It writes the deployments/scale subresource, not the Deployment. A policy that matches only Deployment looks correct in review and is bypassed by the most common way people change replicas. The rule matches both kinds.
  2. Resources are compared per container, by name, with null-safe defaults on both sides. Comparing the container list as a whole flags harmless reorderings; skipping the null default turns “no resources set” into an evaluation error instead of a decision.
  3. RBAC runs before admission, so the test has to prove RBAC first. If the role cannot patch, the request is rejected before Kyverno ever sees it — and a validation run would report “replicas change denied” for the wrong reason. The first step of every rollout is kubectl auth can-i patch deployment --as=<sa>; the policy test only counts once that says yes.

Rolling it out without a surprise

A policy that denies writes is a production change even when it only targets developers, so it went out the same way any risky change should:

  1. Audit — the policy logs violations into policy reports but blocks nothing.
  2. Verify against a disposable test deployment, never a real workload: adding an env var must succeed; changing replicas, resources, and scale must each show up as a violation.
  3. Enforce — flip validationFailureAction, rerun the same checks: env still succeeds, the other three are now rejected with the policy’s message.
  4. Rollback is one patch back to Audit, which keeps the logging and stops the blocking. Deleting the policy is a second, separate step.

Every step is idempotent (kubectl apply, helm upgrade --install) so a half-finished cluster can be rerun from the top. Clusters went one at a time — RBAC alone first, then Kyverno and the Audit policy, then Enforce — and any failed check stopped the rollout at that cluster. A pre-rollout survey of all five clusters turned into a small decision matrix: one cluster already had everything, three needed the ServiceAccount, role, binding and Kyverno from scratch, and the PCI-scoped cluster already ran Kyverno, so only the RBAC objects and the policy were layered on. All five ended in Enforce.

What it does not solve

The honest gap is attribution. Developers reach the cluster through the bastion using a shared ServiceAccount token, so the Kubernetes audit log records serviceaccount:developer-edit, not the person. Who did what lives only in the bastion’s session log. That is acceptable as a first step and wrong as an end state; the upgrade path is per-user identity — impersonation from the bastion, or OIDC — so the API server’s own audit trail names a human.

The policy is also a denylist of two fields. The next tightening, if needed, is the inverse: an allowlist of what developers may change (env, args, image, annotations) and a deny for everything else.

Takeaways

  • RBAC answers “which resource, which verb.” Anything finer — which field, which value — is an admission question. Trying to get field-level control out of role splitting is looking for granularity that does not exist.
  • A policy is only as strong as its narrowest match. Subresources like /scale are separate kinds, and they are exactly where a correct-looking rule leaks.
  • Test the layers in request order. An admission test run against a role that RBAC already denies passes for the wrong reason.
  • Audit before Enforce, with a one-line way back. The cost of a staged rollout is a day; the cost of a policy that blocks a real deploy is an incident.

The sanitized policy, RBAC templates and staged rollout scripts are at github.com/shaorui0/k8s-developer-guardrails.