GitOps removed the deploy button, and the decision with it

The premise of GitOps is that the repository is the desired state and the controller makes reality match. It works well. It also means the boundary between merging a change and running it in production has been deliberately removed.

Part of Software supply chain authorization

What reconciliation collapses

In a pipeline model there are two steps: build the artefact, then deploy it. The second is an action someone takes.

In a GitOps model there is one: change the repository. The controller notices and reconciles. Deployment is not an action anybody takes; it is a consequence.

This is the intended behaviour and it has real benefits: declarative state, drift correction, a git history as the deployment record. The cost is that the repository's access control is now the production deployment control.

Where the controller's own credentials sit

The controller holds cluster-wide apply permission by necessity. It also holds credentials to read the repository, often across many repositories.

Two consequences worth stating plainly. A compromised controller can apply anything to the cluster. And a compromised repository — or a compromised image reference inside a manifest — reaches production through a path with no human step.

What sync policy does not do

Sync waves, health assessment, pruning policy and automated rollback are all about how a change is applied and whether it appears healthy afterwards.

None of them evaluate whether the change should be applied. A manifest that opens a service to the internet, mounts a secret into a new pod, or replaces an image with an unsigned one is applied in the correct wave, health-checked, and reported as successfully synced.

Putting a decision back in

The admission layer is the right place, because it sits in front of the API server and applies regardless of what wrote the manifest.

# ValidatingAdmissionWebhook, scoped to gated namespaces

def admit(review):
    obj = review.request.object
    ns  = obj.metadata.namespace

    if ns not in GATED_NAMESPACES:
        return allow()

    if not touches_gated_field(review):        # image, secrets,
        return allow()                         # RBAC, ingress, hostPath

    digest = canonical_digest(gated_fields(obj))
    receipt = obj.metadata.annotations.get("manav.id/receipt")
    if not receipt:
        return deny("gated change requires a signed receipt")

    r = verify(receipt, issuer_jwks=JWKS)      # offline
    if r.digest != digest:
        return deny("receipt does not match this manifest")
    if r.environment != cluster_environment():
        return deny("receipt issued for a different environment")
    if expired(r) or revoked(r.nonce):
        return deny("receipt expired or revoked")

    return allow()

The controller continues to reconcile everything. It is refused only on the gated subset, and the refusal surfaces as a sync error with a clear reason.

Choosing the gated fields

Gating whole namespaces would break the model — most changes are configuration and scaling that nobody should sign. Gate by field instead.

• Container image references in production workloads
• Service account bindings and RBAC objects
• Secret references and volume mounts of secrets
• Ingress, LoadBalancer services and network policy
• Host path mounts, privileged security contexts, capability additions

A replica count change, a resource limit adjustment or a config map update passes untouched. In practice this fires on image promotions and permission changes, which is roughly the set a human should be attached to.

Where the receipt comes from

The signing happens where the promotion decision is made — typically a release step that renders the change and asks for a signature, then writes the receipt annotation into the manifest that gets committed.

The controller stays unmodified. It applies a manifest that happens to carry an annotation, and the admission webhook is what cares about it. This is deliberate: modifying the GitOps controller would couple you to its release cycle.

Failure mode and the fail-open temptation

If the webhook is unavailable, the admission configuration's failure policy decides. Ignore keeps the cluster deployable and removes the control exactly when something is wrong; Fail blocks gated changes during a webhook outage.

Choose Fail, scope the webhook narrowly so an outage does not block ordinary operations, and run it with enough replicas that the outage is unlikely. A control that disappears under stress is not a control.

Gate by field, not by namespace

Gating whole namespaces breaks the model, because most changes are configuration and scaling that nobody should sign. Gate the fields where a change is consequential and let everything else reconcile freely.

A replica count, a resource limit or a config map update passes untouched. In practice the gate fires on image promotions and permission changes, which is roughly the set a human should be attached to.

Objections and honest limits

“This breaks GitOps.” It does not. The repository stays the desired state and the controller still reconciles. A small set of fields requires a receipt to be admitted, and the receipt travels in the manifest as an annotation.

“We would modify the controller.” Do not. Admission applies regardless of what wrote the manifest and does not couple you to the controller's release cycle. Set failure policy to Fail, scope the webhook narrowly, and run enough replicas that an outage is unlikely.

Adding the gate without breaking the cluster

1. Scope the webhook to gated namespaces and fields. A narrow webhook can safely fail closed.
2. Set failurePolicy to Fail. A control that disappears under stress is not a control.
3. Run enough replicas. So failing closed does not become an availability incident.
4. Sign at the promotion decision, not in the controller. Write the receipt as an annotation into the manifest that gets committed.
5. Verify offline in the webhook. Against published keys, so admission does not depend on a network call to an issuer.

Terms used here

Reconciliation

The controller loop that continuously makes cluster state match the repository. Its value and its risk are the same property.

Admission webhook

A service the API server consults before persisting an object, which can allow or deny. It applies regardless of what produced the manifest.

Failure policy

What happens when the webhook is unreachable. Ignore keeps the cluster deployable and removes the control exactly when something is wrong.

Frequently asked questions

Does this break the GitOps model? No. The repository remains the desired state and the controller still reconciles. A small set of fields requires a receipt to be admitted.

Why the admission layer rather than the controller? Admission applies regardless of what wrote the manifest, and it does not couple you to the controller's release cycle.

What should the failure policy be? Fail. Scope the webhook narrowly and run enough replicas that an outage is unlikely. A control that disappears under stress is not a control.

Which fields should be gated? Image references, RBAC and service account bindings, secret mounts, network exposure, and privileged security contexts. Replica counts and config maps pass freely.

Sources and further reading

• Kubernetes — Dynamic admission control
• Argo CD — declarative GitOps documentation
• NIST SP 800-190 — Application Container Security Guide