{
 "slug": "kubernetes-production-cutover-gating-adding-cryptographic-human-verifi",
 "topic_id": "TOPIC-041",
 "cluster": "CI/CD & Software Supply Chain",
 "tier": "Tier B",
 "title": "GitOps removed the deploy button, and the decision with it",
 "summary": "A GitOps controller reconciles the cluster to whatever the repository says. That is the design's value and it means a repository write is a production deployment.",
 "lede": "The premise of GitOps is that the repository is the desired state and the controller makes reality match. It works well. It also means the boundary between merging a change and running it in production has been deliberately removed.",
 "date": "2024-04-26",
 "category": "Developer",
 "author_id": "margot-reyes",
 "tags": [
  "GitOps",
  "Kubernetes",
  "ArgoCD",
  "Flux",
  "admission control",
  "deployment"
 ],
 "image_title": "GitOps Deploy Decision Gap",
 "schema": "Article",
 "key_takeaways": [
  "Continuous reconciliation makes repository write access equivalent to production deploy access.",
  "Sync waves and health checks control how a change rolls out, not whether it should.",
  "An admission webhook that requires a verified receipt for gated workloads places a decision back in the path without abandoning the GitOps model."
 ],
 "body": [
  {
   "type": "h2",
   "text": "What reconciliation collapses"
  },
  {
   "type": "diagram",
   "kind": "flow",
   "alt": "An admission webhook refuses gated fields without a receipt, whatever wrote the manifest",
   "caption": "The controller keeps reconciling everything. It is refused only on the gated subset, and the refusal surfaces as a sync error.",
   "nodes": [
    {
     "label": "Commit to the repository",
     "note": "= deploy, in GitOps"
    },
    {
     "label": "Controller reconciles",
     "note": "unmodified"
    },
    {
     "label": "Admission webhook inspects gated fields",
     "note": "image, RBAC, secrets, ingress",
     "good": true
    },
    {
     "label": "No valid receipt → denied",
     "note": "clear sync error",
     "good": true
    }
   ]
  },
  {
   "type": "p",
   "html": "In a pipeline model there are two steps: build the artefact, then deploy it. The second is an action someone takes."
  },
  {
   "type": "p",
   "html": "In a GitOps model there is one: change the repository. The controller notices and reconciles. Deployment is not an action anybody takes; it is a consequence."
  },
  {
   "type": "table",
   "head": [
    "Access",
    "Pipeline model",
    "GitOps model"
   ],
   "rows": [
    [
     "Repository write",
     "Can merge; deploy is separate",
     "<strong style=\"font-weight:600\">Can deploy to production</strong>"
    ],
    [
     "Deploy permission",
     "A distinct grant",
     "Held by the controller"
    ],
    [
     "Who decides to ship",
     "A person, at deploy time",
     "Whoever merged"
    ],
    [
     "Rollback",
     "An action",
     "A revert commit, then reconciliation"
    ]
   ]
  },
  {
   "type": "p",
   "html": "This is the intended behaviour and it has real benefits: declarative state, drift correction, a git history as the deployment record. The cost is that the repository's access control is now the production deployment control."
  },
  {
   "type": "h2",
   "text": "Where the controller's own credentials sit"
  },
  {
   "type": "p",
   "html": "The controller holds cluster-wide apply permission by necessity. It also holds credentials to read the repository, often across many repositories."
  },
  {
   "type": "p",
   "html": "Two consequences worth stating plainly. A compromised controller can apply anything to the cluster. And a compromised repository — or a compromised image reference inside a manifest — reaches production through a path with no human step."
  },
  {
   "type": "h2",
   "text": "What sync policy does not do"
  },
  {
   "type": "p",
   "html": "Sync waves, health assessment, pruning policy and automated rollback are all about how a change is applied and whether it appears healthy afterwards."
  },
  {
   "type": "p",
   "html": "None of them evaluate whether the change should be applied. A manifest that opens a service to the internet, mounts a secret into a new pod, or replaces an image with an unsigned one is applied in the correct wave, health-checked, and reported as successfully synced."
  },
  {
   "type": "h2",
   "text": "Putting a decision back in"
  },
  {
   "type": "p",
   "html": "The admission layer is the right place, because it sits in front of the API server and applies regardless of what wrote the manifest."
  },
  {
   "type": "code",
   "text": "# ValidatingAdmissionWebhook, scoped to gated namespaces\n\ndef admit(review):\n    obj = review.request.object\n    ns  = obj.metadata.namespace\n\n    if ns not in GATED_NAMESPACES:\n        return allow()\n\n    if not touches_gated_field(review):        # image, secrets,\n        return allow()                         # RBAC, ingress, hostPath\n\n    digest = canonical_digest(gated_fields(obj))\n    receipt = obj.metadata.annotations.get(\"manav.id/receipt\")\n    if not receipt:\n        return deny(\"gated change requires a signed receipt\")\n\n    r = verify(receipt, issuer_jwks=JWKS)      # offline\n    if r.digest != digest:\n        return deny(\"receipt does not match this manifest\")\n    if r.environment != cluster_environment():\n        return deny(\"receipt issued for a different environment\")\n    if expired(r) or revoked(r.nonce):\n        return deny(\"receipt expired or revoked\")\n\n    return allow()"
  },
  {
   "type": "p",
   "html": "The controller continues to reconcile everything. It is refused only on the gated subset, and the refusal surfaces as a sync error with a clear reason."
  },
  {
   "type": "h2",
   "text": "Choosing the gated fields"
  },
  {
   "type": "p",
   "html": "Gating whole namespaces would break the model — most changes are configuration and scaling that nobody should sign. Gate by field instead."
  },
  {
   "type": "ul",
   "items": [
    "Container image references in production workloads",
    "Service account bindings and RBAC objects",
    "Secret references and volume mounts of secrets",
    "Ingress, LoadBalancer services and network policy",
    "Host path mounts, privileged security contexts, capability additions"
   ]
  },
  {
   "type": "p",
   "html": "A replica count change, a resource limit adjustment or a config map update passes untouched. In practice this fires on image promotions and permission changes, which is roughly the set a human should be attached to."
  },
  {
   "type": "h2",
   "text": "Where the receipt comes from"
  },
  {
   "type": "p",
   "html": "The signing happens where the promotion decision is made — typically a release step that renders the change and asks for a signature, then writes the receipt annotation into the manifest that gets committed."
  },
  {
   "type": "p",
   "html": "The controller stays unmodified. It applies a manifest that happens to carry an annotation, and the admission webhook is what cares about it. This is deliberate: modifying the GitOps controller would couple you to its release cycle."
  },
  {
   "type": "h2",
   "text": "Failure mode and the fail-open temptation"
  },
  {
   "type": "p",
   "html": "If the webhook is unavailable, the admission configuration's failure policy decides. <code>Ignore</code> keeps the cluster deployable and removes the control exactly when something is wrong; <code>Fail</code> blocks gated changes during a webhook outage."
  },
  {
   "type": "p",
   "html": "Choose <code>Fail</code>, scope the webhook narrowly so an outage does not block ordinary operations, and run it with enough replicas that the outage is unlikely. A control that disappears under stress is not a control."
  },
  {
   "type": "h2",
   "text": "Gate by field, not by namespace"
  },
  {
   "type": "p",
   "html": "Gating whole namespaces breaks the model, because most changes are configuration and scaling that nobody should sign. Gate the fields where a change is consequential and let everything else reconcile freely."
  },
  {
   "type": "table",
   "caption": "The gated field set",
   "head": [
    "Field",
    "Why"
   ],
   "rows": [
    [
     "Container image references in production",
     "The promotion decision"
    ],
    [
     "Service account bindings and RBAC objects",
     "Authority changes"
    ],
    [
     "Secret references and secret volume mounts",
     "Access to credentials"
    ],
    [
     "Ingress, LoadBalancer services, network policy",
     "Exposure changes"
    ],
    [
     "Host path mounts, privileged contexts, capabilities",
     "Escape surface"
    ]
   ]
  },
  {
   "type": "p",
   "html": "A replica count, a resource limit or a config map update passes untouched. In practice the gate fires on image promotions and permission changes, which is roughly the set a human should be attached to."
  },
  {
   "type": "h2",
   "text": "Objections and honest limits"
  },
  {
   "type": "p",
   "html": "<strong style=\"font-weight:600\">“This breaks GitOps.”</strong> It does not. The repository stays the desired state and the controller still reconciles. A small set of fields requires a receipt to be admitted, and the receipt travels in the manifest as an annotation."
  },
  {
   "type": "p",
   "html": "<strong style=\"font-weight:600\">“We would modify the controller.”</strong> Do not. Admission applies regardless of what wrote the manifest and does not couple you to the controller's release cycle. Set failure policy to Fail, scope the webhook narrowly, and run enough replicas that an outage is unlikely."
  }
 ],
 "faq": [
  {
   "q": "Does this break the GitOps model?",
   "a": "No. The repository remains the desired state and the controller still reconciles. A small set of fields requires a receipt to be admitted."
  },
  {
   "q": "Why the admission layer rather than the controller?",
   "a": "Admission applies regardless of what wrote the manifest, and it does not couple you to the controller's release cycle."
  },
  {
   "q": "What should the failure policy be?",
   "a": "Fail. Scope the webhook narrowly and run enough replicas that an outage is unlikely. A control that disappears under stress is not a control."
  },
  {
   "q": "Which fields should be gated?",
   "a": "Image references, RBAC and service account bindings, secret mounts, network exposure, and privileged security contexts. Replica counts and config maps pass freely."
  }
 ],
 "sources": [
  {
   "t": "Kubernetes — Dynamic admission control",
   "u": "https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/"
  },
  {
   "t": "Argo CD — declarative GitOps documentation",
   "u": "https://argo-cd.readthedocs.io/en/stable/"
  },
  {
   "t": "NIST SP 800-190 — Application Container Security Guide",
   "u": "https://csrc.nist.gov/pubs/sp/800/190/final"
  }
 ],
 "related": [
  {
   "slug": "infrastructure-as-code-iac-poisoning-gating-terraform-pulumi",
   "title": "The CI runner that can rewrite your cloud estate",
   "category": "Developer"
  },
  {
   "slug": "sigstore-vs-manav-signing-artifacts-prove-approved-deployment",
   "title": "Signing artifacts does not prove who approved",
   "category": "Comparison"
  },
  {
   "slug": "github-copilot-pr-approval-soc2-supply-chain-risk",
   "title": "When an AI approves the pull request",
   "category": "AEO"
  }
 ],
 "image": "https://cdn.twc.sh/images/igcache/GitOps%20Deploy%20Decision%20Gap/1200_630/blog.jpg",
 "wordcount": 994,
 "url": "/blog/kubernetes-production-cutover-gating-adding-cryptographic-human-verifi.html",
 "reading_time": "4 min read",
 "meta_description": "A GitOps controller reconciles the cluster to whatever the repository says. That design means a repository write is a production deployment.",
 "hub": {
  "slug": "topics/software-supply-chain",
  "title": "Software supply chain authorization"
 },
 "answer": "Because continuous reconciliation is the design. Argo CD and Flux make the cluster match whatever the repository says, so deployment is not an action anyone takes — it is a consequence of a commit. Sync waves and health checks control how a change rolls out, never whether it should.",
 "answer_q": "Why does GitOps make repository write access equal to production deploy access?",
 "entities": [
  {
   "name": "Argo CD",
   "type": "SoftwareApplication",
   "url": "https://argo-cd.readthedocs.io/en/stable/",
   "primary": true
  },
  {
   "name": "Kubernetes",
   "type": "SoftwareApplication",
   "url": "https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/"
  }
 ],
 "glossary": [
  {
   "term": "Reconciliation",
   "def": "The controller loop that continuously makes cluster state match the repository. Its value and its risk are the same property."
  },
  {
   "term": "Admission webhook",
   "def": "A service the API server consults before persisting an object, which can allow or deny. It applies regardless of what produced the manifest."
  },
  {
   "term": "Failure policy",
   "def": "What happens when the webhook is unreachable. <code>Ignore</code> keeps the cluster deployable and removes the control exactly when something is wrong."
  }
 ],
 "checklist": {
  "title": "Adding the gate without breaking the cluster",
  "id": "gate",
  "desc": "Five steps.",
  "steps": [
   {
    "name": "Scope the webhook to gated namespaces and fields.",
    "text": "A narrow webhook can safely fail closed."
   },
   {
    "name": "Set failurePolicy to Fail.",
    "text": "A control that disappears under stress is not a control."
   },
   {
    "name": "Run enough replicas.",
    "text": "So failing closed does not become an availability incident."
   },
   {
    "name": "Sign at the promotion decision, not in the controller.",
    "text": "Write the receipt as an annotation into the manifest that gets committed."
   },
   {
    "name": "Verify offline in the webhook.",
    "text": "Against published keys, so admission does not depend on a network call to an issuer."
   }
  ]
 },
 "cta": {
  "title": "Where this fits in Manav",
  "html": "Manav receipts travel as a manifest annotation and verify offline inside the admission webhook. The controller is unmodified, the gate is enforced at the API server, and a manifest whose gated fields changed after signing is denied rather than applied.",
  "href": "../docs.html",
  "label": "See admission gating"
 }
}