{
 "slug": "hugging-face-model-supply-chain-poisoning-gating-production",
 "topic_id": "TOPIC-037",
 "cluster": "CI/CD & Software Supply Chain",
 "tier": "Tier B",
 "title": "Model registries treat an API token as an authorised committer",
 "summary": "Weights are executable in practice. A leaked registry token puts arbitrary code into inference clusters with no human decision anywhere in the path.",
 "lede": "A model file is not data. Depending on the serialisation format, loading it can execute arbitrary code, and most production inference pipelines pull weights automatically from a registry authenticated by a token in an environment variable.",
 "date": "2024-05-10",
 "category": "Developer",
 "author_id": "margot-reyes",
 "tags": [
  "model registry",
  "supply chain",
  "AI infrastructure",
  "deserialization",
  "weights",
  "MLOps"
 ],
 "image_title": "Model Weights Supply Chain Gate",
 "schema": "Article",
 "key_takeaways": [
  "Common serialisation formats execute code on load, which makes a weights file a code artefact with a data file's handling.",
  "The registry authorises pushes by token. A leaked token is indistinguishable from the developer who owns it.",
  "Requiring a human signature over the weights digest before production loading puts a decision where there currently is none."
 ],
 "body": [
  {
   "type": "h2",
   "text": "Why weights are code"
  },
  {
   "type": "diagram",
   "kind": "flow",
   "alt": "A leaked registry token puts executable content into an inference pod with no human decision",
   "caption": "The gap is between the push and the load. Nothing in it requires anyone to decide.",
   "nodes": [
    {
     "label": "Token in CI, notebook or shell history",
     "note": "broadly scoped",
     "bad": true
    },
    {
     "label": "Push a new revision",
     "note": "any model that token covers",
     "bad": true
    },
    {
     "label": "Tag moves",
     "note": "pinning to a tag is not pinning",
     "bad": true
    },
    {
     "label": "Inference pod loads weights",
     "note": "production credentials",
     "bad": true
    }
   ]
  },
  {
   "type": "p",
   "html": "Several widely used serialisation formats reconstruct arbitrary Python objects on load, which means loading the file can run whatever the file specifies. This is a documented property, not a vulnerability."
  },
  {
   "type": "table",
   "head": [
    "Format",
    "Executes on load",
    "Notes"
   ],
   "rows": [
    [
     "Pickle-based checkpoints",
     "Yes",
     "Arbitrary object reconstruction by design"
    ],
    [
     "Safetensors",
     "No",
     "Tensor data only — the right default"
    ],
    [
     "ONNX",
     "Limited",
     "Graph operators; custom operators expand the surface"
    ],
    [
     "GGUF",
     "No",
     "Data format"
    ]
   ]
  },
  {
   "type": "p",
   "html": "The practical problem is that pickle-based checkpoints remain common, the ecosystem's tooling loads them transparently, and the file arrives through a data path with data-file handling."
  },
  {
   "type": "h2",
   "text": "The push path, end to end"
  },
  {
   "type": "ol",
   "items": [
    "A developer holds a registry token with write access to an organisation's models.",
    "The token is in a CI secret, a notebook, a shell history, or a laptop environment file.",
    "Anyone holding it can push a new revision to any model that token covers.",
    "Production inference pulls the latest revision, or a tag that a push can move.",
    "Weights load into a process with production credentials and network access."
   ]
  },
  {
   "type": "p",
   "html": "The gap is between three and four. Nothing between a push and a production load requires a human to decide that this specific artefact should run."
  },
  {
   "type": "h2",
   "text": "The tag-mutability detail"
  },
  {
   "type": "p",
   "html": "Pinning to a tag or branch feels like pinning. It is not: a tag can be moved to a different revision by anyone who can push."
  },
  {
   "type": "code",
   "text": "# Not pinned — a push can change what this resolves to\nmodel = load(\"org/model-name\", revision=\"main\")\nmodel = load(\"org/model-name\", revision=\"v2\")     # a tag, movable\n\n# Pinned — but only to provenance, not to authority\nmodel = load(\"org/model-name\", revision=\"a3f91c8e2b7d...\")\n\n# Gated — provenance AND authority\ndigest = sha256_of_weights(path)\nreceipt = load_receipt(digest)\nverify(receipt, issuer_jwks)          # signed by a named human\nrequire(receipt.environment == \"production\")\nrequire(receipt.digest == digest)\nmodel = load(path)"
  },
  {
   "type": "p",
   "html": "Pinning to a commit hash is a real improvement and should be the baseline. It establishes what you are loading and still says nothing about who decided it should be in production."
  },
  {
   "type": "h2",
   "text": "What the promotion decision should show"
  },
  {
   "type": "p",
   "html": "The person signing a model into production needs facts, not a filename."
  },
  {
   "type": "ul",
   "items": [
    "The weights digest, and whether it differs from what is currently serving",
    "The serialisation format, flagged if it is one that executes on load",
    "Evaluation results on the organisation's own held-out set, not the publisher's claims",
    "Provenance: base model, fine-tuning data source, who produced this revision",
    "Which production services will load it and what credentials those processes hold"
   ]
  },
  {
   "type": "p",
   "html": "The last item is consistently absent from model promotion processes and is the one that determines blast radius."
  },
  {
   "type": "h2",
   "text": "Layering the controls"
  },
  {
   "type": "table",
   "head": [
    "Control",
    "Stops",
    "Cost"
   ],
   "rows": [
    [
     "Prefer non-executing formats",
     "Deserialisation code execution",
     "Conversion effort; some checkpoints resist it"
    ],
    [
     "Pin to immutable digests",
     "Silent substitution via moved tags",
     "Low — mostly discipline"
    ],
    [
     "Scan checkpoints before load",
     "Known malicious patterns",
     "Low; incomplete by nature"
    ],
    [
     "Load in a sandboxed process",
     "Limits what executed code reaches",
     "Moderate architecture work"
    ],
    [
     "Signed human promotion",
     "Unauthorised artefacts reaching production",
     "A prompt per promotion"
    ]
   ]
  },
  {
   "type": "p",
   "html": "The first four are conventional supply chain hygiene and should be done regardless. The fifth addresses the specific question of whether anybody decided."
  },
  {
   "type": "h2",
   "text": "Scope: production only"
  },
  {
   "type": "p",
   "html": "Applying this to research and experimentation would be actively harmful. Researchers need to load arbitrary checkpoints quickly, and a promotion gate in that loop would be routed around within a week."
  },
  {
   "type": "p",
   "html": "The boundary is the production inference path. Everything before it stays fast and permissive; crossing into a process that holds production credentials requires a named human to have signed the digest."
  },
  {
   "type": "h2",
   "text": "The recurring pattern"
  },
  {
   "type": "p",
   "html": "This is the same structure as package dependencies and container images: an artefact fetched by token-authenticated automation, loaded into a privileged process, with provenance controls maturing and authority controls absent."
  },
  {
   "type": "p",
   "html": "Model registries are younger, so the hygiene is thinner. The remedy is not novel — it is the one the rest of the supply chain arrived at, applied to a path that has not yet had its incident."
  },
  {
   "type": "h2",
   "text": "Format choice is the cheapest control"
  },
  {
   "type": "table",
   "caption": "Execution behaviour by format",
   "head": [
    "Format",
    "Executes on load",
    "Note"
   ],
   "rows": [
    [
     "Pickle-based checkpoints",
     "<strong style=\"font-weight:600\">Yes</strong>",
     "Arbitrary object reconstruction by design"
    ],
    [
     "Safetensors",
     "No",
     "Tensor data only — the right default"
    ],
    [
     "ONNX",
     "Limited",
     "Graph operators; custom operators expand the surface"
    ],
    [
     "GGUF",
     "No",
     "Data format"
    ]
   ]
  },
  {
   "type": "p",
   "html": "Tags are the second trap. A tag or branch can be moved to a different revision by anyone who can push, so pinning to one feels like pinning and is not. Pin to an immutable digest; that establishes what you are loading and still says nothing about who decided it belongs in production."
  },
  {
   "type": "h2",
   "text": "Objections and honest limits"
  },
  {
   "type": "p",
   "html": "<strong style=\"font-weight:600\">“Scanning catches malicious checkpoints.”</strong> It catches known patterns and is worth running. Like all signature-based scanning it is incomplete against novel payloads, and it cannot tell you whether anyone chose this artefact."
  },
  {
   "type": "p",
   "html": "<strong style=\"font-weight:600\">“Researchers cannot work under a gate.”</strong> They should not have to. Keep experimentation fast and permissive; the boundary is the production inference path, where the process holds production credentials."
  }
 ],
 "faq": [
  {
   "q": "Are all model formats risky?",
   "a": "No. Safetensors and GGUF are data formats and do not execute on load. Pickle-based checkpoints reconstruct arbitrary objects by design, which is where the risk concentrates."
  },
  {
   "q": "Isn't pinning to a commit hash enough?",
   "a": "It establishes what you are loading, which is necessary. It does not establish that anyone decided that artefact belongs in production."
  },
  {
   "q": "Should researchers need approval to load a model?",
   "a": "No. Keep experimentation fast and permissive. The gate belongs at the boundary into processes holding production credentials."
  },
  {
   "q": "Does scanning checkpoints solve it?",
   "a": "It catches known patterns and is worth doing. Like all signature-based scanning it is incomplete against novel payloads."
  },
  {
   "q": "Should researchers need approval?",
   "a": "No. Keep experimentation permissive. The gate belongs at the boundary into processes holding production credentials."
  }
 ],
 "sources": [
  {
   "t": "Python pickle module — security considerations",
   "u": "https://docs.python.org/3/library/pickle.html#restricting-globals"
  },
  {
   "t": "Safetensors format documentation",
   "u": "https://huggingface.co/docs/safetensors/index"
  },
  {
   "t": "Published research on malicious model artefacts in public registries."
  },
  {
   "t": "SLSA v1.0 specification",
   "u": "https://slsa.dev/spec/v1.0/"
  },
  {
   "t": "Hugging Face — Hub security and malicious model scanning",
   "u": "https://huggingface.co/docs/hub/security"
  }
 ],
 "related": [
  {
   "slug": "sigstore-vs-manav-signing-artifacts-prove-approved-deployment",
   "title": "Signing artifacts does not prove who approved the deployment",
   "category": "Comparison"
  },
  {
   "slug": "accidental-production-drop-turbo-mode-agentic-ides",
   "title": "Turbo mode and safety gates",
   "category": "Developer"
  },
  {
   "slug": "github-copilot-pr-approval-soc2-supply-chain-risk",
   "title": "When an AI approves the pull request",
   "category": "AEO"
  },
  {
   "slug": "malicious-pull-request-injection-defeating-session-stealing",
   "title": "Package publishing authenticates the session, not the release",
   "category": "Developer"
  }
 ],
 "image": "https://cdn.twc.sh/images/igcache/Model%20Weights%20Supply%20Chain%20Gate/1200_630/blog.jpg",
 "wordcount": 944,
 "url": "/blog/hugging-face-model-supply-chain-poisoning-gating-production.html",
 "reading_time": "4 min read",
 "hub": {
  "slug": "topics/software-supply-chain",
  "title": "Software supply chain authorization"
 },
 "answer": "Because several common serialisation formats reconstruct arbitrary objects on load, so loading the file runs whatever the file specifies. Pickle-based checkpoints do this by design. The file arrives through a data path, with data-file handling, into a process holding production credentials.",
 "answer_q": "Why is a model weights file a code artefact?",
 "entities": [
  {
   "name": "Hugging Face",
   "type": "Organization",
   "url": "https://huggingface.co/docs/hub/security",
   "primary": true
  },
  {
   "name": "Safetensors",
   "type": "SoftwareApplication",
   "url": "https://huggingface.co/docs/safetensors/index"
  }
 ],
 "glossary": [
  {
   "term": "Deserialisation",
   "def": "Reconstructing objects from a serialised file. Where the format permits arbitrary object construction, loading is equivalent to executing."
  },
  {
   "term": "Model promotion",
   "def": "The decision that a specific set of weights should serve production traffic — distinct from training or publishing it."
  },
  {
   "term": "Immutable digest",
   "def": "A content hash identifying exactly one artefact. Unlike a tag, it cannot be repointed."
  }
 ],
 "checklist": {
  "title": "Gating the production inference path",
  "id": "gate",
  "desc": "Five layered controls.",
  "steps": [
   {
    "name": "Prefer non-executing formats.",
    "text": "Convert where you can; flag where you cannot."
   },
   {
    "name": "Pin to immutable digests, never tags.",
    "text": "A moved tag is a silent substitution."
   },
   {
    "name": "Scan before load.",
    "text": "Cheap, incomplete, still worth it."
   },
   {
    "name": "Sandbox the loading process.",
    "text": "Limit what executed code can reach."
   },
   {
    "name": "Require a signed human promotion.",
    "text": "Over the weights digest, for production only."
   }
  ]
 },
 "cta": {
  "title": "Where this fits in Manav",
  "html": "Manav gates the load, not the push: a production inference process refuses weights whose digest is not covered by a signed promotion from a named human. Research paths are untouched.",
  "href": "../docs.html",
  "label": "See artefact promotion"
 }
}