Re-performing controls offline: what changes when evidence verifies
Three weeks of an auditor's time goes into requesting tickets, matching them to releases and forming a view on a sample of forty. The same assurance, over five hundred releases, is a two-second command — if the evidence was designed to be checked.
What changes when evidence can be verified by machine?
Sampling stops being necessary. Auditors sample because reading evidence by hand costs money, so a population of 40,000 changes becomes a sample of 25. When each item verifies in milliseconds against a published key, the whole population becomes testable and the finding changes from an inference to a count.
- Sampling is a consequence of evidence format, not an inherent feature of auditing. Verifiable evidence permits population testing.
- Re-performance without access to the audited entity's systems is a stronger form of evidence than inquiry or inspection.
- The exceptions surfaced by full-population testing are usually process defects nobody knew about, not fraud.
Part of Software supply chain authorization
The evidence hierarchy
Audit methodology ranks evidence roughly like this.
| Procedure | Strength | Typical use for change control |
|---|---|---|
| Inquiry | Weakest | Asking how the process works |
| Observation | Weak | Watching an approval happen |
| Inspection | Moderate | Reading tickets and screenshots |
| Re-performance | Strongest | Independently executing the control |
Change control is normally tested by inspection, because re-performing a human approval is not possible. You cannot independently re-execute somebody's decision to approve a release.
A signed receipt changes this. The auditor cannot re-perform the decision, but they can independently re-perform the verification — which is what the control's integrity actually rests on.
What full-population testing looks like
$ verify --population ./receipts \
--issuer-jwks ./jwks.json \
--require-uv \
--require-distinct-approver \
--report csv > exceptions.csv
Population: 517 production releases
Receipts present: 517 (100.0%)
Signature valid: 517
Issuer countersignature valid: 517
Approver credential enrolled: 517
User verification present: 511 (6 exceptions)
Deployed commit == approved: 515 (2 exceptions)
Approver ≠ author: 509 (8 exceptions)
Total distinct exceptions: 14
Network calls: 0 Elapsed: 1.9sNote the last line. No call to the audited entity's systems, no read-only account, no reliance on their logging. The receipts and the published key are the whole input.
Why offline matters for independence
Evidence obtained by querying the audited entity's platform inherits that platform's integrity assumptions. If the platform's records could be altered, so could the evidence, and the auditor is testing the control using the system the control governs.
A signature that verifies against a published key does not have this property. Its validity is a mathematical fact about bytes, independent of the systems that produced them.
There is also a practical dimension. Provisioning auditor access to production systems is a security review, a procurement conversation and a set of accounts to deprovision afterwards. Handing over a directory of files is none of those things.
What the exceptions usually are
Full-population testing finds things sampling misses, and they are rarely dramatic.
- A handful of releases approved by the person who wrote the change — usually a small team where the second approver was unavailable
- Approvals from a credential without user verification, because one platform's default differs
- Deployed commits that differ from approved ones, from a re-tag or a hotfix appended after approval
- A cluster of exceptions in one week, which turns out to be an incident response period
None of these is fraud. All of them are process defects that a forty-item sample would have missed, and they are the kind of finding that improves the control rather than embarrassing anyone.
Being honest about what is covered
Verification answers a bounded question: the receipt is authentic, the approver's credential was enrolled, the signature covers this commit, and the credential performed user verification.
It does not establish that the approver understood the change, that the review was adequate, or that the control design is appropriate. Those remain matters of judgement and always will.
What changes is the split of the auditor's time: less spent establishing that evidence exists and is consistent, more spent on whether the control is the right one. That is a better use of expensive expertise.
Making evidence auditable by design
- Produce the receipt as part of the control, not as a reporting step afterwards. Evidence assembled later is evidence that can be assembled selectively.
- Publish the issuer key at a stable location with a documented rotation policy, so receipts verify years later.
- Retain receipts for the full audit retention period as ordinary files. They are small and format-stable.
- Give the auditor the verification tool and its source, so they are not trusting your implementation of the check.
The fourth point is what distinguishes this from a compliance dashboard. A dashboard tells the auditor what you computed; a verification tool lets them compute it themselves.
A worked example: the exception report
| Sample of 25 | Full population | |
|---|---|---|
| Items examined | 25 | 40,000 |
| Exceptions found | 1 | 31 |
| What is reported | A projected rate with an interval | 31 named changes, each identified |
| Management response | Dispute the projection | Look at 31 specific items |
| Value to the client | A grade | A work list |
The second column is more uncomfortable and considerably more useful. It also removes the most tedious conversation in auditing, which is arguing about whether a sample was representative.
Objections and honest limits
“Full-population testing will surface more exceptions.” It will, and most of them are administrative — an emergency change signed after the fact, a departed employee's key, a batch job under a service identity. Knowing that is the point.
“This replaces the auditor.” It replaces reconciliation, which was never the value. Scoping, judging whether the control addresses the risk, and evaluating the exceptions are all still human work.
Making evidence auditable by design
- Publish the verification key set. So re-performance needs nothing from you.
- Sign at the moment of approval. Not at export time.
- Include what was rendered, not just an id. Otherwise the auditor sees a reference, not a decision.
- Export the full population, not a sample. The cost argument no longer applies.
- Expect and triage the exceptions. Most are administrative; find the ones that are not.
Terms used here
- Re-performance
- An auditor independently repeating a control to confirm it worked, rather than reading evidence about it.
- Full-population testing
- Examining every item rather than a sample, which machine verification makes affordable.
- Evidence hierarchy
- The ranking of evidence quality — externally verifiable outranks client-produced.
Frequently asked questions
Does this eliminate the need for auditors? No. It shifts their time from establishing that evidence exists to judging whether the control is appropriate, which is where expertise actually adds value.
Why is offline verification stronger? Evidence obtained from the audited entity's systems inherits those systems' integrity assumptions. A signature verifying against a published key is independent of them.
What does verification not prove? That the approver understood the change or that the review was adequate. Those remain judgement questions.
What exceptions does full-population testing typically find? Self-approvals in small teams, missing user verification from platform defaults, and commits that drifted after approval. Process defects rather than fraud.
Why do auditors sample? Because manual review costs money. It is an economic constraint, not a methodological preference.
What are the exceptions usually? Administrative: emergency changes signed after the fact, departed employees' keys, service identities. The interesting ones are a minority.
Why does offline verification matter for independence? If re-performance requires the client's systems, it is not fully independent. A published key set removes that dependency.
Where this fits in Manav
Manav publishes a JWKS and an open verifier, so an auditor can re-perform the entire population offline without calling your systems.