{
"slug": "corporate-manslaughter-defense-autonomous-systems-board-members",
"topic_id": "TOPIC-060",
"cluster": "Regulatory & Sovereign AI",
"tier": "Tier B",
"title": "Board oversight of autonomous systems that can cause physical harm",
"summary": "Where an autonomous system controls something physical, the governance question stops being about data and starts being about whether anyone was responsible for the safety envelope.",
"lede": "A board can review an AI governance policy annually and still be unable to answer the question that matters after an incident: who was responsible for the parameters that were in effect at the time, and what did they know?",
"date": "2024-03-30",
"category": "Compliance",
"author_id": "solene-beaumont-adjei",
"tags": [
"board governance",
"corporate liability",
"autonomous systems",
"safety",
"oversight",
"industrial"
],
"image_title": "Board Oversight Autonomous Systems",
"schema": "Article",
"key_takeaways": [
"Oversight duties are generally assessed by whether a monitoring system existed and functioned, not by outcomes.",
"For systems with physical consequences, the operative records are about safety parameters: who set them, when, and on what basis.",
"A board's practical question is whether it can obtain those records on demand, rather than whether a policy exists."
],
"body": [
{
"type": "h2",
"text": "What oversight duties generally require"
},
{
"type": "diagram",
"kind": "flow",
"alt": "What an investigation traces backwards",
"caption": "It works from the harm to a person. Most organisations lose the trail at step three.",
"nodes": [
{
"label": "Physical harm occurs",
"note": "the starting point"
},
{
"label": "System behaviour reconstructed",
"note": "logs usually adequate"
},
{
"label": "Who set the safety envelope?",
"note": "often nobody named",
"bad": true
},
{
"label": "Who was told it was exceeded?",
"note": "a dashboard",
"bad": true
},
{
"label": "Who could have stopped it?",
"note": "undocumented",
"bad": true
}
]
},
{
"type": "p",
"html": "Across jurisdictions, directors' oversight obligations tend to be framed around process rather than outcome: whether the board established a system to surface material risks, and whether it attended to what that system reported."
},
{
"type": "p",
"html": "A board is not usually expected to prevent every failure. It is expected to be able to show that it had a mechanism and used it."
},
{
"type": "h2",
"text": "Why physical systems change the calculus"
},
{
"type": "table",
"head": [
"Dimension",
"Data-affecting system",
"Physically-affecting system"
],
"rows": [
[
"Worst outcome",
"Breach, regulatory penalty, remediation",
"Injury or death"
],
[
"Reversibility",
"Partial",
"None"
],
[
"Investigating authority",
"Privacy and sector regulators",
"Safety regulators, potentially prosecutors"
],
[
"Personal exposure for executives",
"Rare",
"Possible in some jurisdictions"
],
[
"Evidence examined",
"Policies, logs, breach response",
"Safety parameters and who set them"
]
]
},
{
"type": "p",
"html": "The last row is where governance practice is thinnest. Boards review AI policies; safety investigators ask who authorised the operating envelope in effect at 14:22 on the day of the incident."
},
{
"type": "h2",
"text": "The records an investigation will look for"
},
{
"type": "ol",
"items": [
"The safety parameters in effect at the time of the incident, as configured rather than as documented.",
"Every change to those parameters, with who made it and what approval it had.",
"Any override of a safety interlock, with who authorised it and why.",
"Whether alarms or degraded conditions were acknowledged, and by whom.",
"The scope of autonomous operation — what the system was permitted to do without human involvement, and who decided that."
]
},
{
"type": "p",
"html": "Most organisations can produce the first from configuration and struggle with the rest, because change records identify accounts and tickets rather than people and reasons."
},
{
"type": "h2",
"text": "What a board can reasonably ask for"
},
{
"type": "p",
"html": "Not a policy. A demonstration."
},
{
"type": "ul",
"items": [
"For system X, show the current safety envelope and who last changed it",
"Show the last five overrides of a safety interlock, with the authorising individual",
"Show the approval record for the current scope of autonomous operation",
"Show what would be produced if a regulator asked for this tomorrow"
]
},
{
"type": "p",
"html": "The fourth question is the useful one. It is answerable in a meeting, it is not adversarial, and it surfaces the gap without requiring the board to understand the technology."
},
{
"type": "h2",
"text": "The artefact that answers it"
},
{
"type": "code",
"text": "{\n \"system\": \"line-4-autonomous-handling\",\n \"change\": \"safety_envelope.max_velocity\",\n \"from\": 1.2, \"to\": 1.8, \"units\": \"m/s\",\n \"basis\": \"[engineering assessment reference]\",\n \"risk_assessment\": \"RA-2025-0231, reviewed 2025-11-02\",\n \"authorised_by\": \"[named engineer, role]\",\n \"countersigned_by\": \"[named safety officer]\",\n \"effective\": \"2025-11-24T06:00:00Z\",\n \"signatures\": [ \"...\", \"...\" ]\n}"
},
{
"type": "p",
"html": "Two signatures, from two named individuals in different roles. That structure is standard in safety-critical engineering and is frequently absent once the parameter lives in software configuration rather than on a form."
},
{
"type": "h2",
"text": "Where this fits in existing practice"
},
{
"type": "p",
"html": "It is not new governance. Safety-critical industries have management-of-change processes that require exactly this: assessment, authorisation by competent persons, and a record."
},
{
"type": "p",
"html": "What has happened is that as control moved into software, some of these parameters migrated into systems that use ordinary change control — a ticket and a deployment — rather than safety change control. The remedy is to recognise which configuration values are safety parameters and apply the existing discipline to them."
},
{
"type": "h2",
"text": "A note on scope"
},
{
"type": "p",
"html": "This applies to a narrower set of organisations than most AI governance discussion. If your systems affect data and decisions but not physical objects, the personal-exposure dimension largely does not arise and the ordinary governance apparatus is adequate."
},
{
"type": "p",
"html": "If your systems move things, cut things, dispense things or control access to physical spaces, the question is worth putting on an agenda before someone else puts it there."
},
{
"type": "p",
"html": "This describes governance principles at a general level and is not legal advice. Directors' duties and personal liability vary substantially by jurisdiction; take specifics to counsel."
},
{
"type": "h2",
"text": "A worked example: three records, three outcomes"
},
{
"type": "table",
"caption": "What the organisation can produce",
"head": [
"Record available",
"How it reads"
],
"rows": [
[
"Board minutes noting AI adoption",
"Awareness without ownership"
],
[
"A policy document with no named owner",
"A control on paper"
],
[
"An automated alert nobody acknowledged",
"Notice that reached no person"
],
[
"A signed envelope approval by a named director",
"Ownership, with a date and a scope"
],
[
"A signed acknowledgement of a breach alert",
"Notice that demonstrably reached a person"
]
]
},
{
"type": "p",
"html": "The difference between the middle rows and the last two is not diligence, it is whether a person's signature exists over the specific thing. Everything else is an inference someone else will draw."
},
{
"type": "h2",
"text": "Objections and honest limits"
},
{
"type": "p",
"html": "“The board cannot approve every parameter.” It should not. It approves the envelope — the outer limits within which the system may act — and delegates inside it. That is one signature a quarter, not a workload."
},
{
"type": "p",
"html": "“This creates personal exposure for directors.” The exposure exists whether or not the record does. What a signature changes is whether the director can show what they approved and when, rather than having it reconstructed by someone else."
},
{
"type": "p",
"html": "This describes governance practice generally and is not legal advice. Duties and liability vary by jurisdiction and turn on specific facts."
}
],
"faq": [
{
"q": "Does this apply to all AI systems?",
"a": "Primarily to systems with physical consequences. Where the effects are limited to data and decisions, the personal-exposure dimension largely does not arise."
},
{
"q": "What should a board ask?",
"a": "Not for a policy. Ask to see the current safety envelope for a named system, who last changed it, recent interlock overrides, and what would be produced if a regulator asked tomorrow."
},
{
"q": "Is this new governance?",
"a": "No. Safety-critical industries have management-of-change discipline already. The gap is that some safety parameters migrated into software change control."
},
{
"q": "Why two signatures?",
"a": "Authorisation by competent persons in distinct roles is standard safety practice. It tends to be lost when a parameter becomes a configuration value."
},
{
"q": "Does the board approve technical parameters?",
"a": "No. It approves the envelope and delegates inside it — roughly one signature a quarter."
},
{
"q": "Why does a signature beat a minute?",
"a": "A minute records that something was discussed. A signature records that a named person committed to a specific scope on a specific date."
},
{
"q": "Is this legal advice?",
"a": "No. Oversight duties and liability are jurisdiction-specific and fact-dependent."
}
],
"sources": [
{
"t": "Published guidance — directors’ duties and oversight (Cornell LII, corporations)",
"u": "https://www.law.cornell.edu/wex/corporations"
},
{
"t": "IEC 61511 / OSHA process safety management",
"u": "https://www.osha.gov/process-safety-management"
},
{
"t": "Occupational and industrial safety regulator guidance on autonomous equipment."
},
{
"t": "CISA — known exploited vulnerabilities and incident reporting",
"u": "https://www.cisa.gov/known-exploited-vulnerabilities-catalog"
},
{
"t": "ISO/IEC 42001 — AI management systems",
"u": "https://www.iso.org/standard/81230.html"
}
],
"related": [
{
"slug": "eu-ai-act-article-14-human-oversight-playbook",
"title": "Article 14 asks who the natural person was",
"category": "Compliance"
},
{
"slug": "iso-iec-42001-standard-ai-management-systems",
"title": "ISO/IEC 42001 Clause 8.4",
"category": "Compliance"
},
{
"slug": "nist-ai-rmf-1-0-vs-cryptographic-reality",
"title": "Mapping framework language to tests",
"category": "Comparison"
}
],
"image": "https://cdn.twc.sh/images/igcache/Board%20Oversight%20Autonomous%20Systems/1500_900/blog.jpg",
"wordcount": 996,
"url": "/blog/corporate-manslaughter-defense-autonomous-systems-board-members.html",
"reading_time": "4 min read",
"seo_title": "Board oversight of physically-acting autonomous systems",
"meta_description": "When an autonomous system controls something physical, governance stops being about data and starts being about who set the safety envelope.",
"hub": {
"slug": "topics/ai-oversight-regulation",
"title": "AI oversight and regulation"
},
"answer": "The question stops being about data and starts being about whether a named person owned the safety envelope. Investigators look for who set the limits, who was told when they were exceeded, and who had the authority to stop it — and most governance records answer none of those.",
"answer_q": "What changes when an autonomous system can cause physical harm?",
"glossary": [
{
"term": "Safety envelope",
"def": "The outer limits within which an autonomous system may operate without further human authorisation."
},
{
"term": "Named ownership",
"def": "Assignment of a specific responsibility to an identified individual rather than a function."
},
{
"term": "Kill time",
"def": "Elapsed time from a decision to stop a system to the system actually stopping."
}
],
"checklist": {
"title": "Board-level questions worth asking",
"id": "board",
"desc": "Five steps.",
"steps": [
{
"name": "Which autonomous systems can cause physical harm?",
"text": "Start with an inventory, not a policy."
},
{
"name": "Who owns the safety envelope for each?",
"text": "A named person, not a committee."
},
{
"name": "How is a breach escalated to a human?",
"text": "And can you show they received it."
},
{
"name": "Who can stop the system, and how fast?",
"text": "Measure the kill time."
},
{
"name": "Is the approval signed or minuted?",
"text": "A minute records discussion; a signature records commitment."
}
]
},
"cta": {
"title": "Where this fits in Manav",
"html": "Manav turns an envelope approval into an artefact: a named director's hardware signature over the limits they approved, and over each alert they were shown.",
"href": "../docs.html",
"label": "See oversight receipts"
}
}