Skip to content

Articles · · 18 min read

Stop Treating Every AI Decision Like a Level 5 Decision

What twelve years of self-driving car standards can teach your AI governance, and where AI goes past them. A shared scale of autonomy levels turns "should we use AI?" into milestones you can plan, cost and review, and there's a prompt at the end to run the session with your own team.

Sit in on almost any corporate conversation about AI and you'll hear the same unspoken assumption. Someone proposes using AI for a piece of work, and the room quietly pictures it running unsupervised: no human ever checks the output, nothing measures whether it's any good, and there's no way to catch a mistake before it spreads. Nobody says it out loud, but the line between "AI helps with this" and "AI does this on its own" is never drawn.

With the line undrawn, the risk feels unbounded. And when nobody can say what's being approved, the safest move is to approve nothing. The proposal goes to a steering committee, the steering committee asks for more analysis, and months later the company is still doing nothing.

The problem isn't that people are too cautious. It's that they're picturing the most autonomous version of every idea, because there's no shared vocabulary for the versions in between. Every AI decision gets treated like a Level 5 decision.

That causes three predictable problems:

  1. You can't set milestones.

    If the only options are "no AI" and "AI does everything," there are no steps in between to plan, fund or celebrate.

  2. You can't estimate cost or complexity.

    Letting an AI draft a spec and letting an AI deploy to production have wildly different engineering costs. Lumped together, both look expensive and scary.

  3. You can't size the security risk.

    Read-only help with a draft and write access to production deserve different reviews. When everything is reviewed as the worst case, low-risk work stalls and real risks get the same attention as trivial ones.

The auto industry ran into this exact problem more than a decade ago, and it solved it. We can borrow the solution.

What the self-driving world figured out

In January 2014, SAE International published J3016, a taxonomy for driving automation. It has been revised several times since, most recently developed jointly with ISO, and its six levels — 0 through 5 — have become the common language regulators, engineers and journalists use to talk about automated cars.

Twelve years of bounded autonomy

  1. 2014

    SAE publishes J3016

    Six levels, 0 through 5, for driving automation.

  2. 2018

    Tempe, Arizona

    A self-driving test vehicle strikes and kills a pedestrian. The human fallback had stopped being real.

  3. 2021

    J3016 revised with ISO

    Developed jointly with ISO, the levels become the common language for automated cars.

  4. 2023

    The first U.S. Level 3

    Mercedes-Benz Drive Pilot: suitable freeway sections, dense traffic, up to about 40 mph.

  5. 2026

    Bounded Level 4 at scale

    Waymo runs driverless in more than ten U.S. cities, each inside a mapped area.

  6. June 2026

    Authority pulled back

    Waymo recalls about 3,900 vehicles after some drove into construction zones.

Most people know the levels. What's more useful is why the levels work, and that comes down to a few ideas that get far less attention.

The levels describe; they don't rank. SAE is explicit that the levels are descriptive rather than normative. A Level 2 car isn't a failed Level 4 car. It's a different arrangement of responsibility. That single idea would defuse half the AI arguments I sit in on.

Every level is defined by who handles the failure. The jump from Level 2 to Level 3 isn't about the car getting smarter. It's about who is responsible when something goes wrong. At Level 2, the human is driving and must watch the road constantly. At Level 3, the system drives, but a human must be ready to take over when asked. At Level 4, the system handles its own failures — no human takeover needed — but only within defined conditions.

SAE's levels, by who handles the failure

LevelWho drivesWho watches the roadWho handles the failure
0No automation Human Human Human
1Driver assistance Human, with help on steering or speed Human Human
2Partial automation System steers and sets speed Human, constantly Human
3Conditional automation System System Human, when asked to take over
4High automation System System System, inside its boundary
5Full automation System System System, everywhere

The jump that matters is in the last column: from Level 4, the system catches its own failures.

Every level operates inside a boundary. J3016 calls it the operational design domain, or ODD: the specific roads, speeds, weather and conditions a system is designed to handle. Outside that boundary, the system isn't approved to operate.

There's always a safe way to stop. J3016 calls it the minimal risk condition: the state the vehicle falls back to when it can't continue, like pulling onto the shoulder. A Level 4 system must be able to get there on its own.

The safe way to stop, on the road and in production

On the road

Inside the ODD Edge of the ODD Shoulder: the minimal risk condition

In production

Release Error rate over time Flag off Safe state

Both systems reach a safe state without a person. On the road it's the shoulder; in software, as the next section argues, it's a feature flag turned off.

Look at how the real deployments went and you can see these ideas working:

  • The first certified Level 3 system came with a tiny boundary. When Mercedes-Benz certified Drive Pilot in the U.S. in 2023, it was approved only for suitable freeway sections in dense traffic, at speeds up to about 40 mph. Narrow boundary, real autonomy within it.
  • The success story is bounded Level 4, not Level 5. Waymo now runs fully driverless robotaxis in more than ten U.S. cities, each inside a mapped service area. Every new city follows the same staged path: driverless testing, then employees, then the public. Even then, authority gets pulled back when the evidence says so — in June 2026 Waymo recalled about 3,900 vehicles after some drove into construction zones.
  • The handoff is where things go wrong. In 2018, a test vehicle operated by Uber's self-driving unit struck and killed a pedestrian in Tempe, Arizona. The NTSB found the safety driver was distracted, and cited the company's inadequate safety risk assessment, ineffective oversight of operators and lack of mechanisms to address automation complacency. The human was supposed to be the fallback, and the system around her didn't keep that fallback real.

That last lesson matters more for AI than any other, and we'll come back to it.

The levels, applied to software delivery

To make this concrete, I'm going to use the work I know best: building and shipping software. The same pattern applies to finance operations, customer support or anything else with a pipeline from intent to outcome.

Here's the lifecycle of a piece of work: someone has an idea, it becomes a ticket, the ticket gets groomed, someone builds it, it gets tested and reviewed, it's deployed to a test environment, then a staging environment, then production, and then someone watches to make sure it works. Here's how the levels map onto that.

The levels, applied to software delivery

0Human delivery

  1. Intent: a person does it
  2. Ticket: a person does it
  3. Groom: a person does it
  4. Build: a person does it
  5. Test: a person does it
  6. Review: a person does it
  7. Test env: a person does it
  8. Staging: a person does it
  9. Production: a person does it
  10. Watch: a person does it
What the AI does
Nothing in the governed workflow
Where the human sits
The classic scrum team: humans groom, build, review and ship
The fallback
Humans

1Assistance

  1. Intent: a person does it
  2. Ticket: AI suggests, a person decides
  3. Groom: AI suggests, a person decides
  4. Build: a person does it
  5. Test: AI suggests, a person decides
  6. Review: AI suggests, a person decides
  7. Test env: a person does it
  8. Staging: a person does it
  9. Production: a person does it
  10. Watch: a person does it
What the AI does
Turns a messy ticket, email or rambling request into a structured spec; suggests tests; assists code review
Where the human sits
Accepts or rejects every suggestion
The fallback
Humans; the AI only advises

2Supervised execution

  1. Intent: a person does it
  2. Ticket: AI suggests, a person decides
  3. Groom: AI suggests, a person decides
  4. Build: AI owns it
  5. Test: AI owns it
  6. Review: AI acts, a person approves
  7. Test env: AI acts, a person approves
  8. Staging: AI acts, a person approves
  9. Production: AI acts, a person approves
  10. Watch: a person does it
What the AI does
Builds the change and runs the tests locally or in a sandbox
Where the human sits
Approves each consequential step: the pull request, the merge, the deploy
The fallback
Humans approve at every point of consequence

3Conditional delegation

  1. Intent: a person does it
  2. Ticket: AI suggests, a person decides
  3. Groom: AI suggests, a person decides
  4. Build: AI owns it
  5. Test: AI owns it
  6. Review: AI owns it
  7. Test env: AI owns it
  8. Staging: AI owns it
  9. Production: AI acts, a person approves
  10. Watch: a person does it
What the AI does
Owns the chain from build through test environments, code review, staging deploy and verifying its own work in staging — within a defined class of changes
Where the human sits
On call for exceptions; approves promotion to production
The fallback
A human takes over when the AI hits something it can't handle

4Bounded autonomy

  1. Intent: a person does it
  2. Ticket: AI suggests, a person decides
  3. Groom: AI suggests, a person decides
  4. Build: AI owns it
  5. Test: AI owns it
  6. Review: AI owns it
  7. Test env: AI owns it
  8. Staging: AI owns it
  9. Production: AI owns it
  10. Watch: AI owns it
What the AI does
Ships to production behind a feature flag, watches telemetry, and turns the flag off or reverts on its own — within a defined class of changes
Where the human sits
Sets the boundary, reviews periodically, can suspend at any time
The fallback
The system itself: flag off, revert, safe state

5Full autonomy

  1. Intent: a person does it
  2. Ticket: AI owns it
  3. Groom: AI owns it
  4. Build: AI owns it
  5. Test: AI owns it
  6. Review: AI owns it
  7. Test env: AI owns it
  8. Staging: AI owns it
  9. Production: AI owns it
  10. Watch: AI owns it
What the AI does
Everything in Level 4, for any kind of change, in any system — taking raw intent from wherever it lives: a meeting transcript, an email thread, a ticket
Where the human sits
Decides what should be built
The fallback
The system itself, with no boundary on change types

6The AI picks the destinationOurs, not SAE's

  1. Intent: AI owns it
  2. Ticket: AI owns it
  3. Groom: AI owns it
  4. Build: AI owns it
  5. Test: AI owns it
  6. Review: AI owns it
  7. Test env: AI owns it
  8. Staging: AI owns it
  9. Production: AI owns it
  10. Watch: AI owns it
What the AI does
Notices a signal and decides on its own to fix it, improve it or build something new
Where the human sits
Defines what "good" means: the guiding principles, the objectives, the budget
The fallback
A human veto and a hard budget ceiling on self-started work
  • A person does it
  • AI suggests, a person decides
  • AI acts, a person approves
  • AI owns it

Level 6 is ours, not SAE's; it's the subject of a section below. Until then, a person picks what gets built at every level.

A few things fall out of this table.

The feature flag is your shoulder of the road. At Level 4, the system has to reach a safe state without a human. In software, that's usually a feature flag or an automatic revert. Which gives you a concrete, budgetable milestone: you aren't ready for Level 4 on a class of changes until every change in that class can be turned off without a person. That's an engineering project with a scope, not a vibe.

A workflow has a level per stage. In my own work, ticket intake runs close to Level 5 — I can hand an AI a meeting transcript and get back draft specs with priorities, due dates and a list of open questions. But production deploys for some clients stay firmly with humans. "What level are we at?" is the wrong question. "What level is each stage at?" is the right one.

The AI's questions are a takeover request. When the AI drafts a spec and comes back with "I don't know whether this applies to existing customers — ask the stakeholder," that's the intent-level version of a car asking the driver to take the wheel. A good system knows the edge of its own knowledge. That ability is what makes Level 3 possible.

Where a gate lives matters less than whether it exists. Some teams run every quality gate locally and push straight to main. Others use pull requests with CI gates. Both are valid. One operator with a couple of AI agents doesn't need the ceremony a fifty-person team does. Governance should care that the gate is there and what evidence it produces, not which tool runs it.

Level 6: when the AI picks the destination

Here's where AI breaks the car analogy, and I think it's worth breaking on purpose.

In every one of SAE's levels — even Level 5 — a human chooses where the car goes. The passenger picks the destination; the system handles the driving. The same is true of Level 5 in the table above. The AI might take a messy transcript instead of a clean ticket, but a person decided what needed doing.

So we're proposing a Level 6, which is ours, not SAE's: the AI decides what to work on. It notices a signal — an error rate creeping up, a confusing flow in user sessions, a dependency going stale — and decides on its own to fix it, improve it or build something new. Humans no longer pick destinations. They define what "good" means: the guiding principles, the objectives, the budget.

Level 6 changes the nature of the risk, not just its size. Up through Level 5, the failure mode is "it did the task badly." At Level 6, the failure mode is "it did a great job on the wrong thing." That's Goodhart's law: when a measure becomes a target, it stops being a good measure. An agent rewarded for fewer errors in the logs may find that the cheapest way to get there is to log less.

Fewer errors, or less logging?

Errors in the logs, what the agent is scored on

40 a week

Errors customers actually hit

40 a week

The agent's report: no change. Honest, and nothing to celebrate.

A toy with made-up numbers, not data.

Turn down the logging and the number the agent is scored on improves, while nothing customers see does.

So the controls at Level 6 aren't mainly about code. They're about the objective:

  • Who owns the guiding principles, and how changes to them are reviewed and versioned
  • How often a human audits what the agent chose to work on — and what it chose to skip
  • A human veto and a hard budget ceiling on self-started work

Level 6 doesn't have to be science fiction, either. Like every other level, it can be bounded. An agent that decides which flaky tests in one repository are worth fixing is a narrow Level 6. That's probably where most teams will meet it first.

Same scale, different ceilings

Here's what this looks like in practice. At Zero Mission, I run nearly the same delivery pipeline for two very different clients.

The first is a small real estate brokerage. I'm the only technical person, I know the people well, and the software doesn't move money. Work comes in through meetings and emails, the AI turns it into a groomed backlog, and the AI builds, tests and deploys through each environment, verifying its own work along the way. It clicks through the feature in a real browser, takes screenshots and records a demo video, and maps the results to the acceptance criteria. I review that evidence package, ask about any edge cases it missed, and send the change to production — often without ever clicking the feature myself.

The second is a mid-size manufacturer with an ordering system. The pipeline is nearly identical up to the user-acceptance environment. But there, the work goes to a product owner who tests it hands-on, and humans run the production deploy.

Same pipeline, two ceilings

The brokerage

  1. Intent: a person does it
  2. Ticket: AI owns it
  3. Groom: AI owns it
  4. Build: AI owns it
  5. Test: AI owns it
  6. Review: AI owns it
  7. Test env: AI owns it
  8. Staging: AI owns it
  9. Production: AI acts, a person approves
  10. Watch: a person does it

I review the evidence package and send it to production.

The manufacturer

  1. Intent: a person does it
  2. Ticket: AI owns it
  3. Groom: AI owns it
  4. Build: AI owns it
  5. Test: AI owns it
  6. Review: AI owns it
  7. Test env: AI owns it
  8. Staging: a person does it
  9. Production: a person does it
  10. Watch: a person does it

A product owner tests hands-on in user acceptance; humans run the deploy.

  • A person does it
  • AI suggests, a person decides
  • AI acts, a person approves
  • AI owns it

Same tools. Same levels. Different ceilings, set by different risk. The brokerage runs Level 3 with a light production gate. The manufacturer runs Level 3 with a heavier one. Neither runs Level 4 yet, and that's fine — it's an honest description, and it gives both a clear next milestone if they want one.

The same holds for whole organizations. A company in a regulated industry might cap anything touching production at Level 2. An AI-native shop might run Level 3 by default and Level 4 for changes behind flags. When everyone uses the same scale, those differences become documented choices instead of arguments.

Two lessons from the brokerage example are worth pulling out:

The human's job shifts from checking the product to checking the evidence. Approving a production deploy from a demo video sounds like exactly the rubber-stamping that killed the Uber test. It isn't, as long as the evidence is built to be inspected: acceptance criteria mapped to results, recordings of the feature working, test output, telemetry. The answer to the Level 3 handoff problem isn't asking humans to concentrate harder. It's giving them evidence that takes two minutes to check properly.

Evidence is also how you earn the next level. Every verified change adds to a track record. When it's time to argue for Level 4 on a class of changes, you don't argue from optimism. You point at the record.

What your governance document actually needs

Most AI governance documents I've seen are either a one-page list of don'ts or a forty-page binder nobody reads. Here's what I think belongs in one, with the levels doing the heavy lifting.

What belongs in the document

Tick what your own document already covers. It stays in this browser; nothing is sent.

None of this requires a new department. It requires a shared vocabulary and someone who owns keeping the record straight.

Where the analogy breaks

Borrowing from driving works because the core ideas transfer. But AI differs from cars in ways your governance needs to account for:

  1. The road isn't usually trying to trick you. AI inputs can be.

    A web page, an email or a file the AI reads can contain hidden instructions meant to hijack it. That's prompt injection, and it means boundaries have to be enforced by permissions, not good intentions.

  2. Crashes are visible. AI failures can be silent.

    A wrong summary or a leaked record doesn't dent a fender. And some failures — a disclosure, a sent message — can't be rolled back at all.

  3. A car does one job. A company has thousands.

    That's why levels belong to individual workflows, and why "we're a Level 3 company" doesn't mean anything.

  4. Agents can run up the bill.

    A car doesn't spawn copies of itself overnight. An agent stuck in a loop can, and the costs compound with it.

  5. Cars never choose the destination.

    That's Level 6, and it's new territory.

Start the conversation in your own company

Below is a prompt you can paste into Claude or any capable AI assistant to run a structured working session with your AI task force. It walks through the same discovery we run when Zero Mission facilitates these sessions, and it produces a first draft of your level map.

The working-session prompt

Open ClaudeOpen ChatGPT
You are facilitating an AI governance working session for a company's AI
task force. Use an autonomy scale borrowed from SAE J3016 (the self-driving
levels), adapted to company workflows:

- Level 0: Humans only. AI does not take part in the workflow.
- Level 1: AI advises (drafts, summarizes, suggests). A human accepts or
  rejects every output.
- Level 2: AI executes in a sandbox. A human approves every consequential
  action (publishing, merging, sending, deploying, changing records).
- Level 3: AI completes a defined chain of actions inside a written boundary.
  A named human is on call to take over on exceptions and approves the final
  consequential step.
- Level 4: AI operates end to end inside a written boundary and handles its
  own failures by reaching a safe state (for example, turning off a feature
  flag or reverting) without needing a human.
- Level 5: Level 4 without a boundary on the kinds of work. Humans still
  decide what work gets done.
- Level 6: AI decides what work to do, guided by human-defined principles
  and objectives.

Key principles:
- Levels describe a workflow; they don't rank teams. Lower is not worse.
- Levels apply per workflow, and per stage within a workflow.
- Each level is defined by who handles the failure, inside what boundary,
  and what the safe state is.
- Autonomy and risk are separate. A low-autonomy use can still be high-risk.

Run the session in this order. Ask one or two questions at a time and wait
for answers. Push back on vague answers.

1. Context: Ask about the company's industry, size, regulatory and
   contractual obligations, and the team's appetite for risk. Ask who in
   the room owns security, legal, finance and the business outcomes.
2. Inventory: Ask the group to list the AI uses already happening, including
   unofficial ones, and the ones they most want next.
3. Pick three workflows: Choose one that's clearly low-risk, one the group
   is excited about, and one that worries someone. For each, break it into
   stages from intent to outcome.
4. Classify: For each stage, ask what level it runs at today and what level
   the group wants. Ask what data it touches, what actions it can take, and
   what happens if it's wrong. Assign a risk tier (limited, managed,
   significant, restricted).
5. Boundary and fallback: For each stage at Level 3 or above, ask: What
   exactly is the boundary? Who or what catches a failure, and how fast?
   What is the safe state? If the group can't answer, say the stage isn't
   ready for that level and propose a lower one.
6. Evidence: Ask what evidence would convince the group to raise each stage
   one level, and what would cause them to lower it.
7. Human-only decisions: Ask which decisions must always stay with a person,
   no matter how good the AI gets.
8. Cost and limits: Ask who owns the budget for each workflow and how
   spending is capped in practice.
9. Ceilings: Based on everything discussed, propose a default ceiling for
   the company (for example, "Level 2 for anything touching production,
   Level 3 for internal tools") and ask the group to adjust it.

Finish by producing:
- A table of each workflow and stage with its current level, target level,
  risk tier, boundary, fallback and evidence needed to advance.
- The list of human-only decisions.
- The proposed default ceilings.
- The open questions the group still has to answer, with a suggested owner
  for each.

Run it with your task force, argue about the answers, and you'll come out with something more useful than most governance binders: a shared language, a map of where you are, and a list of concrete next milestones.

If you'd rather have someone in the room to facilitate — to push on the vague answers, bring the patterns we've seen at other companies, and help turn the map into an engineering plan — that's exactly the discovery we run at Zero Mission. Get in touch.


References

Get in touch

Questions about your own setup?

Tell me how your people use AI today. I'll tell you what I'd change, whether or not we end up working together.

  1. Dave reads it himself, usually the same day.
  2. He writes back by email. A person, not a mailing list.
  3. Rather write directly? [email protected] goes straight to him.

Goes straight to Dave.