Why AI does not remove accountability—and why organisations keep behaving as though it might.
While researching and drafting my first book, Tales from AI Development Hell, I encountered a familiar occupational hazard: the case studies became rather more thoroughly researched—and considerably more extensively written—than the book required. “Editorial discipline” eventually intervened, as it does, shortly after the word count acquires its own postcode.
But the surplus material was not merely a collection of interesting examples looking for somewhere to live. Across inquiries, judgments and regulatory records, it kept pointing to a related but distinct problem. The first book asked how ambitious AI programmes become organisational failures. These cases raised the question that follows the failure: when a machine has shaped the outcome, who must explain what happened, who is entitled to challenge that explanation and who can put matters right?
There was enough evidence for a second book, but not quite the same book twice. So I narrowed the lens from failure to answerability, followed that question through government, finance, healthcare, aviation and emerging AI agents, and arrived here: AI Made the Decision: Who Answers When Nobody Owns the Outcome?
There is a short question that can reduce an otherwise confident boardroom to the atmosphere of a lift that has stopped between floors.
Who decided this?
The question sounds simple. It ought to produce a name. Instead, it usually produces a guided tour of the organisation.
The proposal went to a committee, which relied on advice, which referred to a supplier’s model. A human remained “in the loop”, although nobody is sure what the human could see or whether they could disagree.
Every step was completed. Everybody did their bit. The question remains unanswered.
It is about consequential decisions being distributed across people, systems, suppliers and committees until nobody holds enough of the decision to explain it.
AI did not create the gap
The book’s thesis is simple:
Artificial intelligence does not create the accountability gap. It reveals the gaps organisations already built, then operates through them at machine speed.
The “problem of many hands” predates machine learning. Large institutions have always divided work. Boards authorise things they cannot operate. Risk teams review processes they do not own. Suppliers know details their customers may never see.
Distribution is not the failure. Disconnection is.
AI makes disconnection easier to reach and harder to see. A model can shape thousands of decisions while each participant performs a defensible task and assumes somebody else has the whole picture. Frequently, nobody does.
Four things that must connect
The word “accountability” has been placed on so many slides that it now risks becoming a decorative font choice. The book gives it a practical test.
An institution remains answerable only where four things exist and connect:
- Knowledge: somebody understands what the system does, what evidence it uses and where it fails.
- Authority: somebody has legitimately permitted the system to shape this decision, for this population, within stated bounds.
- Control: somebody can intervene in the actual case and can suspend the process when necessary.
- Exposure: somebody must answer to a forum able to question the explanation and impose consequences.
One person need not hold all four. Separating them is often sensible, but the chain between them must not terminate in a gap.
An executive may be named as accountable, but if they cannot obtain the evidence, instruct the operator or remedy an outcome, their accountability is ceremonial. They have been given a name badge and an opportunity to apologise.
When “the model decided” becomes organisational grammar
The book examines cases from government, finance, healthcare, aviation and emerging AI agents. Their technologies differ, but their grammar is consistent.
“The model flagged the case.”
“The system recommended enhanced review.”
“A human made the final decision.”
These sentences make actions sound like weather. They conceal who selected the target, accepted the proxy, set the threshold, designed the screen and made disagreement costly.
Australia’s Robodebt programme used an income-averaging method without first establishing its lawful basis. A healthcare algorithm accurately predicted cost when the institution needed to identify need; technical accuracy preserved an unjust policy.
The Apple Card investigation offers another lesson. New York’s financial regulator examined nearly 400,000 applications and did not find a fair-lending violation. Accountability means testing serious claims and producing credible evidence, not assuming the most damning explanation.
An organisation that can discover it is wrong can experiment. One that cannot must be right first time or become very good at denial.
The ceremonial human
Few phrases in AI governance have done more comfortable work than “human in the loop”. It sounds reassuring while specifying almost nothing.
Was the human informed? Could they override the output? Was disagreement recorded? Could they halt the system, or merely add a note for next month’s committee?
A person who clicks “approve” because the interface, workload and incentives all point in one direction is not exercising meaningful control. They are providing the system with a small quantity of legal scenery.
As AI agents move from producing text to taking actions, the question becomes urgent. A tool offers assistance; an agent receives delegated authority. The organisation must define its limits before the agent discovers them experimentally.
From governance artefacts to answerability
The book does not end with another twelve-part maturity model destined to be completed once, coloured green and misplaced during a reorganisation. It proposes three connected instruments.
First, an Accountability Constitution: ten commitments. Consequential outcomes have a named principal. Authority is written, bounded and revocable. Evidence is retained, and affected people can contest outcomes and receive remedy.
Second, a Decision Passport for every consequential process. Its eighteen fields cover purpose, authority, affected population, owners, suppliers, intervention points, appeal routes and exposure. Roughly half require names, which is where the document earns its keep.
Third, an accountability hearing where one outcome owner answers questions and an authoritative panel decides whether the process may continue, face limits, produce evidence, suspend, retire or remediate harm.
The most useful hearing question may be the last: What would make us withdraw authority?
Answer it while the dashboard is green. Once it turns red, everybody will have a sincere reason why this case is different.
Accountability raises the ceiling
Accountability is normally presented as a brake on AI adoption. The book argues almost the reverse.
The amount of authority an organisation can safely give to machines depends on how well it can answer for what they do with it. An organisation that can reconstruct decisions, identify who can act, stop failure quickly and put affected people right can attempt more ambitious things—not fewer.
This is not a case for timidity. It is a case for earned authority.
AI may contribute the score, recommendation or action. It cannot attend the hearing, explain the trade-off, compensate the customer or accept the consequence. The institution still decided. The practical question is whether it arranged itself so that somebody can answer.
If your organisation cannot answer “Who decided this?” with a name, evidence and a route to remedy, the machine is not the most worrying part of the sentence.
Buy AI Made the Decision: Who Answers When Nobody Owns the Outcome? on Amazon.
Preferably before somebody reserves the witness chair.
#ArtificialIntelligence #AIGovernance #ResponsibleAI #AIStrategy #Leadership
