Everybody’s System, Nobody’s Risk: accountability gaps in distributed AI governance

Paper four of six in the Ecaveo Working Papers on the higher-order issues in AI adoption, written as the UK set out its intention to regulate AI through existing regulators rather than a new one.

How does accountability go missing in an AI supply chain?

Through an ordinary sequence in which nobody behaves badly. The paper made it concrete with an invented case, assembled from advisory patterns rather than any real client.

A professional services firm of about 1,400 staff introduces a screening assistant for recruitment. It is sold by a software company that built it on a general-purpose language model licensed from a larger developer. Procurement checked security certifications and signed standard terms describing outputs as recommendations for review by a qualified person. A recruitment manager under pressure to fill 30 roles accepts shortlists with light review. Months later an unsuccessful applicant with a disability complains that the assistant marked down a gap in her employment history.

The supplier cites its human-review terms. The builder says the underlying model was updated without notice and could not be tested. The manager says the firm approved the tool. The data protection officer says the process belongs to human resources, and human resources says the tool belongs to technology. Two regulators could be engaged, each looking at a different part. Everyone in the room behaved reasonably, and the outcome belonged to nobody.

Why is this a structural problem rather than a failure of individuals?

Because responsibility was distributed without anyone holding the whole picture. The philosophical literature has described the problem of many hands for four decades, and identified the barriers that keep it in place: many hands in software development, defects treated as an unavoidable property of complex code, blame settling on the computer itself, and producers claiming ownership while disclaiming liability.

Learning systems add a further gap, because the assumptions of control and foreseeability that ordinary responsibility relies on are weakened when the system changes after deployment. Four kinds of gap are worth separating: culpability, moral accountability, public accountability, and active responsibility. The last is the one an organisation can close by itself, which is why the paper concentrates on it.

A workable test asks three things of any arrangement. Is there an actor obliged to explain and justify? Is there a forum that can question and judge? Is there a consequence? Where any of the three is missing, the gap is open, whatever the policy documents say.

Who controls what in a generative AI supply chain?

The paper mapped four links, with what each controls, what each can see, and what each carries.

1. The model developer controls training and updates, sees aggregate behaviour, rarely sees the use case, and disclaims most of the risk.
2. The application builder controls prompts and interface, cannot see model changes upstream, and usually caps its liability at fees paid.
3. The deploying organisation controls purpose and process, cannot see model internals or update history, and carries most of the operational risk.
4. The professional user controls the final decision, sees one case at a time, cannot see the configuration, and carries risk through internal policy.

The mismatch between the column describing what each party can see and the column describing what each party carries is the accountability gap in commercial form. There is a fifth link outside the chain entirely: staff using public tools that nobody approved, which the paper treats as a governance problem rather than a disciplinary one, because blanket bans push the use out of sight rather than out of existence.

A related concern is the reviewing professional as the place blame settles. That person is rarely given the time, the information or the authority that meaningful review requires. A person in the loop is a safeguard, and it may sometimes be a scapegoat instead.

What can an organisation do without waiting for regulators?

Most of the gap an organisation controls can be closed by three disciplines used together: one named owner for every AI-supported decision, a responsibility map across the chain, and internal audit that tests the seams between parties rather than the parties themselves.

The owner’s role has four parts. They approve the use of AI in the decision and record its purpose. They ensure reviewers have the time, information and authority that real review requires. They receive supplier notifications and incident reports and decide what follows. They answer to a named forum, usually the risk or audit committee.

Six steps put it in place: inventory the decisions, including informal staff use; name the owner; map the chain; set supplier terms; audit the seams; review on change. Six things are worth asking of suppliers: advance notice of material model changes including upstream ones, prompt incident notice, clear terms on whether customer inputs train models and how to switch that off, enough information on known limitations for reviewers to judge when scrutiny is needed, access to documentation sufficient to trace a failure, and a tested exit route.

What has changed since April 2023?

The paper’s central bet was that organisations should not wait for regulators to close these gaps, and that has held. The UK’s non-statutory, regulator-led approach persisted considerably longer than most readers assumed in 2023, so allocation of responsibility within supply chains continued to be settled by contract and existing law, which is what the paper predicted.

What it did not anticipate is that the answer would arrive from outside the UK. The EU’s legislation went on to do what the white paper declined to do, assigning obligations along the chain and placing duties on providers of general-purpose models, so an organisation with European exposure now has a partial external answer.

Named ownership of an AI-supported decision now reads as unremarkable good practice, which is the usual sign that a paper was early rather than wrong. The question ranked last, and described as mattering most, is the one still open. There remains no systematic evidence on whether human reviewers function as safeguards or as scapegoats, even though human oversight has become the load-bearing control in most AI regulation written since.

Nothing here constitutes legal advice, and the provision of legal advice sits outside the terms of any engagement with the author. The material is presented to support discussion and further review by qualified advisers.

Frequently asked questions

What is the unit of accountability for AI?

The decision, rather than the tool. Owning a tool tends in practice to mean owning its licence and its uptime. A decision with several owners has none, so each AI-supported decision above an agreed impact threshold needs one named person answerable to a named committee.

Should an organisation ban public AI tools?

The paper argues against it. A blanket ban moves the use out of sight, which removes the one thing governance depends on. A policy with a fast approval route brings it back into view.

What does auditing the seams mean?

Internal audit usually examines a supplier, a process or a system on its own. Auditing the seams examines the handovers between them, because the failures the paper describes happen in the gaps rather than inside any one party’s remit.

How often should a responsibility map be redrawn?

Whenever a supplier changes its model or its terms, and otherwise on the ordinary review cycle. A map that describes a supply chain as it stood two model versions ago is a record rather than a control.

Does this apply to tools bought rather than built?

Particularly so. The deploying organisation controls purpose and process and carries most of the operational risk, while seeing the least about what sits underneath. That combination is the one the paper is about.