From Assistants to Early AI Agents: an enterprise operating model for action-taking AI

Once a system can take actions, control has to move from reviewing what it produces to governing what it is permitted to touch, what it is permitted to do, and whether what it does can be undone.

What changes when an AI system can act on a business system?

The reviewable artefact disappears. When a model answers a question, a person reads the answer and decides whether to use it, so a wrong answer costs a moment of attention. When a model selects a tool and takes a series of steps, the error lands in a system of record as an action that somebody now has to find and reverse. This paper was published on 13 December 2024, and it argues that an operating model was needed while the capability was still too weak to cause much harm and while the habits were being set. Capability has moved a long way since. The figures that follow record December 2024, and nothing here claims where these systems now sit.

• The paper takes the word agent narrowly, reserving it for a system that selects among tools to pursue a goal, the only one of four current uses of the term that created a new control problem.
• On 22 October 2024 Anthropic released a public beta allowing a model to use a computer, and its research post reported 14.9 per cent on an evaluation of computer-using ability, against a human range of 70 to 75 per cent.
• The vendor’s product post reported 22.0 per cent where the system was given more steps, described the capability as still experimental and imperfect, and encouraged developers to begin with low-risk tasks.
• Prompt injection was named by the vendor itself as a risk, and it sits outside what conventional input validation was built to catch.
• Five rungs of autonomy were in scope and the sixth was excluded, because nothing in the published evidence supported open-ended autonomy.

How strong was the capability evidence in December 2024?

Weak, and the strongest evidence available came from the supplier. The research post stated that computer use remained slow and often error-prone, and reported 14.9 per cent on an evaluation of computer-using ability, against a stated human range of 70 to 75 per cent. The product post published the same day reported 22.0 per cent where the system was allowed more steps. Both figures were the vendor’s own, measured on the vendor’s own system, and that caveat travels with them.

The framing mattered more than the score. A product post that calls its own launch experimental and imperfect, and tells developers to begin with low-risk tasks, says something no external assessment published at that point could say. A supplier has every reason to sell. When one qualifies its own capability in public, the qualification carries weight, and the gap between it and the market conversation of the time is why the paper exists.

Why does prompt injection sit apart from the other risks?

Because the attack arrives inside the material the system is reading. Where a system reads screen content drawn from the internet, it may encounter text that contains instructions, and a system built to follow instructions will treat them as candidates for action. The vendor named this explicitly. Conventional input validation was designed to catch malformed or malicious data arriving at a boundary, and this content arrives well-formed, legitimate to receive, and hostile in what it asks for.

Identity and least privilege therefore sit at the centre of the paper’s control architecture. Where a system’s permissions are scoped to what the task genuinely requires, an instruction reaching it from a hostile page travels only as far as those permissions allow, which changes the problem from detecting hostile content into limiting what any instruction can reach. Scoping the permission is the control that remains available.

Where should an organisation stop on the autonomy ladder?

The paper sets out five rungs and places its ceiling below the sixth. A system can answer, it can recommend, it can prepare an action for a person to execute, it can act with approval, and it can act inside narrow bounds. Each of those five is treated as a deployable position with its own controls. Open-ended autonomy is excluded altogether, on the ground that the published evidence at the time supported nothing of the sort.

The ladder does real work, because it turns a binary argument about whether to use agents into a question about which rung a given process belongs on. Different processes sit on different rungs, and the same process may sit on different rungs for different users. Ownership follows the rung. The paper gives a section to who owns what, since a permission granted by one team, exercised by a second and audited by a third is a permission that nobody owns.

What made reversibility the gate on a pilot?

A pilot is defined by the ability to stop it and undo it. Where an action can be reversed cheaply and quickly, a small group of users can try it, the failures can be observed, and the organisation can decide with evidence in hand. Where an action stands permanently, the language of trial is doing no work. A pilot on actions that cannot be undone is a deployment with a smaller user group, and the paper says so plainly.

That gives a first-quarter test any board can apply. Ask what the system will be permitted to touch, what it will be permitted to do with each of those things, and how an action gets reversed when it goes wrong. Three clear answers make a pilot. One vague answer makes a deployment.

Read now, the figures belong to their moment and the argument still stands. Capability moved on, as everyone involved expected it would, and the questions the paper asks survive that movement, because they concern permissions, ownership and recoverability. An organisation that answered them at the time built the habit of granting an AI system the narrowest access that lets it work, of naming a person who owns that grant, and of knowing in advance how an action gets undone. Those habits become harder to install as systems grow more capable, since the pressure to widen a grant arrives with the first demonstration of value. The case for setting them early rested on the shape of the control problem, which is why it holds whatever the current scores happen to say.

Frequently asked questions

What counts as an AI agent?

The paper, published in December 2024, defines an agent narrowly as a system that selects among tools to pursue a goal. The word was then being applied to at least four different things, among them a chatbot with a persona, a deterministic workflow with a language model inside one step, and a speculative future system that operates without supervision. Only the tool-selecting case created a new control problem.

Where does the 14.9 per cent computer use figure come from?

It was the best capability evidence available in December 2024, and it came from the supplier. Anthropic’s research post of 22 October 2024 reported 14.9 per cent on an evaluation of computer-using ability, against a stated human range of 70 to 75 per cent, and its product post reported 22.0 per cent where the system was given more steps. Both figures were the vendor’s own.

How do you protect an AI agent from prompt injection?

Scope its permissions to what the task genuinely requires. Conventional input validation was built to catch malformed or malicious data arriving at a boundary, and an injected instruction arrives well-formed and legitimate to receive. Identity and least privilege therefore sit at the centre of the December 2024 paper’s control architecture, since an instruction reaching the system from a hostile page travels only as far as its permissions allow.

Who should own the permissions an AI agent is given?

A named person, and ownership follows the rung of autonomy the process sits on. A permission granted by one team, exercised by a second and audited by a third is a permission that nobody owns. The December 2024 paper gives a section to who owns what, on the ground that different processes sit on different rungs and the same process may sit on different rungs for different users.

What should a board ask before approving an AI agent pilot?

Three questions. Ask what the system will be permitted to touch, what it will be permitted to do with each of those things, and how an action gets reversed when it goes wrong. The December 2024 paper treats reversibility as the gate, since a pilot is defined by the ability to stop it and undo it. Three clear answers make a pilot, and one vague answer makes a deployment.

Paul Forrest is a fractional Chief AI Officer and AI strategist working with UK boards, private equity portfolio companies and growing businesses. Read how the fractional CAIO engagement works, or see the full range of Ecaveo AI services.