From Pilots to Copilots: scaling enterprise generative AI with data, controls and measurable value

Scaling generative artificial intelligence is a portfolio decision, taken against baselines and stop thresholds that are written down before any pilot runs.

Nothing in this article constitutes legal advice. It is offered to support discussion and further review with qualified advisers.

How should an organisation decide which AI pilots deserve a budget?

Treat the decision as a portfolio judgement, made against baselines and stop thresholds agreed before the work begins. That was the argument of Paper Four in this series, From Pilots to Copilots, published on 31 August 2023 and describing enterprise practice as it stood that summer. Read it as a record of that moment. Most large organisations were then holding a collection of experiments that had never been designed to be compared with one another, alongside a growing board expectation that something among them should be scaled. The paper set out how to convert that collection into a governed portfolio, using five stage gates, a reference architecture for an assisted workflow, a balanced value scorecard and an explicit decision to scale, hold or stop.

• The paper proposed five gates, running from problem proof through data readiness, controlled pilot and benefit proof to operational readiness, each carrying its own exit criterion.
• It put portfolio discipline ahead of platform selection, because demand intake, a use-case taxonomy and duplication control cost almost nothing and prevent four teams from building the same retrieval assistant in parallel.
• It treated the enterprise terms announced three days before publication as material for a contract negotiation, where each vendor claim could be tested and warranted.
• It proposed a value scorecard covering seven dimensions, being time, quality, throughput, experience, adoption, risk and total operating cost.
• It reported that among surveyed organisations acknowledging any adoption of artificial intelligence, only 21 per cent said they had policies governing employee use of generative tools.

What were the five gates for?

Each gate answered a different question, and each had to be passed in order. Problem proof asked whether the problem was real and worth solving at all. Data readiness asked whether the material the system would draw on existed in a usable and governed state. The controlled pilot asked what happened when the work was done for real, under supervision, with a named owner and a defined population of users. Benefit proof asked whether the measured result cleared a threshold set in advance, and operational readiness asked who would run the thing on a wet Tuesday in February once the project team had dispersed.

The paper observed that the fourth gate was where most portfolios stalled, since a pilot that never captured a baseline has nothing to prove a benefit against. The fifth was where many organisations discovered they had no support model. Written thresholds do the work here. A pilot that cannot say what it would take to stop has not produced the evidence that scaling requires, and the paper treated that as the first test a board should apply.

Did enterprise terms settle the data question?

They changed the procurement conversation and left the evidence where it was. On 28 August 2023, three days before the paper appeared, OpenAI announced an enterprise offering described as carrying enterprise-grade security and privacy alongside expanded capability. The announcement stated that the company does not train on business data or conversations and that models do not learn from usage, and it described encryption in transit and at rest, single sign-on, domain verification, an administrative console with an analytics dashboard, and a compliance attestation.

Every item on that list was a vendor statement about a vendor product, and the paper argued that it should be cited and treated commercially on that footing. The useful move was to carry the list into a contract negotiation and ask which of the claims the supplier would warrant, what happens on breach, and what notice is given when the underlying model changes. A published announcement is a starting point for that conversation. My own reading, which goes beyond what the announcement itself says, is that the distance between a published assurance and a warranted one was where most of the residual risk sat.

What did the paper say about the people doing the work?

A worker-centred section sat in the middle of the document, at the point where the argument could still affect the design. The available international evidence on artificial intelligence in workplaces reported both improved job quality and material concern, and it came from a survey conducted before these systems existed in their current form. That timing governs what the evidence can carry. Used carefully, it supports an argument about consultation and task redesign, both of which concern how work is organised. Its reach stops at the point where someone treats it as evidence about copilots.

Why did the scorecard carry seven dimensions?

One number hides the trade an organisation is actually making. Time, quality, throughput, experience, adoption, risk and total operating cost belonged on the scorecard together, because a scorecard reporting only the first will recommend scaling something that degrades the second. The scorecard then fed an explicit choice to scale, hold or stop, with the stop threshold written before the pilot ran. Writing it afterwards turns a threshold into a rationalisation.

The product detail in the paper belongs to August 2023, and the method it proposed has aged better. Vendor terms change and models change. The announcement examined there has been overtaken more than once since, yet the discipline the paper recommended still holds, which is to establish before any money is committed what the baseline is, what result would justify scaling and what result would stop the work. An organisation that can answer those three questions holds a portfolio and can say so to a board. That was the position at the end of that summer, and it is still the most useful thing to take from a paper now sitting in the archive.

Frequently asked questions

What is a stop threshold in an AI pilot?

A stop threshold is the result, agreed and written down before a pilot runs, that would end the work. From Pilots to Copilots, published on 31 August 2023, held that a pilot unable to say what would stop it has failed the first test a board should apply. Writing the threshold afterwards turns it into a rationalisation, so it belongs in the plan from the outset.

Why should portfolio discipline come before choosing a platform?

Portfolio discipline comes first because demand intake, a use-case taxonomy and duplication control cost almost nothing and prevent the common failure, which the August 2023 paper From Pilots to Copilots described as four teams building the same retrieval assistant in parallel. Platform selection settles which tool gets bought. Portfolio discipline settles which problems deserve one, and it gives a board a single view of everything running.

What should an organisation ask an AI vendor to warrant in a contract?

Ask which of the published claims the supplier will warrant, what happens on breach, and what notice is given when the underlying model changes. From Pilots to Copilots, written in August 2023, treated vendor statements about training on business data, encryption in transit and at rest, single sign-on, domain verification, an administrative console and a compliance attestation as the opening list for that negotiation. Each one belongs in a contract.

How many organisations had policies on staff use of generative AI in 2023?

Among surveyed organisations acknowledging any adoption of artificial intelligence, only 21 per cent said they had policies governing employee use of generative tools. From Pilots to Copilots reported that figure on 31 August 2023 and read it as a sign that the governance layer was thinner than the deployment layer almost everywhere. The number describes the position in that year.

Is a successful demonstration enough to justify scaling an AI tool?

Scaling asks for measured evidence, and a compelling demonstration sits below that bar. The design principle running through From Pilots to Copilots in August 2023 was that scale is a portfolio decision supported by evidence, taken against a baseline captured before the pilot and a threshold agreed in advance. A demonstration can be staged for an audience. A benefit has to show up in live work under supervision.

Paul Forrest is a fractional Chief AI Officer and AI strategist working with UK boards, private equity portfolio companies and growing businesses. Read how the fractional CAIO engagement works, or see the full range of Ecaveo AI services.