Responsible AI Adoption in Health Research: why evaluation capability sets the ceiling

Responsible AI in health research turns on what an institution can measure, and a programme that cannot describe its own baseline is in no position to judge a model.

Nothing in this article constitutes legal, regulatory or clinical advice. It is offered to support discussion and further review with qualified advisers.

What decides how far AI adoption can go in health research?

Institutional evaluation capability decides it. A programme that can state what a workflow costs today, compare it against an assisted version, detect a quality regression before it reaches a publication or a participant, and repeat that exercise when the model changes, can adopt safely. AI research governance follows that capability.

• A systematic review of 4,609 studies of large language models in clinical medicine found that only 19 were prospective randomised trials, and that at least a quarter had samples below 30.
• Across 1,046 head-to-head comparisons, models outperformed humans in 38.4 per cent of exam-style evaluations and in 25.9 per cent of evaluations using real clinical data.
• Comparative studies of literature searching report models missing between 68 and 96 per cent of relevant studies, median 91 per cent, and the omission leaves no trace in the output.
• Human review fails through indifference, credulity and atrophy, and 20 hours of AI literacy training left physicians exposed to deliberately erroneous model output.
• Three regulators now ask what an organisation claims a product does, who holds clinical risk on deployment, and what recourse a person has, questions that sit outside what a model evaluation measures.

Why does measured AI advantage shrink as evaluation becomes more realistic?

A systematic review published in Nature Medicine in March 2026 screened 12,894 records and included 4,609 studies of large language models in clinical medicine, of which 1,048 used real-world patient data and only 19 were prospective randomised trials. Models won 38.4 per cent of exam-style comparisons against 25.9 per cent of comparisons using real clinical data. The review concludes that performance on contrived knowledge-based examinations correlates poorly with real-world clinical practice. Outside medicine the picture is no steadier, since three pooled randomised field experiments across 4,867 developers report a 26.08 per cent gain in completed tasks, while a randomised study of 16 experienced developers in familiar repositories reports a 19 per cent increase in completion time. That spread is the finding, and averaging it away produces a number describing no real workplace.

Which tasks does the evidence support, and which does it warn against?

The clearest positive findings concern ambient documentation. A pre-post survey of 263 ambulatory practitioners across six United States health systems reported burnout falling from 51.9 to 38.8 per cent and 0.90 fewer hours of after-hours documentation, though the design has no control arm and its authors claim only that AI may help reduce administrative burden. Evidence synthesis sits at the far end. Reviewing 19 comparative studies, Clark and colleagues found models missing between 68 and 96 per cent of relevant studies when searching, median 91 per cent, and they recommend against that use specifically. One mechanism explains both poles, because the encouraging evidence arises where verification is immediate, where the verifier owns the consequence, and where the reviewer draws on knowledge held independently of the tool. Searching satisfies none of those three conditions.

Can human review carry the weight AI research governance places on it?

Almost every AI governance framework relies on human review as its principal safeguard, and that safeguard has the weakest empirical support of any control in common use. Goh and colleagues randomised 50 physicians, where median diagnostic reasoning reached 76 per cent with model access against 74 per cent without, while the model working alone reached 92 per cent. A second trial gave 44 physicians, all with 20 hours of AI literacy training, deliberately erroneous output in half the vignettes, and accuracy fell by an adjusted 14.0 percentage points. The training left them exposed. Across four Polish endoscopy centres, adenoma detection in 1,443 unassisted colonoscopies fell from 28.4 to 22.4 per cent after AI arrived. In each study the review step existed, and what failed was the design of the task the human was given, so an auditable control states what the reviewer sees, how long they have and what authority they hold to reject.

How should clinical data governance tier a use case by recall cost?

Most risk frameworks tier by the sensitivity of the input data, and recall cost offers a better spine for clinical data governance in this sector. Recall cost is the expense and the incompleteness of withdrawing an output once it has left the control of the person who produced it. Consider two uses of one architecture. The first summarises pseudonymised measurements inside a trusted research environment for an analyst who checks every figure against the source. The second drafts general health information for a participant newsletter, using no personal data whatsoever. A sensitivity-led scheme scrutinises the first and waves the second through; a recall-led scheme reverses that order, because a published error reaches tens of thousands of people who trusted the sender. Sensitivity remains a modifier that can only raise a tier.

What does model validation in healthcare require before a deployment scales?

An organisation should not deploy above the level of evidence it is equipped to produce, which requires knowing which level it occupies. The five levels run from anecdote, through an instrumented baseline and a controlled comparison, to a pre-registered protocol and finally continuous assurance in production. Most organisations begin at level one while believing they are at level three. Each pilot should open with a one-page protocol written before anyone is given access, fixing the outcome, the comparison method, the quality rubric, the subgroup checks and the stop conditions. Reproducibility in AI research depends on recording model, prompt, corpus, tool and policy versions, since a deployment evaluated once was evaluated against a system that no longer exists. The Five Safes will stop an organisation disclosing what it should not, but whether the tool works falls to a proposed sixth Safe.

Where has the information governance question moved since 2024?

The United Kingdom still has no cross-sector artificial intelligence statute, and the 2026 King’s Speech contained no general AI bill. What has changed is less visible. Sections 67, 68, 80 and 86 of the Data (Use and Access) Act 2025 came into force on 5 February 2026. Section 67 defines scientific research purposes broadly, and section 68 permits consent to an area of research whose purpose cannot be fully specified, while Article 22B keeps a tight gate where special category data is involved. Widening a lawful basis does not widen a participant’s expectations. The Medicines and Healthcare products Regulatory Agency provides that intended purpose derives from claims made in instructions for use, labelling and promotional materials, and that general disclaimers are unacceptable where medical claims appear elsewhere. On my reading, which goes beyond what the guidance provides, marketing copy for an internal tool can create an obligation the engineering never contemplated.

The adoption sequence that follows from this evidence is deliberately unimpressive. Instrumenting a single workflow comes first, because nothing else works without it, and recording an intended purpose in one line comes second, since a regulator reads that line first. Tiering by recall cost, specifying the review and pre-registering the outcome follow in turn. Anything agentic comes sixth.

Responsible AI in health research therefore rests on capability an organisation builds for itself. Model validation in healthcare, reproducibility in AI research and ordinary information governance are one discipline seen from three angles, and each yields a record that a board, a funder or a participant could inspect and challenge. The sector has spent three years buying capability, but has yet to build the measurement that would say whether the capability helped.

A programme that can describe what it changed and whether it worked, with reviewers whose disagreement costs them nothing, has earned the right to let its portfolio grow. The full paper sets out the evaluation ladder, the recall-cost tiering scheme, the extended Five Safes and the 18-item board checklist behind this summary.

Frequently asked questions

What is the biggest constraint on AI adoption in health research?

Institutional evaluation capability constrains it more tightly than model capability does. An organisation needs to establish what a workflow costs today, compare it against an assisted version, detect a quality regression before publication, and repeat that exercise when the model changes. Without it, governance documents describe controls nobody can test.

Is human review a reliable control for AI in clinical research?

The empirical support is weaker than most frameworks assume. Randomised evidence shows physicians gaining two percentage points from model access, losing 14 percentage points when given erroneous output despite 20 hours of literacy training, and endoscopists detecting fewer adenomas unassisted once AI arrived. A review becomes a control once it specifies what rejection costs the reviewer.

Should a research programme use AI for literature searching?

Comparative evidence advises caution. Across the reviewed studies, models missed between 68 and 96 per cent of relevant studies when searching, with a median of 91 per cent, and the reviewers recommend against that use specifically. The omission leaves no trace, so the artefact resembles a complete search while behaving as a biased sampling instrument.

How does the Data (Use and Access) Act 2025 affect health research?

Sections 67, 68, 80 and 86 came into force on 5 February 2026. Section 67 defines scientific research purposes broadly, and section 68 permits consent to an area of research whose purpose cannot be fully specified, subject to recognised ethical standards. Article 22B keeps a tight gate on solely automated significant decisions involving special category data.

When does an AI research tool become a medical device?

Regulatory guidance locates intended purpose in the claims made in a product’s instructions for use, its labelling and its promotional materials, so a retrieval assistant described as helping researchers find datasets sits outside the device perimeter while the same code described as supporting a clinical decision may not. Use in a medical environment alone does not qualify a product as a device.