Prompt Literacy at Work: a practical framework for better human and machine collaboration

Prompt literacy is a verification discipline with a drafting technique attached, and an organisation that teaches the technique alone gets faster production of work nobody has checked.

What is prompt literacy?

Prompt literacy is five things done in sequence. State a goal, supply the context the system cannot infer, constrain the task, specify the form of the output, and verify the result against something other than how the result reads. Paper Two set out that definition on 31 March 2023, 17 days after OpenAI released GPT-4, and what follows describes the position as it stood then rather than the position today. The first four steps were teachable in an afternoon. The fifth was where professional responsibility sat, and no amount of prompting technique reached it.

• The capability frontier moved on 14 March 2023 and the limitations stayed where they were, since the vendor’s own release stated that the model was still not fully reliable, that it hallucinated facts and made reasoning errors, and that social biases and adversarial prompts were known limitations.
• Availability at launch was narrower than the coverage suggested, since image input was demonstrated with one partner and remained unavailable to users.
• A prompt has seven parts and most workplace prompts supplied two, with goal and task present while context, source material, constraints, output form and the success test went missing.
• Verification runs up five rungs of steeply rising cost, and the rung a task requires is set by the consequence of being wrong.
• Assessment works better on observed work than on vocabulary, so a four-level rubric that asks for a verified output under time pressure beats a written test of prompting terms.

Why did this become a workplace question in March 2023?

The release of GPT-4 on 14 March changed the range of tasks worth attempting and left every reason for checking the result exactly where it stood. Until that month the practical question in most organisations was whether these systems were good enough to be worth learning at all, and by the end of March few organisations were still asking it. What replaced it was harder, because it concerned people. Coverage of the release also ran looser than the release itself, since text access on 14 March went to paying subscribers under a usage cap and to a waitlisted developer interface.

OpenAI described the model as a large multimodal model accepting image and text inputs and emitting text outputs, and reported that on internal adversarial factuality evaluations it scored 40 per cent higher than its predecessor, that it was 82 per cent less likely to respond to requests for disallowed content, and that it responded to sensitive requests in accordance with company policy 29 per cent more often. Those were the vendor’s own internal evaluations, and the paper treated them as such. The first figure carried the greatest risk of misreading, because a relative improvement on an adversarially constructed internal benchmark measures one model against another, and an organisation reading it as an accuracy rate would have concluded that four in ten previous errors had gone away. Nobody made that claim.

What was missing from the prompts people were actually writing?

Context, source material, constraints, output form and the success test were the missing parts. Paper Two treated a prompt as having seven parts, and in the workplace prompts it examined the goal and the task were almost always present while the other five were absent. That absence produced the output people then complained about, which made the complaint a fairly accurate description of the input.

Iteration was the work. A first attempt that returns something unusable is an ordinary step in drafting, and the productive response is to identify which of the five missing parts the system lacked and supply it. Treating a second attempt as evidence of failure pushes people towards length when what the prompt needs is the missing part.

How much verification does a task actually need?

The consequence of being wrong sets the level. Paper Two described a ladder of five rungs whose cost rises steeply, starting with surface review, which means reading the output for plausibility and which is nearly free and nearly worthless. Confirming a claim against a source, redoing a calculation, sending the work to someone qualified to judge it, and refusing the task altogether each cost more than the rung below. A task whose failure costs an afternoon sits low on that ladder, and a task whose failure reaches a client sits near the top.

This is where the framing did the damage. The market had fixed on the phrase prompt engineering, which implies that the skill is technical and belongs to specialists, and that implication sent attention towards drafting. What I observed in organisations at the time was well-constructed prompts producing confident output that nobody had checked, which is a question about accountability dressed as a question about technique.

How should an organisation assess prompt literacy?

Watch someone produce a verified output under time pressure. Paper Two proposed a four-level rubric built on observed work, on the argument that a written test of prompting terminology measures vocabulary and vocabulary is cheap to acquire. Someone who can name six prompting patterns and someone who can hand over a checked draft are answering different questions, and only the second tells an employer anything about risk.

The paper closed with a four-week training sprint, followed by end matter setting out method, evidence and limits together with the sources considered and set aside. That last section mattered, because a paper written 17 days after a major release is working with thin evidence, and naming the rejected material is part of showing how thin.

Read now, the factual core of that paper belongs to March 2023, and the figures, the access conditions and the model it discussed have all been overtaken. The structural claim has held up better. Prompt literacy still consists of five steps in sequence, the first four of which remain teachable in an afternoon, and the fifth still carries the whole of the professional responsibility. An organisation that teaches drafting and stops has bought speed at exactly the point where accountability sits, which was the argument then and is the one worth putting to whatever system sits on the desk today.

Frequently asked questions

What is the difference between prompt literacy and prompt engineering?

Prompt literacy is a verification discipline with a drafting technique attached. The market had fixed on the phrase prompt engineering, which implies that the skill is technical and belongs to specialists, and that implication sent attention towards drafting. Paper Two, published on 31 March 2023, defined prompt literacy as five steps, of which verifying the result is the fifth and the one that carries professional responsibility.

What are the seven parts of a prompt?

A prompt has seven parts. Paper Two, written on 31 March 2023, named them as goal, task, context, source material, constraints, output form and a success test. In the workplace prompts it examined, goal and task were almost always present and the other five were absent. Supplying the missing part works better than adding length, and identifying which part the system lacked is the productive response to an unusable first attempt.

What did OpenAI’s 40 per cent factuality figure for GPT-4 actually mean?

It meant a relative improvement over the previous model on an adversarially constructed internal benchmark. OpenAI reported on 14 March 2023 that GPT-4 scored 40 per cent higher than its predecessor on internal adversarial factuality evaluations. The comparison sets one model against another. An organisation reading it as an accuracy rate would have concluded that four in ten previous errors had gone away, and nobody made that claim.

Who could actually use GPT-4 when it launched?

Paying subscribers of the consumer product, under a usage cap, plus developers on a waitlisted interface. Text access opened on that basis on 14 March 2023. Image input was demonstrated with one partner and remained unavailable to users, so availability at launch was narrower than the coverage of the release suggested. Anyone planning work around the launch needed to check what their own staff could reach.

How long does it take to train staff in prompt literacy?

The first four steps of prompt literacy are teachable in an afternoon, which covers stating a goal, supplying context, constraining the task and specifying the output form. Paper Two, published on 31 March 2023, closed with a four-week training sprint, and the fifth step, verification, is where the professional responsibility sits. An organisation that teaches drafting and stops has bought speed at the point where accountability sits.

Paul Forrest is a fractional Chief AI Officer and AI strategist working with UK boards, private equity portfolio companies and growing businesses. Read how the fractional CAIO engagement works, or see the full range of Ecaveo AI services.