The 2020 exam grading algorithm

In March 2020 the government cancelled that summer’s exams in England. Students would still receive A level and GCSE grades, and the regulator, Ofqual, was asked to produce them in a way that kept results broadly in line with previous years.¹ Five months later the method it chose had been abandoned, the Prime Minister was blaming a “mutant algorithm”, and two of the most senior officials in English education had gone.¹

This is not, strictly, an AI story. The model was a statistical standardisation process, not machine learning. That is precisely why it matters to leaders: every one of its failures would have been the same with a neural network inside it.

What the model did

Schools submitted a grade for each student (the centre assessment grade) and a rank order of students within each subject. The model then adjusted those grades so that each school’s results matched its own historical performance, while the national distribution stayed close to previous years.

Where a school had very few students taking a subject, the model trusted teachers instead. Teacher grades were used where five or fewer students from a school entered a subject, and a mix of teacher grades and the statistical model was used for groups of between five and fifteen.²

On its own terms, the logic was defensible. Ofqual argued that teacher predictions were, on average, considerably more generous than past results, and that without moderation 2020 grades would have risen sharply.³

What happened on results day

On 13 August, A level results were published. Of the 718,276 grades awarded in England, 35.6% had been lowered by one grade from the teacher’s assessment, 3.3% by two grades and 0.2% by three.⁴ Only 2.2% were raised.

The pattern of who lost out caused the anger. The share of the most disadvantaged students achieving a C or above fell by 10.42 percentage points during moderation, compared with 8.34 points for the most advantaged.⁵ Meanwhile the proportion of A* and A grades at independent schools rose by 4.7 percentage points on the previous year, against 2.0 points at comprehensives and 1.7 points at academies.⁶

The small-cohort rule explains much of this. Independent schools are more likely to teach subjects in small groups, so more of their students received unmoderated teacher grades. A sensible technical decision produced a visibly unfair outcome.

Four days to reversal

Scotland had already changed course. On 11 August its Education Secretary announced that downgraded results would revert to teachers’ estimates.¹ England held its position for four more days. On 17 August, the government and Ofqual announced that students would receive their centre assessment grades after all. The Education Secretary, Gavin Williamson, said the process “has resulted in more inconsistency and unfairness than can be reasonably resolved through an appeals process”, and added: “I am sorry for the distress this has caused young people and their parents.”⁷

Ofqual’s chief regulator, Sally Collier, stood down on 25 August. The following day, the Department for Education announced that its permanent secretary, Jonathan Slater, would leave, and the Prime Minister told pupils that their grades had been “almost derailed by a mutant algorithm”.¹

Why it failed

Ofqual’s chair, Roger Taylor, gave the clearest diagnosis to the Education Select Committee on 2 September: “the fundamental mistake was to believe that this would ever be acceptable to the public.” He described it as an “impossible task” and said that “with hindsight it appears unlikely that we could ever have delivered this policy successfully.”⁸

Three failures stand out.

The objective was set for the system, not the individual. The model was designed to keep national results consistent with previous years. It achieved that. However, each student experienced it as a decision about them, made on the basis of how other people at their school had performed in the past. Accuracy at the level of the population was never going to feel like fairness at the level of the person.

Appeals were not a safety valve. An automated decision affecting hundreds of thousands of people needs a route to challenge it that works at the same scale. The appeals process could not absorb the volume of grievance, which is what the Education Secretary’s statement conceded.

Accountability was blurred. Taylor told MPs that the Department for Education had been “fully informed about the work we were doing and the approach we intended to take”, and that “the blame lies with us collectively – all of us who failed to design a mechanism”.⁸ When responsibility is shared that widely, nobody owns the decision to stop.

What happened next

Exams were again replaced by teacher assessment in 2021, this time without an algorithm. Ofqual’s experience is now one of the reference cases in UK debates about automated decision-making in public services, and the phrase “mutant algorithm” has become shorthand for the risk of hiding policy choices inside a model.

Lessons for leaders

1. Decide who the system is fair to before you build it. A model optimised for a population will make decisions about individuals. If those two views of fairness conflict, that is a policy choice for leaders, not a technical detail for analysts.
2. Test for legitimacy, not just accuracy. Ask how the outcome will look to the people on the receiving end, and to the public, before launch. Run the results against the groups most likely to be affected, and look for patterns that will be visible on day one.
3. Build a challenge route that matches the scale of the decision. If thousands of people can be affected at once, the appeal process must work for thousands of people at once, or you need a fallback you can switch to quickly.

Questions to ask in your organisation

1. Which of our automated decisions are experienced by individuals as being “about them”, even though the system was designed around averages?
2. If one of those systems produced a visibly unfair pattern tomorrow, who has the authority to stop it, and how quickly could they?
3. Have we tested our models’ outputs against the groups we would least like to see disadvantaged, and would we be comfortable publishing the results?