Amazon’s experimental recruiting engine

In 2014, a team at Amazon began building software to review job applicants’ CVs and pick out the strongest candidates. As Reuters reported in October 2018, the tool gave each applicant a score from one to five stars, much as shoppers rate products on Amazon’s own site.¹

By 2015, the company had found a problem. The system was not rating candidates for software developer and other technical posts in a gender-neutral way.¹

What went wrong

The model had been trained on CVs submitted to Amazon over a ten-year period. Most came from men, reflecting the make-up of the technology industry. The system therefore learned that male candidates were preferable.¹

According to Reuters, it penalised CVs that included the word “women’s”, as in “women’s chess club captain”, and downgraded graduates of two all-women’s colleges. Amazon edited the programs to make them neutral to those particular terms. However, people familiar with the project told Reuters that this was no guarantee the system would not find other ways of sorting candidates that could prove discriminatory.¹

Amazon disbanded the team by the start of 2017, because executives lost hope for the project. Recruiters had looked at the tool’s recommendations when searching for new hires, but never relied solely on its rankings. Amazon declined to comment on the technology’s challenges, but said it was committed to workplace diversity and equality.¹

Why it matters

This case has become a standard reference because it is so clean. There was no malicious intent, no faulty code and no unusual data. The model did exactly what it was trained to do: find patterns in past hiring decisions and reproduce them. The problem was that the past was the problem.

It also shows why the obvious fix does not work. Deleting a protected characteristic, or the words that reveal it, removes the most visible signal. It does not remove the many other features that correlate with it, from the clubs people join to the colleges they attend and the language they use. A capable model will find those proxies.

Finally, it is a case where an organisation stopped. Amazon found the problem during development, tried to fix it, concluded it could not be confident in the result, and closed the project before it was used to make decisions on its own. That is less common than it should be.

What happened next

Reuters reported that Amazon later used a “much-watered down version” of the engine for simpler tasks, such as removing duplicate candidate profiles, and that a new team had been formed to try automated screening again, this time with a focus on diversity.¹ Since then, regulators have taken a closer interest in automated hiring. In the UK, for example, the Equality Act 2010 applies to recruitment decisions however they are made.²

Lessons for leaders

1. Treat historical data as a record of past decisions, not a definition of merit. If those decisions were skewed, a model trained on them will learn the skew and apply it at scale.
2. Expect proxies. Removing protected characteristics from the data is necessary but rarely sufficient. Test outcomes across groups, not just inputs.
3. Make stopping a legitimate outcome. Amazon’s team found the problem and the project was closed. Build the expectation, and the authority, to stop a project into the plan from the start.

Questions to ask in your organisation

1. Where are we using, or planning to use, AI to rank, score or screen people, and what historical decisions was it trained on?
2. Do we test the outcomes of those systems across gender, ethnicity, age and disability, and who sees the results?
3. If our testing found a bias we could not fully remove, who would have the authority to stop the system, and would they feel able to use it?