A few years ago, Amazon quietly scrapped an AI tool it had built to sift job applications. The tool had been trained on ten years of the company’s own hiring decisions, the idea being that it would learn to recognise a good candidate the way Amazon’s best recruiters did. It learned something else. It taught itself that men were preferable, and started marking down CVs that contained the word “women’s” – as in “women’s chess club captain” – along with graduates of a couple of all-women colleges. Amazon tried to correct it, couldn’t be confident they had caught every version of the problem, and shut it down.
Nobody at Amazon set out to build a sexist algorithm. That is the part worth sitting with. The bias was not programmed in. It was learned, faithfully, from the data, and the data was simply a record of how humans had actually behaved.
I think about that example a lot at the moment, because AI is arriving in admissions with a very appealing pitch. It is consistent. It does not get tired, or hungry, or irritable at application 300. It does not carry the unconscious baggage a human reviewer might. Feed it the applications and it will apply the same standard to everyone, fairly and at speed.
The first part of that is true. The word “fairly” is doing an enormous amount of work.
An AI sifting tool learns from examples. Show it your historical admissions decisions and it will hunt for the patterns that separated the offers from the rejections, so it can reproduce them. If those past decisions were entirely free of bias, wonderful. But if they carried any – a slight lean towards certain schools, certain postcodes, certain names, certain backgrounds – the tool has no way of knowing that was the bad part. It sees a pattern, learns it, and applies it to every future applicant, consistently, quickly, and with the quiet authority of a computer. It does not remove human bias. It launders it, and then it scales it.
The obvious response is to take the sensitive information out. Don’t let the model see ethnicity, or sex, or age, and it can’t discriminate on them. Except it can, because it doesn’t need the label. A model stripped of “ethnicity” will cheerfully reconstruct a decent approximation of it from postcode, school, first name, the sports someone played, the phrasing of a personal statement. Remove the protected characteristic and the proxies remain, quietly standing in for it. This is why fairness in these systems is so much harder than it first looks. You are not dealing with one field in a spreadsheet. You are dealing with everything that field was ever correlated with.
Here is where it stops being a technical curiosity and becomes a risk on your desk.
In the UK, admissions sit squarely under the Equality Act. If your process produces a worse outcome for a protected group and you can’t justify it, that is potentially indirect discrimination, and “the algorithm decided” is not a defence anyone should want to test. Nor can you quietly hand the liability to the vendor who sold you the tool. Read the contract and you will usually find they have been careful to leave responsibility for outcomes with you, the institution deploying it. You own the decision. You own the consequences.
And the ground has just shifted underneath all of this. As of this year, the rules on automated decision-making have changed. Solely automated decisions with a significant effect on someone are now permitted where they largely weren’t before, but only with safeguards, including a genuine right for the applicant to contest the decision and get human review. A rubber-stamp does not count. The regulator has been clear that the human has to be able to actually change the outcome. An applicant rejected by your clever new tool now has a clearer route to ask how that decision was reached, and a right to a real answer.
Which brings me to the risk I think is most underestimated, because it isn’t really about any single bad decision. It is about whether you could stand behind your process if someone asked you to.
Picture the question arriving – from a rejected applicant, from the EHRC, from the OfS, from your own governing body. How does this tool reach its decisions, and how do you know it is not discriminating? A lot of institutions, honestly, could not answer that today. The tool is a black box the supplier will not fully open. Nobody tested the outcomes across different groups. There is no documentation, because it all seemed to be working fine. “It seemed fine” is not a position you want to be defending after the fact.
None of this is an argument against using AI in admissions. It is an argument for going in with your eyes open, and treating fairness as something you actively check for rather than something you assume.
In practice that means a few things. Test the tool’s outcomes across protected groups before you trust it, and keep testing, because bias drifts as your applicant pool changes. Do a proper data protection impact assessment and an equality impact assessment, and treat them as genuine interrogation rather than paperwork. Insist that the vendor can explain how the thing makes its decisions, because if they can’t, you can’t defend it, and that in itself should tell you something. Make sure the human in the loop is a real one, with the information and the authority to overrule the machine. And write it down, because the day someone asks you to show your working is not the day you want to start.
This is, more or less, why AI-Ready Admissions exists. Rachel Reeds and I built it because we kept seeing the same gap: institutions adopting AI in admissions with genuine good intentions and no framework for checking that what they had bought was fair, lawful and defensible. The tools are not going away, and used well they can help. But “used well” is a decision, not a default, and it is a great deal cheaper to make that decision now than to explain, later, why nobody did.
Find out more here