There is a particular kind of discomfort that comes from looking at what an AI system has learned and recognising, in its outputs, something uncomfortably human.
Not a glitch. Not a malfunction. Not the robot going rogue. Just a very accurate reflection of the world we gave it to learn from.
The algorithm that learned from us
In 2018, Amazon quietly scrapped a recruitment tool it had been developing for several years. The tool used machine learning to screen CVs, trained on a decade’s worth of successful hires. The idea was straightforward: learn what a good Amazon candidate looks like, and use that to filter applicants faster and more consistently than human reviewers.
It worked. The model learned exactly what a successful Amazon hire had looked like over the previous ten years.
The problem was that successful Amazon hires over the previous ten years had been overwhelmingly male. So the model learned that male was good. It started downgrading CVs that contained the word “women’s” – as in “women’s chess club”, “women’s rugby team”, “president of the women’s society.” It penalised graduates of all-women’s colleges.
Nobody programmed it to do this. Nobody sat down and wrote “deprioritise women.” The algorithm looked at a decade of human decision-making and identified the pattern. Then it replicated it, at scale, faster than any human reviewer could.
Amazon scrapped the tool when they realised what it was doing. But the interesting question isn’t why the algorithm was biased. It’s where the bias came from in the first place.
It’s not always about corporate hiring pipelines
The Amazon case is striking because of its scale and its consequences. But AI bias doesn’t only show up in high-stakes corporate environments.
Consider the automated soap dispenser.
Widely deployed in offices, airports, and public bathrooms everywhere, many of these devices use infrared sensors to detect hands. They were, in many cases, tested primarily on lighter skin tones. The result: they work reliably for some people and intermittently or not at all for others, depending on how much infrared light their skin reflects.
Nobody designed a racist soap dispenser. The bias wasn’t in the intention. It was in the assumption – the assumption that the people doing the testing represented everyone who would use the device. They didn’t. And because nobody thought to check, a mundane piece of everyday technology quietly treats some people differently from others, every single day, in thousands of locations.
Same root problem as Amazon. Very different stakes. The same failure of imagination about whose experience the system was actually built for.
The mirror problem
This is what makes AI bias genuinely difficult to address, and genuinely important to understand.
It would be simpler if bias in AI were a technical problem – something introduced by a buggy algorithm or a rogue parameter that could be identified and patched. It isn’t. It’s a data problem, which means it’s a human problem. AI systems learn from human-generated data. Human-generated data reflects human decisions, human assumptions, human blind spots, and human history. A model trained on that data doesn’t inherit our intentions. It inherits our patterns.
Which means that deploying an AI system without interrogating what it learned – and from whom, and under what conditions – is not neutral. It is a choice to reproduce existing patterns at scale, with the added authority that comes from being able to say “the algorithm decided.”
The algorithm didn’t decide. We did. A long time ago. The algorithm just wrote it down.
What this means in practice
Identifying and addressing bias in AI is not primarily a job for data scientists, though they have an important role. It is a governance question, a leadership question, and ultimately a question about whose experience an institution considers when it builds, procures, or deploys any system that makes or influences decisions about people.
The questions worth asking before deploying any AI tool are straightforward, even if the answers aren’t:
- What was this trained on, and does that training data represent the people it will be used on?
- What patterns might it have learned that nobody intended it to learn?
- Whose experience was centred in its design and testing – and whose wasn’t?
- Has it been evaluated for differential outcomes across different groups of people?
These aren’t technical questions. They’re the kind of questions any thoughtful leader can and should be asking. The fact that an AI system is involved doesn’t make them harder. It makes them more urgent.
A final thought
The soap dispenser isn’t going to end anyone’s career. The recruitment tool might have. But they share a common origin: the assumption that the default experience is universal, and the failure to ask who might be left out.
AI holds a mirror up to that assumption. The reflection isn’t always comfortable. But it is useful – if we’re willing to look at it honestly and ask better questions next time.
Understanding and addressing bias in AI systems is a core part of responsible AI governance. If your institution is deploying or procuring AI tools and hasn’t yet asked these questions systematically, that’s a conversation worth having.