Survivorship bias, the bombers that came home, and the data you never get to see
“They don’t make them like they used to.”
You have heard it, and you may even have said it, usually while admiring a solid old wooden dresser or a building that has stood for two hundred years. The craftsmanship was better back then, the logic goes. Things were built to last.
Except we are only looking at the things that lasted. For every sturdy Victorian dresser still going strong, there were hundreds of flimsy ones that fell apart within a decade and were broken up for firewood long ago. The rubbish didn’t survive to be admired. What’s left gives us a flattering, and completely misleading, impression of the past, because the evidence has been quietly filtered by what happened to make it through.
That filter has a name: survivorship bias. And it is one of the easiest ways to reach a confident, well-reasoned, entirely wrong conclusion.
The bombers that came home
The cleanest example comes from the Second World War.
The Allied air forces were losing a lot of bombers and wanted to add armour to protect them. Armour is heavy, so you cannot cover the whole plane, and the question was where to put it. Analysts examined the bombers returning from raids and mapped the damage. A clear pattern emerged: the bullet holes were concentrated on the wings, the fuselage and the tail. The engines were relatively unscathed. The obvious answer was to reinforce the areas taking the most fire.
The statistician Abraham Wald looked at the same data and reached the opposite conclusion. Put the armour where the holes aren’t. On the engines.
His reasoning was this. They were only looking at the planes that came back. A bomber riddled with holes across its wings had, evidently, survived being hit there. The near-total absence of damage to the engines didn’t mean engines rarely got hit. It meant the planes hit in the engines were not coming home to be examined. They were at the bottom of the Channel. The undamaged areas on the survivors were precisely the fatal ones.
The data everyone needed most, the planes that were shot down, was the data nobody could see. Wald was right, the armour went where he said, and the insight is still taught today.
Why our minds fall for it
Survivorship bias is so persistent because it doesn’t feel like a gap. The returning bombers were real. The damage on them was real. The analysis was careful. Everything present in the data was accurate. The problem was everything that wasn’t there at all.
That is what makes it sneakier than an obvious error. We are wired to reason from what is in front of us, and missing data is, by definition, not in front of us. Nothing flags it. No warning appears. You draw your conclusion from the survivors and never think to ask who didn’t make it into the room.
It is why “successful people all do X” is such shaky advice. The founders who woke at 5am, took the big risk and dropped out of university get to write the book. The ones who did exactly the same things and failed are not on the shelf. You are being shown the survivors and asked to copy them.
Closer to home
Once you start looking, it turns up everywhere, including in the numbers organisations use to reassure themselves.
Consider the student survey that comes back glowing. Overwhelmingly, respondents rate the course highly and say they feel supported. Encouraging, until you remember who is answering. The survey reaches the students who are still enrolled. The ones who struggled, disengaged or quietly left partway through the year are not filling it in, because they are gone. “Our students rate us highly” and “we only asked the students who stayed” can be the same sentence wearing different clothes, and the students you most needed to hear from are exactly the ones the data has filtered out.
The same trap sits underneath a lot of feedback: satisfaction scores from the customers who didn’t cancel, exit data that only captures the people who left in the expected way, graduate earnings drawn from the alumni you happened to stay in touch with.
What to ask instead
You cannot analyse data that isn’t there. But you can learn to notice its absence, and a few questions help:
Who is missing from this? Before reading anything into a result, work out whose experience never made it into the dataset.
Did survival decide who is in the sample? If being included depends on having got through – stayed enrolled, stayed subscribed, come home – the survivors may be telling you the opposite of the whole story.
What would the people who dropped out have said? You may not have their answers, but asking the question is often enough to puncture a too-comfortable conclusion.
Am I studying success, or studying survival? They look identical right up until the moment they aren’t.
The point
Survivorship bias isn’t about bad data. Every number can be accurate and the conclusion still be wrong, because the most important information – the planes that didn’t return, the students who didn’t stay – was never in the dataset to begin with.
So when a set of results looks reassuring, it is worth asking not just what the numbers say, but who wasn’t around to be counted. Because yes, the data might be entirely correct. But what does it actually mean?