Yes, But What Does It Actually Mean? | No. 8
Statistical significance, the pay rise that wasn’t, and why “significant” doesn’t mean what you think
Imagine a colleague tells you they’ve had a significant pay rise. You’d form a picture straight away: a better car, a holiday booked, a certain spring in the step. Then you learn the rise was 40p a month. You would feel, reasonably, that they had misled you – not by lying, but by using a word that carries a promise it didn’t keep. “Significant” means big. Important. Something you’d notice.
Except in one setting it means nothing of the sort, and that setting happens to produce a lot of the numbers you’re asked to believe.
In statistics, “significant” has a technical meaning that has almost nothing to do with size or importance. The gap between the everyday word and the statistical one is responsible for a remarkable amount of confident nonsense.
What “significant” really means
When a study reports a “statistically significant” result, it is making a narrow, specific claim: that the result it found is unlikely to be down to pure chance. That is the whole of it. A statement about flukes, not about size.
The machinery behind it is the p-value, and you don’t need the maths, only the idea. Suppose you’re testing whether something has an effect – a new teaching method, a drug, a change to a website. You run your study and see a difference. The p-value asks: if there were no real effect at all, how often would a difference this big show up by random luck alone? If the answer is “hardly ever” – by convention, less than 5% of the time – the result is declared statistically significant. The thinking is that a fluke this size would be rare, so this probably isn’t one.
Notice what that does and doesn’t tell you. It tells you the effect is probably real. It tells you nothing at all about whether the effect is large, useful, or worth a moment of your attention. “Probably not a fluke” and “big enough to matter” are different questions, and only the first one has been answered.
The two ways this misleads
The first trap is the significant result that means nothing. Significance is heavily influenced by how many people are in the study, and with a large enough sample, almost any difference becomes significant, however trivially small. Study two million people and you can turn up a statistically significant link between some food and some illness that raises the risk by a fraction of a percent – real, not a fluke, and of no consequence to anyone’s life. The significance is sound. The importance is imaginary. The larger the study, the more easily this happens, which is the reverse of most people’s instinct.
The second trap runs the other way. A promising result from a small study can fail to reach significance, not because the effect isn’t real, but because the study was too small to prove it either way. A pilot with twenty students might show an encouraging improvement, come back “not statistically significant”, and get shelved as a failure, when the honest conclusion was “promising, needs a bigger trial”. Absence of significance is not evidence of absence. It is often evidence of a small sample.
Why the word trips us up
Part of the problem is the word itself, a poor choice from the moment it escaped the textbook. In every other context, “significant” signals importance, so when it’s attached to a finding, importance is what people hear. The technical meaning – “clears a particular threshold for being unlikely by chance” – never stood a chance against the everyday one.
That threshold deserves a note of suspicion too. The 5% line is a convention, not a law of nature. It means a result at 4.9% gets to call itself significant and one at 5.1% does not, despite there being next to nothing between them. Plenty of decisions hinge on which side of that arbitrary line a result lands, as though nature recognised the distinction.
Closer to home
This one reaches you most often as a claim about something that works.
A supplier tells you their tool delivers a statistically significant improvement in student retention. A pilot of a new intervention reports a statistically significant gain. A report announces this year’s satisfaction scores are significantly up on last year’s. In each case the word is doing the persuading, and in each case it has answered only the smaller question.
Because “statistically significant improvement in retention” is entirely compatible with an improvement so small it would never justify the cost of the tool. If the study was large enough, a rise of a fraction of a percentage point clears the bar. The number is real. Whether it is worth having is a question the word “significant” has been careful not to address – and the one you most need answered before you sign anything.
What to ask instead
The fix is to stop letting “significant” stand in for “important”, and ask the questions it skips past:
How big is the effect, in plain terms? Not whether it’s significant – how large. A real effect can still be far too small to care about.
How many people were in the study? Large samples manufacture significance from trivial differences; small ones hide real effects. The size changes how you should read the result.
How close to the line was it? “Significant” at 4.9% and “not significant” at 5.1% are the same result wearing different labels. Treat the threshold as a guide, not a verdict.
And the one that cuts through: if I take this seriously, what would I do differently? If the honest answer is “nothing, the effect is too small to act on”, then significant or not, the finding doesn’t change anything.
The point
Statistical significance is a useful idea doing an unglamorous job: telling you a result is probably more than chance. That is worth knowing. It is not the same as telling you the result matters, and the shared word tricks us into hearing a claim about importance that nobody ever made.
So the next time a finding arrives wrapped in the word “significant”, ask the question the word is hoping you’ll skip: significant, yes, but how big, and so what? Because yes, it might be statistically significant. But what does it actually mean?
