Why the One Striking Result May Not Be a Signal
Imagine measuring many things about the same group: sleep, diet, mood, exercise, and dozens of other details. If you compare each measure with an outcome, most comparisons may show no meaningful relationship. Yet random variation can make one look unusually strong simply because so many chances were offered for a result to stand out.
This is the multiple-comparisons problem. A statistical test estimates how surprising a result would be under a particular assumption, such as there being no real relationship. That estimate applies to the test being run—not automatically to a whole collection of tests. With enough tries, an apparently rare result becomes less surprising: it may be the statistical equivalent of getting an unlikely roll after rolling the dice repeatedly.
How a Coincidence Becomes a Claim
The danger is greatest when researchers make choices after seeing the data. They might try different outcomes, divide people into different groups, or adjust which factors to account for, then emphasize the version that produced the clearest result. Each choice can be defensible on its own; taken together, they create more opportunities to find a pattern that will not hold up elsewhere.
That does not mean a surprising result is false. It means its apparent strength can be overstated if readers are not told how many comparisons were made. Statistical adjustments can make evidence harder to overinterpret when many tests are planned. Registering the main question and analysis in advance also makes it clearer which tests were central and which were exploratory.
The Test Is Whether It Holds Up
A pattern found while exploring data can be a useful lead. The next step is to check it with new data, ideally using a plan set before the results are known. If the relationship appears again, confidence grows; if it disappears, the first result may have been a coincidence rather than a discovery.
The practical lesson is simple: a compelling result is not just the one that emerged. It matters how many other possibilities were searched before it did.
