How P Values Are Misunderstood

A p-value of 0.03 does not mean there is a 3 percent chance the null hypothesis is true, and it does not mean there is a 97 percent chance the result is real. Those are the two most common ways researchers, journalists, and readers misread the single most-cited number in quantitative science. The American Statistical Association (ASA) considered the confusion serious enough to issue a formal statement on it in 2016, its first policy statement on a specific statistical practice in the organization's history. This article breaks down what a p-value actually measures, why the 0.05 threshold became a trap instead of a tool, how p-hacking exploits that trap, and why "statistically significant" so often gets mistaken for "true," "important," or "proven."

What a p-value actually measures

A p-value is the probability of observing data at least as extreme as what was actually measured, calculated under the assumption that the null hypothesis is exactly true. That is a mouthful, so it helps to slow down and unpack the direction of the logic. A p-value answers the question "if nothing real were happening here, how surprising would this data be?" It does not answer the question "given this data, how likely is it that something real is happening?" Those two questions sound similar but point in opposite directions, and swapping one for the other is called the inverse probability fallacy, sometimes referred to as the transposed conditional.

To make the distinction concrete, imagine testing whether a coin is fair by flipping it 20 times and getting 16 heads. The p-value tells you how unusual 16-or-more heads would be if the coin really were fair (50/50). It does not tell you the probability that the coin is fair given what you observed. Calculating that second, more intuitive question requires additional information the p-value alone does not provide, specifically some prior sense of how likely a biased coin was to begin with. Confusing the two is not a rounding error. It is a category mistake, and it is baked into how p-values get reported in press releases, news stories, and even some peer-reviewed abstracts.

The ASA's 2016 statement, authored by Ronald Wasserstein and Nicole Lazar and published in the journal The American Statistician, laid out six formal principles for interpreting p-values correctly. Among them: p-values do not measure the probability that the studied hypothesis is true, and p-values do not measure the probability that the data were produced by random chance alone. They also state plainly that a p-value, or statistical significance, does not measure the size of an effect or the importance of a result. A study can report p less than 0.001, an extremely small p-value suggesting the observed pattern would be very unlikely under the null hypothesis, for an effect so tiny it changes nothing in practice. Precision about the phenomenon being measured and practical relevance of that phenomenon are two completely separate questions, and a p-value only speaks to one small piece of the first.

It is worth being explicit about what the null hypothesis itself is, since the whole concept of a p-value hangs on it. The null hypothesis is the default assumption in a statistical test, typically that there is no effect, no difference, or no relationship between the variables being studied. A clinical trial testing a new drug against a placebo starts by assuming the drug does nothing different from the placebo; the p-value then measures how compatible the observed data are with that assumption of "nothing going on." Reject the null hypothesis, and you are saying the observed data would be unusually rare if that "nothing going on" assumption were correct, not that you have proven the opposite is true.

Why the 0.05 threshold became a trap

The conventional 0.05 significance threshold, meaning a p-value below 0.05 or 5 percent, traces back to the statistician Ronald Fisher and his 1925 book Statistical Methods for Research Workers. Fisher proposed the threshold as a loose, informal, context-dependent benchmark for deciding when a result was worth a second look, not as a permanent bright line separating "real" findings from "fake" ones. He explicitly treated it as a convenience, one that could and should shift depending on the field, the stakes, and the plausibility of the hypothesis being tested.

Over the following decades, and especially as null hypothesis significance testing spread into psychology, biomedicine, and the social sciences through the mid-20th century, that flexible benchmark hardened into something closer to a pass-fail exam. Below 0.05, a result was publishable, fundable, and citable. Above 0.05, it often disappeared into a file drawer, unpublished and unseen, a phenomenon researchers now call the file-drawer problem or publication bias. The threshold Fisher offered as one useful heuristic among several became, in practice, the single gate a study had to clear to count as science.

The American Statistical Association's 2016 statement on p-values was the first formal policy statement the organization had ever issued on a specific statistical practice, reflecting how widespread the misunderstanding had become even among trained researchers publishing in peer-reviewed journals.

That gate created a predictable incentive problem. When a career, a grant renewal, or a publication depends on crossing a specific numeric line, researchers, even careful and well-intentioned ones, start making analytic choices that nudge results toward that line. This behavior has a name: p-hacking, sometimes also called data dredging or the garden of forking paths, a term coined by statisticians Andrew Gelman and Eric Loken to describe how many small, reasonable-seeming analytic decisions can combine to produce a false-positive result without anyone intending to cheat.

Does a p-value below 0.05 prove an effect is real?

No. A p-value below 0.05 means the observed data would be unusual if the null hypothesis were exactly true, under the specific assumptions of the statistical model used. It does not rule out that the result came from chance, a flawed study design, an unmeasured confounder, or a biased sample, and it says nothing about whether the effect is large enough to matter in the real world.

What common analytic choices count as p-hacking?

Testing multiple outcome variables and reporting only the ones that crossed the threshold, checking results partway through data collection and stopping early once significance appears, excluding "outlier" data points without a pre-registered rule, and trying different statistical models until one produces a low p-value are all documented forms of p-hacking. None require deliberate fraud; each is often justified in the moment as reasonable analytic flexibility.

Statistical significance versus practical significance

Statistical significance and practical significance are answering two different questions, and treating them as interchangeable is one of the most consequential misreadings in applied research. Statistical significance asks whether an observed difference is unlikely to have arisen purely from random sampling variation, given the study's sample size and design. Practical significance, sometimes called clinical significance in medicine, asks whether the size of that difference is large enough for anyone to actually care about it, act on it, or change a decision because of it.

Sample size is the variable that most often drives a wedge between the two. A clinical trial with tens of thousands of participants has enough statistical power, meaning the ability of a study to detect a real effect if one exists, to detect a genuinely tiny effect, such as a systolic blood pressure reduction of 1 millimeter of mercury (mmHg), and report it with a very low p-value, sometimes far below 0.001. That result is statistically significant in the technical sense: it is unlikely to be pure sampling noise. It is not necessarily something a doctor or patient should change behavior over, because a 1 mmHg shift is far smaller than the natural day-to-day variation in a single person's blood pressure reading and carries no established clinical benefit at that magnitude.

The reverse problem happens too, and it is arguably more dangerous because it hides real effects. A study with a small sample size can fail to reach statistical significance even when a meaningful, real-world effect is present, simply because the study lacked the statistical power to detect it reliably. Readers who equate "not statistically significant" with "no effect exists" routinely make this error, when the more honest reading is often "this study was underpowered to tell either way."

This is why methodologists increasingly argue that effect size, a standardized measure of how large or meaningful a difference or relationship actually is, and the confidence interval, a range of plausible values for that effect given the observed data, matter more for real-world decisions than the p-value in isolation. An effect size tells a reader the magnitude of what was found; a confidence interval tells a reader how much uncertainty surrounds that magnitude, given the sample. A p-value alone collapses both of those pieces of information into a single binary signal: crossed the line, or did not.

Concept	What it tells you	What it does not tell you
P-value	How compatible the observed data are with the null hypothesis	The probability that the hypothesis itself is true
Effect size	The magnitude of the observed difference or relationship	Whether the study was well designed or free of bias
Confidence interval	A range of plausible values for the true effect	A guarantee the true value falls inside that specific range
Statistical power	The study's ability to detect a real effect if one exists	Whether an effect that was detected is practically meaningful

None of these four numbers alone answers "should I care about this result?" — that judgment requires reading the effect size, the confidence interval, and the study's design together, not the p-value in isolation.

P-hacking and the reproducibility problem

P-hacking would be a minor academic quirk if it did not have a measurable downstream cost: results that do not hold up when other researchers try to reproduce them. The Reproducibility Project: Psychology, a large, pre-registered, multi-lab collaboration published in Science in 2015, attempted to replicate 100 studies originally published in three major psychology journals. The project found that only 36 percent of the replications produced a statistically significant result in the same direction as the original study, and that the average effect size in the replications was roughly half the size of the effect reported in the original papers (Open Science Collaboration, 2015). The project's authors were careful to note that a failed replication does not automatically prove the original finding was false, since replications themselves carry their own uncertainty, but the scale of the shortfall was still striking to researchers across the field.

This finding, along with parallel replication difficulties documented in cancer biology and other fields, contributed to what is now widely described across the sciences as the replication crisis, a documented pattern in which a substantial share of published, peer-reviewed findings fail to reproduce when independently retested under similar conditions. P-hacking is not the sole cause of that crisis. Small sample sizes, weak theoretical grounding, and publication bias toward positive results all play a role. But threshold-chasing behavior around p less than 0.05, whether through flexible stopping rules, selective variable reporting, or undisclosed multiple comparisons, is one of the most well-documented mechanical drivers, because it directly inflates the rate at which false positives make it into print.

Statisticians have proposed several partial remedies. Pre-registration, publicly committing to a study's hypotheses, sample size, and analysis plan before collecting data, closes off much of the flexibility that enables p-hacking after the fact. Reporting exact p-values and confidence intervals rather than just "p < 0.05" or "significant" gives readers more information to judge a result's strength. Some journals and fields have also experimented with lowering the conventional threshold itself, or abandoning a single bright-line cutoff altogether in favor of describing results along a continuum of evidence strength, an approach closer to what the ASA's 2016 statement itself recommended.

The multiple comparisons problem hides inside a single study

P-hacking is often described as if it requires a researcher testing dozens of variables and cherry-picking the one that worked. But statisticians Andrew Gelman and Eric Loken showed that the problem runs deeper than that in their influential paper on the garden of forking paths. Even a single, honestly pre-specified hypothesis can pass through many small analytic decisions on its way to a p-value: which covariates to control for, how to handle missing data, whether to log-transform a skewed variable, which subgroup to highlight. Each choice seems reasonable in isolation, and a researcher may make only one final choice rather than testing several. The problem is that the choice was still contingent on the data, meaning a different dataset would likely have nudged the researcher toward a different analytic path, and that hidden flexibility inflates the effective number of comparisons far beyond the single test that gets reported (Gelman and Loken, unpublished manuscript, Columbia University).

This is distinct from, but related to, the more familiar multiple comparisons problem, which shows up whenever a study explicitly tests many hypotheses at once: comparing a drug against a placebo across a dozen outcome measures, or screening thousands of genes for an association with a disease. Running 20 independent tests at a 0.05 significance threshold means that, on average, one of those tests will cross the threshold purely by chance even if every null hypothesis is exactly true. Multiply that logic across a genome-wide study testing tens of thousands of comparisons, and false positives become close to guaranteed unless the threshold is adjusted.

The most common fix is the Bonferroni correction, a statistical adjustment that divides the significance threshold by the number of tests being run, so a study running 20 comparisons would require each individual p-value to fall below 0.0025 (0.05 divided by 20) rather than 0.05 to be considered significant. The correction is deliberately conservative. It controls the family-wise error rate, the probability of making at least one false-positive claim anywhere across the whole set of tests, at the cost of making it harder to detect real effects when many tests are involved. Less conservative alternatives, such as the Benjamini-Hochberg procedure, instead control the false discovery rate, the expected proportion of false positives among the results a study calls significant, which many researchers in genomics and other high-throughput fields consider a better tradeoff when hundreds or thousands of comparisons are unavoidable.

None of these corrections fix the garden-of-forking-paths problem on their own, because that problem lives in undisclosed analytic flexibility rather than in an explicit, countable list of tests. The more durable defense against both problems is the same one statisticians recommend against ordinary p-hacking: pre-registering the specific comparisons a study will run, and the specific correction method it will apply, before looking at the data.

Bayesian statistics as an alternative to p-value thresholds

A growing number of statisticians argue the deeper fix is not a better correction to the p-value framework but a different framework entirely. Bayesian statistics treats probability as a degree of belief that gets updated as new evidence arrives, rather than as a long-run frequency across hypothetical repeated experiments, which is the frequentist logic that p-values are built on. Instead of asking "how surprising would this data be if the null hypothesis were true," a Bayesian analysis asks "given this data and everything I believed beforehand, how much more or less plausible is each hypothesis now?"

The central tool for that comparison is the Bayes factor, a ratio that quantifies how much more strongly the observed data support one hypothesis over another, such as an effect existing versus not existing. Unlike a p-value, which can only ever argue against the null hypothesis, a Bayes factor can also provide positive evidence in favor of the null, letting researchers distinguish "we found good evidence there is no effect" from "we simply failed to detect one," a distinction ordinary null hypothesis significance testing structurally cannot make (Keysers, Gazzola, and Wagenmakers, PMC, 2020).

Bayesian methods are not a free lunch. They require specifying a prior, a starting assessment of how plausible each hypothesis was before seeing the new data, and critics point out that different researchers can reasonably choose different priors and land on different conclusions from identical data. Interpretive labels attached to Bayes factor thresholds, such as calling a certain ratio "strong evidence," have also drawn criticism for recreating the same bright-line, box-checking behavior that made the 0.05 p-value threshold a trap in the first place. Even researchers who advocate for Bayesian approaches generally describe them as a genuine complement to, not a guaranteed cure for, the interpretive problems that surround p-values, and most working statisticians today recommend judging a Bayesian or frequentist result the same way, by its effect size, its uncertainty, and whether it replicates, rather than by whether it clears any single numeric bar.

Why this keeps causing real-world confusion

Journalists and press-release writers routinely translate "p less than 0.05" into "scientifically proven" or "confirmed," collapsing a narrow, technical, conditional statement into a certainty the underlying data never actually supported. This is not a purely academic semantic quibble. It shapes which nutrition studies get covered as dietary breakthroughs on a single underpowered trial, which experimental drugs get described as more effective than the trial data actually justifies, and which contested policy claims get treated in public discourse as settled science rather than as evidence with real, quantifiable uncertainty attached.

The confusion compounds when a single study's statistically significant result gets reported without any mention of effect size, confidence interval, or whether the finding has been independently replicated. A reader encountering "new study finds X causes Y (p < 0.05)" has almost no way, from that headline alone, to judge whether X causes a meaningful amount of Y, a trivial amount of Y, or whether the finding might simply evaporate under a larger, better-designed follow-up study. The p-value in that sentence is doing far more rhetorical work than its actual statistical content supports.

Part of the fix has to happen upstream, in how researchers themselves report results, favoring exact p-values, effect sizes, and confidence intervals over a bare "significant" or "not significant" verdict. But part of the fix is also on the reading side: treating any single p-value as one data point about compatibility with a null hypothesis, not as a verdict on truth, and asking three follow-up questions before accepting a headline claim. How big was the effect? How wide was the uncertainty around it? Has anyone tried to reproduce it?

The takeaway

A p-value answers one narrow, specific question: how surprising would this observed data be if nothing real were going on, under this particular statistical model? It does not tell you the probability that a hypothesis is true, it does not tell you how big an effect is, and it does not tell you whether a study will replicate. The American Statistical Association said as much explicitly in 2016, and the replication difficulties documented since then, including in the Reproducibility Project: Psychology, back up why that distinction matters in practice. Reading past the p-value into the effect size, the confidence interval, and the study's design and sample size is the difference between actually understanding a result and simply reciting a threshold that a statistician a century ago offered as a rough starting point, not a final verdict.

---

Read next:

• What A Confidence Interval Really Means
• Why Randomized Trials Are Powerful
• Why Sample Size Changes Confidence

FAQ

Q: What does a p-value of 0.05 actually mean?

It means that if the null hypothesis were exactly true, data this extreme or more extreme would occur about 5 percent of the time by chance alone, under the assumptions of the statistical model used. It is a statement about the data's compatibility with an assumption, not a probability that the assumption itself is true or false.

Q: Why did the American Statistical Association issue a statement about p-values?

In 2016 the ASA published a formal statement in The American Statistician because widespread misuse and misinterpretation of p-values, including treating them as a stand-in for effect size, proof of truth, or the probability that a result happened by chance, had become common even in peer-reviewed research across multiple fields.

Q: Can a statistically significant result still be meaningless in practice?

Yes. Very large sample sizes can make even a trivially small, practically irrelevant effect statistically significant, because larger samples give a study more statistical power to detect small differences. Practical importance depends on the effect size and confidence interval, not on whether the p-value crossed the 0.05 threshold.

Q: What is p-hacking?

P-hacking describes analytic practices like testing many variables and reporting only the significant ones, adjusting an analysis after seeing preliminary results, or stopping data collection once a result crosses 0.05, all of which inflate the chance that a false-positive finding gets reported as if it were a real effect.

Q: Does a p-value above 0.05 mean there is definitely no effect?

No. A result that fails to reach statistical significance can simply reflect a study that was underpowered, meaning it had too small a sample size to reliably detect a real effect even if one exists. "Not statistically significant" and "no effect" are not the same claim, and conflating them is one of the most common misreadings of null results.

Q: How is a p-value different from a confidence interval?

A p-value is a single number testing compatibility with one specific null hypothesis value, usually zero effect. A confidence interval is a range of plausible values for the actual effect size, giving more information about both the magnitude of an effect and the uncertainty surrounding it than a p-value alone can provide.

Q: What is the Bonferroni correction and when is it used?

The Bonferroni correction divides the significance threshold by the number of statistical tests being run in a single study, making each individual test harder to pass. It is used whenever a study tests multiple hypotheses at once, such as comparing several outcomes or screening many variables, to keep the overall chance of a false-positive finding from climbing as more tests are added.

Q: Are Bayesian methods better than p-values?

Bayesian methods, particularly the Bayes factor, offer a different kind of answer: they can provide direct evidence for or against a hypothesis, including evidence favoring the null, which p-values cannot do. They are not automatically more reliable, however, since they depend on a chosen prior and can be misread through the same rigid threshold-checking behavior that undermined the 0.05 cutoff.