Why Sample Size Changes Confidence
Photo: Lukas Blazek / Pexels
A national poll of 1,000 registered voters produces roughly the same margin of error as a poll of 1,000 people drawn from a town of 50,000, because precision depends on how many people were sampled, not on how large the total population is. Quadrupling that sample from 1,000 to 4,000 only cuts the margin of error in half, not to a quarter, and doubling it barely moves the number at all. That counterintuitive relationship, driven by a single square-root term in the standard error formula, explains why polling organizations rarely spend the money to survey tens of thousands of people, and why a bigger sample size alone can never rescue a poll that asked the wrong people to begin with.
The square-root relationship behind every margin of error
The standard error — a measure of how much a sample statistic, like a sample mean or a sample proportion, is expected to vary from the true population value across repeated samples — is calculated by dividing a measure of variability by the square root of the sample size. Written out, the formula for the standard error of a sample proportion is the square root of p(1−p) divided by n, where p is the proportion being estimated and n is the sample size. The margin of error reported in most political polls is just that standard error multiplied by a constant tied to the desired confidence level, typically about 1.96 for a 95 percent confidence level.
The part that trips people up is the denominator. Because n sits inside a square root rather than standing alone, the relationship between sample size and precision is not linear. To cut a margin of error in half, you don't need to double your sample, you need to quadruple it: √4 = 2, so quadrupling n produces exactly half the standard error. To cut the margin of error to a third of its original size, you need nine times the sample. This is a textbook property of the square-root function, not an approximation or a rule of thumb specific to polling, and it holds for any statistic whose standard error follows the standard √n scaling, from opinion polls to lab measurements to A/B test conversion rates.
Pew Research Center's own methodology work makes the diminishing-return curve concrete rather than abstract. In a 2018 analysis of survey variability, Pew reported that for an unweighted sample size of 2,000 respondents, the average margin of error across estimated proportions was about plus or minus 1.3 percentage points; pushing the sample up to 8,000, a fourfold increase, brought the average margin of error down to about plus or minus 0.4 percentage points (Pew Research Center, "Variability of survey estimates," 2018). Roughly consistent with the square-root prediction: quadrupling the sample from 2,000 to 8,000 should shrink the margin of error to about half, and the observed drop, from 1.3 to 0.4 points, is even steeper once real-world weighting and design effects are accounted for, but the direction and the diminishing-returns shape match the underlying math exactly.
That diminishing-returns curve is also why national polling organizations, from Pew to Gallup, have converged on samples in the roughly 1,000-to-1,500 range for most standard public-opinion surveys. Below that range, margins of error widen quickly enough to make single-poll results unreliable for close races. Above it, each additional thousand respondents buys progressively less precision for progressively more fieldwork cost, which is why a pollster chasing a tighter margin of error is usually better off improving the sampling method than simply recruiting more people.
Why a bigger population doesn't require a bigger sample
One of the most persistent misunderstandings about sample size is the assumption that a study of a country with 330 million people needs a proportionally larger sample than a study of a city with 330,000 people. It doesn't, and the reason traces back to the same standard error formula. The formula for the standard error of a proportion, √(p(1−p)/n), contains no term for the size of the underlying population at all. Precision is a function of how many independent observations were actually collected, not what fraction of a larger population those observations represent.
There's a technical caveat here, and a real one: the standard formula assumes the sample is small relative to the population, generally under about 5 percent of it. When a sample gets large relative to a small, well-defined population, such as surveying 2,000 people out of a town of 10,000, statisticians apply a downward adjustment called the finite population correction factor, which slightly tightens the estimate further because sampling without replacement from a small pool removes proportionally more uncertainty. But for anything approaching national scale, this correction is negligible. Surveying 1,000 out of 10 million and surveying 1,000 out of 330 million produce almost identical margins of error, because in both cases the sample represents such a tiny fraction of the population that the correction factor rounds to essentially 1.
This is exactly why national polls and state-level polls often report similar margins of error even though the underlying populations differ by tens of millions of people. A pollster surveying likely voters in a single U.S. state with a population of 6 million, and a pollster surveying likely voters nationwide across a population of roughly 240 million eligible voters, can both use samples of around 800 to 1,000 respondents and land on comparable, roughly plus-or-minus-3-to-4-point margins of error. What differs the required sample size is not the population size, but factors like how much the outcome varies across the population, how confident the researcher wants to be, and how tight a margin of error the study needs, not what fraction of the total population gets interviewed.
Is a sample of 1,000 really enough to represent a country of millions?
Yes, for most standard opinion or attitude estimates, as long as the sample is drawn using genuinely random or properly weighted probability methods. Once a sample reaches roughly 1,000 respondents, further increases in the size of the underlying population have essentially no effect on the required sample size, because precision depends on the number of independent observations collected, not on what share of the total population those observations represent.
Why bigger samples cannot fix a biased sample
Sample size and sample representativeness are two entirely separate properties of a study, and conflating them is a common, consequential mistake. A larger sample size reduces random sampling error — the natural variability you'd expect to see across different random subsets drawn from the same population, purely due to chance. It does nothing whatsoever to correct systematic bias, a consistent skew introduced when the method used to select respondents favors certain kinds of people over others, regardless of how many of those non-representative people get surveyed.
The 1936 Literary Digest poll remains the field's textbook cautionary tale precisely because it isolates this distinction so cleanly. The magazine mailed out roughly 10 million ballots and received more than 2 million back, a genuinely enormous sample by any measure, and confidently forecast that Republican challenger Alf Landon would defeat incumbent Franklin D. Roosevelt. Roosevelt carried 46 of 48 states. The Digest's sampling frame, built from telephone directories, automobile registration lists, and its own subscriber rolls during the depths of the Great Depression, systematically overrepresented wealthier Americans who could afford a telephone or a car at a time when both were far from universal, and wealthier Americans in 1936 leaned more heavily Republican than the electorate as a whole. No number of additional mailed ballots, drawn from that same skewed list, could have corrected the underlying bias, because every additional respondent was being pulled from the same non-representative pool.
This is the reason methodologists insist on distinguishing "not enough sample" from "the wrong sample." A poll that is merely small but genuinely randomly drawn will have a wide, honest margin of error — the number itself signals the uncertainty. A poll that is enormous but systematically skewed will often report a deceptively narrow, falsely confident margin of error, because the standard error formula only accounts for random variation, not for a sampling frame that structurally excludes or underrepresents entire segments of the population it claims to describe. A narrow margin of error next to a large sample size can look more trustworthy than it actually is if nobody checks how the sample was assembled.
Modern polling failures echo the same underlying lesson even where the mechanics differ from 1936. Post-election reviews of the 2016 and 2020 U.S. presidential polling, conducted by industry bodies including the American Association for Public Opinion Research, identified issues like differential nonresponse — certain kinds of voters becoming systematically less likely to answer any poll at all — as a meaningful contributor to polling error in those cycles, a bias problem that additional sample size alone would not have solved, because the missing respondents weren't randomly missing; they were missing in a pattern correlated with how they intended to vote.
What statistical power actually measures, and why underpowered studies matter
Sample size doesn't only govern how tight a survey's margin of error is. In experimental research, it governs something related but distinct: statistical power, defined as the probability that a study will correctly detect a real effect, given that the effect actually exists at a specified size. A study with 80 percent power, a common target in study design, has an 80 percent chance of finding a statistically significant result if the true effect is exactly as large as the study was designed to detect. The remaining 20 percent of the time, an underpowered or unlucky sample will produce a result that fails to reach statistical significance even though the real effect was there all along, a mistake known as a false negative or a Type II error.
Statistical power depends on the same square-root relationship that governs margins of error, plus the size of the effect being searched for. Smaller true effects require larger samples to detect reliably, because a subtle signal is easier to lose inside random noise than an obvious one. This is why a clinical trial looking for a modest reduction in blood pressure needs thousands of participants, while a trial testing a treatment with an overwhelming, unmistakable effect can sometimes reach clear conclusions with far fewer.
The consequences of skipping this calculation are documented, not hypothetical. A widely cited 2013 review by Katherine Button and colleagues, published in Nature Reviews Neuroscience, examined the statistical power of studies across several major areas of neuroscience by analyzing a large set of published meta-analyses. The review found the median statistical power of the studies examined was about 20 percent, with plausible estimates ranging from roughly 8 to 31 percent depending on the subfield (Button et al., "Power failure: why small sample size undermines the reliability of neuroscience," Nature Reviews Neuroscience, 2013). A field running at 20 percent median power is missing roughly four out of every five real effects it goes looking for, on average, and the paper additionally noted that low-powered studies inflate the exaggeration of the effect sizes they do manage to detect, because a small, underpowered sample only reaches statistical significance when it happens to produce an unusually large effect estimate by chance.
Statisticians calculating a required sample size before a study begins run a process called power analysis, working backward from four inputs: the desired statistical power (commonly set at 80 or 90 percent), the significance threshold the study will use to judge a result, the amount of natural variability in the measurement, and the smallest effect size the researchers consider meaningful to detect. Skipping this step, or running it after data collection has already begun, is one of the more quietly common ways a well-intentioned study ends up too small to answer the question it was designed to answer.
| Factor in the calculation | Effect on required sample size |
|---|---|
| Desired precision (a narrower margin of error) | Larger sample needed |
| Higher confidence level (99% instead of 95%) | Larger sample needed |
| Higher desired statistical power (90% instead of 80%) | Larger sample needed |
| More natural variability in the outcome being measured | Larger sample needed |
| Smaller true effect size being searched for | Larger sample needed |
| Larger total population being studied | Little to no effect, once the population is much bigger than the sample |
Sample size calculations weigh precision, confidence, power, variability, and effect size — total population size, notably, barely factors in at all.
How is statistical power different from a margin of error?
A margin of error describes the precision of a single estimate, like the percentage of people who support a candidate. Statistical power describes a study's ability to correctly detect a real difference or effect between groups, given that the effect actually exists. Both depend on sample size through the same square-root relationship, but one measures precision of a single number and the other measures a study's ability to reliably tell "real effect" apart from "no effect."
How margin of error actually shrinks as sample size grows
The abstract square-root relationship is easier to hold onto with concrete numbers attached to it. The table below uses the standard formula for a survey estimate near 50 percent, where variability is at its maximum and the margin of error is widest, at a 95 percent confidence level.
| Sample size (n) | Approximate margin of error (±) | Sample size needed to halve the previous margin |
|---|---|---|
| 100 | 9.8 percentage points | — |
| 400 | 4.9 percentage points | 4× the previous sample |
| 1,000 | 3.1 percentage points | roughly 2.5× the previous sample |
| 1,600 | 2.5 percentage points | 1.6× the previous sample |
| 4,000 | 1.5 percentage points | 2.5× the previous sample |
| 10,000 | 1.0 percentage point | 2.5× the previous sample |
| 40,000 | 0.5 percentage points | 4× the previous sample |
Every halving of the margin of error requires roughly quadrupling the sample size — the pattern holds at every point along the curve, which is why polling budgets rarely chase margins below about ±1 point.
Reading down that table, the pattern is consistent: going from 100 to 400 respondents (a 4x increase) cuts the margin of error roughly in half, from about 9.8 to 4.9 percentage points. Going from 10,000 to 40,000 (again a 4x increase) does the same thing at the other end of the scale, from about 1.0 to 0.5 percentage points. The absolute gain in precision from that last jump, adding 30,000 more respondents to shave half a percentage point off the margin, is why almost no organization runs surveys at that scale for routine polling. The cost of recruiting tens of thousands of additional respondents rarely justifies a margin-of-error improvement measured in fractions of a point, especially once other sources of survey error, like nonresponse bias or question-wording effects, are typically larger than that remaining sliver of sampling error anyway.
Common confusions about what a larger sample actually buys you
A recurring mistake is treating sample size as a single dial that fixes every kind of uncertainty in a study, when it only addresses one specific kind: random sampling variability. It's worth being explicit about a few of the confusions that follow from that mistake, because each one shows up regularly in how survey results and study findings get reported.
The first confusion is assuming a huge sample size automatically means a trustworthy result. As the Literary Digest case demonstrates, and as post-election polling reviews from AAPOR have continued to document in more recent cycles, a large sample drawn through a flawed method can be less accurate than a small sample drawn through a genuinely random one. Sample size answers "how much random noise is in this estimate," not "was this the right group of people to ask."
The second confusion is assuming that a study reporting "no statistically significant effect" has proven that no effect exists. As the discussion of statistical power above makes clear, a small or moderately sized study can easily fail to detect a real effect simply because it lacked the power to find it reliably, a distinct outcome from a large, well-powered study that specifically rules an effect out. The companion articles on what a confidence interval really means and how p-values are misunderstood both cover this "absence of evidence is not evidence of absence" trap from a different angle, since it's ultimately the same underlying sample-size problem viewed through different statistical tools.
The third confusion is assuming precision and accuracy are the same thing. A large, well-executed random sample can be both precise (a narrow margin of error) and accurate (centered near the true population value). But it's entirely possible for an estimate to be precise and wrong at the same time, if a systematic bias shifts every observation in the same direction regardless of sample size, which is exactly the mechanism behind the Literary Digest failure and a mechanism the standard error formula, by design, has no way to detect.
The takeaway
Sample size shrinks random error according to a strict, well-established square-root relationship: quadruple the sample, halve the margin of error, and every further halving costs four times as much fieldwork as the last. That relationship, documented in Pew Research Center's own survey methodology data and built directly into the standard error formula, is a genuine mathematical fact, not a rule of thumb. But it only ever addresses random variability. It says nothing about whether the sample was drawn from the right people in the first place, which is why the 2.3-million-response Literary Digest poll remains, nearly ninety years later, the clearest historical proof that a bigger number sitting next to "n =" is not automatically a more trustworthy one, and why a well-designed study of 1,000 respondents can outperform a poorly designed study of a million.
Read next:
FAQ
Q: Why does a national poll only need about 1,000 people to be reasonably accurate?
Because the precision of a random sample depends on the sample's size, not on what fraction of the total population it represents. Once a sample reaches roughly 1,000 well-selected respondents, further increases in the underlying population's size have almost no effect on the required sample size, since the standard error formula contains no term for total population at all.
Q: Does doubling a sample size cut the margin of error in half?
No. Precision improves with the square root of the sample size, so quadrupling the sample size, not doubling it, is needed to cut the margin of error in half. Doubling the sample only reduces the margin of error by a factor of about 1.4, the square root of 2.
Q: Can a large sample size still produce a wrong result?
Yes, if the sampling method itself is biased. The 1936 Literary Digest poll surveyed over 2 million people yet wrongly predicted the presidential election's outcome because its sampling frame systematically overrepresented wealthier Americans, a bias sample size alone could not correct.
Q: What is statistical power, and why does it depend on sample size?
Statistical power is the probability that a study correctly detects a real effect, given that the effect actually exists at a specified size. It depends on sample size because larger samples reduce random noise, making a true underlying effect easier to distinguish from chance variation; a 2013 review found the median power across a large set of neuroscience studies was only about 20 percent.
Q: Is there ever a reason to use a sample much larger than 1,000 to 1,500 people?
Yes, when a study needs to detect a small effect size, compare results across many subgroups, or achieve an unusually tight margin of error for high-stakes decisions. Genome-wide association studies, large clinical trials searching for modest treatment effects, and surveys designed to report precise state-by-state or demographic breakdowns all typically require samples well beyond the range that suffices for a single national topline estimate.
Q: What is the difference between a small sample and a biased sample?
A small sample produces a wide, honest margin of error that signals its own uncertainty, and that uncertainty shrinks predictably as the sample grows. A biased sample can be any size, including very large, and produces a systematically wrong estimate that the standard margin-of-error calculation cannot detect, because that calculation only accounts for random sampling variability, not for flaws in how respondents were selected.
Follow new articles
Add SciHub101 to your feed reader to see new science and technology stories.
About the Author
SciHub101 is an independent science and technology publication founded and edited by Buğra Sözeri, a software engineer and app developer based in Türkiye. AI tools assist with drafting and research. Source documentation in the existing archive is uneven, and articles are being reviewed and corrected. The Editorial Team byline identifies editorial responsibility; it does not certify specialist or independent human review. Where we compare products, anything built by our own team is labelled as such — see the editorial guidelines and About page for how corrections and disclosures work.
Related Articles
What Is DuckDB? In-Process Analytics SQL
DuckDB is an embedded analytical database — SQLite for analytics. Query Parquet, CSV, and Arrow with SQL, no ETL needed.
