A/B Testing for Data Scientists — Statistical Fundamentals

An A/B test is a randomized controlled experiment run on live users. Half see the current version, half see a change, and statistics decide whether the difference in a metric such as conversion rate is larger than chance alone would produce. The arithmetic is simple. The discipline is not. Most invalid results come from decisions made before or during the test, such as skipping the sample-size calculation or checking results every morning. This article walks through the statistics with worked numbers and covers the seven mistakes that most often invalidate experiments.

How an A/B test works and why randomization is the point

A/B testing (also called split testing) randomly divides users into a control group that sees the current version and a treatment group that sees the new one. You measure a key metric for both groups over a defined period, then use a statistical test to judge whether any difference is real or just noise.

Randomization is what makes the comparison fair. Think of shuffling a deck before dealing: if the shuffle is honest, any difference between the two hands is luck, not design. If you instead let iOS users land in control and Android users in treatment, you are dealing the good cards to one player and then crediting them for winning. Randomization spreads every other factor, including the ones you did not think to measure, evenly across groups on average. That is why a randomized experiment can support a causal claim while an observational comparison cannot. For the longer argument, see our explainer on why randomized trials are powerful and on how correlation differs from causation.

Without an A/B test: "I think the orange button converts better."
With an A/B test:    "The orange button raised checkout conversion from 9.8% to 12.5%
                      (95% CI for the difference: +1.3 to +4.0 points, n = 4,200 per group)."

The second statement can be audited and challenged. The first is a guess.

A short history

Controlled comparison in marketing is old. According to Wikipedia's summary of the history, Claude Hopkins used promotional coupons to compare advertising campaigns in the early 1920s, though without formal significance testing. Google engineers ran their first online A/B test in 2000, on how many search results to show per page, and the same source reports that Google ran more than 7,000 such tests in 2011.

A well-documented modern example comes from Bing. In 2012 an idea about how to display ad headlines sat untested for more than six months because it was rated low priority. When an engineer finally ran a simple A/B test, it showed a 12% revenue increase without hurting key user-experience metrics, which Harvard Business Review later estimated would exceed $100 million a year in the United States alone. The test even triggered a "too good to be true" alert, which usually signals a bug; here the analysis confirmed the result. The lesson usually drawn is that intuition about which changes matter is unreliable, and cheap experiments are the corrective.

Step 1: Write a hypothesis you can fail

A good hypothesis is specific and falsifiable. It names what changes, what you measure, which direction you expect, and how large the effect must be to matter.

Bad:  "The new design will be better."
Good: "Changing the checkout button from blue to orange will raise the
       checkout click-through rate by at least 2 percentage points."

Three decisions follow from that sentence:

• The change: one button color, not a full redesign. Bundling many changes means you cannot tell which one worked.
• The primary metric: one number named in advance, here checkout click-through rate.
• The minimum detectable effect (MDE): the smallest lift worth acting on, here 2 percentage points. This is a business judgment, not a statistical one, and it sets the cost of the whole experiment.

Writing the MDE down is the most underrated habit in experimentation. If you cannot say how large an effect would change a decision, the test cannot tell you anything useful, whatever its p-value.

Step 2: Calculate sample size before launch

This is the step most teams skip, and skipping it invalidates the experiment. Without a pre-computed sample size you will either stop early on a lucky streak or run far longer than necessary.

Sample size depends on four inputs: the baseline rate, the MDE, the significance level alpha (the false-positive rate you will tolerate, conventionally 0.05), and the statistical power (the chance of detecting a real effect of at least the MDE, conventionally 0.80).

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

baseline = 0.10   # current conversion rate: 10%
target = 0.12     # smallest lift worth detecting: +2 percentage points

# Cohen's h: effect size for comparing two proportions
h = proportion_effectsize(target, baseline)

n_per_group = NormalIndPower().solve_power(
    effect_size=h,
    alpha=0.05,
    power=0.80,
    ratio=1.0,
    alternative="two-sided",
)

print(f"Cohen's h: {h:.4f}")                      # about 0.064
print(f"Users per group: {n_per_group:,.0f}")     # about 3,800

With a 10% baseline and a hoped-for lift to 12%, you need roughly 3,800 users in each group, or about 7,700 in total. Run until you reach that number, not before.

A handy cross-check, suggested in Evan Miller's "How Not To Run an A/B Test," is the rule of thumb n = 16σ²/δ² per group, where δ is the minimum effect you want to detect and σ² = p(1−p) for conversion rates. With p = 0.10 and δ = 0.02 that gives 16 × 0.09 / 0.0004 = 3,600, close to the 3,800 from the exact calculation. The rule is a sanity check, not a substitute for the full computation.

Turning sample size into calendar time

Sample size converts to duration through traffic. If 2,000 eligible visitors arrive per day and you split them evenly, 7,700 users take roughly four days. You would still run for at least one full week, so that every day of the week appears in both groups. Traffic is the binding constraint: if only 200 eligible visitors arrive daily, the same test takes more than five weeks, and you may decide the MDE is too small to be worth chasing.

How the minimum detectable effect drives cost

Sample size grows with the inverse square of the effect you want to detect. Halving the MDE roughly quadruples the users needed. The table shows approximate per-group sample sizes for a 10% baseline at 5% significance and 80% power, computed with the same two-proportion method as above.

Target conversion	Absolute lift (MDE)	Approx. users per group
15%	+5 points	680
13%	+3 points	1,770
12%	+2 points	3,800
11%	+1 point	14,700

The practical consequence is blunt. Small sites cannot reliably detect small effects. A change that lifts conversion by one point needs nearly 15,000 users in each group, so a team with modest traffic has to either test bolder changes or accept that some questions cannot be answered by experiment.

Step 3: Randomize and run the experiment

Assignment must be random with respect to everything that could affect the outcome. In practice most platforms assign deterministically by hashing a user identifier together with an experiment name, so a returning user always sees the same variant while assignment still looks random across users.

import hashlib

def assign_group(user_id: str, experiment: str, salt: str = "v1") -> str:
    """Deterministic, effectively random 50/50 assignment."""
    key = f"{user_id}:{experiment}:{salt}".encode()
    bucket = int(hashlib.sha256(key).hexdigest(), 16) % 100
    return "treatment" if bucket < 50 else "control"

print(assign_group("user_12345", "checkout_cta_color"))

Including the experiment name in the hash keeps assignments independent across experiments. If you reused the same hash for every test, the same users would always be in treatment together, and the experiments would contaminate one another.

Check that the split is what you intended

Before reading any result, confirm the groups are the size you planned. If you set a 50/50 split and see 5,000 versus 4,600 users, something is broken, perhaps a redirect that drops some users or a bot filter that applies to only one variant. This failure is known as a sample ratio mismatch, and it makes any outcome comparison untrustworthy. A chi-square goodness-of-fit test catches it:

from scipy.stats import chisquare

observed = [5000, 4600]
expected = [sum(observed) / 2] * 2    # intended 50/50 split
stat, p = chisquare(observed, f_exp=expected)
print(f"p = {p:.5f}")                 # about 0.00004: investigate before analyzing

For the 4,100 versus 4,050 split used in the worked example later, the same test returns p of about 0.58, which is unremarkable.

Run the experiment for at least one full business cycle, usually a week, so that weekday and weekend behavior are both represented.

Step 4: Analyze the results once, at the planned end

When the pre-computed sample size is reached, run the test you planned. For conversion rates, a two-proportion z-test is standard. (A chi-square test on the 2×2 table is equivalent in the large-sample case; note that SciPy's chi2_contingency applies a continuity correction to 2×2 tables by default, which makes it very slightly more conservative.)

import numpy as np
from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep

conversions = np.array([523, 412])    # treatment, control
users = np.array([4200, 4200])

rate_t, rate_c = conversions / users
z, p_value = proportions_ztest(conversions, users)
low, high = confint_proportions_2indep(
    conversions[0], users[0], conversions[1], users[1],
    method="wald", compare="diff",
)

print(f"Control {rate_c:.1%}, treatment {rate_t:.1%}")            # 9.8% vs 12.5%
print(f"Absolute lift {rate_t - rate_c:+.2%}, relative {rate_t / rate_c - 1:+.1%}")
print(f"p = {p_value:.5f}")                                       # about 0.0001
print(f"95% CI for the difference: {low:+.2%} to {high:+.2%}")    # about +1.3 to +4.0 points

In this example the control converts 412 of 4,200 users (9.8%) and the treatment 523 of 4,200 (12.5%). That is an absolute lift of about 2.6 percentage points, a relative lift of about 27%, and a p-value near 0.0001. The 95% confidence interval for the difference runs from roughly 1.3 to 4.0 percentage points.

Step 5: Decide with effect size and intervals, not p alone

A p-value answers only one question: how surprising is this data if there were truly no difference? It says nothing about whether the difference is large enough to matter. The decision should weigh:

• Effect size: a 0.1-point improvement with p = 0.001 may not justify the engineering cost of shipping.
• The confidence interval: the plausible range of the true effect. A lower bound near zero means you may have gained very little.
• Guardrail metrics: did anything important get worse, such as page load time, refund rate, or unsubscribes?
• Segments, carefully: whether the effect holds across devices or regions can inform follow-ups, but see the warning below about slicing.

Result pattern	What it suggests	Typical action
Significant, interval well above your MDE	Real and meaningful lift	Ship, monitor guardrails
Significant, interval spans trivial to useful	Real but possibly tiny effect	Weigh cost of shipping against the lower bound
Not significant, narrow interval around zero	No effect large enough to matter	Stop; drop the change
Not significant, wide interval	Inconclusive, underpowered	Collect more data or redesign the test

What a p-value does and does not say

The p-value is the probability of seeing a result at least as extreme as yours if the null hypothesis (no true difference) were correct. If p = 0.03, then under no true effect a difference this large would appear by chance about 3% of the time.

The American Statistical Association's 2016 statement on p-values lists six principles. Three matter most for experimenters:

1. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
2. Business and policy decisions should not be based only on whether a p-value passes a specific threshold.
3. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.

The common misreading, "p < 0.05 means a 95% chance the effect is real," is exactly what principle 1 rules out. Our article on how p-values are misunderstood goes deeper, and the ASA statement itself is short and readable.

Any threshold is a convention. A stricter alpha such as 0.01 is reasonable when a wrong call is costly, for example a change to pricing or a safety-relevant feature, and 0.05 is the usual default for low-stakes interface tests.

Statistical power in plain terms

Power is the probability that your experiment detects a real effect of the size you specified. At 80% power, if the effect truly equals your MDE, you will find it four times out of five and miss it one time in five. A miss is a Type II error (a false negative), the mirror image of the false positive controlled by alpha.

Power, sample size, and effect size are linked. For a standardized effect of 0.2 (a "small" effect in Cohen's convention) with a two-sided test at alpha = 0.05, the approximate per-group sample sizes from the normal approximation are:

Power	Users per group (effect size 0.2)
70%	309
80%	393
90%	526
95%	650

These come from the formula n ≈ 2(z₁₋α/₂ + z₁₋β)² / d², where the z-values are standard normal quantiles. More power costs more users, and 80% is the standard compromise. For more on how sample size changes what your data can support, see why sample size changes confidence.

Confidence intervals: the range, not just the verdict

A 95% confidence interval is built by a procedure that, across many repeated experiments, would contain the true value about 95% of the time. For a difference in conversion rates, an interval that excludes zero corresponds to a two-sided test significant at 0.05, but it carries more information: the lower and upper ends tell you how small and how large the true lift could plausibly be.

For a single proportion, statsmodels.stats.proportion.proportion_confint provides several methods. Its default is the simple normal approximation, which behaves poorly when counts are small or rates are near zero or one; the Wilson method (method="wilson") is a safer choice there. For the treatment group above, 523 conversions out of 4,200 users gives a 95% interval of roughly 11.5% to 13.5%. For a fuller treatment of what these intervals mean and do not mean, read what a confidence interval really means.

Choosing metrics: primary, guardrail, and diagnostic

An experiment needs a small, deliberate set of metrics.

Metric type	Purpose	Example	How many
Primary	Decides the experiment	Checkout conversion rate	One
Guardrail	Detects harm	Page load time, refund rate, unsubscribe rate	A few
Diagnostic	Explains why	Click-through on each step of the funnel	As needed, exploratory only

The primary metric is the only one for which the test is designed and the only one that should drive the ship decision. Guardrails protect against a change that wins on one number while hurting the business elsewhere. Diagnostics help you understand mechanisms but should be treated as hypothesis-generating, because looking at many of them invites the multiple comparison problem described next.

A related trap is choosing a metric that responds quickly but does not track what you care about. Clicks are cheap to move; revenue per user or retention are what the business actually wants, and the two can disagree.

The seven mistakes that invalidate A/B tests

Mistake 1: Peeking and stopping early

Standard significance calculations assume the sample size was fixed in advance. Checking repeatedly and stopping when the result first crosses p < 0.05 breaks that assumption. Evan Miller's classic analysis gives a concrete example: with a 50% baseline conversion rate and a change that does nothing, stopping at the first sign of 5% significance, or after 150 observations, produced a false positive 26.1% of the time, more than five times the nominal 5%. To keep a true 5% false-positive rate, he calculates that after 10 peeks the reported significance level must be about 1.0%.

Fix: commit to a sample size and an analysis date. If you genuinely need to monitor results continuously, use a method built for it. Sequential designs borrowed from clinical trials set planned checkpoints with corrected thresholds, and some Bayesian designs allow valid stopping at any time. Do not apply an ordinary fixed-sample test repeatedly.

Mistake 2: Testing many metrics and keeping the winners

If you test 10 independent metrics at alpha = 0.05 on a change that has no effect, the chance that at least one shows p < 0.05 is 1 − 0.95¹⁰, about 40%. Report the "significant" one and you have a false discovery.

Fix: name one primary metric in advance and treat the rest as exploratory. If several metrics must count as primary, correct for the number of tests. The Bonferroni correction divides alpha by the number of tests (0.05 / 5 = 0.01), which is simple but conservative; Holm's method and false-discovery-rate procedures are less strict alternatives.

Mistake 3: Ignoring seasonality and short windows

Behavior differs between weekdays and weekends, paydays, and holidays. A test that runs only Monday to Thursday may not generalize to the weekend.

Fix: run in whole-week multiples. For high-stakes decisions, run two full weeks and check that the effect is consistent from one week to the next.

Mistake 4: The novelty effect

A new element can attract extra clicks simply because it is new, and the lift fades as users habituate. The reverse also happens: existing users may initially resist a change.

Fix: for visible interface changes, run long enough for the effect to stabilize, and compare early and late periods. If the lift decays steadily week over week, the long-run effect is smaller than the average suggests.

Mistake 5: Broken randomization

Randomization can fail quietly: a redirect that drops slow connections, a bot filter applied to one arm, a caching bug that serves treatment to returning users only. The groups then differ in more than the treatment.

Fix: run the sample ratio check described above on every experiment, and compare pre-experiment characteristics (device, country, tenure) across groups. Running an A/A test, where both groups see the identical experience, validates the whole pipeline: with a correct setup it should show no significant difference, apart from the roughly 5% of A/A tests that cross p < 0.05 by chance.

Mistake 6: Reading an underpowered "no result" as "no effect"

A non-significant result from a small test is weak evidence of absence. With only a few hundred users per group, even a real 2-point lift is likely to be missed.

Fix: look at the confidence interval. If it is wide and includes both zero and meaningful gains, the test was inconclusive, not negative.

Mistake 7: Slicing into segments after the fact

Cutting results by country, device, browser, and new versus returning users produces many comparisons. Some will look significant by chance, and a story can be fitted to any of them.

Fix: pre-specify the few segments you care about, treat any other segment finding as a hypothesis for a follow-up test, and apply multiple-comparison corrections where you do analyze several.

Faster experiments: variance reduction and sequential designs

Small effects need large samples, so teams look for ways to reduce noise rather than only add users. One widely used technique is CUPED (Controlled-experiment Using Pre-Experiment Data), described by Alex Deng, Ya Xu, Ron Kohavi, and Toby Walker at WSDM 2013. It adjusts each user's metric using their behavior before the experiment began, which removes variation that has nothing to do with the treatment. In their Bing experiments the authors report a variance reduction of about 50%, equivalent to getting the same statistical power with half the users or half the duration.

The intuition is a noisy scale. If you weigh people on a scale that wobbles, you gain accuracy by subtracting each person's baseline weight from their later reading, because the baseline differences between people are not what you are trying to measure. CUPED does the same thing for user behavior.

Sequential testing, mentioned above, addresses a different problem: the need to look early. It lets you check results at planned points or continuously while controlling false positives, at the cost of a somewhat larger sample if the effect is small.

A full worked example

An e-commerce team wants to test a shorter checkout form. Current checkout completion is 8%, and they decide a 2-point improvement is the smallest lift worth shipping.

Plan. With a baseline of 8%, a target of 10%, alpha = 0.05, and power = 0.80, the two-proportion calculation gives about 3,200 users per group. They add a safety margin, set a 14-day window to cover two full weeks, and name checkout completion as the primary metric, with refund rate and page load time as guardrails.

Run. After 14 days, 4,100 users are in control and 4,050 in treatment. The sample ratio check returns p of about 0.58, so the split is acceptable.

Analyze. Control completes 328 of 4,100 (8.0%) and treatment 412 of 4,050 (10.2%).

import numpy as np
from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep

conv = np.array([412, 328])      # treatment, control
n = np.array([4050, 4100])

z, p = proportions_ztest(conv, n)
low, high = confint_proportions_2indep(conv[0], n[0], conv[1], n[1],
                                       method="wald", compare="diff")
print(f"p = {p:.4f}")                          # about 0.0006
print(f"Difference: {conv[0]/n[0] - conv[1]/n[1]:+.2%}")   # about +2.2 points
print(f"95% CI: {low:+.2%} to {high:+.2%}")    # about +0.9 to +3.4 points

Decide. The lift is about 2.2 percentage points, a relative increase of roughly 27%, with p near 0.0006. The 95% interval runs from about 0.9 to 3.4 points. The result is statistically clear, but the lower end is below the 2-point threshold the team set, so the honest summary is "very likely a real improvement, probably between one and three and a half points." If guardrails are clean, shipping is reasonable, and the team should expect the realized effect to land somewhere in that range rather than exactly at the observed value. Estimates from experiments that crossed a significance threshold tend to run a little high, which is one more reason to prefer the interval over the point estimate.

Tools for running and analyzing experiments

The statistics above need only open-source libraries. Platforms mostly add traffic splitting, targeting, and reporting.

Tool	What it is for	Note
Python (SciPy, statsmodels)	Power calculations and analysis	Free; enough for analysis
Optimizely	Website and product experimentation	Named by Google as a partner after Optimize
VWO	Website testing and optimization	Named by Google as a partner after Optimize
AB Tasty	Experimentation and personalization	Named by Google as a partner after Optimize
LaunchDarkly	Feature flags with experimentation	Useful when tests ship as flagged code
Firebase A/B Testing	Mobile app experiments	Experiments started after October 23, 2023 use frequentist inference
Google Optimize	Simple web experiments	Discontinued September 30, 2023

Google's own help page confirms that Google Optimize became unavailable on September 30, 2023, and that Google worked with AB Tasty, Optimizely, and VWO to integrate with Google Analytics. For a wider look at analysis software, see our comparison of the best data science tools.

When not to run an A/B test

Some questions do not suit experiments:

• Too little traffic. If the sample size for your MDE would take months, test a bolder change or use other evidence.
• Interference between users. In marketplaces and social networks, one user's variant can affect another's experience, which violates the independence assumption behind the simple analysis.
• Rare, long-horizon outcomes. Lifetime value or annual retention may take too long to measure; teams often rely on earlier proxy metrics, which must be validated.
• Obvious fixes. A confirmed bug or a legal requirement does not need a test.
• Ethical limits. Withholding something beneficial or exposing users to harm requires review beyond statistics.

Experimentation answers "does this change move this metric for these users right now?" It does not answer why, and it cannot replace understanding the product.

Frequently Asked Questions

Q: How long should an A/B test run? A: Until it reaches the sample size computed before launch, rounded up to whole weeks so each weekday appears equally often. Many tests land between one and four weeks. Stopping early because a result looks significant inflates false positives; in one published example, such peeking produced a 26.1% false-positive rate against a nominal 5%.

Q: Is p < 0.05 always the right threshold? A: No. It is a convention. The American Statistical Association's 2016 statement says decisions should not rest only on whether a p-value passes a specific threshold. Use stricter alpha when errors are costly, and always look at effect size and the confidence interval.

Q: Can I test multiple changes at once? A: Yes, with multivariate designs or by running independent experiments on separate traffic, but each additional comparison raises the risk of false positives and requires more users. With 10 independent metrics at alpha = 0.05, the chance of at least one false positive on a null change is about 40%.

Q: What if I cannot get enough users for significance? A: Options include testing a larger change, accepting a larger MDE, extending the duration, or reducing metric variance with a method such as CUPED, which reduced variance by about 50% in Bing experiments. Treat a small underpowered test as inconclusive rather than as evidence of no effect.

Q: What is an A/A test and why run one? A: An A/A test gives both groups the identical experience. A correct setup should show no significant difference, apart from about 5% of tests crossing p < 0.05 by chance at alpha = 0.05. It validates randomization and measurement before you trust real experiments.

Q: What is a sample ratio mismatch? A: It is a split that deviates from the intended allocation by more than chance allows, for example 5,000 versus 4,600 users in a planned 50/50 test. It usually signals a bug in assignment or logging, and results from such an experiment should not be trusted until the cause is found.

Q: Should I use a chi-square test or a z-test for conversion rates? A: For two groups and a binary outcome, the two-proportion z-test and the chi-square test on the 2×2 table are equivalent in large samples. Differences are tiny, mostly from continuity corrections, so the choice rarely changes a decision.

The takeaway

An A/B test is only as trustworthy as the choices made before the first user is assigned. Fix the hypothesis, the primary metric, the minimum effect worth acting on, and the sample size in advance. Check the split, analyze once, and read the confidence interval alongside the p-value. The same few habits explain why experiments at companies like Bing can find a 12% revenue gain that intuition missed, and why undisciplined experiments produce "wins" that vanish after launch.

The most useful question to ask of any experiment result is not "is it significant?" but "how would I know if this were wrong?" If the plan, the randomization checks, and the interval can answer that, the result is probably worth acting on.

Sources

• Miller, E. "How Not To Run an A/B Test." https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Kohavi, R., and Thomke, S. "The Surprising Power of Online Experiments." Harvard Business Review, September-October 2017. https://hbr.org/2017/09/the-surprising-power-of-online-experiments
• Deng, A., Xu, Y., Kohavi, R., and Walker, T. "Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data." WSDM 2013. https://www.exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf
• American Statistical Association. "Statement on Statistical Significance and P-Values" (press release and six principles), March 7, 2016. https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf
• Statsmodels documentation, NormalIndPower. https://www.statsmodels.org/stable/generated/statsmodels.stats.power.NormalIndPower.html
• Google Optimize Help, "Optimize sunset." https://support.google.com/optimize/answer/12979939
• Wikipedia, "A/B testing" (history and citations to Kohavi and Thomke 2017 and others). https://en.wikipedia.org/wiki/A/B_testing

The sample-size and confidence-interval figures in this article were computed from the standard two-proportion formulas and match the code shown; run the snippets to confirm them in your own environment.

Related Articles

• SQL for Data Science — Queries You Need to Know
• Data Cleaning: 80% of Data Scientist's Job

Last verified: October 2026