AB Testing: The Complete Guide

A/B testing — also written AB testing, or split testing — gets taught as the easiest idea in product: show half your users version A, half version B, keep whichever wins. That framing is why so many teams run tests for a year and learn nothing. The mechanic is trivial; the discipline is not. An A/B test is a controlled experiment, and an experiment without a sample size, a significance threshold, and a stopping rule set before you look at the data isn't a test — it's a coin flip you talked yourself into trusting. Almost everything worth knowing about A/B testing lives in the rules teams skip, so that's where this guide spends its time.

What A/B testing actually is

A/B testing is a method for comparing two versions of something — a page, a headline, a checkout flow — by randomly splitting live traffic between them and measuring which drives a chosen outcome. It comes straight from the randomized controlled trial, the same design medicine uses to test a drug against a placebo. The random split is the whole point: it's what lets you claim the change caused the difference, not the weather, the day of the week, or a marketing email that happened to land Tuesday.

Nielsen Norman Group defines it plainly: A/B testing "consists of comparing two versions of a webpage or app to determine which one performs better" by showing them to users at random and analyzing the results (NN/g). The word doing the work is random. Without random assignment you have two audiences, not one audience seeing two designs — and any difference you measure might just be the difference between those audiences.

The version everyone already sees is the control (A). The change you're proposing is the variant (B). You pick one primary metric — conversion rate, sign-ups, add-to-cart — before the test runs, and you commit to judging on that. Everything after this is about protecting that comparison from the ways it can lie to you.

How an A/B test actually works

Strip out the tooling and every legitimate A/B test is the same five steps, in this order.

1. Form a hypothesis. Not "let's try a green button." A hypothesis names a change, an expected effect, and a reason: "Moving the price above the fold will raise checkout starts, because users currently bounce before they find it." If you can't state the "because," you don't have a hypothesis. You have a hunch you want to confirm.

2. Pick one primary metric. One. The moment you have five metrics you're allowed to celebrate, one of them will move by chance and you'll call it a win. Secondary metrics are fine to watch; they're not allowed to decide the test.

3. Calculate the sample size first. Before a single user sees the variant, you decide how many you need — based on your current conversion rate and the smallest improvement worth caring about. This number tells you how long to run. Skipping it is the single most common way A/B testing goes wrong.

4. Split traffic randomly and run to the plan. Users are assigned to A or B at random and stay in their bucket. You run until you hit the pre-computed sample size — not until the graph looks good.

5. Read the result against your threshold. Did the primary metric move, by enough, with enough statistical confidence to rule out chance? If yes, ship. If no — and "no" is a real, useful answer — you've learned the change doesn't matter, which is worth knowing before you build ten more like it.

The steps look bureaucratic until you've watched a team ship a "20% lift" that evaporated in production. The order is the protection.

The statistics you can't skip

This is the part most guides wave past, and it's the part that decides whether your results mean anything.

Statistical significance answers one question: if the variant were actually no better than the control, how likely is a difference this big by pure luck? The conventional bar is a p-value under 0.05 — a 5% chance you're being fooled by randomness. Hit it and you have evidence; miss it and your "winner" is inside the margin of noise, no matter how much you like it.

Sample size is significance's twin. A 3% lift on 200 visitors is meaningless — the confidence interval is so wide the true effect could be anything from "much worse" to "much better." The same 3% on 40,000 visitors is a finding. Small tests don't produce small truths; they produce loud guesses. This is why step 3 comes before the test, not after: the sample size you need depends on your baseline rate and the effect you're hunting, and you have to know it to know when to stop.

Peeking is the trap that ruins good tests. It's tempting to watch the dashboard and call it the moment significance flickers green. Don't. When you check repeatedly and stop at the first significant reading, you dramatically inflate your false-positive rate — Evan Miller's How Not To Run An A/B Test shows the error rate climbing well past the 5% you thought you were holding. A test significant on day two and flat by day seven was never significant; you just caught it mid-wobble. Set the sample size, run to it, then look once.

Most changes don't move the needle. Microsoft, Google, and Bing teams report that a large share of their carefully-designed experiments produce no improvement or a negative one (HBR — The Surprising Power of Online Experiments). If your A/B testing program declares a winner every week, it isn't beating those odds — it's miscounting.

What's worth testing

A/B testing rewards leverage. Testing the shade of a link on a page 40 people visit a month will take years to reach significance and change nothing when it does. Point the method where traffic and money concentrate.

• High-traffic, high-intent pages. The checkout, the pricing page, the sign-up form, the homepage hero. Enough volume to reach significance quickly, and enough at stake to matter.
• One meaningful change at a time. Headline or layout or the number of form fields — not all three. If B changes five things and wins, you've learned that some combination works, not which one. That's not a finding you can build on.
• Changes with a mechanism. Test things you can explain why about. "Fewer form fields → less friction → more completions" is a testable chain. "Rounder corners feel friendlier" usually isn't worth the traffic.

And some things shouldn't be A/B tested at all. Rare events (enterprise deals, annual renewals) never reach sample size. Obvious bug fixes don't need a test — just ship them. Brand and trust decisions play out over months, not the two weeks a test runs. A/B testing is a scalpel for high-volume, measurable changes; it's the wrong tool for slow, qualitative, or low-frequency questions, where usability testing and research earn their keep instead.

Where A/B testing goes wrong

The failures are predictable, and almost none of them are about the tooling.

• Calling it early. Stopping the day significance first appears. The most common and most expensive mistake — see peeking, above.
• Ignoring the business cycle. Running a test for four days that spans a weekday-only B2B product, then generalizing. Run full weekly cycles so weekday-vs-weekend behavior averages out.
• Testing trivia. Button colors and comma placement on low-traffic pages. You'll get "results" that are pure noise and a program that feels busy while learning nothing.
• Chasing the local maximum. A/B testing optimizes the page you have; it won't tell you the whole flow is wrong. It's great at "which of these two checkouts is better" and useless at "should this be a checkout at all." Pair it with qualitative research so you're not perfecting the wrong thing.
• No hypothesis. Testing variants because you can, with no expected effect written down. When there's nothing to be wrong about, every result gets rationalized into a win.

Every one of these is a discipline failure dressed up as a data problem. Which is the theme of A/B testing generally: the math is easy, and the honesty is the hard part.

AB testing FAQ

What is A/B testing, in one sentence? Randomly splitting live users between two versions of something and measuring which one better drives a metric you chose in advance — a controlled experiment, not a guess.

Is AB testing the same as split testing? For practical purposes, yes — "split testing" is just another name for the two-version A/B test. It differs from multivariate testing, which varies several elements at once to study their combinations and needs far more traffic to reach significance.

How long should an A/B test run? As long as it takes to reach the sample size you calculated up front, and at minimum one or two full business cycles (usually two weeks) so day-of-week effects wash out. Not "until it looks significant."

What's a good sample size for A/B testing? There's no universal number — it depends on your baseline conversion rate and the smallest lift worth detecting. A tiny expected effect on a low-converting page can need tens of thousands of visitors per variant. Use a sample-size calculator before you start.

Why did my winning test stop winning in production? Almost always because it was called early or judged on too small a sample — the "win" was noise that regressed to the mean once real volume arrived. Set the sample size first, run to it, and look once.

Take it further

A/B testing answers "which version performs better." It can't answer the deeper question underneath it: can a person tell what your interface is doing and whether it's the right thing to do at all? That's the UX Clarity framework, and it's what we score in a Full UX Audit — the qualitative counterpart that tells you what to test before you spend traffic finding out. For the deeper pieces — sample-size strategy, significance in plain English, and the case studies behind this guide — the A/B testing hub collects them.

Sources: NN/g — A/B Testing 101 · HBR — The Surprising Power of Online Experiments · Evan Miller — How Not To Run An A/B Test.

Ready to find out whether you're optimizing the right page — or perfecting the wrong one? Apply for a Full UX Audit →