What Is a Control Group?

A product team ships a new onboarding checklist on the first of the month. Four weeks later, activation (the share of new signups who complete a first project) has gone from 31% to 36%. The checklist gets the credit.

The same month, marketing ran a webinar series that brought in a wave of signups from larger companies, and those signups activate at a higher rate regardless of what onboarding looks like. The 5-point jump contains both effects, and the dashboard can't separate them.

What is a control group?

A control group is a set of participants who are drawn from the same population as everyone else in a study but don't receive the change being tested. They see the current design, the old flow, or no intervention at all. Everything else about their experience runs in parallel with the group that does get the change: same time period, same traffic sources, same product outside the part under test.

The group that receives the change is the experimental group (in A/B testing, the treatment or variant). The control group gives you a baseline for comparison. Whatever happens to the control group over the study period, whether seasonality, a campaign, an outage or a pricing change, also happens to the experimental group. So you can attribute the difference between the two groups to the change itself.

Rerun the onboarding example with a control. New signups are randomly split 50/50 for the month: half get the checklist, half get the old onboarding. At the end, the control group activates at 34% and the checklist group at 36%. The webinar lifted both groups by about the same amount, and the checklist's own contribution is 2 points. That's still worth shipping, but it's well short of the 5 points the before/after comparison claimed.

Control group vs experimental group

The two groups differ in exactly one respect: the thing you changed, which is your independent variable. Everything you measure afterward is a dependent variable, and the comparison only holds if the groups started out alike.

	Control group	Experimental group
Receives the change	No	Yes
Drawn from, and when	Same population, same assignment rule, same weeks	Same population, same assignment rule, same weeks
What its result tells you	What would have happened without the change	What happened with it

The row that matters most is "drawn from." If the experimental group is users who opted into a beta and the control group is everyone else, the groups differ in something besides the change. Beta volunteers are more engaged to begin with. The comparison then measures the change plus the difference between the kinds of people in each group, and there's no way to pull the two apart afterward.

Why random assignment matters

Random assignment makes the two groups comparable. When a coin flip (or a hash of the user ID) decides who lands in which group, every trait of the user base, measured or not, gets spread across both groups roughly evenly. Plan tier, device, how motivated someone is that day: the randomization balances all of them without you having to list them.

Any other assignment rule lets something slip in. Splitting by signup date puts weekday and weekend users in different groups. Splitting by region confounds the change with language and market. Letting the sales team choose which accounts get the new feature almost guarantees the experimental group is the friendlier accounts.

If one variable matters a lot and your sample is modest, you can randomize within groups (for example, split iOS users 50/50 and web users 50/50 separately), which keeps the platform mix identical in control and variant. That's stratified randomization, the experimental cousin of stratified sampling.

Types of control groups in UX research

Current-design control. The standard A/B setup: the control sees the product as it is today. This answers the question most product teams have, which is whether the new version beats the existing one.

No-treatment control. The control gets nothing at all, such as no onboarding email, no tooltip or no nudge. Use this when you need to know whether an intervention does anything, before arguing about which version of it is best.

Holdout group. A small share of users, often 1–5%, kept away from a feature or a whole program of changes for weeks or months. Short tests capture the first-week reaction. A holdout captures the cumulative effect, which is the one that matters for things like notification programs or personalization, where the value (or the annoyance) builds up over time.

A/A test. Both groups get the identical experience. An A/A test checks that your assignment and logging work: if two identical groups show a significant difference, the testing setup is broken, and every A/B result it produced is suspect.

Historical control. Comparing against last month's or last year's numbers. This is the before/after comparison from the opening, and it's weak for the same reason. Use it only when a concurrent control is impossible, and label the comparison as historical.

Control group vs control variable

A control group is a set of people who don't receive the change.

A control variable (or controlled variable) is a factor you hold constant for everyone so it can't explain the result. In a usability study comparing two checkout designs, you'd keep the task wording, the test device and the product catalog identical across sessions. Those are control variables. You'd still want a control condition, meaning the current checkout, to compare the new one against.

A study can have control variables and no control group (a single-design usability test with a fixed script), or a control group and poorly controlled variables (an A/B test where the variant also loads 800 ms slower because of an unrelated bundle change). The second case is common and easy to miss, because the page speed difference ends up counted as part of the design's effect.

Setting up a control group for an A/B test

1. Define the population. Decide who is eligible before assigning anyone: new signups only, users who reach the pricing page, accounts on a given plan. Both groups come from this pool.
2. Pick the unit of assignment. Usually the user. If users share a workspace and would see each other's experience, assign by workspace instead, or the groups will contaminate each other.
3. Assign randomly and consistently. Hash a stable ID so a returning user always lands in the same group. Re-randomizing on each visit means users drift between groups and the comparison is meaningless.
4. Fix the split and the duration in advance. 50/50 gives the most statistical power for a given amount of traffic. Choose the sample size up front and run at least one full weekly cycle, since behavior on Monday and Saturday usually differs.
5. Check the split before you read the result. If you planned 50/50 and ended up with 50,000 users in one group and 48,800 in the other, that gap is too large to be chance: a chi-square test puts it at p ≈ 0.0001. This is a sample ratio mismatch, and it usually means one variant is dropping users through a bug, a redirect or a bot filter. Fix it before trusting the numbers.
6. Compare on the metric you chose before launch. Significance, peeking and test length are in the A/B testing guide; choosing that metric is its own problem, covered in how to choose a dependent variable.

How control groups fail

Contamination. Control users see the treatment anyway: a colleague shares a screenshot of the new dashboard, a support article describes the new flow, or a cached page serves the wrong version. Each leak narrows the measured gap between groups, so a real effect can read as none.

Uneven exposure. The experimental group gets the change, and it also gets extra attention: a launch email, an in-app announcement, a faster server. The measured effect then includes all of it.

Novelty effects. Users click on new things because they're new. A two-week test can show a lift that disappears by week six, which is one of the better arguments for a holdout that runs past launch.

Too small to detect anything. A control group of 200 users can't show a 1-point difference in conversion with any confidence. Estimate the sample size you need before you start. If the traffic can't support it, test a bigger change or use a qualitative method instead.

Stopping when the number looks good. Checking the result daily and stopping on the first significant reading inflates false positives well past the nominal 5%. Set the duration in advance, or use a sequential testing method that's designed for repeated looks.

Frequently asked questions

What is a control group in simple terms?

It's the group in an experiment that doesn't get the change. Everyone else in the study is treated the same, so comparing the control group with the group that got the change shows what the change did on its own.

What is a control group in a UX experiment, specifically?

In most UX experiments, the control group is the set of users who keep seeing the current design while another randomly chosen set sees the new one. In an A/B test, "A" is conventionally the control.

Does every UX study need a control group?

No. Interviews, usability tests on a single design and other qualitative methods look for problems and reasons, and there's nothing to compare against. A control group becomes necessary when you want to say that a change caused a difference in a number. The research methods hub covers which method fits which question.

How big should the control group be?

In a standard A/B test the control is usually the same size as the treatment, because an even split gives the most power per user. Holdout groups are the exception: they're kept small, often 1–5%, because they're held out for a long time and you want most users to get the improvement.

Can the control group be last month's users?

It can, but that's a historical control, and it can't separate your change from anything else that differed between the two periods. Use a concurrent, randomized control whenever the product allows one.

What is the difference between a control group and a placebo group?

In clinical research, a placebo group receives a fake treatment so participants can't tell which group they're in. UX experiments rarely need one, because users in an A/B test rarely know a test is running. The A/A test is the nearest equivalent: it checks the measurement setup rather than participants' expectations.