Stratified Sampling in UX Research

Suppose your product has 20,000 active accounts: 16,000 on the free plan, 3,400 on Pro, and 600 on Enterprise. You send a satisfaction survey to a random 400 of them. On average about 12 of those 400 will be Enterprise accounts — and on a bad draw, fewer. Twelve answers is not enough to say anything about the customers who pay most of the bills, and the free-plan majority will set the headline number whether or not that is what anyone wanted to learn.

Stratified sampling is the standard fix. You divide the population into groups that don't overlap, draw a random sample inside each group, and decide in advance how many people each group gets.

What stratified sampling is

A stratum is a subgroup of the population defined by something you already know about every member before you sample: plan tier, platform, region, account age. Stratified sampling has three steps:

1. Split the full list of eligible users (the sampling frame) into strata. Every user belongs to exactly one.
2. Choose a sample size for each stratum.
3. Draw a simple random sample within each stratum, independently.

Step 3 is what separates it from recruiting to a quota. Inside each group, every member has a known, non-zero chance of selection. That property is what lets you compute margins of error and weight the results back to the whole population later.

Stratifying pays off only when the strata answer differently on the thing you're measuring. Then it removes the luck of the draw from how many of each group you hear from, and the overall estimate gets more precise for the same number of responses.

Choosing the strata

The variable you stratify on has to pass two tests.

It must be known for everyone in the frame before you sample. Plan tier from billing data works. "Power user" does not, unless you've written a rule that classifies every account from logged behavior. A stratum you can only identify from the survey answers themselves can't be sampled from.

It should plausibly relate to the outcome. Good candidates in product research are the ones that change how someone uses the product: plan tier, role (admin vs member), platform (iOS, Android, web), tenure (under 30 days vs longer), and region when the product is localized. Signup month or email domain usually are not.

Keep the number of strata small. Crossing plan tier (3 levels) with platform (3) with tenure (2) gives 18 cells, and a 400-person survey spread over 18 cells leaves some of them with a handful of people. Pick the one or two variables you actually expect to report on.

Proportional vs disproportionate allocation

Once the strata exist, you decide how many people to draw from each.

Proportional allocation samples each stratum in proportion to its size. With the account mix above and a sample of 400:

Stratum	Accounts	Share	Proportional sample
Free	16,000	80%	320
Pro	3,400	17%	68
Enterprise	600	3%	12
Total	20,000	100%	400

This is simple random sampling with the variance taken out: you are guaranteed exactly 12 Enterprise responses instead of "about 12, maybe 6." The overall estimate needs no weighting, because each group already appears at its true share. But 12 is still 12.

Disproportionate allocation (sometimes called oversampling) gives small strata more than their share so each one can stand on its own:

Stratum	Accounts	Disproportionate sample	Weight (accounts ÷ sample)
Free	16,000	200	80
Pro	3,400	100	34
Enterprise	600	100	6

Now Enterprise has 100 responses, enough to report its own result with a usable margin of error. The price is that the raw sample no longer resembles the population, so any all-users number has to be weighted.

In most product surveys the small segment is the reason the survey exists, so oversample it. Proportional allocation is the better fit only when the report leads with one overall figure and no segment gets its own line.

Weighting the results back

Each response in a disproportionate sample stands in for a different number of real users. In the table above, one free-plan answer represents 80 accounts and one Enterprise answer represents 6.

Say the share of respondents who report being satisfied comes back as 60% for Free, 70% for Pro, and 40% for Enterprise.

The unweighted average of all 400 responses is (120 + 70 + 40) ÷ 400 = 57.5%.

The weighted estimate multiplies each stratum's rate by its population size: (16,000 × 0.60 + 3,400 × 0.70 + 600 × 0.40) ÷ 20,000 = 12,220 ÷ 20,000 = 61.1%.

The 3.6-point gap comes entirely from Enterprise being overrepresented in the sample. Reporting 57.5% as the all-users figure would understate satisfaction across the user base, because the unhappiest group makes up 25% of the sample and 3% of the accounts. Only combined figures need the weights. Put both the per-stratum and the weighted numbers in the report and label which is which.

Stratified sampling vs quota, cluster, and simple random sampling

These get confused in research plans, often because a quota was run and then described as stratified sampling.

Method	How people are picked	What it supports
Simple random	Everyone in the frame has an equal chance	Unbiased estimates; small groups may barely appear
Stratified	Random draw inside each predefined group, fixed count per group	Precise estimates overall and per group, with weighting
Quota	Fill a fixed count per group with whoever is available	Coverage of groups; no valid margin of error
Cluster	Randomly pick whole groups (teams, offices, schools), then study everyone or a sample inside them	Cheaper field work when members are geographically or organizationally bunched

Most UX recruiting is quota sampling: "we need four admins and four members," filled from a panel or whoever answers the intercept first. That is fine for qualitative work, where the goal is to hear from each kind of user. It becomes a problem only when the results are later presented with percentages and confidence intervals, which assume a random draw that never happened.

Cluster sampling wants the opposite shape from stratified sampling. Strata should be internally alike, while a cluster works best as a small, mixed copy of the whole population.

Where it goes wrong

Out-of-date stratum data. If your CRM marks accounts as Enterprise that downgraded two months ago, some respondents land in the wrong stratum and the weights are off. Refresh the frame from the system of record the day you draw.

Different response rates per stratum. You can sample 100 Enterprise admins and hear back from 18, while free-plan users respond at three times that rate. Weight on completed responses rather than invitations, and report the response rate for each stratum. Low response in one stratum is a fielding problem. Send reminders to that stratum or extend the window.

Strata that overlap. An account that is both "mobile" and "web" has to be assigned to one stratum by a written rule (primary platform by sessions over the last 30 days, say). Otherwise it gets two chances of selection and the weights no longer mean anything.

Stratifying an eight-person study. With eight interviews, there's nothing to stratify in the statistical sense. Pick participants on purpose to cover the segments that matter, and call it purposive or quota selection in the write-up.

Frequently asked questions

What is stratified sampling in simple terms?

You split your users into non-overlapping groups based on something you already know about them, then take a random sample from each group. Every group is guaranteed to be represented, and you choose how many responses each one gets.

When should UX researchers use stratified sampling?

For surveys and other quantitative studies where a meaningful segment is small relative to the whole — Enterprise customers, admins, users of a newer platform — or where you plan to compare segments. It's also useful when you want a tighter overall estimate from the same number of responses. For interviews and usability sessions, quota or purposive selection fits better.

What is the difference between stratified and quota sampling?

Both fix how many people come from each group. Stratified sampling fills each group by random draw from a complete list; quota sampling fills it with whoever is convenient. Only the stratified version supports margins of error and generalizing to the population.

Do I always need to weight stratified sample results?

Only when the allocation was disproportionate and you're reporting a combined figure. Proportional allocation produces a self-weighting sample, and per-stratum results never need weighting.

How many strata should a survey have?

As few as the questions require — usually one variable with two to four levels. Each additional variable multiplies the number of cells, and cells with only a few respondents produce unstable numbers.

Can I use stratified sampling for A/B tests?

Yes, as stratified randomization: split users by a variable such as platform, then randomize into control and variant within each stratum so both arms get the same mix. It reduces the chance that an imbalance between arms explains the result. The A/B testing guide covers the rest of the setup. The research methods hub covers when a survey is the wrong instrument in the first place.