Selection Bias in User Research

In 1936 the Literary Digest mailed roughly ten million straw-poll ballots and got about 2.4 million back. The poll predicted that Alf Landon would beat Franklin Roosevelt. Roosevelt won 46 of 48 states. The usual explanation is that the mailing list came from telephone directories and car registrations, which skewed wealthy in the Depression. A 1988 re-analysis of survey data by Peverill Squire found the list was not the whole story: Landon supporters were also more likely to mail the ballot back.

Product teams repeat that mistake at a smaller scale every quarter. A feedback survey goes out, a few hundred people answer, and the result gets presented as what "users" think.

What selection bias is

Selection bias is the error that appears when the process that puts people into your study is related to the thing you are measuring. The sample stops resembling the population on the variable you care about, and no amount of extra responses fixes it, because each new response comes through the same filter.

It is a property of the recruiting process, separate from sample size, and that makes it harder to notice than ordinary sampling error. Margins of error shrink as responses come in, whatever the recruiting did.

Selection bias is also distinct from bias in how questions are asked or answers are read. A neutral questionnaire and an analyst with no confirmation bias still produce a skewed result if the respondents were filtered on the outcome.

Five filters between the user base and the data

Every study passes its participants through a sequence of filters. Selection bias can enter at any of them, and naming each one is the fastest way to audit a research plan.

1. The frame. This is the list you recruit from. An in-app survey can only reach people who open the app. A customer-email list leaves out users whose admin set up the account. A panel provider reaches people who signed up to be paid for surveys.
2. The invitation. Of the people on that list, some get asked. Triggering a survey after a successful checkout invites people who succeeded. Showing it on the third session invites people who came back three times.
3. The response. Who chooses to answer. This is where self-selection lives, covered below.
4. Completion. Who finishes. A 25-question survey keeps the patient and loses the busy, and in a usability study the participants who give up halfway often disappear from the notes.
5. Analysis. Who you keep. Dropping "outliers," excluding accounts younger than 30 days, or analyzing only users who reached a feature all narrow the group after the data is in.

The group that survives all five is rarely written down next to the finding.

Self-selection bias

Self-selection bias is the subtype where participants decide for themselves whether to take part, and that decision correlates with their answers. It is the default condition of almost every feedback channel a product owns.

Optional surveys draw from the ends of the distribution. People who are very pleased or very annoyed have a reason to spend two minutes on a form, and the people in the middle mostly skip it. App-store reviews, feature-request boards, support tickets, and churn surveys all share the shape.

Self-selection also runs the other way. People who have already stopped using a product rarely answer its surveys. A satisfaction score collected only from active users excludes the users with the most to say about why they left, and it will drift upward as unhappy users churn out of the frame even if nothing about the product improves.

Incentives change who self-selects without removing the effect. A gift card pulls in the people who want one, and that group is not a random slice of the base either.

Convenience sampling

Convenience sampling means recruiting whoever is easiest to reach: colleagues, friends, the first people to reply on a community forum, last month's participants. It is cheap and fast, which is why so much product research runs on it.

For early qualitative work it is often the right call. When the goal is to find out whether a prototype's navigation makes sense at all, five reachable people will surface the obvious breaks, and the interview or session is only a first pass.

Convenience samples share traits that matter in UX research: they skew toward people who know the product well, who are comfortable with technology, who already like the team, and who live in the researcher's time zone. Internal dogfooding is the extreme case, since every participant knows how the feature is supposed to work.

Label it in the readout: "six participants recruited from our beta community" tells the reader what kind of claim they are looking at.

Survivorship bias in product analytics

Survivorship bias is selection bias where the filter is survival: you study only what made it through a process and treat it as representative of everything that entered.

In analytics it usually shows up as a cohort mistake. Analyzing the behavior of current power users to decide what drives retention looks only at people who retained. If 90% of power users set up an integration in week one, that may be a cause of retention, or it may be something only people who were going to stay anyway bother to do. The users who set up the integration and churned are not in the dataset to settle it.

The fix is to start the analysis from everyone who entered, such as all signups in a given month, and follow them forward. Comparing the retention of users who did and did not set up the integration in that starting cohort still does not prove cause, but it removes the survivor filter. A properly randomized A/B test is the stronger answer when the question allows one.

Checking a sample against the user base

You cannot see the opinions of people who did not answer, but you can usually see their attributes. Before reading any results, compare the respondents with the population on variables that exist in your own data: plan, platform, tenure, region, activity level.

Here is an invented example. A product has accounts split across platforms like this, and an in-app survey gets 400 responses:

Primary platform	Share of active accounts	Share of respondents
iOS	45%	30%
Android	40%	20%
Web	15%	50%

Web users are 15% of the accounts and half of the respondents. In this example, say the widget renders better on desktop. Any overall figure from this survey describes web users more than anyone else. If the team's question is about the mobile apps, the survey mostly cannot answer it.

When the gap is large, there are three options, in order of preference.

The first is to re-field to the underrepresented groups, with reminders or a channel that reaches them, until the mix is closer. The second is to weight the responses back to the population shares, as with a stratified sample, which fixes the mix on the weighted variables and nothing else. The third is to report each platform separately and leave out the combined number.

If no attribute differs, that is evidence against selection bias on those variables. It says nothing about variables you did not check, such as mood or recent experience. Report the response rate anyway so readers can judge how much of the frame answered.

How to reduce selection bias before fielding

Most of the work happens in the research plan, before a single invitation goes out.

Write down the population the finding is meant to describe, in one sentence, and then write down the frame you can actually reach. Any gap between those two sentences is a known limit and belongs in the readout.

Trigger surveys on neutral events where you can. A survey shown at a random point in a session reaches a broader group than one shown after task success or after a cancel button. If the trigger has to be tied to an event, say which one in the results.

Sample from the full list rather than waiting for volunteers. Inviting a random set of accounts and following up with non-responders produces a different and more representative sample than a permanent "give feedback" link, even with the same number of answers.

Keep the incompletes. Partial responses and abandoned sessions are data about who drops out and where. Recording them makes the completion filter visible.

More methods and the tradeoffs between them are collected in the research methods hub.

Frequently asked questions

Is selection bias the same as sampling bias?

The terms overlap and are often used interchangeably. Sampling bias usually refers to a flawed frame or draw, the first two filters above. Selection bias is the broader term and also covers who responds, who finishes, and who is kept in the analysis.

What is the difference between selection bias and sampling error?

Sampling error is random: a different random sample would give a slightly different answer. Selection bias is systematic and pushes results in one direction. A bigger biased sample narrows the confidence interval around the wrong value.

What is self-selection bias?

It is selection bias caused by participants choosing whether to take part. In product research it affects every opt-in channel: feedback widgets, optional surveys, reviews, and feature-request boards. People with strong opinions answer at higher rates than people without them.

Is convenience sampling always wrong?

No. Convenience sampling is a reasonable way to find obvious usability problems early, when any participant will do. The problem starts when a beta-community or dogfooding result is reported as what the whole user base thinks.

How do you know if a survey has selection bias?

Compare respondents with the full user population on attributes you already have, such as plan, platform, tenure, and activity. Large differences are a warning. Also look at response and completion rates by group; a group that answers at a much lower rate is likely underrepresented in the opinions too.

Can weighting fix selection bias?

Partly, and only for the attributes you weight by. If the angriest users in every segment are the ones who answered, weighting by segment leaves that untouched.

Source for the 1936 poll re-analysis: Peverill Squire, "Why the 1936 Literary Digest Poll Failed," Public Opinion Quarterly, 1988.