Confirmation Bias in UX Design: How Teams Prove What They Already Decided

Robert Millikan measured the charge of the electron in 1913 and got a number slightly too low, because he was working from a wrong value for the viscosity of air. What happened next is the useful part. Plot every subsequent measurement of that constant against time and you get a slow climb: each one a little higher than the last, creeping toward the true value over decades rather than correcting in one jump.

Richard Feynman's account of why is blunt. When physicists got a number well above Millikan's, they assumed something was wrong with their apparatus, and went looking until they found a reason. When they got a number close to Millikan's, they didn't look nearly as hard (Cargo Cult Science, 1974).

Every one of those physicists was doing honest work. The distortion sits entirely in the asymmetry of scrutiny: contrary evidence had to clear a bar that agreeable evidence never faced. It took the field decades to work the error out of the literature.

That asymmetry is confirmation bias, and it runs a UX readout at least as efficiently as it ran a physics constant.

What confirmation bias actually is

Confirmation bias is the tendency to seek, notice, and weight evidence that fits what you already believe, while holding disconfirming evidence to a higher standard. The important word is unwitting. Raymond Nickerson's review of the literature makes the case that this is not usually motivated deception or even self-deception, but a difference in how easily belief-consistent information gets processed, observable in people with no stake in the outcome (Nickerson, 1998).

The cleanest demonstration is Peter Wason's rule-discovery task from 1960. Participants were given the number triple 2–4–6 and told it followed a rule. They could test as many triples of their own as they liked, getting only "fits the rule" or "doesn't fit" in response, and were asked to announce the rule once confident.

Almost everyone formed a hypothesis, usually ascending even numbers, then tested it by proposing 8–10–12. 20–22–24. 100–102–104. All confirmations. All useless. The real rule was any three ascending numbers, and the only route to it was proposing something you expected to fail, like 1–2–3 or 6–4–2. Most participants announced a wrong rule with high confidence, having collected a great deal of evidence perfectly consistent with it (Wason, 1960).

Wason's participants never generated a single observation that could have contradicted them, and read the accumulating passes as strength. That is the shape to carry into product work: a team running test after test that its hypothesis passes by construction.

It enters the process long before the analysis

Teams police the last step, where results get interpreted. By then most of the damage is upstream and invisible, because the upstream decisions all looked like ordinary planning.

The question. "Does the new flow work better?" has the answer built into its shape. Compare it to "where do people get stuck in each flow, and what does the stuck look like?" The first asks for a verdict. The second asks for observations that can embarrass either version.

The screener. Recruiting criteria are where confirmation bias is cheapest to install and hardest to see afterward. Screen for people who "use tools like ours weekly" and you have selected for fluency in the conventions your design already assumes. The users who would have failed are disqualified before the session exists, and the study comes out valid on a population that was never the risk.

The task wording. "Find the export button" has told the participant there is one and that it is a button. "Get a copy of this report to send to your accountant" leaves the mechanism to them. The gap in success rate between those two phrasings is not the interface changing. NN/g's case against leading questions is usually made about interviews; it applies at least as hard to task scenarios.

The moderation. A moderator who wants the flow to work says "great" when someone clicks correctly and "hmm, take your time" when they don't. Neither is a neutral signal. Participants read the room and adjust.

The note-taking. Two observers watching the same three-second hesitation will write "paused briefly" or "didn't know where to go" depending on what they walked in expecting, and both notes are honest.

The readout is where a data point becomes a false insight

The distance between what happened and what gets acted on is where the bias converts into product decisions. A finding has to survive compression into a slide, and compression is a selection process. Someone chooses which observations earn a line and which become context.

That selection runs on the same asymmetry the physicists ran on. Evidence pointing where the team already leans goes into the deck as a clean statement. Evidence cutting against it needs a caveat, a qualifier, an "although we only saw this once," and caveated findings are what gets cut for time. The deck ends up honest sentence by sentence and wrong in aggregate.

This is the difference between a data point and an insight, and it is why the boundary matters for product insights. An insight claims to explain why, and an explanation is exactly the kind of object confirmation bias can manufacture out of thin material. "Users loved the simplified nav" is an explanation. What was observed is that four of six participants finished a task and two never mentioned the nav.

A practical discipline, unglamorous and effective: in the readout, record the count and the denominator above the interpretation. Four of six completed; two of six abandoned at step three; one required intervention. Then the reading, labeled as a reading. Stakeholders will argue with "4 of 6" on the merits. "Users loved it" gives them nothing to grip.

The tell: a study that couldn't have lost

Before running anything, ask what result would have made the team abandon the design.

If the honest answer is "nothing, really, we'd have tweaked the copy and shipped it," that study is a ceremony. It will consume two weeks and produce a slide that makes the decision harder to revisit later, because now it is validated.

The counter is a falsifiable prediction written down in advance, in the shape of a decision:

If fewer than 5 of 8 participants reach the confirmation screen unaided within four minutes, we don't ship this flow. We return to the step-three form and retest.

Two things happen once that sentence exists in writing before collection. The threshold gets argued about while arguing is still cheap, and afterward nobody can quietly move it. Hindsight is very good at making whichever number arrived feel like the number you were always looking for, which is why the dependent variable and its decision rule belong on paper before the first session rather than in the analysis meeting.

Countermeasures that cost something

The ones that work impose a real cost on somebody. Anything free is a ritual.

The strongest is to make disconfirmation somebody's assignment. One named person's job for the study is to build the case that the design fails. This works because it converts finding problems from a social risk into a deliverable, and the difference between that and a general instruction to stay objective is the whole intervention.

Around it, three habits that cost little to run and a lot to skip. Every claim in the synthesis carries the strongest counter-example anyone observed, or an explicit note that none was found; claims with neither are the ones to distrust. Notes get taken in two columns, what the participant did and what we think it means, and only the left column counts as evidence. Moderation goes to someone who didn't design the thing, because authorship is the strongest predictor of a leading prompt and intending to be neutral does not reliably suppress it.

Two more that operate before and after the sessions. Recruit at least two participants who should struggle: a screener producing only fluent users cannot generate the evidence that matters, which is the usability test equivalent of proposing 1–2–3. And before analysis, draft the slide that says the design failed. If it can't be written because no observation could support it, you learned that before spending a week on synthesis.

The quantitative version

Instrumented work inherits all of this and adds a mechanism of its own: a large enough search space always contains something that agrees with you. Check a running experiment daily and stop when it crosses significance in the direction you wanted, and noise gets converted into a result, because a continuously monitored test will eventually cross the line on its own. Instrument twenty events against one change and whichever event moved becomes the headline. Both are the same move that produced Millikan's slow climb, executed faster.

The prevention is identical in both cases: fix the primary measure, the sample size, and the stopping point before looking, then read the number once (A/B testing).

Post-hoc segments deserve their own warning. "It worked, for returning users on mobile" arrives feeling like a conclusion because the explanation and the data showed up together. It is a hypothesis, and the honest move is to label it as one and run it as the next study.

Where the bias meets its cousins

Confirmation bias rarely operates alone, and its neighbors matter because each one supplies belief early and cheaply for it to then protect.

Bias	What it does	Why it compounds confirmation bias
Dunning–Kruger	Low performers overrate their own ability	Self-reported "that was easy" becomes confirming evidence for a flow that failed
Halo effect	One strong trait colors judgment of unrelated ones	A beautiful interface reads as a usable one, and the study goes looking for support
Anchoring	The first number or idea sets the frame	The initial hypothesis becomes the thing every later observation gets tested against

NN/g's overview of confirmation bias in UX covers more of the family.

What to change on Monday

Three edits, in order of what they return:

1. Add a decision rule to the next study brief. One sentence, with a number and a consequence, written before recruiting starts.
2. Split the notes template into observation and interpretation. One change to one file, and it survives everyone forgetting the reason for it.
3. Name the disconfirmation owner for the next round, and put their three minutes on the readout agenda in writing.

None of these makes anyone less biased. Each one raises the cost of acting on the bias, which is the intervention that has historical evidence behind it: physics did not fix the electron charge by asking researchers to be more objective. It fixed it by making discrepant results something you had to report rather than explain away.

Frequently asked questions

What is confirmation bias in UX design?

The tendency to design, run, and read research in ways that support what the team already believes about the product: favoring supportive evidence, discounting contrary evidence, and never generating the observations that could have proven the design wrong. It shows up in screeners and task wording as much as in analysis.

What's a real example of confirmation bias in a UX study?

A team runs six usability sessions on a redesign it already favors. Two participants struggle, and both get explained away, one as "not the target user" and one as "having a bad day." Neither explanation is unreasonable on its own, and neither would have been offered about a participant who succeeded.

How is confirmation bias different from the Dunning–Kruger effect?

They act at different points in the pipeline. The Dunning–Kruger effect is people misjudging their own ability, which corrupts self-reported data at the source. Confirmation bias is how any evidence, self-reported or observed, gets filtered afterward. A study can take both hits: a participant over-reports how easy the task was, and the team keeps that quote because it agrees with them.

Can training or awareness eliminate confirmation bias?

No, and the research on this is older and more discouraging than most teams expect. Nickerson's review documents the effect in trained scientists working on questions where they had no personal stake in the answer, which rules out the comfortable reading that it's a motivation problem solvable by caring more about being right. It operates below the level of intention, so a workshop on cognitive bias reliably produces people who can name the bias and still commit it that afternoon. Awareness does buy one thing: it makes the structural fixes legible as fixes rather than as bureaucracy, which is usually what determines whether a team keeps them past the second sprint. The fixes themselves are pre-committed decision rules, a named disconfirmation role, observation separated from interpretation, and moderation by someone who didn't design the thing.

Does a bigger sample size protect against it?

No. Sample size fixes sampling error, and this is a selection-and-weighting problem that scales right along with the data.

Does confirmation bias affect quantitative research too?

Yes, usually as peeking at a running test and stopping at a favorable moment, or as running many metrics and reporting whichever moved. The standard defense is to pre-register the primary measure, the sample size, and the stopping rule.

Does a more diverse team fix it?

Partly, and only if disagreement is cheap to voice. Different backgrounds generate different priors, which is genuinely useful, but a team where contradicting the design lead carries a cost will converge anyway regardless of who is in the room.

Take it further

Judging an interface by what people observably do, rather than by how confident anyone in the room feels, is the discipline underneath the UX Clarity framework. A Full UX Audit is what it looks like when an outside party applies it to work you're too close to. For more on turning observations into decisions that hold, keep reading in product insights.

Sources: Feynman, 1974 — Cargo Cult Science · Wason, 1960 — On the Failure to Eliminate Hypotheses in a Conceptual Task · Nickerson, 1998 — Confirmation Bias: A Ubiquitous Phenomenon in Many Guises · NN/g — Confirmation Bias in UX · NN/g — Avoid Leading Questions.

Think a recent "validated" result might not have been able to lose? Apply for a Full UX Audit →