Run a heuristic evaluation

Outcome

One merged list of usability problems in the flows you scoped. Each finding names the screen, what goes wrong, the heuristic it breaks and a severity from 0 to 4. The list is sorted by severity, duplicates are collapsed, and every finding a single reviewer saw is still on it.

When to use it

• You need a list of likely problems in days, before or instead of testing with users.
• A redesign or a release is about to ship and nobody has looked at it with fresh eyes.
• You want to clear the obvious problems first, so a later usability test spends its sessions on the ones only users can show you.

When not to use it

• The question is whether real users can complete a task, or why they abandon it. Reviewers predict problems; a usability test observes them.
• Only one person is available and the result will be presented as an evaluation. One reviewer finds roughly a third of the problems; call it an expert review and say so.
• The product is aimed at specialists whose domain the reviewers do not know, such as clinical or trading software. The reviewers will flag what is unfamiliar to them rather than what is wrong.

Before you start

• The scope. Two to four flows, each written as a goal, such as a new customer adds a payment method. "The whole app" is not a scope.
• Reviewers. Three to five people who did not design the flows. Usability experience helps; knowledge of the domain helps more than seniority.
• The heuristics. Everyone works from the same ten, starting with visibility of system status. Agree in advance that a problem which fits none of them is still a finding.
• A capture sheet. One row per finding: flow, screen, what happens, heuristic, a screenshot. Each reviewer gets their own copy.
• Time. About two hours per reviewer, then two hours together to merge.

Steps

Step 1: Write the flows you will evaluate

List the flows as goals with a starting point, such as from the pricing page, start a trial on the team plan. Note who the user is for each one and what they know when they arrive.

You should now have: two to four written flows, each with a user, a starting point and a goal.

Step 2: Brief the reviewers and send them off alone

Give every reviewer the flows, the heuristics and an empty capture sheet. Ask them not to discuss the product, share notes or read each other's sheets until the merge. The first opinion said aloud anchors everyone else, and the reason to use several reviewers is that they notice different things.

You should now have: three to five reviewers, each holding the same brief and working independently.

Step 3: Walk each flow once to learn how the product works

Each reviewer goes through every flow end to end without recording problems, except anything that blocks them completely. The aim is to learn the product's vocabulary and model so the second pass can tell "I did not expect that" from "that is inconsistent".

You should now have: reviewers who have completed every flow once and know what the product is trying to do.

Step 4: Inspect each screen against the heuristics

On the second pass, each reviewer goes screen by screen and records every problem as a row: where it is, what happens, what the user would need instead, and the heuristic it breaks. One problem per row, with a screenshot. Problems that fit no heuristic go in with the label other.

You should now have: one capture sheet per reviewer, one problem per row, each with a location and a heuristic.

Step 5: Merge the lists and collapse duplicates

Bring the sheets together. Group rows that describe the same problem in the same place, keep the clearest description and note how many reviewers found it. Keep every finding only one person saw; singletons are the normal case, not noise.

You should now have: one merged list without duplicates, each row showing how many reviewers found it.

Step 6: Rate severity on what the problem costs the user

Each reviewer rates every merged finding from 0 (not a problem) to 4 (stops the task), without seeing the others' ratings. Base the rating on how often the problem occurs, whether users can get past it, and whether it happens every time. Use the average.

You should now have: a merged list where every finding has an averaged severity from 0 to 4.

Step 7: Write each finding as something a team can fix

Sort by severity. Rewrite each row as one sentence on what happens, one on why it matters to the user, and a suggested direction for the fix. Hand the list to the team that owns the flows, with the flows and the reviewers' names at the top.

You should now have: a severity-sorted list of findings, each with a location, the problem, its cost and a direction, handed to the owning team.

Worked example

A team is about to launch a self-serve upgrade from a free to a paid plan.

1. Flows: from the usage-limit banner, upgrade to the paid plan; from billing settings, change the card; from the plan page, downgrade back to free.
2. Four reviewers get the brief: a designer from another team, a support lead, a front-end engineer and a researcher.
3. On the first pass, two reviewers get stuck on the downgrade flow because the button is under Danger zone. They note it and carry on.
4. The second pass produces 23, 17, 14 and 19 rows.
5. The merge leaves 31 distinct findings. Only four were seen by all four reviewers; 15 were seen by one.
6. Highest severity (3.75): after paying, the banner still says the account is at its limit until the page is reloaded, a failure of system status. Lowest (0.5): the plan names use two different capitalisations.
7. The team fixes the four findings rated 3 and above before launch and files the rest. The usability test two weeks later starts from the fixed build.

Checklist

• The flows are written as goals with a user and a starting point.
• Three to five reviewers took part, and none of them designed the flows.
• Every reviewer worked alone until the merge.
• Each reviewer walked the flows once before recording problems.
• Every finding names a screen and either a heuristic or other.
• Duplicates are collapsed and single-reviewer findings are kept.
• Severity was rated independently and averaged.
• Each finding states the problem, its cost to the user and a direction for the fix.

Common mistakes

• Reviewing as a group. A shared document or a kickoff discussion turns several reviewers into one opinion, and you lose the reason to have more than one.
• Scoring every heuristic on every screen. A grid of scores looks thorough and tells nobody what to fix. Record problems, not ratings.
• Rating severity by how annoying it felt. Severity is what the problem costs users: how often, how bad and whether they can get round it.
• Treating the result as user evidence. Reviewers predict problems. Some will not bother real users, and users will hit problems no reviewer saw.
• Dropping findings that fit no heuristic. Accessibility, performance and content problems are real even when they map badly onto the ten.

Related

• UX Snapshot: an automated first pass over one page, with a score and top findings, before the reviewers start.
• Visibility of system status: the first of the ten heuristics.
• Consistency and standards: the heuristic behind most findings in redesigned products.
• Error prevention: the heuristic for problems the design could have stopped.
• Write a usability test script: test the problems that only users can show you.
• Heuristic evaluation course: the method in lessons with knowledge checks.
• Heuristic Evaluation: A Practical Walkthrough: why independent passes and severity matter.