Writing Closed Questions in Research: Getting Answers You Can Count

A team ships a survey with one question on it: Are you satisfied with the new dashboard? Yes / No. Sixty-one percent say yes. The number goes in a deck, the deck goes to a review, and the redesign is declared a success.

Nobody can say what the 39% wanted instead, whether "satisfied" meant the dashboard or the week they'd had, or how many people picked yes because no felt like a complaint. The question produced a number with no information in it.

That's the specific risk of a closed question. It always returns something countable, whether or not it measured anything.

The trade you make when you close a question

An open question hands the respondent a blank field and asks them to supply both the answer and the frame. A closed question supplies the frame and asks them only to pick. Everything good and everything bad about closed questions in research follows from that one move.

What you buy: answers that compare. Two hundred people picking from the same seven options produce a distribution you can track release over release, split by segment, and put on a chart without a coding pass. Response rates go up, because tapping an option costs seconds and typing a sentence costs minutes.

What you pay: the answer set is now your hypothesis, not their reality. If the option they'd have written isn't on the list, they don't tell you — they pick the nearest thing, and you never find out that they compromised. Open text at least fails loudly, with a blank field or an off-topic reply. A badly framed closed question fails silently, and silent failure is the expensive kind.

So the drafting job starts before the phrasing does. You are deciding what the complete set of possible answers is, and a set that's missing one is not something careful wording can rescue.

The forms, and what each one can carry

"Closed" covers a family of formats, and picking the wrong member is a common way to lose data that was available for free.

Dichotomous (yes/no). One bit per respondent. Use it for genuine binaries — did you complete the purchase? — and nothing else.

Single-select multiple choice. One answer from a mutually exclusive list. The list has to be exhaustive as well as exclusive, which is why Other (please specify) exists and why the contents of that box are the first thing worth reading.

Multi-select (checkbox). Cheap to write, and the one that most reliably produces a chart nobody can interpret. Ask which of these did you use? across nine features and you get a stacked bar summing to 340%, at which point the only honest reading is per-option: 61% checked exports, 12% checked the API, and those two numbers have no relationship to each other. Worse, an unchecked box is ambiguous by construction — it might mean "I don't use that," or "I stopped reading at item four," or "I wasn't sure if the CSV thing counts." Multi-select can't tell them apart, so a low count is never evidence of low usage. Reach for it when you want a rough presence list. Don't reach for it when someone's going to compute a share.

Rating scale. A numeric or labeled range — 1–5, 1–7, 0–10 — and comparable only if the endpoints and the number of points stay identical across questions and across quarters (NN/g on rating scales).

Likert items. A statement plus an agreement scale: a specific kind of rating scale with its own conventions and its own failure modes. Worth treating as its own format rather than a synonym for "any 1–5 question."

Ranking. Collapses past seven items. Below that, real preference data.

Numeric entry. Technically open, functionally closed. Ask for a count instead of a frequency label wherever you can — 3 is unambiguous where often is not.

A quick heuristic: if you can't say which of these formats a question is before you write the options, you haven't decided what you're measuring yet. That decision is the same one you'd make when choosing a dependent variable for any study.

Closed ended questions examples, before and after

Most broken questions are recognizable on sight once you know the shapes.

Double-barreled. How satisfied are you with the speed and accuracy of search? Someone who finds it fast and wrong has no honest answer. Split it into two questions and accept the extra row.

Leading. How helpful was our improved onboarding? The premise is already in the sentence, and agreement is the path of least resistance (NN/g on leading questions). A leading question is any question whose wording makes one answer easier to give than the others. Rewrite to How would you rate the onboarding? (Very unhelpful → Very helpful).

Missing option. Which plan are you on? Free / Pro / Enterprise, asked of an audience that also contains trial users and lapsed accounts. Everyone outside the list picks something wrong, invisibly.

Overlapping ranges. 0–1 year / 1–3 years / 3+ years. Someone at exactly one year flips a coin. Make the ranges touch without overlapping.

Unbalanced scale. Excellent / Very good / Good / Fair / Poor has four positive options and one negative. The mean drifts up by roughly half a point, which nobody notices, because half a point is also what a real improvement looks like. Balance the poles: equal counts either side of a neutral midpoint.

Assumed knowledge. Do you use the API rate-limit headers? Anyone who doesn't know what those are will answer anyway, usually no, and you'll record ignorance as a preference. When knowledge is in doubt, screen for it in a question before you ask about behavior.

The response options are the measurement

Two questions with identical wording and different option sets are different questions, and the option set is where the sneaky errors live.

Use an odd number of points. A midpoint costs you a little discrimination and buys honest answers from people who genuinely have no view; forcing a lean manufactures opinion that you then report as signal. Go even only when you're measuring a choice the respondent has already made in real life, where indifference isn't a real state. Whichever you pick, don't switch mid-questionnaire — mixed parities make the items incomparable.

Label every point if you can, or at minimum both ends. A bare 1–7 with no anchors means whatever each respondent decides it means, and they don't all decide the same thing.

Keep Don't know separate from the middle of the scale. They aren't the same state, and collapsing them turns absent knowledge into moderate opinion.

Randomize option order where the list is long and unordered — position bias is real and it favors the top. Don't randomize anything that has an inherent order, which includes every scale.

And keep the scale itself fixed across time. A 1–5 in Q1 and a 1–7 in Q2 cannot be compared, no matter how the numbers get normalized afterward. If a scale has to change, say so on the chart.

Where closed and open questions belong together

Order decides this. Open first, closed second, and the reason is boring: you can't write options for vocabulary you haven't heard yet.

Open questions come first, in the exploratory phase, when you don't yet know the answer set. This is where you learn the words respondents actually use — the phrases they reach for unprompted, which then become your options. Interviews and focus groups are for exactly this. They decide nothing. They tell you what the alternatives are.

Closed questions come second, once the space is mapped, to measure how the population distributes across it. That's the whole reason the sequence matters — writing the options before you've heard how people talk means writing your own assumptions into the response set and then measuring them.

One open box at the end of a closed questionnaire earns its place as an escape valve. Anything we didn't ask about? is where you find the option you should have listed. Read those before you read the charts. It's the same inductive-then-deductive arc that governs any research program — generate the categories, then test how they distribute.

A closed questionnaire example, end to end

Say you want to know whether a new export feature is working. Five questions, in order:

1. Screener. Have you exported data in the last 30 days? Yes / No. No routes out.
2. Behavior, counted. How many times? Numeric entry, not a frequency label.
3. Outcome. Did your most recent export produce the file you expected? Yes / No / I'm not sure. The third option is doing real work — it separates failure from uncertainty.
4. Intensity. How would you rate the export experience? 1–5, both ends labeled, balanced.
5. Escape valve. What would have made it better? Open.

Note what isn't there: no question about whether the feature is "valuable," no rating of a thing the respondent hasn't used, no double-barrel. Every closed item has an answer set that covers the real population, and the one open field exists to catch what the four closed ones assumed away. For pre-written items with their scales already chosen, the survey question bank is a faster starting point than a blank page.

Pretest with eight people reading aloud

Nothing on this page catches as many errors as watching eight people answer the draft while narrating what they're thinking.

Sit with each one, ask them to read each question out loud and say what they think it's asking before they answer, then ask why they picked what they picked. You're listening for four things: a question they had to read twice, an option they hovered over and rejected as "close enough," a term they defined differently than you do, and any moment they asked a clarifying question — because your respondents won't be able to.

Eight is enough — the same curve Nielsen ran on usability tests holds here. The first five surface most of it and three more catch the stragglers. The fixes are almost always the same three: split a double-barrel, add a missing option, define a term.

Closed questions in research get judged on the wording, but they fail on the answer set. A pretest is the only cheap way to find out whether yours is complete before two hundred people quietly round themselves to the nearest available option. More on study design across the rest of the research methods hub.

FAQ

What are closed questions in research?

Questions where the respondent selects from a fixed set of answers the researcher wrote in advance — yes/no, multiple choice, checkboxes, rating scales, Likert items, rankings. Brevity has nothing to do with it. The answer set is fixed before the question is asked, and that's what makes the responses countable and comparable across people.

What is an example of a close ended questionnaire?

A short post-purchase survey: Did you complete your order today? (Yes/No) · How many items did you buy? (numeric) · How would you rate the checkout? (1–5, both ends labeled) · Which payment method did you use? (list, plus Other). Every item has a predefined answer set, so the whole thing tabulates without a coding pass.

When should I use closed rather than open questions?

Use closed when you already know the plausible answers and need to know how a population distributes across them. Use open when you don't know the answer set yet, or when the interesting part is the reasoning rather than the count. Running closed questions in research before any exploratory work means measuring your own assumptions.

How many points should a rating scale have?

Five or seven for agreement and satisfaction; 0–10 when you need finer discrimination or you're matching an external benchmark. Below five you lose sensitivity to small changes, above eleven respondents stop distinguishing adjacent points. The specific number matters far less than keeping it identical across every wave you intend to compare.

What makes a question leading?

Wording that makes one answer easier or more socially acceptable to give — a positive adjective in the stem (our improved checkout), an unbalanced set of options, or a premise the respondent hasn't agreed to (how much time did the new layout save you?). The test: read the question and try to guess which answer the author is hoping for. If you can, rewrite it.

Should I include a neutral or "don't know" option?

Treat them as two separate decisions. A neutral midpoint is a legitimate position and belongs on the scale when genuine indifference is plausible. Don't know is not a position at all — it's absent knowledge, it belongs off the scale as its own choice, and merging the two records ignorance as moderate opinion.