Read our Free UX Assessment Self-Help Guide
A field guide to the most common usability testing myths and how to think about each one instead.
Created on
July 30, 2026
Usability testing has been one of the most reliable ways to avoid shipping the wrong product. It's also one of the most misunderstood practices in user experience design.
Most teams that skip usability testing don't think they're skipping it — they believe they simply don't need it. Meanwhile, teams that do invest in usability testing often misunderstand what it can and can't do. After years of working with stakeholders across different products and industries, we keep seeing the same misconceptions emerge. This article breaks them down.
"Our interface looks clear. We have a professional designer and data on product usage. Why do we need to spend more time and money on usability testing?"
Why it's wrong: Design expertise and analytics data don't replace real users. Analytics tell you what users do: where they drop off, what they click, where they stop. The why remains invisible. Designers apply best practices, which are generalizations and recommendations, not guarantees. And "the interface looks clear" is almost always said by the person who created it, not the person trying to use it. Users bring a completely different mental model, prior experience, and zero context about your design decisions. The gap between how you expect them to behave and how they actually behave is the usability problem.
What it costs: Without testing, you can end up spinning cycles optimizing the wrong things, A/B testing button colors while a broken error message kills conversions, or shipping designs that make perfect sense internally but confuse everyone externally.
How to address it: Use analytics to find where problems exist. Use usability testing to understand why. Used together, they give you the full picture.
"Five users aren't statistically significant. And even if that were true, results depend on who runs usability tests and how they interpret those results."
Why it's wrong: Yes, bias exists in qualitative usability research. Moderator bias, confirmation bias in analysis, unnatural participant behavior — these are real risks. And five participants is a small number by statistical standards. Both concerns come from the same place: applying quantitative thinking to a qualitative method. That's the mistake.
Qualitative usability testing is not a survey or A/B test. Statistical significance is irrelevant because you're not measuring frequency, you're observing behavior. Jakob Nielsen's research established that five users uncover approximately 85% of the most critical usability issues in a single round. This is about discovery efficiency, not statistical representation. It says nothing about how many users in your broader audience will encounter those issues. That's a separate question, and it needs a different method.
Modern usability testing methodologies don't eliminate bias, but they have well-established controls, neutral task phrasing, think-aloud protocol, behavioral observation over self-report, and structured analysis frameworks. It's not a bias-free process, but it's far more controlled than relying on gut feel or making decisions without any research at all.
What it costs: Waiting for a statistically perfect study delays testing by weeks, inflates budget, and often means no study happens at all.
How to address it: Run multiple rounds of five users instead of one round of 50. Iterate between rounds rather than waiting for a single large report. It won't tell you how many users are affected, but it will tell you exactly what to fix.
"Can you test whether users will pay €49 for this feature? We want to understand purchase intent."
Why it's wrong: Qualitative usability testing is designed to evaluate how people interact with an interface, not to measure willingness to pay or validate business strategy. Those questions require different methods, such as surveys, pricing research, diary studies, or concept testing.
What it costs: Money gets spent on a study that wasn't designed to answer the question. Results come back inconclusive or misleading enough to drive a real design decision in the wrong direction.
How to address it: Be explicit about what question you're trying to answer before choosing a method. Usability testing answers the questions: "Can users complete this task?" — "Where do they get confused?" — "What do they misunderstand?" — and everything else needs a different research approach.
"The product is almost ready to go live, so we can test the solution."
Why it's wrong: Testing after code has been written is like proofreading after a thousand printed copies. By the time a product reaches pre-launch testing, the interface is built, the logic is coded, and the team is deep in delivery mode. Every critical issue discovered becomes a negotiation about scope rather than a straightforward fix.
What it costs: Fixing a usability issue in a prototype is a design decision. Fixing the same issue after development is significantly more expensive, and the fix now has to compete with deadlines and sprint priorities. Most often, it doesn't get fixed properly, the issue ships, and becomes someone else's problem in the next cycle.
How to address it: Test early prototypes. A clickable wireframe of a checkout flow can reveal critical errors before a single line of code is written. The earlier the finding, the cheaper the fix.
"We just want to validate that the design is good before we present to leadership."
Why it's wrong: This is a dangerous misconception because it shapes everything: the brief, the task design, the questions, and how findings get interpreted. Usability testing is not a validation tool. It's a failure-finding tool. The entire point is to surface where real users get confused, stuck, or make errors, not to collect evidence that the team did good work. When teams go in expecting confirmation, they unconsciously design tasks that are too easy, avoid edge cases, and dismiss friction as "user error."
What it costs: The outcome is a study that protects the design instead of improving it. You get false confidence, polished results, and problems that only show up once real users arrive.
How to address it: Frame every study around risk, not reassurance. The core question isn't, "Did users succeed?" — the right question is, "Where might this design fail, and why?" That reframe changes everything from how tasks are written to how findings are presented.
"We want to make sure everyone understands it, so let's test it with anyone randomly."
Why it's wrong: "Everyone" is not a target audience. A design built for everyone satisfies no one, and a study works the same way. Testing across all user types at once, whether novice or expert, young or old, power user or occasional visitor, produces contradictory findings and insights too generalized to act on. Different user groups have distinct mental models, goals, and pain points.
What it costs: You end up with a report full of "some users found X easy, others didn't." It doesn't tell you who the interface is actually failing and why. Decisions get delayed waiting for more data, or are made based on averaged findings that don't represent any real segment.
How to address it: Define a primary user segment for each study. If you genuinely need to test multiple groups, run separate sessions with clearly defined recruiting criteria per group. Focused research produces findings you can act on.
"While we have users, let's test the onboarding, the dashboard, the settings page, and the new checkout."
Why it's wrong: A usability study with too many goals is a study with no goals. The most common reason to do this broad usability testing is to avoid a second round. But participants need time to orient and work through tasks genuinely, while sessions have a hard limit, typically around 60 minutes of core testing. When sessions run too long or cover too much, quality drops because participant focus degrades sharply, or tasks get rushed.
What it costs: A tired or rushed participant produces biased data and not enough depth to understand why something is wrong. Biased or surface-level insights lead to surface-level fixes, or incorrect ones.
How to address it: Limit each study to core tasks or research questions. If you have more to test, run a second round. Shorter, focused studies are faster to run, easier to analyze, and produce sharper recommendations.
"A participant said it was really easy and intuitive, so the design must be working."
Why it's wrong: What users say and what users do are often completely different things. This is one of the most well-documented phenomena in usability research. Users are polite. They want to be helpful. They don't want to make the designer feel bad. So when asked, "Was that clear?" — many will say yes, even after spending three minutes visibly confused, clicking the wrong button twice, and arriving at the right screen by accident. This is sometimes called the social desirability bias.
What it costs: The design gets treated as validated when it isn't. Issues get dismissed, wrong conclusions get drawn, and fixes get made based on what users said rather than what they actually did.
How to address it: Always prioritize observed behavior over stated opinion. Record sessions and track what actually happens, where users hesitate, fail, or take wrong turns. Verbal feedback is context, not conclusion.
"Only one out of six users had that problem, so it's probably not a real issue."
Why it's wrong: Applying statistical logic to qualitative usability testing produces false confidence. When one out of six participants fails on a task, that's not a 17% failure rate; it's a signal worth investigating. In a real user base of thousands, even a problem affecting a small percentage of users can still represent a significant volume of failures, lost conversions, or support tickets.
Completion rate also tells you less than it appears. "All six participants completed the task" doesn't mean the task was easy. It means all six completed it. They may have struggled, taken longer than expected, expressed frustration, or found a workaround. Completion rate without behavioral context is a shallow metric.
What it costs: Design decisions get made based on frequency rather than actual impact. Real problems get dismissed, wrong things get fixed, and development time gets spent in the wrong direction.
How to address it: Evaluate findings by severity and impact first, not frequency. Ask: If this issue went unfixed, what would it cost the user and the product? A single critical path failure is a high-priority finding regardless of how many participants encountered it.
"That participant was confused, but they're clearly not our typical user."
Why it's wrong: When a user fails to complete a task, the instinct is often to disqualify them somehow as having the wrong age, wrong tech literacy, or wrong background. It happens frequently, and it almost never holds up. If a participant was recruited against your target persona criteria and still failed, the design has a problem. The recruiting criteria exist precisely to ensure the right people are being tested. Using "wrong audience" as an explanation after the fact is a way of deciding the outcome before analyzing the data.
What it costs: Real issues get explained away. The product ships with known friction, and the surprise comes later, when real users struggle in exactly the same ways as the participant who was dismissed.
How to address it: Treat every failure as a signal. The impulse to disqualify a participant is worth examining. And before writing anyone off, watch the remaining sessions first.
Every misconception on this list comes down to misunderstanding why to use usability testing, when to use it, and how to use it. That misunderstanding leads to wasted budget, missed issues, and fixes that either don't happen or shouldn't have.
These aren't edge cases. If even one sounds familiar, it's worth addressing before the next study begins. Testing works when it's applied at the right time, for the right purpose, and done properly.
Sergey Pilkevich is a Strategic designer at The Norm, Coherent Solutions' UX practice, working across product design, usability research, and testing.
If you're planning a usability study or looking for product design support, get in touch.
