see discussion overview

Related

What are the limitations of usability testing?

Categorised in:



The low sample size makes it untrustworthy for drawing conclusions about subjective user preferences

Usability testing is often confused with market research, but the two are very different.
Usability problems can easily be detected from a sample of about five test participants. Very different kinds of people will tend to encounter the same usability problems. More than seven participants is just a waste of time.
Market research issues, things like design-preferences and whether they would actually want to use the product, are completely different. These vary wildly from person-to-person and different personalities/backgrounds. You need big sample sizes to judge them accurately.

People behave differently when they know they're being observed



It's difficult to simulate the appropriate context

Many users perform tasks while performing several unrelated tasks on the same computer at the same time. This may have a strong influence on how they interract with a given product, but it is difficult to simulate.
Similarly, users may normally interact with a product at particular time of the day when they are subject to certain distractions or a certain state-of-mind. These may also have a significant effect on the task but are difficult to simulate.

Test participants are assigned motivations; this will cause different behavior to genuine motivations

For example, a participant will be much more careful about a purchase they are genuinely making than about a purchase they are pretending to make for the sake of a usability test.

You can't test long-term experiences

Usability testing typically last 45 minutes, but many web interactions take place over several days, especially when they involve interacting with an online community

more