When designing a qualitative study with users, a question that often arises is: “how many participants are needed?” Often, the most common answer heard is “5 users are enough.” However, the answer is not as obvious as it seems, and the question itself is not that simple.
In this article, we will explore in-depth the question of how many participants are needed in a qualitative study, reviewing the most common theories, and describing how researchers approach it in our day-to-day work.
Nielsen and the 5 users rule.
The “5 users rule” is one of the best-known, and also controversial, rules in user research. This rule, popularized by Jakob Nielsen in his famous 2000 article, is based on the observation that most problems are detected during the first sessions of a user study, and that as more participants are added, the number of new problems decreases progressively.
Through his user testing, Nielsen was able to calculate the exact impact of including more participants in a study. His conclusions are reflected in a famous graph that shows two axes: the vertical axis represents the percentage of problems present in an interface (from 0% to 100%), while the horizontal axis represents the number of research participants.
As you can see in the graph, there is a point, around 85% of the problems, where the line stops growing significantly and begins to flatten out. This point corresponds to 5 participants. According to Nielsen, after 5 people, finding new problems that have not already arisen with previous participants would cost us more and more.
In summary, if 5 users can find 85% of the problems, adding another 5 would only allow us to find the next 10%. Therefore, doubling our effort would have a very small impact on the research findings and may not be worth it in most cases.
Although the 5 users rule is mainly applied to usability problems, it is important to highlight that it can be applied to any type of findings in a research project.
5 total or per profile?
The decision of whether the 5 users rule is sufficient or not depends on the profile of the participants to be recruited in the research. Do we want to study a homogeneous group of people or different groups with specific characteristics? In the second case, if we expect each group to behave differently, a sample of 5 users is not enough.
For example, in a usability test of a website involving both customers and non-customers, we should recruit at least 5 from each group, especially if they perform different tasks.
In an interview about travel habits with couples, families with children, and business travelers, the opinions and needs of each group are likely to be different. In this case, we should also work with a larger sample to obtain a more complete and accurate understanding of each profile.
The limitations of the 5 users rule
A myth based on little data?
The 5 users rule is not without controversy. In a long Twitter thread, Jared Spool, another reference in User Experience, criticizes the assertion that with 5 users we can find 85% of the problems in an interface.
In summary, Jared Spool criticizes that the idea that 5-8 users are sufficient to obtain meaningful results is based on data from very few studies that cannot be extrapolated to a wider context. Spool asserts that this rule is a myth and is not suitable for research in the current context where products have millions of users in different cultures and do thousands of different things.
He also argues that this rule was based on cost limitations from the time it was created. At that time, usability testing was expensive and required expensive facilities, specialized personnel, and recruiting participants was very expensive. Optimization of the number of participants was necessary to reduce overall costs. However, today, user research is much more economical, and test prototypes are cheaper. Therefore, there is no need to worry so much about optimizing the number of users per iteration.
With 5 users, we only find what is easy to find
One of the most well-known contributions to the 5 users rule comes from Jeff Sauro, a UX consultant specializing in applying statistics to research.
In his 2010 article, Sauro analyzed Nielsen’s rule from a statistical point of view and added an important nuance: if it is true that with 5 users we can find 85% of the problems in an interface, this only applies to those problems that occur more frequently. To be exact, those that occur more than 31% of the time.
To understand what the probability of a problem occurring means, let’s imagine that our shopping cart has a problem that affects 31% of users who try to complete their purchase. If we test our shopping cart with 5 participants, we’ll have an 85% chance of detecting the problem during our study. However, if the problem is less frequent and only affects 10% of purchase attempts, we would need to involve more participants, precisely 18, to have the same chance of finding it.
While researching with 5 users and finding 85% of interface problems may be sufficient for many products, there are situations where solving a problem or detecting a need that affects only a few users can have a significant economic impact, especially when the product is used by millions of people. In this regard, it’s also important to weigh how much we value finding these “small” discoveries and what the acceptable risk is of not finding them.
In conclusion, how we do it in our daily practice.
In our daily practice at TeaCup, we carry out most of our qualitative studies with a number of participants ranging from 8 to 12, regardless of who makes the decision: whether it’s us or our clients (other agencies or large companies with UX research teams).
The range of 8 to 12 users allows us to accommodate most of the needs (and budget limitations) of our clients.
However, there are exceptions worth highlighting:
- When we conduct iterative studies, the most common practice is still to involve 5 participants, as this allows us to optimize efforts and results. In this sense, Nielsen’s recommendation still stands: if we have resources to involve 10 participants, it’s better to test twice with 5 than once with 10.
- Another exception occurs when we want to involve different profiles, in which case we can increase the number of participants to between 5 and 8 per profile if we want to obtain consistent data without increasing costs too much.
- In the case of international studies repeated in different countries, the sample of 8-12 participants is applied to each of the investigated markets.
- The number of participants may increase when we are interested in obtaining as much data as possible and we need to delve deeper into our findings. For example, if we’re using very exploratory techniques, such as User Diaries, it’s common to involve between 20 and 40 users to investigate their specific experiences more extensively before delving deeper by interviewing 8-12 of them.
It’s important to note that all of the above applies to qualitative studies. In the case of quantitative studies, the ideal number of participants is higher, as our goal is to obtain reliable metrics. For example, we recommend involving at least 30 people for Card Sorting or Eye Tracking studies.
