Qualitative usability testing: focus on problems, not success rates

mm

The terms qualitative and quantitative are often used incorrectly in user research.

Many researchers tend to call qualitative usability testing those studies with small samples, and consider quantitative those that employ larger ones.
However, the two terms are not referring to sample sizes, but to the type of data you are collecting.

Most research methods collect both quantitative (success rates, completion times or satisfaction scores) or qualitative (problems encountered, users’ feedback) data, but that does not mean that they should have the same importance in a usability report.

Qualitative (formative) user testing are aimed at discovering usability problems and are often carried out during the early stages of the development process.
On the contrary, quantitative (summative) user testing evaluate the usability of a system by comparing the results against usability metrics.

Success rates are indeed important since they provide an easy to understand estimation of how easy/difficult a certain task was, but focusing too much percentages may be misleading when you are performing a qualitative study:

  • A task may have a low success rate caused by a single and easy to solve issue, while another task may obtain the same success rate due to a number of different and complex problems.
  • Success rates in a user test do not necessarily correspond to the ones of a real usage of a webpage.
    Samples are generally too low and the margin of error can be quite large (See this tool to estimate the confidence interval for a completion rate).
    Moreover, when designing our piece of research, we may want to add some extra difficulties aiming to discover whether certain features present usability issues.
  • Stakeholders have better tools to quantify the real success rates of their users.
    It is not so important for them to know that 60% of the participants managed to book a flight during a formative user test.
    They need to know why the other 40% was not able to do it and a good usability test report should prioritize those reasons over success rates.

When you are building your qualitative research report, focus on errors discovered and what caused them, not percentages.
Sure you still have to provide success rates or completion times, but try to replace your stunning charts with a well reasoned analysis.

We are experts in User Research. We test with real users to help you create usable products and great user experiences.
Contact us for a free consultation about your project.

 

 

Photo by Helloquence on Unsplash

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Privacy Preference Center

Performance cookies

Estas cookies recopilan información sobre como los visitantes usan nuestra web.
Usamos esta información para crear informes y mejorar nuestra web.
Las cookies recopilan información de forma anónima sobre el número de visitantes a nuestra web, desde dónde han venido y qué páginas han visto.

_ga, _gid, _gat

Functionality

Utilizamos estas cookies para conocer el idioma de tu navegador y poderte enseñar una web en tu idioma.

_icl_visitor_lang_js

Strictly necessary cookies

Esta cookie sirve para que los usuarios registrados puedan iniciar sesión en la web.

PHPSESSID