Recording what happens during a usability test is a crucial task as your notes are the basis of your later analysis.
I’ve been collecting a list of tips on taking notes based on my experience of years testing with users.
Ideally, two kinds of professionals should be involved in a user test: a moderator and a note taker.
In practice, due to budget constraints, many users tests are run with one facilitator who moderates the sessions, takes notes and writes the final report.
This makes note taking a quite difficult task and you may likely miss some important finding since you can’t focus your attention to the user and to your notes at the same time.
Whether you are in charge of everything or you are just taking notes, a well-structured template will indeed be one of your most important tools during a user test.
Use a template
Preparing a template for your notes is a task that some find time-consuming and boring.
However, a template requires only a little effort, it can save you plenty of time when writing the final report and it helps you to explain your findings in a more effective way.
For example, by adopting a template, I’ve managed to cut my reporting times by half.
A structured template has several advantages VS free-form writing:
- It guides you through the study, so you won’t forget to take notes on any important topic.
- You can easily retrieve a note on a specific aspect at any time after the study is over.
- It helps compare between participants.
- It makes it easier to calculate quant data.
Choose the right format for your notes.
I usually take notes in my laptop. I find it faster, I don’t have to decipher my ugly handwriting and it helps me maintain my notes tidy.
I’ve started with a word document but now I’m using an Google Drive spreadsheet which I believe it can be a good solution for most people.
It allows me to keep your notes well arranged and the autosave feature is definitely a plus that can save me hours of work.
There is not a single template or format that can work for everybody. My advice is to try different solutions and see which one you feel more comfortable with.
Be open to change or tweak it. Each usability study is different and a good note-taker template is often the result of incremental changes.
Build your template around the testing script.
Let’s take a step back.
When you decide to run a user test you do it because you have some critical questions regarding your product or service.
For example, are my customers able to use feature x? How many usability issues does my website have?
Whatever your critical questions are, your testing guide -which is what your participants will do during the study and the questions you will ask them- will be aimed to get answers to them.
For example, if you want to test whether your forms are easy to use, you will ask your participants to perform a task that requires filling in some forms.
Or, if you want to discover the users’ opinion regarding a certain feature, you will ask some questions about that feature, how easy was for them to use it, how it could you improve it, etc…
Pretty obvious, indeed.
And It is pretty obvious too that your notes should record all the information needed to provide the answers you are looking for.
However, I’ve seen many usability testing reports that were not delivering any useful insight partly because the notetaker failed to record all the relevant info.
When you are preparing your template, make sure you are covering all the key aspects you are investigating.
Build your template around the testing script and make sure it allows you to record any relevant data that might not be included in the guide but is related to the study goals.
Perform a detailed inspection of what your are testing.
Whether you were the one who prepared the testing script or you just received it from someone else, you should inspect the website/app you are testing prior to the test.
If you do it, you will probably find many aspects you believe will turn into an issue for some of your participants.
Since you are a UX expert and, hopefully, you ran many other user tests in the past, your predictions can be right and that means you will have to report those issues in your notes.
You should be ready to take notes about those aspects during the test by making room for them in your template.
Don't record more information than you need.
Years ago I ran a user test with 36 participants over three different locations and, because of this almost quantitative approach, I prepared a long excel template in which I collected usage data of many interface elements my participants could use during the test.
Did I use all of them for my report? No.
When I analyzed the results I finally found that only 1/3 of them provided valuable insights on the overall user experience and could be used to enrich my explanation of the issues I discovered.
Another 20% (Such as “How many times did they use the filters?”) was added to the report to provide a few quantitative data that were well welcomed by the client but had little to contribute to my analysis.
The remaining 50% was simply a waste of time: I could have reduced by half my effort when taking notes during the test.
So, what did I do wrong?
- I forgot that I was running a qualitative user test and that I should provide few, relevant, quantitative data such as success rates and the frequency of each problem.
- I thought that it might be useful for my client to have lots of usage data, while this is not the scope of a user test. My client could have gathered more precise information with his analytics tool.
- I collected lots of data just because I was able to do it, not because I really needed them.
When you are preparing your template, make sure you are not recording more information that the one you really need.
Once again, stick to the study goals and the technique you are using.
Reduce the amount of data to be recorded.
When I started testing with users, my template was structured by generic topics (such as “Forms”, “Search box”, “Error messages”) and, for each of them, I used to write a description of the problems encountered by each participant.
That was very inefficient for three reasons:
- I had to write a lot each time I discovered an issue on a topic, explaining what happened and why. This meant that I was not able to pay too much attention to the user and sometimes I had to check the videos from the sessions since I was not fast enough to write down everything.
- Many issues were recurrent, but I could describe the same issue in different ways. When I was analyzing the results I had to read all my notes over and over again to spot the problems.
- Since my descriptions did not follow any standard, it was hard to quantify how many times the same error occurred.
I soon realized that, since some issues could be predicted in advance or were likely to happen, I could rather record them as a yes/no answer to a series of questions.
For example, the topic “error messages” can become three questions:
- Does the user see the error message?
- Does the user understand the error message?
- Is the user able to recover from the error?
Plus, of course, a “Comments/More” field available to provide further details.
By doing so I was able to summarize what happened with error messages by using only a few words and I could see how often they occurred at a glance.
However, even if you do an extensive revision of the website you are testing, you should always foresee a space for any other issue that may arise during the sessions or for your comments.
Test your template
A note-taker template is a living document that can (and has to) be adapted to your discoveries while you are testing users.
You may probably find out a recurring issue that you were not considering or, on the contrary, discover that some of the problems you were expecting did not happen at all.
Updating a template can be difficult while you are running a test, with little time available between a user and a client that wants to debrief after each session.
That’s why I suggest to have your template ready for the pilot session and not to built it as a result of it.
You will have the opportunity to put it to the test and make most of the changes immediately.