The first and most important thing when we do an online survey is to be clear about what we can get from it; what is the purpose of surveying at this time and with these users?
Mainly, surveys serve to:
- Discover. In the early stages of design or redesign, surveys can reveal the opinions and experiences of our users on specific topics. These results can inform further qualitative research to deepen the findings.
“Data can provide those larger-scale “Hey we notice this…” observations, and we then use qualitative methods to dive deeper and explore what might be going on and why.” Greg Marinelli – JP Morgan
- Sizing. When we do qualitative research and get a series of findings and insights, it can be interesting to do a survey to understand the statistical implication among our users. It’s a way to confirm those findings; as Nielsen Norman says, “you can’t trust the data if it doesn’t represent your users.”
- Evaluate. When we have launched a new design or redesign, surveys can be used to compare experiences and opinions concerning previous designs.
“Surveys can help you test your users’ specific feelings about a certain feature, and if you keep running them continuously, you can establish a baseline of user satisfaction. Knowing this baseline lets you see how changes directly impact your users.” Mithila Fox, senior UX researcher at Stack Overflow
A clear and achievable goal should be easier to design the survey. However, we must be careful not to fall into the following mistakes:
1. Avoid unnecessary questions
It seems obvious but you’d be surprised how many times “useless” questions are added to a survey. Sometimes they are demographic questions that you don’t really need, sometimes they are questions inherited from other phases of the research. These questions unnecessarily lengthen the survey, lowering the response rate and driving up the cost.
So, two tips:
Write on a post-it note the objective of the survey and don’t lose sight of it! Add only questions that will help you meet your objective.
Every time you write a new question, try to answer this one: How do the answers to this question help me in my objective of [survey objective]? And evaluate whether or not it is worthwhile or if there is perhaps another, more direct way to ask it.
2. Avoid surveys that are too long
The longer your survey is the lower your participation rate and the lower the quality of the answers (and the higher the economic cost). Therefore, prioritize and select only those questions that add value. Don’t be afraid to cut back. The good and the brief… you already know.
As Jakob Nielsen says: “Question overload is a natural consequence of working with a diverse group of marketing managers, all of whom want customer feedback on their specific issues. Please resist the temptation to collect all the information anyone could want. You’ll end up with no information (or misleading information).”
3. Avoid confusing questions
Do not use complex linguistic structures or overly specific vocabulary. We must speak the same language as the people who will respond to the survey. Be direct and clear about what you are asking. Do not leave room for possible interpretations or double meanings.
This will help the participants:
Don’t have to make so much effort to understand; they won’t get tired so much/so soon.
Be clear about what they have to do
Respond more accurately/correctly and in less time.
For this, it is very helpful to ask a colleague to read your questions without having the context of the project and to point out if there are any questions that might be confusing.
4. Open-ended questions...in moderation!
There are some occasions when we need to ask an open-ended question (i.e. when we want an explanation for an opinion or behavior). Especially in projects where we want to discover. But open-ended questions require more effort and time on the part of the participant and therefore, used in excess, can be detrimental to the results of our survey.
Think about whether you can get the same information with a closed-ended question.
5. Avoid dead-end questions
Normally, participants try to answer our questions correctly and in good faith. But what if you can’t find an answer that applies to you among the options? Chances are they will either answer the next best option or give up. And with that, we will see a lower quality in our data.
We must make a great effort to be 100% inclusive and not forget about all possible realities of our users.
Make sure that when you set up the answers, you offer all the necessary options. Consider whether a participant might:
- need to select more than one option from the list → multiple responses vs. single response
- not want to select anything → to add a N/A or I prefer not to say option
- not find their answer in the list → add an “other” option with an open field to specify
6. Do not use uniform questions
Try to use different types of questions (multiple choice, Likert’s, rankings, etc.) so that participants do not “fall asleep” and lose interest in your survey. Also, avoid using questions in which only one or two words vary.
If the questions are similar to each other, they are more likely to answer the same thing, and therefore, you will lose the quality of the answers.
7. Don't forget to test your survey
Sometimes we are in a hurry to launch surveys before we have reviewed them. And the reality is that it is very easy for a typo to slip out, for some logic not to work well or for an important answer to be missing.
For this reason, apart from doing an internal check, it is advisable to do a “soft launch” with 5-10% of your sample. It is a good opportunity to confirm that the questions are understood and that the answers make sense.
8. Not just any user will do
Asking the wrong users can only lead to useless, invalid data. So why waste time? Make sure from the beginning that you define well the segment of the population you are asking.
It is not the same to survey current users of an app as potential users who may never have heard of the app; it is not the same to ask people of all ages and socioeconomic segments as it is to ask only a portion of them.
“There’s a type of survey that asks everyone: it’s called census”.
Defining the participants is as important as the survey itself. The value of the answers and therefore the insights we extract depends on it. Don’t overlook it!
9. Avoid limiting access from certain devices
In many cases, it makes no difference whether participants access the survey via their cell phone, computer, or tablet. Offering as many alternatives to participate as possible helps to increase the response rate, lowering your costs. Therefore, whenever possible, try not to limit the device from which the survey can be completed.
It is true that in some cases, such as when you want to test certain designs/journeys, viewing the prototype on one device or another may have an impact on the way you respond. But as a general rule, before limiting access to mobile or desktop, ask yourself if it is really necessary.
10. Don't forget to set expectations
As we said in point 3, it is essential to be clear and concise in a survey. And this also applies to the survey presentation or invitation itself. At this point, participants need to understand:
- What the survey is about.
- What is expected of them?
- How long it will take
- What they will get in return
Everything else can be complementary. Encouraging or motivating our participants in the introduction can be interesting, but it cannot be the only thing we explain. We must ensure that we directly convey these four points as the core of the presentation or invitation. In this way, we avoid negative surprises among our participants and contribute to lowering the abandonment rate.
With these 10 tips and a clear objective, you should be able to create an online survey that is enjoyable and clear to participants and therefore returns quality data with which to draw valuable insights. Good luck!
References:
https://www.nngroup.com/articles/keep-online-surveys-short/
https://www.voxco.com/blog/writing-survey-questions
https://www.qualtrics.com/blog/10-tips-for-building-effective-surveys/
https://surveykiwi.com/blog/article/9-tips-to-create-the-perfect-online-survey-and-not-die-trying
