Learn tricks of the trade for filtering the right participants into (and out of) your user research study
Researchers seeking to design screener surveys to expertly capture high value participants
You’ve determined who would fit into your ideal audience, and now you want to go out and find them. Enter the screener survey. Start with what must be true about your participants, then build a set of questions to identify the people who meet your criteria. If you take away one piece of advice it's this: don’t over-screen.
A screening survey is quite simply one of the most useful tools at a researcher’s disposal.
User research takes time and money, which is why you need a screener survey to ensure your research is built on insights from ideal participants. A screener survey is like a sieve that captures the people who hit all your ‘must have’ criteria and filters out the ones who don’t fit the bill.
Once you’ve determined who you want to include in your study, it’s time to write a screener survey that will allow you to go out and find those people.
Tip: If you haven’t already got it down, review the previous chapter on finding the right participants for your study before moving on.
If you want the right participants, you’ve got to design smart screening questions.
That’s not always as straightforward as you might think. You have to ask questions in a somewhat roundabout way to avoid leading people to certain responses, but also in a clear way to make sure you’re universally understood.
The devil is in the details, but luckily there are some pro strategies that anyone can learn and put to use right away.
These guiding principles will help get you there.
Once you know the characteristics of your target participants, and you’ve broken that down into specific criteria for how you’ll identify the people who qualify, it’s time to write questions that specifically address your must-have criteria.
Note: recall the tips and techniques for finding the right audience and identifying your criteria in the previous chapter.
If you’re developing organic food delivery app and you want to discover how and why your target audience orders food you might want to include people who use apps to order food and prefer organic food and exclude people who aren’t willing to pay more for premium food products.
Be as precise as possible with the criteria you choose to include.
Imprecise criteria: People who use food delivery apps regularly
Precise criteria: People who have used food delivery apps 2+ times per month
Then craft the perfect question to zero in on on those people who meet your criteria.
Demographics and geographics are the low hanging fruit of screener surveys, but these characteristics have their limitations.
Where people live is important if you’re doing an in-person study, or if your app will only serve certain locations, but if you don’t have a clear reason to target on geography, don’t. Our assumptions about demographics are prone to bias, and don’t waste valuable screener questions on criteria that aren’t absolutely essential.
On the other hand, if you want to test for accessibility with a mix of men and women, age ranges and educational backgrounds, adding demographic criteria will allow you to target a diverse audience.
Not every question on your screener has to result in an automatic in or out, but can be used to filter for a variety of participants as a final step. Accept anyone who could be a fit based on any given question.
Tip: Some recruiting services (User Interviews included) will automatically provide basic demographic, geographic and technographic information for you, so you don’t need to include it in your screener survey. Win for you and your participants!
Screening for psychographics and behaviors lets you group people based on how they live, what they value and how they relate to your product or category. That’s the juicy stuff!
Jumping back to our food delivery app example, you might be able to get information about the advantages and disadvantages of your competitors’ products by interviewing people who regularly use them. Or you could tap into motivations for ordering organic food over alternatives by targeting people who identify that as a preference. These are the kind of insights that you can use to inform your product and marketing strategy.
They’re also the kind of insights that can only come from targeting people who have the requisite experience.
Rule of thumb: Worry less about how people are categorized on a census and more about how they think, feel and behave.
Ask questions that will easily weed people out first.
One way to do this is to take your essential criteria and rank them in order of priority and interdependency.
If you’re doing an in-person study, ask about location right away. Location here is a must and must-have criteria go first.
Before diving into questions about how people use apps on their smartphones, find out if they use a smartphone at all. Start higher level and get more niche.
Then move on to the questions that tap into specific behaviours, interests and preferences.
If you’re not working with a recruiting service that gathers demographics for you, ask any demographic questions that you need to ensure you get a a range of different participants (age, gender, income) toward the end.
You know how people have a tendency to add a ‘right?’ at the end of every sentence so that you’ll agree, right?
Don’t do that. Leading questions will influence people to answer in a certain way.
A survey is not the place to try to validate your assumptions. You’ll end up with skewed results or the wrong kind of participants.
Double check your questions for any hint of a leading tone. Here’s an example:
Another useful trick to avoid leading questions is to provide unrelated options as answers.
Some participants may look for clues as to the “right” answer and will respond in the way that they believe will let them participate in the study and receive the incentive.
Your screener survey should be carefully written to avoid hinting at the answers you’ll accept or reject.
Be aware that binary questions (i.e. yes/no, true/false) tend to look like clues. Whenever possible, replace these questions with multiple choice options or provide a scale for degree of agreement with a given question.
Here’s an example of a binary question:
And here’s a better way of asking the same question:
Exception: In cases where a black and white answer is required, for example when asking if a person is willing or able to participate under the conditions of you study, a binary question will be your best bet.
If you create multiple choice responses, don’t assume that you’ve presented the user with every possible option. Even the best survey designers have their limitations.
Include a ‘none of the above,’ ‘I don’t know,’ or ‘other’ option to account for any outliers. Otherwise you could end up with someone in your survey who doesn’t belong there because they were forced to choose an answer that didn’t apply to them. Likewise, you might screen good participants out because they didn’t quite fit the answers you provided.
The language you use is important. Avoid double negatives, keep it short and sweet, skip the fancy prose and leave out industry jargon (unless knowledge of it is a requirement for participation).
The more clear and specific you are, the less likely participants will be to get confused and answer a completely different question than you intended to ask. Leave no room for misinterpretation!
Being clear in your responses is just as important as being clear with your questions.
Have you ever taken a survey where, on a certain question, you found yourself forced to choose between more than one answer that applied to you?
To avoid putting your audience in that position, make sure your answers have clear borders without any overlap.
When asking for numerical values (age, size, frequency etc.), make sure your values are mutually exclusive:
For less definitive answers or for answers that can’t be made mutually exclusive, ask for the answer that is most true or give your participants the option to choose all that apply, rather than just one.
The whole point of using screener surveys is to get more value for your time and money on a per-participant basis. Sometimes that means excluding certain people who otherwise perfectly fit your ideal audience profile.
If you’re looking for people who will show up to your study willing and able to share freely and communicate clearly about their experience, you have to filter out people who are very shy or unable to articulate themselves.
Adding an open ended question to your screener survey can give you clues as to how much insight a participant will provide in your actual study. One word responses, illogical rambling or cagey answers can all indicate that a participant will provide a low return on investment.
There are certain questions that are useful almost across the board. Here are a few example to get you started.
Ask employment questions when you want to screen for people with a certain level of familiarity with a particular industry, or exclude those who work for competitors.
If you need to test with novices, experienced users, or some combination of each, ask about familiarity with a given product.
Just because someone meets your screening criteria doesn’t mean they’re actually going to be willing to participate. Ask participants directly if they’re willing and able to answer personal questions.
Once you know who you want to target, you’ve figured out the right carrot to dangle and you’ve written a screener survey to capture your desired participants, there’s one more important step. Actually finding recruits.
You have two choices: do-it-yourself or use a service to do it for you. We’ll look at both.
If you have a lot of time to dedicate to finding participants and your budget is limited, you may need to take the whole recruiting process into your own hands. Additionally, if you’re targeting your existing users, you’re in a good position to recruit them on your own.
You can look for prospective customers or non-users in the following ways:
For existing customers, be sure to emphasize how their input will improve the product (a clear benefit for them). You can recruit your own users in the following ways:
In your communications, be clear about incentives and mention any requirements for participation including location, NDAs, and technical/network specifications.
The key disadvantage of the DIY approach of course is that it can be time-consuming and challenging, particularly if the audience you’re after requires targeting by occupation or other niches.
Also keep in mind that you take full responsibility for abiding legal and ethical requirements for recruitment, consent, payment and communication with your audience.
If you have a bit more wiggle room with your budget, using a service to find recruits who match your criteria can save you time and effort. It can also make it easier to run a blind study by completely hiding the identities of your company, your product, and yourself.
User Interviews specializes in connecting researchers with participants. We have a panel of over 75,000 general population and occupation specific participants for both in-person and remote studies. The process typically take 3-5 days and we charge a low fee of $20-$60 on a per-recruit basis. That covers screening people based on demographics we’ve already collected plus your own screening questions, as well as scheduling, sending confirmations and reminders and distributing incentives. Check us out if you haven’t already.