Best Practices

How to Write Screener Surveys to Capture the Right Participants

Save time and money by creating a screener survey that gets you the perfect study participants

Lauren Leonardi
/
November 2, 2017

What is a screener survey and when do you need one?

Surveys and testing take time and money, which is why it makes sense to design a screener survey right from the start that captures the users you want. Maybe you want high-octane users who really get the web, your product, your vertical. Maybe you want beginner level peeps who are just beginning to dabble in whatever it is you offer. Maybe you want a broad swath because you’re testing something that you want to have broad appeal.

Whatever the desire, know your intended audience for your study. Imagine your absolute ideal participant. Know the “must haves,” and the things that must be true about them. Then build a screener survey to capture those people and filter out the rest.

Capturing the right audience with smart screener questions.

Screening questions get you the participants you’re looking for. They’re a few questions designed to weed out the folks who just aren’t your intended audience and capture the ones who are. A super simple example is filtering out men if your intended audience is women. But there’s way more to screener surveys than demographics. Keep an open mind about who these perfect candidates might be.

Here are some guidelines that will help get you there.

Guiding principles of building a smart screener survey 

1. Get the big, easy questions out of the way early.

Identify unsuitable candidates early. Think of the process as a funnel. You’re refining your participants, and refining them further. Or, think of it like weeding a grown-over garden. The biggest, tallest, most obvious weeds come out first, simply because they’re the easiest to grab. Then comes the hands-and-knees part where you pull out the littler ones.

If you know that you absolutely want experienced gamers to be part of your UX testing, ask some specific questions about how users interact with games right out of the gate. Users who answer appropriately move on, users who don’t get dismissed (and maybe saved for future studies for newbie gamers.)

introductory screener questions

2. Screen for behaviours over demographics

Demographics are the low hanging fruit of screener surveys, and it might work to include a few demographics questions either at the beginning or at the end of your survey to really cut the chaff. But don’t let age, gender, and location questions be the end-all be-all.

In many cases, a person’s gender or how much they earn per year won’t determine how they interact with a product. You’ll want to identify which demographics, if any, really define your audience conclusively. Don’t waste valuable screener questions on things that don't definitely map to who you need to talk to. You know who your audience is and isn’t. Just remember not to discount folks who could provide surprising insights, even though they might fall outside of a certain demographic threshold.

If you’re screening users for a web-specific study, the most basic non-demographic information most user experience professionals go to are: knowledge of the internet, and knowledge of or experience with your specific area.

Knowledge of the internet can include information on how comfortable a certain person is with terms like “cookies” or “cache” or “IP address.” You can ask these questions specifically if you have specifics in mind.

To gauge experience with your topic or task, you might include specific questions around basic familiarity. Let’s say you’re a baking website, and you want to create a survey for the real baking VIPs, the sort of people who you want to become your power users. To filter out this group, you might ask questions about familiarity and comfort with baking terms like “fold” and “beat” and “peak” (yum).

Or, you can ask about baking behaviors. For example:

screen for behaviors

Our sample answers are tongue in cheek, but determining these classifications will help you identify whether a user ranks high or low for each category you create. Depending on what you want (power users vs. beginners, for example), you can filter from there.

A screener survey, and later your user study, and ultimately your product design, should really tap into what people want, and demographics alone won’t get you there.

3. Avoid asking leading questions

There’s a conversational tendency some folks employ by adding a “right?” at the end of every sentence, so that you have no choice but to nod or shrug in agreement. Right?

This is an example of “leading.”

A leading question will indirectly nudge a user toward answering a question in a certain way. You want to avoid this in your studies and especially in your screener surveys. This isn’t the time to cajole people into answering how you want them to answer. You’ll end up with study participants who aren’t ultimately what you’re looking for. A good way to identify whether a question might be leading is if it includes a hint or excludes possible answers.

Examples of a leading question:

  • Leading: On a scale of 0-100, how much do you hate pistachio ice cream?
  • Not leading: On a scale from 1-10 where 1 is gross and 10 is delicious, rank how you regard pistachio ice cream

Another way to avoid leading questions is to provide a series of unrelated options as answers. For example, if you want to screen users who have a high level of concern around internet privacy issues, rather than diving right into questions about internet privacy, you can create a question like this to get to the people who really care:

don't use leading questions in screeners

Likewise, avoid yes or no questions which tend to be leading. Users might answer in the way they believe will entitle them to participate in the study.

Make like Law & Order, and when you spot a leading question in your own screener survey, object!

4. Always provide a “None of the above,” “I don’t know,” or “Other” option

As you create answer options, it helps not to assume that you’ve presented the user with every possible option for an answer. You might not know what they’re thinking, and if you don’t include one of these extra catch-all answers, you could end up with someone in your survey who doesn’t belong there because they were forced to choose an answer that didn’t apply to them. Likewise, you might screen good people out if you aren’t able to provide all possible answers.

5. Provide clear and distinct answers that don’t overlap

We’ve all taken a survey where, on a certain question, we find ourselves searching for that “all of the above” option. It’s frustrating when we’re asked to choose just one answer when the real answer is “a bit of this, a bit of that.” If you force your respondents into choosing just one that doesn’t really hit the nail on the head, you’re not getting the most helpful insight into your product. The solution to this is making sure your answers have clear borders between them without any overlap.

Be sure not to overlap ranges.

For example, when asking for ages or sizes or anything with numerical value, make sure your values are mutually exclusive:

  • Correct: 0-3, 4-7, 8-12
  • Incorrect: 0-3, 3-7, 7-12

Likewise, for answers less definitive or for answers that can’t be made into mutually exclusive options, perhaps offer a “choose all that apply” option rather than a “choose just one” option.

avoid overlapping answers in screeners

6. Ask precise questions

Asking good questions is just as important as offering clear and distinct answer choices. A mushy question where users are required to interpret what it is you’re looking for will yield weird mushy answers and land you with survey participants who might not be right.

Instead of:

Try asking about a specific aspect of user experience. “Findability” of content or answers or buttons is one thing UX designers are often striving to improve:

7. Know who you don’t want to take your survey

Another way to put this is, get more value for your time and money on a per-participant basis. If you’re looking for people who will show up to your study ready to share freely and communicate clearly about their experience of your product, maybe conceive of a way to filter out people who aren’t able to articulate their challenges with a product or process. If manual dexterity is important to user testing a gaming console, maybe filter out people with debilitating hand ailments. If you need people for your study who are avid crafters but NOT in a certain discipline like beading, for example, be sure to create questions designed to specifically exclude those folks who you absolutely don’t want.

Common screener questions and themes

There are certain questions, themes, and formats that will suit just about any screener survey. Here are four to help get you started.

1. Industry or occupation

Useful: When you want to include or exclude people who have a certain level of

familiarity or relationship to a certain industry.

Question format example: What industry do you work in?  

Answers: A list of industries like retail, tech, healthcare, art, etc.

2. Familiarity with a product in terms of level of experience

Useful: when you need to understand whether a participant will be useful for a specific task in your test

Question format example: please rank your experience with {name of product}

Answers: A range from expert to novice

3. Comfort to share personal information

Useful: when you need to know personal details about a tester like body type, health

issues, income, habits, lifestyle, or other issues that might feel private

Question format example: This test will require you to share openly about {examples}.

Do you agree to share honestly about these subjects?

Answers: Agree or disagree

4. Frequency of executing specific tasks

Useful: when screening for users who regularly do a specific task, or who used to

behave in a certain way and then stopped

Question format example: Please tell us how often you {name the task}

Answers: A range of time from often to rarely. Consider defining terms like often (every

day) and rarely (once a year) so there’s no guesswork.

Common Problems and what to do about them

Be aware of the common challenges that can come up during screener surveys, and do what you can to prepare.

Communicate with the folks taking your screener survey, and manage their expectations

Make sure your participants are clear about what they’re doing, and what stage of the process they’re at. The screener survey is a sort of dress rehearsal, and it will help the participant to know they’re not yet in the final round. Be sure the candidate knows what they’re in for if they do make it. If there are any possible deal-breakers (like NDA agreements, for example) let them know up front. And of course, be clear that they won’t be paid until they make it through to complete the actual survey.

Prep for and avoid no-shows

People who promise to show up for your study and don’t will cost you time, and likely a moment of discomfort with colleagues and bosses who are forced to sit around waiting. Save yourself some trouble and work to prepare your participants and yourself for this sometimes unavoidable pitfall of user testing.

  • Get your users contact information including email and cell phone.
  • Send reminder emails from an individual not a general group name.
  • Give participants your number or the number of the testing office so they can get in touch if they’ll be late.
  • Send great instructions for how to get where they’re going.
  • Most of all, stress the importance of their participation so that they’re incentivized to show up because they’re able to perceive their own value in the project.

Also consider being prepared with on-deck participants. Folks you can call in it the last minute, if need be.

Do you need additional screening?

Once you’ve chosen your study participants, consider pilot test screening before you delve into the official study. Do this especially if there’s someone in your organization who you suspect of being questioning of your ideas. A great way to discount someone else’s survey results is to suggest that they had no value because they were the “wrong subjects.” Run a pilot test and include your colleagues in the process. This gives you the opportunity to have others sign off on your subjects beforehand, which could bulletproof your final results.

Long story short, envision your ideal participant

Know who they are, know who they aren’t, and build a screener survey that’s a sieve that allows the wrong participants to fall away but captures the gems. The specifics of how to get there are outlined above. Just remember to keep an open mind as to who these study participants might be, and don’t limit yourself with prejudgements mired in demographics.

Ready to launch a research project and easily build your screener survey? Get started here.

Want to contribute to User Interviews content? Here’s how.

P.S. Can we email you?

Get "Fresh Views," our weekly newsletter, delivered to your inbox. We feature interviews and point-of-views from UXers, product managers, and research nerds of all stripes.

Subscribe
Lauren Leonardi

Lauren Leonardi is a globetrotting writer. When she's not writing about UX for User Interviews, she uses digital marketing to support the growth of local, independently owned businesses. She also writes about global travel through the lens of social, economic, and ecological sustainability.

Recent Posts

Subscribe to FRESH VIEWS newsletter

Weekly interviews and point of views all about UX research

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.