Grade My Screener Survey

How do I create an effective screener survey? Follow these best practices

If you want the right participants, you’ve got to design smart screening questions. Here are the 11 tips that informed our Grade My Screener Survey tool, based on our team’s 10+ years of expertise and adapted from our UX Research Field Guide.

1. Review your goals 

Clearly defined research goals and objectives are must-have requirements for any user research project. These goals should be hammered out well before you start writing your screener surveys.

2. Define target audience criteria

Your targeting criteria will typically be defined by using a mixture of:

• Psychographics: Activities, hobbies, interests, and opinions
• Behaviors: What they do (e.g. ‘regularly commutes by car’)
• Demographics: Age, gender, education, income, marital status, etc.
• Geographics: Country, city, or region

3. Screen for behaviors and psychographics over demographics

Take a closer look at your targeting criteria. Do they include demographic criteria like age, gender, race, income, etc? Do they need to?

Not every question on your screener has to result in an automatic in or out, but can be used to filter for a variety of participants as a final step. Accept anyone who could be a fit based on any given question. 

4. Write carefully-worded questions

When writing screener questions:

• Avoid double negatives.
• Keep the questions short and sweet.
• Leave out industry jargon (unless knowledge of it is a requirement for participation).
• Be specific.

The more clearly worded and specific your questions are, the less likely participants will be to get confused and answer inaccurately.

5. Put your screener questions in the correct order

Don’t make prospective participants complete your entire screener before finding out they don’t qualify. Eliminate unqualified people early.

The easiest way to do this is to write out your questions, rank them in order of importance, and look for any interdependencies. 

6. Avoid leading questions

Leading questions have no place in user research—and definitely not in your screener survey. This is not the place to try to validate your assumptions. You’ll end up with skewed results or the wrong kind of participants.

Example:

• Leading: Would you like it if there was a feature that did [X]?
• Not leading: Are there any features that don’t currently exist in the product that would help you do [X]? If so, what are they?

A good way to identify whether a question might be leading is if it includes a hint or excludes possible answers.

7. Provide a catchall alternative option

If you create multiple choice responses, don’t assume that you’ve presented the user with every possible option. Include a ‘none of the above,’ ‘I don’t know,’ or ‘other’ option to account for any outliers. 

Otherwise, you could admit otherwise unqualified participants forced to choose an answer that didn't apply to them. Likewise, you might screen good participants out because they didn’t quite fit the answers you provided.

Need to assess your survey tool options? Companies such as Lyssna, Typeform, and Qualtrics offer a range of options. We also put together a list of 13+ survey tools you can use right now.

8. Include an open-ended question

Screener surveys help you get more value for your time and money by indicating the quality of participants. Sometimes that means excluding certain people who otherwise perfectly fit your ideal audience profile because they cannot articulate the feedback you're looking for.

Screen for expressive participants by asking ‘articulation questions.’ These are open-ended questions designed to test a user’s capacity to communicate. If a person can express their ideas with depth of thought, they’re likely to be a helpful participant. 

Including open-ended questions also helps weed out “professional participants” who are just looking to make a quick buck by qualifying for any and every study.

9. Don’t give too much information away

A screener survey is meant to help you find the candidates who are a perfect fit for your study. Giving away too much information about the purpose of your study can devalue the screening process and make your research less effective.

10. Manage expectations

The screener survey is a sort of dress rehearsal, and it will help the participant to know they’re not yet in the final round. If there are any possible deal-breakers (like NDA agreements, for example) let them know up front. 

And of course, be clear that they won’t be paid until they make it through to complete the actual study.

11. Keep it brief

Finally, keep your screener surveys short and sweet. We’ve seen some screener surveys get so long that participants mistake them for a (paid) research survey! If you’re looking for a rough guideline on length, try to keep your screener to fewer than 10 questions.