SUBSCRIBE TO OUR NEWSLETTER
July 2, 2020
Decide on the right screening criteria, write non-leading questions, and identify the right mix of great research participants.
Before you write your first screening question, think through who belongs in your study and who doesn’t. Here are a few activities we recommend to help you consider your criteria:
When you’re finished, you should have a clearer picture of what you’re screening for and against.
For example, let’s say you’re developing an app to help disabled diabetes patients manage their health by teaching them exercises they can do while sitting.
So you might rule out anyone who is able to walk on their own. You might decide to include potential subjects who have had an amputation in the last six months of their medical history. You would, of course, only want to include diabetes patients.
But there are other traits where the answers aren’t as clear. Are you looking for people who have tried and failed to exercise consistently in the past? Do you want recently diagnosed patients, or can study participants have obtained their diagnosis years ago? How motivated to get healthier should they be?
As you dig into these questions, you might find it helpful to segment your criteria according to four categories: demographics, geographics, psychographics, and behaviors.
Demographics include objective facts about the prospective subject, such as ethnicity, gender, age, education level, marital status, occupation, and income. Geographics cover where your users live and work. Psychographics include their activities, interests, and opinions. Behaviors include how and why they would interact with your product or service.
The first two are easy to screen for. Continuing the previous example, if your training app is intended for adults in the U.S. and Canada who speak English or Spanish, then you’ll want to look for traits like:
If income and occupation don’t matter, don’t create screening questions for them. Leaving other demographics questions open will help you select a diverse set of research subjects later in the process.
Screening for psychographics and behavior is more nuanced. We’ll get into the details of how to write those questions later in the post, but here’s an example of each:
Behavioral screening question example:
How often in the past six months have you broken a sweat while working out?
Psychographic screening question example:
When you think about spending 30 minutes per day working out, how does that make you feel? (Free Response)
The behavior question attempts to determine their history of exercise without giving away the “right” answer. If you’re looking for people who haven’t exercised much, then A) and B) might be your preferred answers. But if you want someone who’s made a consistent effort to exercise, then C) or D) would be appropriate.
The psychographic question digs into how they feel about exercise: Does it seem daunting to them? Does it seem doable? If you want people with a certain view regarding exercise, this is the way to find them.
One of the dangers to avoid when designing your screening process is overscreening. It’s easy to accidentally screen out prospective participants by making your criteria too precise. By staying open to people with different demographic, psychographic, and behavioral backgrounds, you can ensure that your research represents the diversity of your possible users.
One of the best ways to ensure you have a diverse and inclusive pool of participants in your study is to comb through your screener responses and approve participants manually until you have the spread you want.
Having a diverse mix of participants is important for generative research and usability studies alike. Otherwise, the product your team makes may not be usable by certain segments of the population. That’s not to say you should include demographics that are clearly not a good fit for your work (e.g. talking to low-income participants when you’re developing luxury goods), but it is worth taking the time to ensure that a mix of ethnicities, educational backgrounds, and so forth are represented in your study, especially groups who have been historically underrepresented but may be great users and customers for your product!
To learn more about historical, systemic failures to include women and people of color in medical research and product development, consider listening to this episode of 99% Invisible. The consequences of a failure to include diverse participants can be concrete — money, time, and even lives can be at stake. For our part, we are researching implicit bias in our product to help foster more diversity and inclusion in research sessions.
Something else to consider is testing with folks who have different situational, temporary, or permanent disabilities that affect how they access your product. What if they’re blind? Missing a limb? Constantly distracted? How can you design the experience to be easier and more intuitive for them?
There are other types of diversity to consider, too. If you’re doing usability testing for a web app that still displays well on phones, for example, then you might talk to users of different browsers and operating systems (iOS, Android, Chrome, Firefox, Edge, etc.).
Finally, one of the best ways to ensure you’re inclusive is to also make sure the screener questions are as free from bias as you can manage; we’ll chat about best practices for writing those questions next.
Here are a few of our favorite practices for writing unbiased screening questions to help you find the best participants for your study.
We recommend writing at least one question per eligibility criteria you’ve identified. That said, we’ve noticed that researchers more often write screener surveys that are too long versus too short.
The longer your screener, the less likely potential participants will make it until the end. And if your screener is too long, you may confuse participants who thought that they were already participating in your research by answering the screening questions!
If you’re struggling with survey length, try to make your criteria less restrictive and eliminate any repetitive questions.
A leading question contains language that subtly influences the choice of the participant. It might make an assumption about the respondent’s opinion or use non-neutral language. Here’s an example that researcher Els Aerts shared with us from Twitter:
The question asks, “Do you love working remotely?” The question is phrased in a highly positive way, thereby encouraging highly positive responses.
It’s all too easy to bake your hoped-for response into the manner in which you ask the question. Returning to the exercise app example from earlier in this post, which of these phrases is leading vs non-leading?
The first question, “Do you need assistance to walk?” could imply that you’re looking for participants who are not fully ambulatory. The second question uses neutral language and offers answer choices that don’t reveal the intended response.
If you’re struggling to determine whether or not your questions are leading, ask yourself:
Here’s one last example to help. If you want to talk to diabetes patients, it’s important not to state that outright, if you can. So you might ask a question like this:
Select all chronic health conditions (if you have any) that you have been diagnosed with.
This kind of questioning helps you identify folks who have diabetes without leading respondents to say they have it.
Screening surveys are not the place to use flowery, fancy language. Avoid less widely used words—for example, “ambulate.” Also, avoid idioms and slang (such as “beat around the bush” or “cut corners”); these phrases are often difficult for non-native speakers to understand.
We recommend including at least one open-ended question that forces respondents to articulate their opinion or behavior. It’s a good way to see who can give detailed and insightful responses versus folks who maybe won’t be the most helpful when you’re asking them questions in the study itself.
It’s also a good way to know if someone is who they say they are; if they’re able to speak about a topic knowledgeably, they’re likely not just answering whatever they think you want to gain access to the study incentive.
Here are some example articulation questions:
Even when you’re finished crafting your screening survey, the way you talk about your research to potential participants matters.
Revealing the name of your company (if you’re not communicating directly with existing users) or the exact purpose of the study can bias participant responses. In the case of the exercise app for diabetes, you might tell participants it’s a study related to exercise habits or a lack thereof. That way they have some context (which can help them decide to click into the screener), but they don’t know what type of exercise habits you’re looking for.
This will make it harder for professional testers to guess what you want, making it more likely you’ll get authentic responses.
If you want, your screening procedures can include a phone call to the potential candidates who seem most promising. Double-screening like this often isn’t necessary, but it’s a good way to be absolutely sure that you’re getting the right people to talk to during your study.
We’ve seen researchers use double-screening when the study they’re doing is high-profile (visible to important stakeholders in their organizations) or when they have a highly specific research need. Sometimes, they make the calls just to know that they won’t have inarticulate or untrustworthy participants.
If you use User Interviews for double-screening, you will be given the participants’ phone numbers or email addresses after they’ve completed the screener.
If you’re conducting medical research, the recruitment process and screening process are considered separate activities. Recruitment — in which you reach out to research candidates and tell them about the planned study — is a pre-screening activity that can be done without informed consent. But even your pre-screening process may have to be submitted to your Institutional Review Board (IRB) before you can proceed.
Before you gather protected health information or obtain medical records to determine study eligibility, you’ll need patients to sign a consent form to proceed with screening activities. Your screening script for interacting with possible participants and gathering information also has to be submitted for IRB review.
For more information about IRBs, refer to the FDA website. For your institution’s specific rules regarding screening and research procedures, refer to its specific IRB.
Getting your screener in front of the right people can take weeks (along with some clever advertising and outreach) unless you use a service or platform to do the work for you. At User Interviews, we offer fast research recruiting combined with high-quality participants. Our median time to match researchers with their first participant is just two hours.
We offer two recruiting solutions: Recruit (which pairs you with participants who aren’t necessarily existing users) and Research Hub (which helps you manage research with existing users of your product).
Recruit includes our panel of over 350,000 vetted professionals and consumers; if your niche is too narrow for our existing base, we’ll recruit more. You can quickly find participants from 500 different professions, in locations throughout the U.S. and Canada, of any ethnicity and income level, and more. Use our screening tool to look for specific psychographics and behaviors.
But if you already have users and just need a better way to manage doing research with them (without stepping on your team’s toes), consider Research Hub. We provide automated tracking of research activities (like who’s been contacted, whether they participated, and what they’ve been paid in incentives). Reminders for participants, incentive payouts, and scheduling are all automated, just like in our Recruit tool.
Best of all? Research Hub is free forever for the first 100 participants you add to the platform.
To get started on Recruit or Research Hub, create a free account (and find your first three participants for free). Launching a study is completely free; you only pay for completed research sessions with the participants we find.
Olivia is a content strategist at Grow & Convert who loves science, cats, and swing dancing. She enjoys a mix of writing, editing, and strategy in every work week.