Find Good Research Participants

The Goldilocks Method

If you’ve ever created a screener survey, you’ve probably asked yourself the same question every researcher asks:

How do I create a survey that’s not so narrow it weeds out the people we do want, not so broad it captures the people we don’t want... a survey that's just right.

Whether you’re a research veteran or a first timer, that question can be a head scratcher.

Fortunately, even rookies can hone their intuition and use tried and true strategies to get the results that they need.

It’s part art, part science, part stone cold trust in your own intuition.

The Goldilocks method is all about figuring out the audience that’s the best fit for your study.

The best audience is representative of your eventual users/customers.

The best audience is representative of your users/customers. Of course there’s no one-size-fits all formula, but you don’t need one. You need a plan.

Start with your learning goals for your research (i.e. what questions you want to answer) and then determine the best audience to provide relevant insight.

To get started, ask yourself the following questions.

What is the goal of your research?

What you want to learn from your study will determine who you should study. Consider what insight would be most useful, then work backward to figure out who can best provide that insight.

Are you in the discovery, testing and validating, or post-launch phase of product development?

Research can be useful at every stage of the product development cycle. In the early stages, think broad. Further along in the process, a more precise target audience will provide the richer insight you need to optimize your product for current users.

All of these variables will influence the type of research participants you should recruit and how precisely you should target them.

How many participants do you need?  

Before Sara Blakely launched Spanx, she only needed insight from one person—a sales associate at Neiman Marcus—to understand that there was a market fit for her new pantyhose design. That one person told her that several customers had been making a homemade version of the product she wanted to create because there was nothing else on the market at the time that met their needs. Running with that one piece of insight, Blakely was able to create a prototype, and guess who her first customer was? Neiman Marcus.

That’s an extreme example, but it does illustrate the importance of going for quality over quantity in qualitative research (hence qualitative). Five good participants is generally an ideal number to shoot for; beyond that you’ll get diminishing returns, unless of course you recruited the wrong people in the first place. So don’t do that.

The better targeted your audience, the more value your results will provide.

When to Use Existing Users or Potential Users

Start with the goals of your research before deciding whether it makes sense to use your existing customers, recruit a representative audience, or use a mix of both. There are several pros and cons that will inform your decision.

More often than not, if you’re updating an existing product with new power features, your existing users will be your best audience. If you’re building a brand new feature or product and need a fresh perspective, non-users will be a good resource.

Research with Existing Users

Your current customers can provide a wealth of insight. You may want to use them when:

• You’re updating an existing product
• Your research requires extensive experience with your product
• You want to test for usability among experts rather than novices (for example to test advanced features for power users)

Of course, if you choose to work with your existing customers, a lot of the work of finding a representative audience is done for you. That’s a big benefit.

It may be possible to access recruits by emailing existing customers, posting a request for participants on your website or social media, or having account managers, sales people, or customer service reps make direct requests—just be mindful not to inundate your valued customers with too many requests for input, and don’t forget to make it clear what’s in it for them. In some cases you may not need to offer financial incentives to your own customers. Instead, consider emphasizing how much their feedback will improve the product that they’re already using or offering early access to the new feature

In terms of your results, using existing users can give you a deeper understanding of how your product is used in the real world, including insights that depend on ongoing use of a product—something that’s difficult to replicate in a lab situation.

For usability studies, be aware that existing customers may have been using your product habitually (i.e. without thinking), making it harder for them to articulate their processes.

Long-term users also tend to overcome usability challenges and can even learn to love a flawed product the way it is.

Research With Potential Users

Potential users (or non-customers/users) can provide several advantages that existing customers may not. Non-users can be useful when:

• You’re developing a brand new product
• You need to test for usability among novices (who are more likely to stumble on usability issues that an experienced user of your product might not)
• You want to test potential new customer groups 
• You want to understand your competitors’ customers

Before recruiting non-users for a study, ask yourself: will these participants be able to get sufficiently familiar with the product and provide the level of insight you need in a limited amount of time?

If so, looking beyond existing customers opens up a larger pool for recruiting and makes it possible to explore new target user groups or use cases.

Non-users are an excellent resource if you want to understand the people who might benefit from your product if your product isn’t on the market yet. They also tend to be useful for identifying usability issues that people familiar with the product may have already overcome.

Talking to your competitors’ customers may help you understand how to adapt your product to cater to a gap in the market or gain a competitive edge based on your respective strengths and weaknesses.

Tip: Research with existing users and potential users doesn’t have to be mutually exclusive.

Given the resources and the need (think back to your research goals), it can be useful to test both groups to get a broader range of insights. Just be sure to differentiate between groups and adapt your methods accordingly.

Demographics & Other Ways to Filter People

There are a few different ways to think about segmenting your ideal audience. The most common categories are demographics, geographics, psychographics, and behaviors. Together these make up your target audience profile.

First, some definitions.

Demographics

Demographics are the basic characteristics of a population: age, gender, education level, income, occupation, ethnicity, marital status, or other categories of the like. They’re the kind of questions you’d find on a census. When used individually you get a massive potential audience, but by combining several demographics you can zero in on a specific niche.

Identifying the demographics of your audience can be helpful for ensuring a diverse set of perspectives. For instance, unless your product is for women or men only, it can be useful to target a combination of both men and women for your research.

Geographics 

Geographics are used to segment people based on where they are rather than who they are.  You can target geographics by locations such as country, city, region or a radius from a specific location. Only go as narrow as you really need to.

Geographics don’t always have to zero in on a specific location. If your product depends on a type of climate for example, you might target people across several regions based on the weather in their location. The population density or size of a person’s city or town can also be useful for grouping people geographically.

A good screener will include demographics and geographics to quickly screen out people who are definitely not a good fit.

If you’re designing a mobile app for rural farmers, don’t recruit your friends and colleagues who work at tech startups in San Francisco.

Without filtering for demographics and geographics, you’ll likely end up with a broad swath of participants who can provide little insight at best or false signals at worst.

Keep in mind there are pros and cons to targeting your exact target demographic. The Interaction Design Foundation explains:

Deciding on the right number and type of criteria for research participants is a balancing act. You don’t want so few criteria that your participants don’t represent your end users...On the other hand, you don’t want to add criteria that might make it difficult to recruit participants, at least not unless you really have to. The more specific the criteria you have, the more effort you must put into recruiting, so you should carefully consider the necessity of each criterion.

Casting a wider net also means you’ll get a greater diversity of insight. That saves you from replicating the same research with multiple niche audiences. You can segment your results by various demographics and geographics after the fact if those are meaningful to you.

Rule of Thumb: When in doubt, err on the side of going broad, rather than too narrow.

The same rule applies to psychographics and behaviors. These categories are subjective, which makes them trickier to pin down. The upside is that they enable you to gather incredibly valuable insight into what actually makes your audience tick.

Psychographics 

Psychographics are the activities, interests and opinions (also known as “AIO’s”) of an individual or group. The goal of psychographic segmentation is to understand how your target audience lives their lives and what they value.

Say you’re developing an organic food delivery app. By targeting people who have a preference for organic products you’ll be able to get feedback on whether your target customers might purchase your product versus alternatives.

The key benefit of using psychographics to segment your audience is to gain deeper insight into the general motivations and likely behaviors of a group of consumers who would be a good target for your product.

Behaviors

Behavioral segmentation takes this a step further and helps you understand if, how and why your audience would use your product.  People are commonly grouped according to benefits sought, awareness, usage, loyalty, buying occasion or attitude in relation to a product.

For the food delivery app study, you could target people who regularly order takeout in order to understand how they go about it.

Here’s an example of an audience profile for this particular study:

Demographics

• Range of ages 18-65
• Men and women

Geographics

• Lives in an urban area

Psychographics

• Smartphone user
• Prefers organic products

Behaviors

• Orders takeout food
• Often pays more for higher quality food

The demographics and geographics will filter in the right kind of people and weed out the ones who are not a good fit at a high level. The psychographics and behaviors will drill down further to pinpoint people who can provide richer insight into the eventual customers’ wants and needs.

Your Ideal User or Cast a Wide Net

We know audience segmentation is important for understanding your eventual customers, but how do you figure out how to segment your ideal audience in the first place?   

If it seems like a chicken and egg scenario, that’s because it kind of is.

Your understanding of your ideal audience will grow as your product matures and you move through additional phases of research.

If you don’t have data or existing research to go on, use what resources you have to create personas. If you don’t have the budget for preliminary research to define your personas, you and your team will have to use whatever evidence you can gather to determine who you think your users could be.

Carol Barnum of UX Firm emphasizes the importance of defining your user personas:

Don’t let the answer that ‘everyone is a potential user’ stand. You have to break down ‘everyone’ into subgroups of potential users, and each subgroup needs its own list of characteristics that make up that subgroup’s user profile. Depending on the size of your research budget, you should recruit participants from one or more of these subgroups of users.

If you’re in the very early stages of development, the ideal audience for your study will be broader. You may be able to gather insight into your competitors’ customers, but in some cases you’ll have to rely on your own conjectures about who your potential customers could be.

Just be sure to avoid making too many assumptions or eliminating too many potential users before testing things out.

Later on in the research process, when your product has launched you’ll be able to use analytics, surveys, and conversations with existing customers and customer-facing folks at your company to determine the average user’s profile.

From there you can refine your personas with more and more relevant insight.

Tip: Don’t assume that your existing customer profile represents your only target audience.

Remember, the goal when recruiting research participants is to test your product with an audience that’s representative of your end users.

That’s important.

But also remember that there’s a spectrum of individuals that could use your product.

Design consultancy IDEO’s methodology for testing products is based on recruiting extremes and mainstreams. That is, targeting the kind of people who fit squarely within your target audience (the mainstreams) and the people who fall on opposite ends of the spectrum (the extremes).

If you only test on your ideal user, you might miss out on opportunities to take your product from good to great.

By seeking insight from people who are a clear fit for your product as well as people who are somewhat less typical, you can get a broad perspective on opportunities to expand your offering or improve usability.

Bottom line: Focus your resources on your core audience, but avoid limiting yourself to a narrow view of what your audience could be.

One final tip. For broad consumer products, sometimes you need look no further than your coworker down the hall or a random passerby on the street for insight that could vastly improve your product. Before recruiting your ideal participants, save yourself time and money by asking friends, family or random strangers to help out with very early stage research.

If your product is designed for a specialized audience skip the general public and go right for the experts. If not, the “outsider’s perspective” of random volunteers can surface glaring issues that your team has missed.

Who You Don’t Want

Some experienced researchers seem to be able to spot a poor quality participant a mile away. Sure, experience helps, but you don’t have to have a sixth sense. Using a few tricks of the trade you can weed out people who would provide limited insight, skew your research, drain your resources or cause other problems.

Who you should exclude will depend on the nature of your product.

For example, Michael Margolis, User Research Partner at Google Ventures, explains that for his usability studies for Google’s consumer products he will, “commonly exclude folks who are under 18 years old, work for competitors, or are unusually technical, including engineers, designers, and product managers.”

Weeding out people who work for your competitors is a pretty good idea across the board, but beyond that it’s up to you to decide on general criteria for exclusion. Conducting research for a brewery? Filter out anyone under the legal drinking age. Designing a new children’s app? Then you’ll definitely want to include kids.

Tip: Be aware of professional user research participants.

To keep professional participants in check, User Interviews limits the number of studies each participant can join.

Keep in mind, even if they fit your ideal user profile, not all research participants are created equal. You’ll generally get more insight from participants who are open, honest, articulate and capable of providing information under the conditions of the study.

These are all characteristics that you can test for with a well-designed screener.

Putting it All Together: Defining Your Audience Criteria

One of your most important tasks as a researcher is to find an audience that is representative of your ideal target user.

To do that you need to translate your target audience profile into precise criteria that indicate whether a person is a good fit or not.

Demographics

Translating demographics and geographics into criteria is pretty straight forward. You can simply turn these characteristics into criteria for inclusion and exclusion. Keep in mind that you might be looking for a mix of both.

Let’s look back at our audience profile for the food delivery app.

• Ages 18-65 —> exclude people under 18 or over 65
• Men and women —> Include at least 2 men and 2 women

Geographics

• Lives in an urban area —> exclude people who live in rural or suburban areas

Psychographics

Defining criteria can become more nuanced when you get into psychographics and behaviors. These characteristics are often subjective or may be true to a certain degree. Err on the side of being inclusive, but know where you to draw the line.

• Smartphone user  —> include people who use iOS, Android or Windows operating systems on their mobile phone 
• Prefers organic food  —> include people who rank purchasing organic food very or extremely important

Behaviors

• Uses apps to order food —> include people who typically use apps to order food 2+ times per month
• Will pay more for higher quality food —> exclude people who rank cheap price as most important

The next step is to write a great screener that will filter the target audience you’ve chosen into your study.