All posts

UX Research Topics

Podcasts

Field Guide

SUBSCRIBE TO OUR NEWSLETTER

Thank you! You are all signed up.
Oops! Something went wrong while submitting the form.

FOLLOW US

BlogResearch Methods & Deliverables

June 4, 2020

  • Last Updated:
  • June 12, 2020

Looking to Try Different Usability Test Methods in Your Next Study? Consider These.

There are many methods you can use to beef up your usability testing efforts. Here are 10 to try.

Jenny Smiechowski

Usability testing is no longer a nice-to-have; it’s a must-have. It can save you time and money by helping you find problems when they are easier and less expensive to fix. It can help you get inside the minds of your customers and uncover weak elements that are contributing to customer dissatisfaction and an increasing burden on your support team.

Thanks to these and other benefits, more and more businesses are making usability testing a regular part of their development process. Some of these businesses don’t clearly distinguish between usability testing and user testing, but we think the difference between the two is important enough to mention:

  • User testing is a broad term used to describe any research you conduct to learn more about your user’s experience with your site or product. This includes research methods such as customer interviews, focus groups, surveys, A/B Testing, etc.
  • Usability testing is a type of user testing that focuses on your user’s experience with the technical elements of your site or product. It answers the question: Is my site or product easy and intuitive to use?

In this article, we’re going to focus on usability testing specifically. We’ll share details on 10 different testing methods you can use to get valuable usability data for your business:

  • Performance tests
  • Open card sorts
  • Expectancy tests
  • Closed card sorts
  • Visual affordance tests
  • Five-second tests
  • Tree testing
  • Eye tracking
  • Functional salience tests
  • Free exploration tests.

  • Before we dive into these different testing methods, we’ll go over the different types of usability testing. Determining what type of usability testing you need or want to do can make selecting the right usability test methods easier.

    Note: Looking for a specific audience to participate in your user research? User Interviews offers a complete platform for finding and managing participants in the U.S. and Canada. Find your first three participants for free. Or, streamline research with your own users in Research Hub (forever free for up to 100 participants).

    The best stories about user research

    Thank you! Your submission has been received!
    Oops! Something went wrong while submitting the form.

    Exploring Different Types of Usability Testing

    If you hope to incorporate new usability testing methods into your research, you need to decide what type of testing to do. Here’s a quick overview.

    Moderated vs. Unmoderated Testing

    Whether you need to conduct a moderated or unmoderated usability test depends on what type of information you’re trying to get.

    • Moderated tests are better when you want in-depth information because they allow you to ask follow-up questions. It can answer the “why” and not just the “what.” If a testing method is complicated or there’s a high probability of confusion, moderated tests are also better to prevent users from getting frustrated.
    • Unmoderated tests are best when you just want to observe a certain behavior pattern but don’t need to know the context behind why the behavior occurred. They’re also good if you need quantifiable data.

    Cost is also an important consideration. Moderated tests tend to be more expensive because they require

    1. Moderator time, which is expensive;
    2. Higher incentive payments to motivate users (since they are committing to participating in research at a specific time rather than at any time that’s convenient for them).

    Qualitative vs. Quantitative Testing

    Quantitative usability research produces numeric data. Qualitative usability research produces descriptive, narrative data. It’s rare for a test to only produce quantitative data. You can almost always make qualitative observations, too. But certain test methods are better for securing one type of data over the other, so keep that in mind when selecting one (we’ll point them out in the methods listed).

    Remote vs. In-Person Testing

    Remote and in-person testing each have their own unique perks. In-person testing allows researchers to closely observe users’ facial expressions and body language, which can be an important source of qualitative data. Also, some testing methods can only be performed effectively in a lab environment.

    Remote usability testing, on the other hand, gives you the ability to test users in their natural environment, pay lower participation incentives, and get research done faster. Due to the COVID-19 pandemic, remote testing will be the default choice for many businesses for quite a while. 

    Formative vs. Summative Testing 

    Formative tests are performed early in the development phase. They’re less formal and used to get quick insights that can help shape a design that’s in progress but not finished. Usability testing performed on an early wireframe of a new website would be an example of formative testing.

    Summative tests are performed at the end of the development stage. They’re more formal and used to validate the usability of a finished (or close to finished) product. Any testing that’s conducted on a fully designed website or even a high-fidelity prototype would be an example of summative testing.

    10 Usability Testing Methods to Try in Your Research

    Now let’s dive into some usability test methods you can try in your research. These are just 10 of many different usability test methods out there, but hopefully they’ll provide some helpful inspiration for your next usability study.

    If you’re looking for tools to help you perform the tests, consult our UX Research Tools Map. Some popular usability testing platforms include User Testing, TryMyUI, Lookback, and more.

    Performance Test

    A performance test is a usability test in which a user is asked to accomplish a specific task and be recorded or watched as they do it. 

    Here’s an example: A researcher for a flight deals website might ask a user to record themselves as they book a flight from Chicago to Phoenix. The researcher would then document how long it takes for the user to perform this task, as well as how many clicks it takes.

    It sounds simple enough, but there are many different approaches you can use when performance testing. It can be performed in person or remotely, with or without a moderator, and can be used to get qualitative or quantitative data.

    The flight deal website test we described above is a good example of a remote, unmoderated performance test that gets quantitative data. If you wanted to get qualitative data from a performance test, you could conduct a moderated test (remotely or in person) where the facilitator asks the user follow-up questions about the task they completed. In fact, asking follow up questions is a great way to help test participants express their feedback in performance tests and other usability tests, too.

    Performance tests are best for summative testing, since they’re usually used to evaluate the performance of a site or product that’s fully developed (or close to it).

    Expectancy Test

    An expectancy test is a usability test in which researchers ask users what they think a prototype or design does to see if their expectation matches up with what it actually does.

    You could perform expectancy tests in a variety of ways. You could show users an app you created and ask them to tell you what they think it does (without clicking on anything). Or you could use an expectancy test to hone in on specific elements of a product or design by asking users what they’d expect to find under a certain heading or page.

    This type of usability test is great for weeding out unfamiliar or confusing elements in your design. Because it dives into users’ expectations, it’s also a good source of qualitative data.

    To get the most valuable qualitative data, you’ll want to run this one as a moderated test with a facilitator who can ask questions. It’s best used early on or mid-way through the design process to ensure what you’re creating does what users think it does. If the design doesn’t meet users’ expectations, you’ll need to make some changes.

    Here’s one other suggestion: When conducting expectancy tests (and most usability tests, for that matter), don’t tell a user if their answer is wrong. Let them share their initial impressions without giving away what the design or product is intended for.

    Visual Affordance Test

    In a visual affordance test, researchers give users a mockup of the page they’re testing and ask them to circle what they think is clickable. For example, you could show users a printed (or static digital) version of your ecommerce website homepage and ask them to circle everything they believe is clickable. You could then give them another copy of the page and ask them to circle everything they think isn’t clickable. This helps you refine the clickability and tappability of your user interface.

    Whether you want to conduct this type of test moderated or unmoderated and remote or in person depends on what information you’re trying to uncover with your test. You should recognize, however, that unless you ask follow-up questions, visual affordance tests only provide quantitative data. Also, like the expectancy test, a visual affordance test is best used in the formative phase of development to reveal gaps between user expectations and actual functions.

    Tree Testing 

    A tree test is just one of many usability test methods you can employ while developing and finalizing your work.

    Tree testing helps refine a site’s architecture. That may be why it’s known as reverse card sorting (which we cover next). In a tree test, you ask users to look at a tree-like sitemap and tell you where they would go to find something. For example, you could ask them, “Where would you go to view your account settings?” And then, depending on if they answer correctly, you could gauge whether the content on your site is organized in a user-friendly, intuitive way.

    Tree testing is useful as formative or summative testing. In the formative stage, it can help shine a light on any navigation issues while it’s still relatively easy to fix them. In the summative stage, it can let you know whether your existing structure and labeling are easy to understand and use.

    Tree testing mainly provides quantitative data unless you ask follow-up questions. If you’re planning on using tree testing in your research, you may want to try a tool like Optimal Workshop’s Treejack, a software designed for setting up remote tree tests.

    Open Card Sorts

    Card sorts can play a critical part in making your site’s navigation user friendly. Open card sorts in particular are helpful for nailing down the organization of content on your site.

    In an open card sort, users are asked to sort a list of items into categories they develop. So, for example, you could create an online test for users where you present them with a list of “cards,” each one representing a piece of content from your website. You could then ask them to sort those cards into groups that make the most sense to them, and ask them to give each group a name. 

    Open card sorts make the most sense as formative testing that helps you develop your site’s architecture. They could also be helpful during a site redesign. They’re often conducted with a moderator who can ask follow-up questions to uncover some valuable qualitative data, but they don’t have to be.

    Open card sorts are a good next step when a tree test shows there’s an issue with the sitemap. And they’re often followed up with closed card sorts, which are next on our list.

    There are many tools that can help you conduct a virtual card sort, such as Optimal Sort, software designed specifically for card sorting, or wireframe tools such as Whimsical, which include the features you need to conduct a card sort. You could also conduct a card sort in person with actual cards, sticky notes, or whatever else you have available.

    Here’s one suggestion for conducting card sorts: Limit the number of cards you ask users to sort, especially in an open card sort. Don’t exceed 30-40 cards. Asking users to sort too many cards can cause fatigue, which will impact the quality of your results. 

    Closed Card Sorts

    In closed card sorts, users are asked to sort a list of items into predefined categories. So instead of letting users come up with categories themselves like you do in open card sorts, you give them the categories at the start of the study. Closed card sorts are often done as a follow up to open card sorts. They help you confirm whether the navigation labels you’ve set up make sense.

    When conducting a closed card sort, you can focus on data such as the number of cards a user sorted into the correct category, the amount of time it took them to answer, and how confident they were in their answers. If you choose to have a moderator, you could also ask follow up questions during closed card sorts, like “Why did you put the ‘FAQs’ card in the ‘About Us’ category?”

    Closed card sorts could be used as formative testing to help you develop your site architecture or summative testing to verify that the site architecture you’ve already set up is user friendly.

    Another suggestion for card sorts in general and closed card sorts in particular is to make sure cards are presented in a random order, so users aren’t tipped off to the relationships between certain cards.

    Functional Salience Test 

    In functional salience tests, users are asked what site functions are most important to them. If you were conducting a functional salience test for a meal kit delivery site, you would give users a list of possible site functions (or show them your sitemap) and ask them choose the three functions that were most important to them. They might choose functions like “changing my delivery schedule,” “viewing nutritional information,” and “contacting customer support.” 

    Whatever they choose should help inform the structure of your site. The most important functions in the mind of your users should be front and center if you want your site to have good usability. In fact, many user researchers use the results of a functional salience test to expand upon their sitemap and create something called a task or user flow map, which not only outlines a site’s architecture but highlights the user flow (the path a user takes to complete a task) as well. 

    Sketch offers a free user flow diagram template (pictured above).

    This type of test would be most helpful during the formative stage, when you’re still working out the details of your site’s architecture. That could mean using it when you’re developing a new site or when you want to verify the usability of an existing site’s structure during a redesign.

    It’s mainly going to provide quantitative data, unless you choose to ask follow-up questions. If you’re looking to turn the results of your functional salience test into a user flow map, you might want to look into the wireframing tool Whimsical, which offers simple options for that and other diagramming.

    Five-Second Test

    You may have heard that it only takes most users five seconds to look at your website or app, decide it isn’t for them, and click away. The five-second test was developed around this piece of data (although some research shows you may have as little as 0.05 seconds). Five-second tests gauge what users can remember about a website page, app, etc. after looking at it for five seconds.

    Typically, in a 5-second test, you’ll ask a user a question like, “What element on the page stuck out to you most?” or “What audience is this page targeted toward?” Then, you’ll let them look at the page for five seconds. When five seconds is up, you let them answer the question.

    Another approach could be to ask users to jot down everything they remember about the page once they’re done looking at it. Then you could ask a few follow up questions to dive into their first impressions. 

    Five-second tests are a great source of qualitative data. They allow you to capture that first reaction to your site or app, which is an important indicator of whether it will attract or repel the right users in the future.

    These tests are helpful in the formative phase because they’re relatively fast and easy to set up and provide good early feedback about the usability of your design. But they could also be helpful in the summative phase to test important elements such as graphics and copy once they’re more formalized.

    Five-second tests are best moderated so you have the ability to ask follow up questions if needed. They’re also better for evaluating web pages that don’t require a lot of reading and for answering simple rather than complex usability questions, since they’re so quick.

    Eye Tracking

    For eye tracking tests, researchers use a pupil-tracking device to study users eye movements. This helps answer questions like:

    • What attracts and holds users’ attention?
    • How long did users focus on a specific area of the screen?
    • In what order did users look at items on the screen?

    These tests also help determine what users scan and what they read, and they’re great for testing the effectiveness of layout and design elements. This type of test is ideal in the formative stage, because it’s better to run it with a detailed design. It’s also one of the few testing methods that needs to be done in person because it involves special equipment. 

    As a result, this is one of the more expensive testing methods, since it requires special equipment, software, and a lab space.

    Also, if you’re considering eye tracking, you should know that accurately interpreting the results of eye tracking tests is challenging. Eye tracking only provides quantitative data (unless you choose to ask follow-up questions), and this quantitative data requires very specific forms of analysis. The most common analysis techniques are heat maps (graphic representations that use a range of colors to show you what areas of a design receive the most attention) and scan path visualizations (graphic visualizations that plot individual eye movements using straight lines and fixations on a certain spot using circles).

    If you’re not a usability specialist, you may want to work with one to help you conduct this test and interpret the data to get the most from it. You could also participate in eye tracking training or self-study to develop your eye tracking knowledge. Read this article to understand eye tracking more in-depth. 

    Free Exploration Test

    With the free exploration test, users get a specified amount of time to freely explore a site or application. They’re also asked to vocalize their actions as they make them. For example, a moderator might say something like, “You have five minutes to look through anything you’d like on this website. Please express your thoughts out loud as you explore. I’ll let you know when your time is up.”

    As you can probably guess, this type of test is meant for summative research. You should do it once your site is close to finalized and navigation issues have already been worked out. Since the user is speaking aloud while they explore your site, these tests are a rich source of qualitative data. They allow you to get inside the user’s head while they’re navigating your site and help you uncover any remaining usability issues.

    Continuing Education

    If you’d like to explore these and other usability test methods in more detail, you may want to look into the User Experience Professionals Association (UXPA). The UXPA holds yearly conferences, publishes the Journal of Usability Research, and has developed a community of usability professionals that is a valuable source of information and advice. The UXPA even has a free Facebook group you can join to ask questions and get feedback on your usability test methods. 

    Once you get your usability testing strategy worked out, you also have to find people who match your target audience to participate in your tests and manage those participants during the testing process. That’s where User Interviews comes in. We can help you identify candidates, screen and schedule participants, manage incentives, and log participant information, making the process of conducting user research less complex and labor-intensive. 

    A look at Research Hub, where you can organize and manage research with your own users (free for up to 100 participants!).

    At the time of this publication, User Interviews has a pool of over 300,000 pre-registered participants, and we’re recruiting more all the time. If you’re ready to get going on a new usability study right away, we can help make that happen. It only takes a median time of two hours to match you with your first study participant. 

    The participants we recruit for you are highly targeted, too. You can use screeners to filter participants by profession, geography, demographics, and behavior, to find participants who closely match your real users. User Interviews also syncs calendars for scheduling participants, processes incentive payments, issues 1099s, and acts as a CRM for tracking participant data — streamlining the entire participant recruitment and management process for you.

    Ready to give User Interviews a try for your next usability test? Find your first three participants for free.

    Jenny Smiechowski

    Jenny is a Chicago-based freelance copywriter who works with health, SaaS, senior living, and green living companies. When she’s not creating marketing content, she loves hiking with her dogs, doing yoga, and binge-watching true crime shows.

    More from this author