Qualitative vs. Quantitative Research

Knowing when to use which type of research is an essential instinct for UX researchers to hone. In the last chapter, we gave an overview of the different types of user research methods. In this chapter, we’re going to take a closer look at qualitative and quantitative research, in particular.

Why? Because combining qualitative and quantitative UX research methods is the most effective way to create a comprehensive portrait of your customers’ wants and needs. 

Properly acquired and analyzed, the high-volume of data in quantitative research helps you uncover what is happening—trends, issues, and opportunities—and prove hypotheses. 

Qualitative research, on the other hand, adds a layer of humanity to user data, delivering details that add depth and a deeper understanding of not only what’s happening, but why it’s happening. 

The trick to choosing the right user research method is knowing what kind of data you need to answer your research question. In this chapter, we’ll go over the differences between qualitative and quantitative UX research methods, the pros and cons of each one, and how they complement each other in a mixed methods approach. 

‍

In this chapter:

• The differences between qualitative and quantitative research
• When to do qualitative vs. quantitative user research
• Research methodologies for both qualitative and quantitative research
• Recruiting participants for qualitative vs quantitative studies
• Qualitative vs quantitative data analysis

The differences between qualitative and quantitative research 

The primary difference between qualitative and quantitative UX research lies in the nature of the data—which impacts how it is collected and then analyzed.

In qualitative research, researchers gain insight through direct observation of a small group of people, often engaging in follow-up questions that dig deeper into the “why” behind certain attitudes and behaviors. Qualitative research can also be adapted on the fly as researchers adjust study protocols based on participant engagement and response. 

Common qualitative research methods include interviews, focus groups, field studies, qualitative usability tests, and co-design sessions.

Quantitative research, on the other hand, usually involves collecting large quantities of numerical data from a much larger group of people, often indirectly through surveys or analytics tools. Insights are derived using mathematical analysis of the data to quantify a problem by answering the questions, “How much?” and “How many?” Because statistical significance is a key goal of qualitative research, testing protocols are rigid and not subject to change over the course of a study. 

Common quantitative research methods include user tests, surveys, click tests, card sorts, and A/B tests.

Let’s take a closer look.

‍

Qualitative vs quantitative UX research methods

‍

Whatever kind of product you’re working on, and wherever you are in the product development cycle, it’s important to know which kind of UX research will get you the answers you need. 

‍

When to do qualitative user research

We love qualitative UX research at User Interviews. Qualitative methods can provide valuable insight when you are trying to:

• Come up with new ideas.
• Gain deeper insight into customer needs.
• Identify design problems.
• Formulate hypotheses.

The use cases for qualitative research stretch across the product development and design cycle—when you’re brainstorming, in discovery, validating a concept, testing the usability and desirability of a finished product, preparing a go-to-market strategy, iterating post launch, doing a redesign… every step of the way.

Qualitative research brings the human element to bear on whatever question you’re exploring. It lets you hear what people think in their own words. It helps you compile detailed information that is both valuable in itself and also a great jumping off point for additional research directed at solving specific challenges. 

‍

When to do quantitative user research

Stakeholders love hard data. Compiling data to drive a strategic research agenda with management is far from the only reason to engage in quantitative UX research, but it is often the catalyst for major research efforts. 

Quantitative research can help you to:

• Establish critical benchmarks.
• Identify problem areas.
• Begin to define specific problems.
• Elevate one solution over another.
• Prioritize projects.

It can also lay the foundation for (and help you make the case for) doing the qualitative research you need to find solutions to problems. 

Quantitative UX research is most commonly applied when you already have a working product and are trying to evaluate its usability. It can also be very effective at uncovering the answers to broad, high-level questions through statistical analysis to either validate or disprove your hypothesis. 

‍

Qualitative research methodologies

Qualitative research methods—which may be behavioral or attitudinal, moderated or unmoderated, remote or in-person—can be further categorized into five different types. 

Some kinds of qualitative research (like grounded theory) are infrequently utilized in a UX research context. They’re more commonly used by researchers in the fields of sociology, anthropology, and other social sciences from which they originated. But that’s not to say you couldn’t use grounded theory for UX research, if a problem called for it. 

Five types of qualitative research

‍

1. Ethnographic research

In ethnographic research methods, researchers observe user behavior in participants’ natural environment. This approach is a powerful way to understand context, observe real-world product use, and gain deep cultural insights about participants’ day-to-day lives. 

2. Narrative research

Humans are wired for story, which is why narrative research often packs an outsized punch. In this approach, researchers run in-depth interviews with a very small number of participants in order to create a cohesive narrative that reveals themes and patterns.

3. Phenomenology

Phenomenology uses a combination of interviews, observation, documentation, video recordings, etc. to help researchers understand the what, how, and why behind a particular phenomenon or event. This method helps describe and interpret lived experiences, which in turn helps uncover participants’ perceptions and motivations.

4. Grounded Theory

While phenomenology aims to describe an event, grounded theory aims to explain why an event happens, by uncovering the social and psychological processes behind it. This is typically done through a combination of user interviews with 20 to 60 participants and in-depth document research.

5. Case Study

The purpose of the case study is to relate, in detail, a real example of a specific type of experience. The nature of a case study can be either explanatory or exploratory, but the aim in either case is to gain access to deep understanding of how things happen in the real world. 

‍

Quantitative research methodologies

As with qualitative research, quantitative methods can be broken down into categories—which type of quantitative approach is most appropriate depends on the nature of the issue you are researching, the type of information you are after, and the study protocols involved. 

The following decision tree illustrates the process:

Four types of quantitative research 

‍

1. Descriptive 

Descriptive research uses a wide variety of methods to identify the characteristics and frequency of a study topic while also looking at associated trends and categories. It relies on observation and measurement to deliver insights into the what, where, when and how something happens.

2. Correlational

Correlational research looks at how two or more variables that are similar and interdependent relate to each other. This type of research uses mathematical analysis to show how each variable affects the other. Results are often presented using diagrams or statistics.  

3. Quasi-experimental (aka causal-comparative)

This type of research looks at the cause-and-effect relationship between two variables—one dependent and one independent—that are not related to each other.

4. Experimental

Used to verify an argument, experimental research takes a theoretical approach that focuses on a theory and then helps researchers identify whether a given statement is right or wrong. 

‍

Recruiting participants for qualitative vs quantitative studies

Quality research—whether it’s qualitative or quantitative—requires quality participants.  But the challenges of recruiting are different depending on which research methods you’re using.

‍

In quantitative research, you start with a large ‘population’—the entire group you want to study—and then you create a ‘sample,’ which is a subset of the specific individuals who will participate in the study. This process is often called ‘sampling.’  For this to work, you need two things: 1) a large sample size, and 2) a solid sampling design that guarantees a random sample, which is critical to ensure truly accurate results. 

‍

In qualitative research (especially for B2B or other highly targeted studies) the key is finding the perfect participants—people who fit your customer profile exactly in terms of not only demographics and geographics, but also psychographics, behavior, and specific criteria relevant to the study parameters. Recruiting participants for qualitative studies involves non-random sampling and appropriate screening to deliver the best results.

‍

User Interviews can help with both types of recruiting, but we’re especially great at helping researchers fill qualitative studies that require vetted, good-fit participants who meet the specific criteria of your study.

Qualitative vs quantitative data analysis

Once you’ve collected your data, what next? 

Regardless of whether your data is qualitative or quantitative, it isn’t much use to anyone until it’s been analyzed and synthesized. The methods you use to analyze your data will depend on the methods you used to gather it—and unsurprisingly, there’s quite a lot of difference between quantitative and qualitative research analysis.

We’ll be covering user research data analysis in more detail in a later chapter. But here’s what you need to know in a nutshell.

‍

Analyzing qualitative data

Qualitative data to yield a wealth of information, but not all of it is relevant to your research goals. Qualitative analysis involves sifting through the raw data to find patterns, themes, and stories that tell you something meaningful about the product, the user, or both.

Qualitative findings are:

• Estimates, impressions, and interpretations
• Informed by the knowledge and experience of the researcher
• Unstructured or semi-structured in nature

They are used to determine:

• General strengths and weaknesses of a design or process
• Average scores across multiple participants
• Themes and trends
• Correlation or causation between two or more variables

They are analyzed using:

• The application of metadata to help organize unstructured data
• Qualitative content analysis, which tracks instances, positioning, and intended meaning of specific words and phrases
• Thematic analysis, which identifies themes and patterns
• Discourse analysis, which looks at the nature of communication within a specific social context

Analyzing quantitative data

Quantitative analysis is comparatively (deceptively) straightforward. It involves a lot of number crunching—but what you are doing at heart is trying to understand how people use a certain product, what problems they may be experiencing, and where improvements could be made by looking for patterns in the data.

‍

Quantitative findings are:

• Statistically meaningful 
• Replicable
• Measurable data points expressed in numerical form
• Structured data that is easy to organize and search using a relational database

They are used to determine:

• How many
• How much
• How frequently

They are analyzed using:

• Mathematical and statistical analysis

‍

📖 Read more about UX Research Analysis and Synthesis

Comparing advantages and disadvantages

We know there’s a lot of information in this chapter, so before we wrap up with a brief discussion of mixed methods, let's take a moment to summarize what we’ve learned about each type of research with a list of pros and cons.

‍

Advantages of qualitative research

• Gives you first-hand insight into what people are really thinking and feeling
• Reveals nuances and subtleties that quantitative data conceals
• Uncovers the cause-and-effect connections that shape experience and drive behaviors
• Works with a smaller sample size
• Allows for a flexible and nuanced approach 
• Provides study participants with a more expansive way to express their feelings and share their experience
• Delivers emotionally driven, narrative-style evidence that can be very persuasive
• Reveals why something is happening 
• Helps identify solutions to UX problems 
• Inspires future studies

Disadvantages of qualitative research

• Incremental expense—can have a higher upfront cost than quantitative research since most teams already run some sort of analytics by default
• Time required—can take a long time to coordinate and run (though getting help with recruiting, one of the most time-consuming parts of the process, can streamline things substantially)
• Risk of researcher bias, which can influence conclusions consciously or subconsciously
• Non-traditional validation—the subjective, open-ended nature and small participant pools do not align with conventional standards for reliability and validity such as statistically representative data
• Non-replicable—neither the study nor the results can be replicated since you cannot control for variables like the context, conditions, researcher knowledge, researcher approach, etc.
• Narrow context—you cannot extrapolate results to make confident generalizations about a broader audience.
• Challenging analysis—accurate interpretation requires expert knowledge of both the subject matter and qualitative research methodology. 

Advantages of quantitative research

• Provides some level of statistical significance
• Reduces random noise
• Allows for fast results
• Lends itself to rapid analysis 
• Delivers comparatively more objective results*
• Supports helpful data visualizations
• Simplifies connecting the dots between UX improvements and team objectives
• Enables test replication

*Caveat: The aim of quantitative research is to provide unbiased and objective results. But, despite best intentions, there are always variables that can influence a study. Quantitative methods do offer a greater opportunity to find statistical significance, but only when the data is collected, analyzed, and presented correctly. For example, if a survey design is biased—even unintentionally—the data will also be biased. Excellence in quantitative research requires a high level of expertise and knowledge about each step in the process.  

Disadvantages of quantitative research

• Limited insight—quantitative research can reveal the what, but it can’t tell you the why. It can tell you that there is an issue, but it can’t necessarily identify the specific problem, and it doesn’t provide a solution.
• Risk of confirmation bias—researchers may overlook certain generative insights due to an over focus on evaluative testing.
• Limited context—participants’ response options are narrowly defined. They do not have the opportunity to explain their choices or ask for clarification about questions. 
• Reliance on researcher assumptions—the context-bound nature of responses means that researchers need to make their own assumptions about how participants interpreted questions and why they chose certain responses.
• Large sample size requirement—statistical significance requires a large pool of participants.
• High analytical skills required—an inadequate knowledge of how to apply statistical analysis can influence data interpretation and negatively affect the accuracy of results.

‍

For best results, mix methods

Oftentimes, the best way to answer your research question with both certainty and nuance is to take a mixed methods approach. 

Mixed methods user research is exactly what it sounds like—the practice of using both kinds of research methods in a single study. Side note: If you’re blending multiple qualitative methods (and you often will) or multiple quantitative methods—but not the two together—that’s referred to as hybrid research. Similar, but different.

Many UX researchers opt for a mixed method approach. That’s because quantitative and qualitative methods are actually highly complementary. Quantitative research helps you identify specific problems by measuring the ‘what’ and the ‘how’, and provides hard data that can be quickly analyzed and understood. Meanwhile, qualitative data helps you uncover the ‘why’ behind an issue or opportunity. Each type of data helps fill in the gaps left by the other, giving you a holistic picture of exactly what’s happening, why it’s happening, and how you should address it.

Mixing methods also helps you avoid making false assumptions that can cause you to travel down the wrong path, wasting time and resources on developing a solution that users don’t really need. 

For example, if quantitative research shows that a particular step in your conversion journey takes twice as long as any other part of the process, you might assume that that step is simply more time-consuming. However, if you then run a qualitative test to observe how people do that step and collect direct feedback about the experience, you may learn that the instructions are confusing and users are getting hung up on trying to interpret them. 

The key to successful UX research is knowing which methods will give you the answers you need, and how they can be combined with other methods to give you the most complete and accurate picture possible. 

Which is an excellent segway into the next chapter, How to Choose a User Research Method, which is all about the frameworks user researchers use to decide on the specific methodologies their research question requires.