Analysis in UX Research

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Note to the reader:

This part of the field guide comes from our 2019 version of the UX Research Field Guide. Updated content for this chapter is coming soon!

Want to know when it's released?
Subscribe to our newsletter!

Congratulations, you’ve completed your study! Maybe you conducted surveys with existing users, or had conversations with potential users. Regardless of the type of study you did or how many participants you recruited, you’ve gathered a lot of information and data.

Regardless of whether your data is qualitative or quantitative, the next step after completing any study is analysis

In this chapter, we’ll go over everything you need to know about analyzing your data, teasing out meaningful insights, and synthesizing research to share with your stakeholders.

In this chapter:

  • What is user research analysis?
  • When should you do analysis in UX Research?
  • Challenges and common mistakes in analyzing data
  • Qualitative vs. quantitative analysis
  • How to do data analysis in UX research
  • Tools and software for research data analysis

What is user research analysis?

Research analysis is the umbrella term used to define the process of classifying, organizing, and transforming raw data into valuable information, and eventually a conclusion. When performed correctly, your analysis will generate the building blocks you’ll need to construct your research deliverables.

As Nomensa’s Director of UX Research JJ Knowles said

“It’s the act of taking raw data and turning it into something useful.”

Since data can be interpreted an infinite number of ways, part of your job as a researcher is to decide how to analyze your data and use it to tell a compelling story. The methods you use to analyze your data will depend on the methods you used to gather it

Qualitative interview analysis and quantitative survey analysis are two very different beasts! Yet the ultimate reason for doing research analysis is the same: 

“Regardless of whether you’re dealing with a numeric dataset or a verbal interview, you’re always on the lookout for patterns and themes that can tell you something meaningful about the user, the product, or both.”  – Career Foundry

Research analysis vs. synthesis

Research analysis, as defined above, is the process of sorting and categorizing data. 

Synthesis involves interpreting research data and pulling out insights and key findings that can be used to impact decisions.

Synthesis may follow once all the analysis is done, or the two processes might happen more or less in tandem, depending on the methods you use at this stage.

Together, research analysis and synthesis are key processes that create meaning from raw data. With this meaning, you can make better, more informed decisions

what is research analysis diagram - define disassemble, evaluate, decide
Source

When should you do analysis in UX Research?

This chapter comes late in our Field Guide, since you need to have data from user research in order to analyze it.

However, you should not be thinking about analysis for the first time after having already collected your data. Good research analysis starts at the very beginning of a project, before research even begins. As you’re creating your user research plan, you should be thinking about the types of analysis you want to do, what you want to learn, and how you’re going to use the data post-study. 

Conducting some analysis and synthesis as you go will save you time at the end of a project, and gives you valuable snapshots to share with stakeholders.

In other words, analysis should occur:

  • During the planning phase of a research study
  • Periodically, as the study progresses
  • After the study is completed

Always be analyzing, folks.

Before conducting research

As we’ve said time and time again throughout this Field Guide—the first thing you need to do is define your goals. 

What are the objectives of this study? What do you want to learn? What are the key research questions?

Keep the scope of your research narrow, and then dig in. During the planning stage, think about how you will categorize and catalog your data. What kinds of themes do you expect to emerge? 

Brainstorm tags that you can assign to data and session notes as you go—this will save you mountains of time at the end, and give everyone involved in the research a shared system for coding notes and insights.

During research

You don’t have to wait until you’ve completed your study to begin your analysis. In fact, it’s often helpful to think about what your data might look like, and what it is starting to look like, as it’s being collected.

Periodic analysis may help you discover that you’re asking the wrong questions or even building the wrong product or feature. Analyzing your data, your variables for analysis, and the assumptions behind them means you’ll be able to catch mistakes and anomalies early on—which can end up saving the whole team a lot of time and money.

Analyzing and synthesizing as you go is also more efficient. It ensures that you don’t miss important details that might become the bedrock of your work’s final quality.

The things you jot down during a session identify what was most important to you, the client, and other stakeholders in the moment. Use your predefined list of themes and tags pre-defined to code your notes and data in real time. Then, give yourself a buffer of 15 minutes or more after each participant to review, analyze, and discuss the session. 

By the time you sit down to complete the final deliverable, you will have already created much of the framework for your entire analysis. 

To go beyond real-time coding, Roberta Dombrowski, VP of User Research at User Interviews, suggests the following workflow for analyzing, synthesizing, and sharing findings over the course of a research project:

  • After each user interview, create a snapshot of the interview in a collaborative tool like Miro
  • Share the snapshot and key takeaways in a project Slack channel to give stakeholders visibility as the study progresses.
  • Once you’ve concluded your interviews, do a “final synthesis” using the takeaways you put together along the way. 
  • This synthesis can be shared with stakeholders as a slide deck,  a written summary, or both. 

At the end of a project

Most research analysis happens at the end of a project. This is when you have all your raw data and (if you’ve followed our advice) tagged notes and initial analysis from individual sessions in hand.

At this stage, you’re looking for patterns and themes that exist between participants and data sets. If you’ve been synthesizing as you go, this will be a lot easier! Either way, you’ll still need to synthesize the results of your project, look for answers to your initial research questions, and provide stakeholders with actionable insights that enable further decision making. 

Challenges and common mistakes in analyzing UX research data

Before you begin analyzing your data, you should be aware of some of the most common mistakes people make during analysis, including:

  • Presenting a large volume of unassimilated and uncategorized data in an effort to be “perfectly objective”.
  • Withholding initial biases or assumptions about the study focus, and thus misrepresenting the relevant and important data.
  • Over-reduction, or “flattening” of the data collected into close-ended survey responses (i.e. binary “yes or no” questions).
  • Jumping to decisions based only on statistical numbers (e.g. redesigning a full page because of a high bounce rate—when the real issue might be as small as a poorly-placed button).
  • Getting lost in the details and simply rehashing information collected during the study, without applying any real analysis to that information. 

Of course, analysis is challenging. With large quantities of rich, sometimes contradicting data, you’re bound to mishandle the analysis process without taking an informed, systematic approach. 

Let’s dive deeper into that analysis approach, and how it differs based on the type of data you’re working with. 

Qualitative vs. quantitative research analysis

Qualitative and quantitative data are two very different beasts—and for that reason, the analysis frameworks you apply to them will be quite different as well. 

comic about quantitative vs qualitative methods: quantitative (1 in 30 take the free ice cream), qualitative (what did you feel when you saw the free ice cream)
Source

Analyzing quantitative UX research data

In quantitative UX analysis, you are looking to develop insights, through patterns in that data you’ve collected, about the how and the why of people using that product. You might also have, as part of your project, the task of gauging the quality of the overall user experience through survey or behavioral UX data.

Large datasets are often analyzed with tools like R, Python, or SPSS, or (for smaller datasets) in a spreadsheet. Common variables that are analyzed in quantitative UX data include:

  • Success rates
  • Task times
  • Error rates

In addition to answering questions about how people are using a product, you might also have the task of gauging the quality of the overall user experience through attitudinal surveys or behavioral UX data.

Demographic and geographic data is then layered into the analysis, in case they are helpful in determining patterns among certain groups of users.

It’s a lot of number crunching—but what you are doing at heart is trying to understand how people use a certain product, what problems they may be experiencing, and where improvements could be made. 

Questions that quantitative analysis can help to answer include:

  1. How long does it take the average user to complete a certain task?
  2. What features do they use?
  3. Are there any user needs not being met by the product?
  4. What features need the greatest attention for improvement? Why?
  5. Do certain users have different experiences using the product? How so?

Analyzing qualitative UX research data

Because qualitative data can be wildly diverse in format and subjective in nature, there are very few hard-and-fast, agreed-upon rules as to how this data should be handled. 

We’ll discuss some of the different data reduction methods in the “How to do data analysis in UX research” section below, along with a framework for doing qualitative analysis.

But regardless of which methods you use to collect and analyze qualitative data, there are some questions and practices that will make the process more focused and a lot less daunting.

When analyzing qualitative data, ask yourself the following questions:

  1. What are the major patterns and common themes in users’ responses?
  2. Did any findings surprise you, your colleagues, and/or the client? How so?
  3. In what context did users express the greatest emotional response to questions?
  4. What interesting user stories emerged from the responses?
  5. How do people view this product overall and how does it fit into their daily lives? How indispensable is this product to them? Why?
  6. What features were most important to these users?
  7. What did they like most about this product? What did they like least about this product? Why?
  8. What values are most important to these users?
  9. How are these users different from other users?
  10. Are there any use-cases not adequately supported by the current interface?

These questions should be in the back of your mind the second you start collecting data. You may even want to make yourself a little print out or index card to keep on you as a reminder. 

How to do data analysis in UX research

In this section, we’ll go over everything you need to know about analyzing your data and using it to tell a meaningful story. Specifically, we’ll discuss the following process for conducting UX research analysis:

  1. Make a plan and set objectives for analysis.
  2. Take notes immediately after sessions. 
  3. Review all the data upfront. 
  4. Organize your data. 
  5. Identify trends and synthesize your findings. 
  6. Write recommendations. 

The following steps and recommendations are geared more toward qualitative research than quantitative—for example, quantitative data may not require any kind of thematic coding. However, many of the basic principles (setting objectives, identifying significant trends, synthesizing, and writing recommendations) still apply. We’ll make note of essential differences where applicable. 

1. Make a plan and set objectives for analysis.

Setting goals and analysis objectives ahead of time helps focus your study—and prevents you from collecting too much “noise.” 

Lucy Denton, Product Design Lead at Dovetail, experienced this overload of “noisy” data while running a large-scale research project: 

“We decided to interview 45 people…. I'm not sure in hindsight if speaking to so many people was the right approach. It made the analysis process really overwhelming and difficult.”

If she could do the project over again, she thinks she’d find ways to narrow the scope of the project—and much of that scoping could’ve taken place during the planning phase of the project. 

By setting learning goals and carefully designing your study, you can collect just enough data to find meaningful insights without overwhelming yourself during analysis. 

💡 Great plans lead to better analysis. Head to Module 2 of the Field Guide to learn more about Planning for User Research.

2. Take notes immediately after sessions. 

If you’re conducting data analysis for interviews, multiple rounds of focus groups, or ethnographic fieldwork—you can improve your efficiency by reviewing notes, videos, transcripts, or other materials from each session and jotting down initial impressions immediately after. 

This process is called periodic analysis and its benefits include:

  • Preventing the wasted time and effort of redundant work 
  • Preventing your memory of different sessions from blending together
  • Reducing the risk of you missing important details that might become the bedrock of your work’s final quality
  • Identifying what was most important to you and other stakeholders in the moment

Ultimately, these benefits help you answer important research questions as efficiently and thoroughly as possible. Work smarter, not harder!

If you’re conducting user interviews with other team members, reconvene with your team after each conversation. Have a discussion about how your participants’ responses fit into your research questions. Does the whole team agree? Maybe you missed something that your teammate picked up on. 

Team Interview Notes Using The Rainbow Spreadsheet

📌 Pro tip: Give yourself a buffer of 15 minutes or more after each participant to review, analyze, and discuss the interview. Scheduling this time as part of your interview process will help reinforce analysis as part of your regular research practice, and before you know it, it will be second nature.

Periodic analysis for quantitative research

Periodic analysis is also useful in quantitative research, as going into a study with the wrong questions, metrics, or ranges can lead to big headaches down the line during analysis. By analyzing your variables for analysis, the assumptions behind them, and your data, as you go, you’ll be able to catch mistakes and anomalies early on, some of which may lead you to adjust your study.

For example, say your final data should be a bimodal distribution like the graph below. 

Bimodal Distribution Graph
Bimodal Distribution Graph

See how the graph has two peaks with a range of -4 to 4? Well, what if you assumed that the range you should be testing is -4 to 0? You would just have a normal distribution curve and be missing one of your peaks because you limited your range from the get-go.

Considering the implications of certain anomalies and outliers early on in the process can save you lots of time and money. But the only way to catch these things early on, or sometimes at all, is if you are analyzing your data at each phase of the study.

3. Review all the data upfront. 

Before tagging, organizing, or analyzing anything, scan through the entirety of your dataset to see what jumps out at you. 

Just looking through your data ≠ analysis. But taking a moment to slow down and orient yourself to what’s there can make a massive difference in your ability to understand and apply analytical frameworks to the data. 

Once you’ve gotten familiar with what’s there, you can start to sort it into an easier, more manageable form. 

4. Organize your data. 

Qualitative data tends to yield a wealth of information, but not all of it is meaningful to your research goals. As the evaluator, it’s your job to sift through the raw data and find patterns, themes, and stories that are significant in the context of your research question.

This process of organizing your data is known as “qualitative data reduction.”

What is qualitative data reduction?

Data reduction is the process of thickening and intensifying the flavor of a qualitative data by simmering or boiling.

Oh wait, no… that’s how you make a reduction sauce. Let’s try that again...

Data reduction is the process of transforming raw data into a simplified, ordered, and categorized form. 

Basically, you’re reducing the volume of data into a summarized and more meaningful format… kind of like turning juice into a flavorful syrup by boiling away all the boring water.

There are several common ways to organize qualitative research data. The most common methods in a UX research context are thematic analysis, content analysis, and narrative analysis. Discourse analysis, framework analysis, and grounded theory, while less commonly used in user research, are two other methods worth noting.

Thematic analysis 

Thematic analysis is a systematic approach to grouping data into themes that represent user needs, motivations, and behaviors. In some cases, these themes may be directly adapted from your learning goals and research questions, while in others, you may see these themes emerge after the data is collected. 

Thematic Analysis process and examples for UX research
Thematic analysis: Process and examples

Content analysis

Content analysis is a structured organization of large amounts of textual data using codes for certain words or themes. By assigning codes to different pieces of qualitative data (in a process called qualitative coding), you can begin to identify patterns and interpret their meanings. 

For example, if you’re doing a study about how college students choose their majors, you might create codes for passion, salary considerations, and family ties, and identify how often these themes come up in participant responses. 

Narrative analysis

Narrative analysis is a framework for understanding the stories people tell and the ways in which they’re told. For example, you could use narrative analysis to understand the importance of specific content to their participant, their motivations behind certain actions, and their perspectives. 

⚠️ Pro tip: Be mindful of the biases you might bring with you in the narrative approach. Although bias can influence the results of any analysis, it’s particularly a risk in narrative analysis because of the broad, human, and subjective nature of storytelling. 

Discourse analysis

Discourse analysis is a method for drawing meaning from conversations, either in written or spoken language. As opposed to content analysis, which may involve analyzing textual data from a participant’s written survey responses, discourse analysis looks at text within its social context. 

For example, analyzing the way an entry-level employee speaks to an executive (and vice versa) can tell you about the culture and power dynamics of the company. 

Framework analysis

Framework analysis is an advanced, systematic method which involves five stages: familiarizing, identifying themes, coding or ‘indexing’ themes, charting, and interpreting. 

Because framework analysis is more systematic and prescriptive than other types of analysis, it’s popular for applied research and researchers who need to adhere to high, strict quality standards. 

Grounded theory

Grounded theory is an analysis method which involves analyzing a single set of data to form a theory (or theories), and then analyzing additional sets of data to see if the theory holds up. Instead of approaching the data with an existing theory or hypothesis, grounded theory analysis allows the data to speak for itself—requiring the analyst to develop their theory from the “ground” up. 

Synthesis frameworks

You can also organize your data using various synthesis frameworks—structured analysis methods that help you visualize your raw data to find common themes. Some common synthesis frameworks include affinity mapping, matrices, and spectrums.

5. Identify trends and synthesize your findings. 

Regardless of whether you are analyzing your data quantitatively, qualitatively, or both, you will be looking for trends and keeping a count of problems or themes that occurred across participants.

Each theme and finding should be prioritized by severity and importance. You should always go back to the original research objectives at this point—hopefully you took notes about what’s most important for this project to address. Use that overall understanding of project objectives as the backdrop for your data, as you rank the most meaningful patterns, themes, and stories you’ve found thus far.

Synthesis is the process of breaking down everything you learned into small, bite-sized insights (sometimes referred to as ‘atomic research nuggets’) that can be easily shared with your team. 

Keep in mind the 3 parts of a key insight and how to communicate them

  1. A statement of what you learned: As succinctly as possible, communicate an important takeaway from your analysis. 
  2. Tags that describe the insight: You likely used tags or codes to organize your data during the analysis. Include these tags with the insight, so folks can search for other, related insights if necessary. 
  3. Supporting data and evidence: Provide links to the data, notes, videos, or other materials that demonstrate what you learned.

For example, a key insight from a project about college students’ motivations for picking majors might be: Students who prioritize earning potential are worried about being able to pay off student loans. #salary #student-loans — with links and attachments to the data which supports this claim. 

You can take several approaches to synthesizing data, including:

  • Telling stories (or ‘user scenarios’) to break down how participants think and behave. Stories are a great way to humanize and contextualize user behavior.
  • Building empathy maps or mental model diagrams to demonstrate participants’ thought processes, values, and feelings. 
  • Creating comics or storyboards to visualize (and empathize with) the participants as they move through a user flow. 
  • Running brainstorming sessions or workshops with your team to collect other ideas and perspectives. 

6. Write recommendations. 

You’ve analyzed the data and boiled it down to several key insights—now, what do you actually do with those insights?

The final step of analysis is providing insight-backed recommendations for next steps. Final recommendations take your analysis one step further and allow your stakeholders to quickly understand the big takeaway from this project, and take the appropriate actions in response to those findings.

It would be a shame, after all, if stakeholders funded a whole study, and then didn’t do anything with it, because they simply didn’t know what they were supposed to be doing with that information.

Your recommendations might look like these:

  • Place the “SHOP NOW” button in the top ⅓ of the page, and use greater color contrast between the button and background, as customers have a hard time locating it.
  • Eliminate the error window that pops up during the form, as it is preventing participants from completing the registration process.
  • Offer a search bar in the upper right corner of the landing page, as people often look for obscure content that is not relevant for other site users.

Tools and software for UX research data analysis

Analysis, whether qualitative or quantitative, takes time—and many busy researchers do not have the capacity to spend as much time as they’d like on analysis. Fortunately, there are special tools and programs built to help you accelerate and improve the analysis process. 

These programs can help you do things like organize your data sets based on demographics or descriptors, code your data to help with organization and pattern recognition, and collaborate with other researchers on your team.

Some of the most popular analysis tools include:

🧙✨ Explore these tools and more in the 2022 UX Research Tools Map, a fantastical guide to the UXR software landscape.

Since every research team and project is unique, we recommend looking into the different software options to determine which program or combination of programs would best fit your research, team structure, and workflow.

In a nutshell

Your approach to analysis can make or break your overall impact as a user researcher. From raw data to actionable insights, the process can be complicated and sometimes overwhelming. We hope you use this chapter of the Field Guide as a reference while you dig into your next dataset—and remember, always be analyzing!

Head to the next module to move onto the next step in your research journey: Creating effective research reports and deliverables

Find legit participants for your research study

Sign up for free

Subscribe
X

Master UX Research

Get the full course and exclusive pro tips delivered straight to your inbox. (New content dropping in 2022)!
Thanks! See you in your inbox 👋
Oops! Something went wrong while submitting the form.
We have updated our Privacy Policy. By continuing to use our services, you confirm that you have read and agree to the updated Privacy Policy.
OK