Analyzing User Research

Congratulations, you’ve completed your study! Maybe you conducted surveys with existing users, or had conversations with potential users. Regardless of the type of study you did or how many participants you used, you’ve gathered a lot of information and data.

‍

Now, what do you do with all of this information? How do you communicate your findings so that your stakeholders can actually understand them the way you do, and use them to improve the user experience?

‍

That’s where research deliverables come in.

‍

In this module, we’ll go over everything you need to know about analyzing your data, using it to tell a meaningful story, crafting a deliverable that is most appropriate for your stakeholders, and best practices for creating the deliverables themselves.

‍

Whether you’re an experienced researcher or you just completed your very first study, if you’re looking to improve how you communicate your findings, this module is for you.

‍

Analyzing Your Data

Before you decide how you want to communicate your findings, you have to understand your data so you know exactly what it is you’re communicating. Regardless of whether your data is qualitative or quantitative, the first step after completing any study, is analysis.

‍

The way you end up communicating your findings will be determined by your analysis.

‍

Analysis is the umbrella term used to define the process you take to transform raw data into valuable information, and eventually a conclusion. When performed correctly, your analysis will generate the building blocks you’ll need to construct your deliverables. Data can be interpreted an infinite number of ways; it’s how you decide to analyze your data and use it to tell a compelling story that will determine the ultimate quality of the results of your study.

‍

Analysis Begins At The Beginning

Hopefully, however, you’re not thinking about analysis for the first time after having already collected the data. Your methods of analysis ideally begin right when you begin designing the research itself. It always helps to go back to Step 1 – your project objectives. Do you want to understand who your target user is, and their main motivations for using the project? Then you’ll be trying to look for patterns in the data along demographic, attitudinal, and lifestyle indicators. Do you want to stress-test the new version of your app right before launch? Then of course you’ll be looking to identify and evaluate the severity of each and every pain point, big and small. It is useful to consider your potential variables of interest and possible hypotheses, or theories behind what you think might be going on, and use those to guide your main axes of analysis.

‍

This also means performing periodic analysis, and not simply waiting until the end of the study to analyze the data you’re collecting. You don’t have to wait until you’ve completed your study to begin your analysis. In fact, it’s often helpful to think about what your data might look like, and what it is starting to look like, as it’s being collected.

‍

If you take the time do to periodic analysis during the actual study, you may discover that you’re asking the wrong questions or even building the wrong product or feature. Figuring this out early in the research and development process can help ensure that you’re asking the right questions and building the right product for your users, and save you a lot of time and money in the event that you’re not.

‍

Using Data For Discovery

User Research, and UX design as a whole, begins with a lot of discovery! Discovery is the process of conducting research to figure out what your product should be, what its functions should be, and what the goals of its main users would be as they relate to your future product.

‍

A rigorous discovery process could get your team on the right track from the start, saving you serious time and the costs of additional product iterations down the road. Here is one scenario where discovery benefits your UX design:

‍

Your client thinks a product is most useful for one particular use-case, and your team finds through discovery that there is actually another unmet need that was not originally envisioned for that product, but that would be easy to design for and that is in high demand, and one that your competitors are unable to address.  

‍

This is why it’s hugely beneficial to be open to letting discovery inform your design. What does this mean for your plan for discovery? Well, it means that you might initially start off with a few basic exploratory questions, adapted to your research context:

• What are the potential benefits of this product/site/experience to the users?
• Where are the clear or potential pain points?
• What emotions might users experience while using various parts of this product?

‍

When you’re in discovery, there is a general process when it comes to whom you’ll talk to first. You’ll want to begin by checking in with your stakeholders. What is their vision for this product? How does it align with the business goals? What are the key performance indicators going to be for the success of this product?

‍

After speaking with stakeholders, you’ll turn to potential customers and users. You might create personas, and journey or experience maps, to describe in detail who they are and how they relate to the problem that your product seeks to solve.

‍

Next, you should look to align your stakeholders’ vision of your users, and the picture that your findings from your discovery phase have yielded. There might be a gap between these two pictures, so you’ll need to communicate to your stakeholders exactly what new information they need to know about their target users.

‍

Always Be Analyzing

Taking notes during fieldwork makes for a significant portion of the overall analysis and can provide strong framework for the final deliverable. Even if you’re a research superhero, chances are there are little things you’re going to miss while you’re conducting the study, especially if you’re doing everything on your own.

‍

This is why it will make a huge difference to review each interview and jot down any notes and initial impressions immediately after, while the conversation is fresh in your memory. If you’re conducting user interviews and collecting qualitative data, reconvene with your team after each conversation. Have a discussion about how your participants’ responses fit into your research questions. Does the whole team agree? Maybe you missed something that your teammate picked up on. That’s why we work in teams!

‍

‍

If you happen to be a lone wolf without a team, review the notes, videos, transcripts, or any other materials you may have after each interview.

‍

Most researchers conduct 5 or more sessions in a day, so immediate review will help prevent the interviews from blending together. Incorporating this into your practice will help you be more thorough in your research and ensure that you don’t miss important details that might become the bedrock of your work’s final quality.

‍

Importantly, taking notes during a study also saves you the time and the effort of doing redundant work. Those details you jotted down in the moment identify what was most important to you, the client, and other stakeholders in the moment, and so these usually help answer important research questions. This way, by the time you sit down to complete the final deliverable, you have already created much of the framework for your entire analysis.

‍

A great way to bake this practice into your process is to give yourself a buffer of 15 minutes or more after each participant to review, analyze, and discuss the interview. Scheduling this time as part of your interview process will help reinforce analysis as part of your regular research practice, and before you know it, it will be second nature.

‍

Periodic analysis is also useful in quantitative research, as going into a study with the wrong questions, metrics, or ranges can lead to big headaches down the line during analysis. By analyzing your variables for analysis, the assumptions behind them, and your data, as you go, you’ll be able to catch mistakes and anomalies early on, some of which may lead you to adjust your study.

‍

For example, say your final data should be a bimodal distribution like the graph below. 👇

‍

‍

See how the graph has two peaks with a range of -4 to 4? Well, what if you assumed that the range you should be testing is -4 to 0? You would just have a normal distribution curve and be missing one of your peaks because you limited your range from the get-go.

‍

Considering the implications of certain anomalies and outliers early on in the process can save you lots of time and money. But the only way to catch these things early on, or sometimes at all, is if you are analyzing your data at each phase of the study.

‍

Prioritizing Areas of Analysis

Have you ever stared at a dataset, and become hypnotized by its daunting size? Oftentimes, we deal with so much data that it may be hard to figure out where even to begin. The analysis thus should be focused on organizing the data into discrete areas of analysis, and then on prioritizing the analysis by areas of attention.

‍

Take this example: You may have found something interesting for the next version of a product, but you found this through a discussion for a different product under the same brand altogether. That would fall to the bottom of the priority for analysis pool.

‍

Again, the findings with the greatest impact on the UX of your product will go to the top of the list of analysis. And there may be many “interesting” findings that simply don’t make it to the final deliverable because they are “nice to haves” rather than “must-haves” for the target user.

‍

Analyzing Quantitative Data

In quantitative UX analysis, you are looking to develop insights, through patterns in that data you’ve collected, about the how and the why of people using that product. You might also have, as part of your project, the task of gauging the quality of the overall user experience through survey or behavioral UX data.

‍

Chances are, you’ve either received a large dataset and analyze it in R, Python, or SPSS, or, for smaller datasets, you may have entered data manually into a spreadsheet. Regardless, most quantitative UX studies are examining a number of common variables: success rates, task times, and error rates are just a few examples of important analytical UX metrics. Quantitative UX research also tends to involve attitudinal measures, gauged by questionnaire ratings of satisfaction with the experience and various aspects related to it. Finally, involved in analysis are the participants’ demographic data, in case they are helpful in determining patterns among certain groups of users.

‍

Despite how many numbers and variables you are crunching during your analysis, what you are doing at heart is trying to understand how people use a certain product, what problems they may have in using it, and what could be working differently for them.

1. How long does it take the average user to complete a certain task?
2. How long is the average user experience? What does this say about their level of engagement?
3. What features do they use?
4. Are there any user needs not being met by the product?
5. What features need the greatest attention for improvement? Why?
6. Do certain users have different experiences using the product? How so?

‍

Analyzing Qualitative Data

Because qualitative data can be wildly diverse in format and subjective in nature, there are very few agreed-upon ground rules as to how this data should be handled. But there are certain questions and practices that will definitely make the process less daunting no matter what the format or context.

‍

While analyzing qualitative data, ask yourself the following questions:

1. What are the major patterns and common themes that emerged in the user responses?
2. Were there any findings that surprised you, your colleagues, and/or the client? How so?
3. Where did users express the greatest emotional response to questions?
4. What interesting user stories emerged from the responses?
5. How do they view this product overall and how does it fit into their daily lives? How indispensable is this product to them? Why?
6. What functions were most important to these users?
7. Are the features most appropriate for the context of use? (While composing a review for a product would work in a desktop browser, it might be seldom done on a mobile phone.)
8. What did they like most about this product?
9. What did they like least about this product? Why?
10. What values were most important to these users?
11. How are these users different from other users?
12. Are there any use-cases not adequately supported by the UI?

‍

These questions should be in the back of your mind the second you start collecting data. You may even want to make yourself a little print out or index card with the questions to keep on you as a reminder. There is also a series of steps that user researchers can follow to guarantee thorough qualitative analysis.

‍

Qualitative Data Reduction and Organization

Qualitative data tends to yield a wealth of information, but not all of it is meaningful. As the evaluator, it’s your job to sift through the raw data and find patterns, themes, and stories that are significant in the context of your research question.

‍

There are two common ways to organize data: thematic analysis and content analysis.

‍

Thematic analysis groups the data into themes that will help answer the research questions. These themes may be either directly evolved from the research questions and were pre-set before data collection even began, or naturally emerged from the data as or after the study was conducted.

‍

Content analysis, on the other hand, is a more mathematical organization of long stretches of text that involves coding the data for certain words or content, identifying their patterns, and interpreting their meanings.

‍

Chances are, you’ll find both thematic and content analysis useful, and they’ll likely inform or supplement each other.

‍

Here are a few ways you can begin your data reduction and organization process:

‍

Organize Your Data in an Excel Spreadsheet

Using your research questions as a basis, you should make an excel spreadsheet with your insights to help visually and numerically highlight the problems, themes, and patterns you came across during your study.

‍

This method is useful for any type of qualitative data, because you can organize it to meet the specifications of your unique project. For example, in the image below, the spreadsheet is organized by task and by the issue that participants had with a given task. It then quantifies the frequency and severity of each, which is instrumental in finding patterns and deciding what to prioritize.

‍

‍

While this particular example from a usability text may seem more easily quantifiable than other qualitative data, it still works for analyzing more conversational studies, such as user interviews.(P.S. If you want to read more about this method, check out the book Quantifying the User Experience by Lewis and Sauro.)

‍

Try Affinity Diagramming

Affinity diagramming can help you organize your data in a visual way and use it to identify meaningful patterns. This is especially useful if you’re working on user personas, information architecture, or discovery. It’s also a great tool for the many UX researchers that work in teams. In fact, if you’re working on a team you may want to try affinity diagramming in silence— that way no single voice overpowers any others, allowing everyone to interpret the data independently before being influenced by the thoughts of others.

‍

Four Simple Steps to Affinity Diagramming

1. Write all of the qualitative data points on Post-It notes. There will likely be dozens, but that’s okay. Use as many post-it notes as you need for your project.
2. Post all of the Post-Its on to a wall or whiteboard of some sort.
3. Start a timer; we recommend 20 -30 minutes depending on how many post-it notes you have, and organize the data thematically based on what each data point suggests about the different parts of the problem you’re trying to solve.
4. Organize and reorganize into meaningful categories until the timer is up, or until everyone seems to be in agreement. Ideally, at the end of this process, everyone will be able to come to a consensus. In the event of disagreement, even after the timer, hold an organized discussion and make sure everyone is heard.

‍

‍

While these four steps have proven useful to many UX researchers, it’s not the only way to create an affinity diagram or even to use Post-It notes as a means of organization. You may find it helpful to nix the timer, and start the wall of post-its as you collect data, and organize the points into different categories as you go. Different things work for different projects and different teams of researchers!

‍

Find a Software Program to “Code” Your Qualitative Data

You probably have all of these notes, videos, and transcripts from your interviews. That data would also probably take forever to organize and sort through, especially if you want to analyze your data using multiple descriptors.

‍

As mentioned earlier, content analysis usually involves coding the data for certain words, content or patterns. Fortunately, there are special programs that were built for researchers who need to analyze qualitative data regularly.

‍

These programs can help you do things like organize your data sets based on demographics or descriptors, code your data to help with organization and pattern recognition, and collaborate with other researchers on your team.

‍

‍

Some of our favorite programs are Delve, MAXQDA, Condens, and Dedoose. Since every research team and project is unique, we recommend looking into the different software options to determine which program or combination of programs, would best fit your research, team structure, and workflow.

‍

Qualitative Analysis Mistakes to Look Out For

Now that you know how to analyze your qualitative data like a pro, here are a few common mistakes to look out for throughout the process:

‍

1. Presenting a large volume of unassimilated and uncategorized data in an effort to be “perfectly objective”.
2. Withholding initial biases or assumptions about the study focus, and thus misrepresenting the relevant and important data.
3. Over-reduction, or “flattening” of the data collected into close-ended survey responses (i.e. binary “yes or no” questions).

‍

It should be noted that this step does not happen consecutively; it occurs naturally during your data collection and analysis. The actual reduction is properly done when you avoid recording everything that occurs during collection, and instead only note what you feel is most meaningful, usable and important. You are also synthesizing or reducing data by looking for themes and patterns throughout this process.

‍

Synthesizing Your Findings

Regardless of whether you are analyzing your data quantitatively, qualitatively, or both, you will be looking for trends and keeping a count of problems or themes that occurred across participants.

‍

Each theme and finding should be prioritized by severity and importance. You should always go back to the original research objectives at this point – hopefully you took notes about what’s most important for this project to address. Use that overall understanding of project objectives as the backdrop for your data, as you rank the most meaningful patterns, themes, and stories you’ve found thus far.

‍

Writing Recommendations

The last step of your analysis is coming up with recommendations. Final recommendations take your analysis one step further and allow your stakeholders to quickly understand the big takeaway from this project, and take the appropriate actions in response to those findings.

‍

It would be a shame, after all, if stakeholders funded a whole study, and then didn’t do anything with it, because they simply didn’t know what they were supposed to be doing with that information.

‍

Your recommendations might look like these:

• Place the “SHOP NOW” button in the top ⅓ of the page, and use greater color contrast between the button and background, as customers have a hard time locating it.
• Eliminate the error window that pops up during the form, as it is preventing participants from completing the registration process.
• Offer a search bar in the upper right corner of the landing page, as people often look for obscure content that is not relevant for other site users.

‍

Okay, that covers the bases of data analysis; now you’re ready to move onto the next step: constructing a research deliverable!