User Analytics for UX Research

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Note to the reader:

This part of the field guide comes from our 2019 version of the UX Research Field Guide. Updated content for this chapter is coming soon!

Want to know when it's released?
Subscribe to our newsletter!

User analytics gives you insight into how your product is actually being used, out in the wild.

Most product teams are already collecting some form of user analytics, perhaps in the form of passive usage data or website behavior. But what many teams aren’t doing yet is using user analytics data for UX research too! 

Here, we’ll provide concrete, actionable advice for using the user analytics that teams are already collecting to answer user research questions and support continuous (qualitative) research.

In this chapter:

  • What is user behavior analytics?
  • Why does user analytics matter?
  • Benefits and limitations of user analytics
  • How to get started with user analytics for UX research
  • User analytics tools and software

What is user behavior analytics?

User analytics (a.k.a user behavior analytics) is a form of continuous, quantitative data tracking and evaluation that occurs post-launch. Teams use analytics tools to passively collect data about users’ interactions with their product, app, or website. Then, they analyze this data to better understand user engagement and sentiment. 

UX researchers and PwDRs (people who do research who aren’t part of the core UXR team, such as marketers, UX designers, product managers, and engineers) can use user analytics to:

  • Identify issues with the product
  • Investigate hypotheses about design or technical issues
  • Monitor the user journey at key moments like activation
  • Quantify the user experience
  • Target users at specific moments in their journey for user research
  • Persuade data-oriented stakeholders

The difference between user analytics and UX research

User Analytics vs. UX research venn diagram. Under the user analytics bubble: Quantitative, answers questions like "how much" and "how many", large datasets, collected passively and continuously. In the UX research bubble: qualitative, answers questions like "why", smaller datasets, collected through direct observation during a set period of time. In the overlapping section: Reveals issues, opportunities, & other insights to inform decisions - by User Interviews
User analytics vs. UX research

To understand the difference between user analytics and user research, it’s helpful to review the difference between qualitative and quantitative research:

  • Quantitative data is a measure of numerical values and answers questions like “how many” and “how often.” It usually involves collecting large volumes of data indirectly from analytics tools. 
  • Qualitative data is categorical or thematic, taking the form of stories, observations, thoughts, motivations, and feelings, and it answers questions like “why.” Typically, qualitative data is gathered through direct observation of a small group of people.

User analytics is quantitative; it gathers numerical data from large sample sizes, usually passively through tools like Qualtrics or Google Analytics. User research, on the other hand, is qualitative, digging into the reasons and motivations behind user behavior. 

As Yaron Cohen, Senior UX researcher at RBC, says in his article about collaboration between user researchers and analytics specialists:

“Analytics professionals spend their days extracting transactional and behavioral data from databases, and analyzing data that was recorded by electronic systems and not provided by the customers themselves (implicit data)...

UX researchers, on the other hand, work with explicit data provided by customers in the form of surveys, interviews, and usability studies.”

Despite (and because of) their differences, UX research and analytics make a great pair. Used together, insights from each can inform holistic customer personas, pinpoint the causes of churn, and reveal opportunities to improve retention rates, product usage, and conversions. 

Why does user analytics matter?

Quantitative user analytics are important to all organizations, with different departments and functions monitoring what is most relevant to their needs. 

For example, the C-Suite is likely watching top-line metrics, like daily, weekly, and monthly revenue, plus whatever key metrics drive that revenue. Marketers are focused on metrics that drive revenue throughout the entire funnel. Product teams are focused on product usage and user-centric metrics that help drive that usage (and ultimately revenue).

Whoever the person or team, the beauty of rallying around quantitative metrics is everyone can speak the same language in a pretty objective way. In the context of ongoing listening following a product release, you probably have some historical benchmarks to watch post-launch, focusing on the areas you were trying to impact. Mapping user goals, to product goals, to revenue goals, to quantitative metrics is an important way to align goals across an organization.

In other words, user analytics can help you:

  • Align goals and metrics across the organization
  • Understand actual user behavior with the product
  • Improve product design and development
  • Identify and predict trends 
  • Retain and upsell existing customers 

Note that often, pairing quantitative methods with qualitative methods like user interviews can reap the best, most well-rounded results.

Benefits and limitations of user analytics

Here at User Interviews, we’re pretty enthusiastic about all forms of research, including user analytics. However, like any research method, user analytics comes with both pros and cons. 

As our own VP of Analytics, Utsav Kaushish, has said in his article, “I’m a Data Scientist and Here’s Why Quantitative Data Isn’t Enough”:

“I know firsthand how valuable analytics can be for a company; I’ve literally made a living from it. But I also know of its limitations; there are times when you have to dig deeper than possible with a query or a regression model. And often the best way to do this is to simply talk to the people you want to use your product. A truly data-driven organization will complement their analytics with user research, and use learnings from one to power the other.”

Benefits of user analytics include: 

  • Saves time and cost: Because user analytics data is collected passively (i.e. without a researchers’ oversight and independent of a research study), it’s a cost-efficient source of insight for resource-constrained teams. 
  • Reveals real-world user behavior “outside the lab”: A common challenge with research is that users sometimes behave differently than they normally would when they know they’re partaking in a study. User analytics data is collected without users’ knowledge, painting a more accurate picture of their behavior. 
  • Access to large sample sizes: User analytics data, such as web browsing behavior, can be collected by a sizable number of users quickly, easily, and (relatively) cheaply, allowing for statistically significant insights. 
  • Limited researcher bias: Because user analytics is quantitative data collected passively, the data itself can’t be skewed by a human’s perspective or interference. As always, however, the human analysis of this data may be at risk for bias. 
  • Informs prioritization: User analytics can help you decide which bug fixes, product developments, and research projects to prioritize in your roadmap. 

Limitations of user analytics include:

  • A lack of information about user motivations and sentiment: Like most quantitative methods, user analytics doesn’t tell you the “why” behind user behavior, which is important in determining the best way to respond to that behavior. 
  • Risk of making assumptions about the “why” behind user behavior: Because user analytics doesn’t tell you “why” users took certain actions, the person analyzing the data might introduce bias by making assumptions about the “why.” This risk can be mitigated by pairing user analytics with qualitative research (we’ll talk more about this in the next section).
  • Difficulty connecting data with actionable insights: As Jen Cardello said on NN/g: “The biggest issue with analytics is that it can very quickly become a distracting black hole of ‘interesting’ data without any actionable insight.”
  • Large amounts of data: While large sample sizes allow you to achieve statistical significance, they may also make it difficult to differentiate between meaningful data and noise. 
  • Many analytics systems are not purpose-built for user researchers: Instead, these systems are typically built for marketers, product people, and designers—but this is one of the reasons we believe it’s valuable for UXR to work with these teams!

How to get started with user analytics for UX research

You’re probably already collecting some form of user behavior data at your organization, in which case, some of these steps will be review for you:

  1. Start with user goals.
  2. Determine the analytics metrics you want to track.
  3. Set up a system for measuring analytics.
  4. Set a cadence for reviewing analytics.
  5. Identify trends, user segments, and other patterns in the data.
  6. Conduct formalized UX research to explore trends, questions, and opportunities.

However, let’s start from the beginning for folks who want a refresher.

1. Start with user goals.

Begin by understanding your user’s goals. 

For example, if you’ve created an experience in a fitness app for someone who wants to get lean, you need to understand which actions are the most important for meeting this goal, then track the ease with which folks can complete those actions.‍ 

So, how do you know when someone’s having trouble? Sometimes users give you feedback directly through a survey or support interaction. Other times the writing is on the wall in the form of quantitative analytics—like Google Analytics for web, Mixpanel for product events, or other tools in your analytics stack.

2. Determine the analytics metrics you want to track.

Depending on the goals of your product launch or feature updates, you may want to focus on some of the common quantitative metrics below. 

Survey analytics

Fictional NPS data over time via Promoter.io

NPS, CSAT, or CES scores

These survey metrics give you an idea of how your customers feel about your company and particular touchpoints with your company, such as the support experience. 

If these scores are changing (positively or negatively) following a launch, this can be an indication of how your changes are being received. 

Drilling into your data with a focus on key segments and cohorts, and validating with further data will help you uncover insights.

Qualitative survey data

Often the above surveys include a free-form question that can help you understand some of the whys, the motivations behind the positive or negative scores. 

As you combine quantitative data with qualitative and segment it by meaningful customer groups, you should start to form hypotheses you can then validate through testing and further research.

Product analytics

Example of retention data from Mixpanel

  • Feature use: Which features are used and the most? Are your new features, or updates, getting used?
  • Recency and frequency: How recently did someone use your product? How frequently do they return?
  • Value of use: Are people who use a given feature more valuable, happy, or otherwise positively impacted by using it?

Website analytics

High level web analytics data from Google Analytics

  • Time on site: This can be a great area to drill into, especially for a content driven experience, or one where the revenue model is closely tied to visit duration.
  • Visits: How many people visited your experience overall? How are visits changing over time? More isn’t always more, but all things being equal, it is.
  • Unique visitors: How many different users are interacting with the experience?
  • Goal completion: From leads generated, to purchases completed, to buttons clicked, if you can tag it, you can track it.
  • Pages visited: Which pages did an individual or group of individuals visit? Are those pages connected to key buyer/user journeys?
  • Traffic source: How did the user enter your experience? Was it from a certain campaign, organic traffic, or another source?
  • Path to conversion: Before converting, which pages do users visit? How long do they stay on site? Is this path as direct as possible? Is there an opportunity to improve your site navigation? Evaluate across different personas, lifecycle stages, or user stories. Not every user has the same goals.

‍By understanding and tracking your baseline against the metrics that matter to your goals and your users’ goals, you’ll quickly notice where a product update or launch is having an impact.

3. Set up a system for measuring analytics.

Once you've determined the metrics you need to track, audit the user analytics systems teams are already using and look for any gaps in capabilities. 

Some user researchers build out custom dashboards, integrating Google Analytics with Microsoft Excel, or taking advantage of one of the many business intelligence tools now available. 

Tracking data doesn’t have to be a heavy lift. We recommend automating as much as possible your dashboards to streamline your workflows. This way, you can spend less time collecting data and more time analyzing and acting on it. Make sure to share your dashboards with other stakeholders too!

A good dashboard:

  • Prominently highlights the most important data
  • Illustrates change, anomalies, data worth noticing
  • Is connected to deeper data for drilling where necessary to understand what’s “really going on”

4. Set a cadence for reviewing analytics.

Perhaps you have a weekly department meeting where key metrics review is a recurring agenda item. 

Or maybe you have a monthly OKR meeting. 

Or maybe you are the lead researcher on a particular project you’re very invested in, and you need to know what’s happening by the day or hour on something that has just launched. 

Depending on your individual, team, or company situation, set up a regular cadence for reviewing the metrics that matter to you the most. Of course, it may make sense to review different metrics at a different cadence. 

We recommend a weekly check-in of your top level key metrics at a minimum, and then build from there based on your needs. Give yourself a recurring task or calendar event to make this ongoing review a habit or take advantage of automated email reports for your services that offer them.

5. Identify trends, user segments, and other patterns in the data.

As you’re reviewing your user analytics data, you may uncover patterns and other points of interest that can inform, spark, and eliminate the need for future qualitative studies.

Be sure to:

  • Look for points where users drop off. These drop-off points often signal areas of confusion or other UX issues. Make note of them and explore them later with UX research. 
  • Identify the most popular and least popular features. The most-used features tend to be those that allow users to perform intended actions as simply as possible, while the least-used features tend to have poor usability or low value to users. Dig deeper into why users do or don’t use these features in future studies. 
  • Segment user personas. Do different types of users interact with the product differently? If these groups are distinct enough, your team may benefit from using this data to inform user personas. 
  • Outline user journeys. What path (or paths) do users take as they’re using the product? Can any steps be removed to make the process smoother for users? 
  • Benchmark behavioral patterns to understand change. As NN/g’s Kate Moran talks about in this Awkward Silences episode, analytics data can help you benchmark current usage and behavioral patterns to keep track of changes in those patterns over time. 
  • Develop hypotheses regarding user motivations and product improvements. Analytics data can only tell you what users did—it can’t tell you why they did it. Using what you already know about customers, develop hypotheses about their motivations and evaluate the validity of these hypotheses with qualitative research. 

6. Conduct formalized UX research to explore trends, questions, and opportunities.

User analytics data will likely leave you with a list of unconfirmed hypotheses about product improvements and unanswered questions about user motivations. 

Qualitative research comes in to answer those questions. AP Intego’s Cat Anderson explains this interplay between quantitative and qualitative data on Awkward Silences:

“We use quantitative to inform our qualitative round. An example of that is when we draw from FullStory, the platform where you can record the screen as somebody's using your app or your website…. 

After watching hundreds of people struggle on this one interaction, we were then able to design a qualitative research round that really put it into a more holistic perspective and context for us: Why? Why is this a difficult step for you? Oh, well, it's because of all these other things that are happening around it in a small business owner's life or in a small business owner's journey. 

That was an example of… the quantitative informing the qualitative. And then, there's just a whole lot of complex interplay between those moving forward as we do new rounds of research.” 

User behavior analytics tools and software

If you’re already collecting user analytics data at your organization, you might already be familiar with the most popular user behavior analytics software, such as:

🏰 ✨ To explore these and other user analytics tools in more depth, check out the 2022 UX Research Tools Map, a fantastical guide to the user research software landscape, with 230+ tools for recruiting, usability testing, surveys, moderated sessions, analysis, and more.

In a nutshell

User analytics and UX research make a great team.

By analyzing quantitative user analytics data—and pairing it with qualitative data from UX research—you can identify areas for improvement, create exceptional user experiences, and ultimately increase retention and revenue. 

Start doing customer research today

Try Research Hub free

Subscribe
X

Master UX Research

Get the full course and exclusive pro tips delivered straight to your inbox. (New content dropping in 2022)!
Thanks! See you in your inbox 👋
Oops! Something went wrong while submitting the form.