All posts
UX Research Topics
Podcasts
Field Guide
SUBSCRIBE TO OUR NEWSLETTER
FOLLOW US
How are researchers using AI? Insights from a survey about the tools, use cases, ethical concerns, and guardrails for AI in UX research.
%20(1).png)
According to the 2023 State of User Research Report, 20% of researchers are currently using artificial intelligence for research, with another 38% planning to join them in the future.
With the majority of researchers either using or planning to use AI at work (as of May 2023, when the State of User Research survey was conducted), we wanted to learn more about the role of this new technology in our industry.
So in August 2023, we sent out a short survey to learn:
In this article, you’ll find an overview of our discoveries about the role of AI in UX research, based on 1,000+ survey responses from User Researchers, ReOps Specialists, and people who do research (PwDRs).
The AI in UX Research Report was created by Product Education Manager (former Content Marketing Manager) Lizzy Burnam, with strategic guidance and support from Lead UX Researcher Morgan Mullen, Senior Data Scientist Hayley Johnson, Career Coach and UX Research Leader Roberta Dombrowski, Head of Creative Content & Special Projects Katryna Balboni.
✍️🎨This report was authored by Lizzy, and brought to life by Illustrator Olivia Whitworth.
We used SurveyMonkey to create the survey (with 3 screener questions, 10 single- or multi-select questions and 2 open-response questions, to get a fair mix of quant and qual data) and distributed it to User Interviews audiences via our social media pages, our weekly newsletter, and a banner on our website. We also shared the survey in relevant UX research communities on Linkedin, Facebook, and Slack.
Between August 14th and 18th of 2023, we collected 1093 qualified responses from User Researchers, Research Ops Specialists, and people who do research (PwDRs) as part of their jobs. (An additional 1091 people took our screener but did not qualify for our survey based on their responses.)
In most of the questions, we gave participants the option to write-in an “other” response. Where applicable, we omitted redundant answers (e.g. when the response would be covered by a future question) or merged them with existing categories. For the two open-response questions, we eliminated exact duplicates and comments without meaning (e.g. “asdfghjk”).
More than half of our audience self-identified as User Researchers (30.4%) and Product/UX Designers (20.8%). Only 5.9% of our audience identified as Research Ops professionals—we were surprised by this relatively low response rate, since ReOps is typically heavily involved in exploring the value of new technologies like AI for their team. But perhaps we just weren’t as successful in reaching this audience.
Write-ins for the “other” category included Founders/CEOs, Content Designers/Strategists, Developers, and CX/UX Strategists or Consultants.
Our main takeaways from the survey include:
We’ll dive deeper into these takeaways below.
Obviously, not everyone’s using AI in their research—but among those who are, how often are they using it?
Note: There is probably some self-selection bias in our data for this question; because our survey was focused on AI in UX research, we likely attracted a higher proportion of folks who are open to using AI in their research than would be present in a general population of researchers.
That said, here’s what our audience said about how often they use AI.
The majority (77.1%) of our audience said they’re using AI in at least some of their research projects, while a minority (22.9%) said they’ve tried AI but don’t regularly use it in their work.
An additional 8.0% said they have never used AI. Since our goal with this survey was to learn more about how researchers are using AI, we filtered these folks out for the rest of the questions.
If you’ve reviewed your tech stack recently, you’ll notice that many UX research tools have robust AI features built into their platforms. In our survey, we distinguished between these UXR tools and AI-specific tools because some folks may use the AI features in these popular UXR tools (either knowingly or unknowingly), and we wanted to capture this usage as well as the intentional use of AI-only tools.
More than 10% of our audience said they’re using the AI features in the following tools:
.jpg)
A small portion of our audience (4.2%) said they aren’t using any of these tools. About 2.8% of folks provided write-in responses for tools such as Miro, Zoom, Great Question, Google/Microsoft, Condens, Trint, Notably, and Grammarly.
✉️ P.S.—If you’re interested in UX research tools, then you should know we’ll be publishing a 2023 UX Research Software & Tooling Report very soon. Subscribe to our newsletter, Fresh Views, to be notified when that report drops!
As one of the newest and most advanced AI tools, ChatGPT has garnered attention in almost every industry, from academia and advertising to entertainment and education (and even US courtrooms!).
Given this widespread buzz, we weren’t surprised to see that the majority (51.1%) of our audience said they use ChatGPT in their research. That’s more than 3x the number who said they use ClientZen, the second-most popular AI-specific tool in our survey, which is used by just under 15% of our audience.
A little over 5% of our audience wrote-in additional tools. Common write-ins included Bard, CoNote, Claude, Azure, TL;DV, Fireflies, RewatchAI, and Read AI.
Notably, 5 write-ins said they’re using internally-built/proprietary AI tools. In a later question, some people listed proprietary tools as a guardrail for safe, ethical use of AI in UX research, since they don’t allow public access to the tools or data.
4.1% of our audience said they aren’t using any of these tools. And despite the popularity of ChatGPT among our audience, some responses to our later question about AI’s limitations noted serious concerns about the role of ChatGPT (and language learning models in general) in UX research:
“I am very skeptical about the role that AI should directly play in research because it violates the core epistemologies (especially reproducibility) that underlie qualitative methods[…] LLMs are probabilistic agents and the fact that the same dataset can get a different analysis each run through the model makes it an inappropriate tool for the job.”
📚 More reading: 20+ Powerful AI-Based Tools for UX Research Toolkits in 2023
Recruiting-related tasks are notoriously one of the most painful aspects of a researcher’s job. In fact, 97% of researchers reported experiencing challenges during recruitment in the 2023 State of User Research Report.
(👋 That’s why User Interviews exists: to connect researchers with quality participants quickly, easily, and affordably).
Where there’s pain, there’s potential for the right technology to provide relief—but is AI the right tool for the job? Let’s look at what the data said about AI’s impact on recruiting.
According to our survey, about a third of researchers say they use AI for signature collection, scheduling, and screening applicants.
(⏸️ Speaking of scheduling—did you know that User Interviews just revamped our scheduling feature for moderated research?)
In general, though, it seems like AI hasn’t influenced recruitment quite as much as the other aspects of research; the percentage of folks who said they do not use AI for recruitment (24.2%) was significantly higher than those who said the same for conducting research (4.7%), analysis (8.8%), or synthesis (9.6%).
There are different potential reasons for this—maybe folks don’t trust AI for the delicate tasks of recruiting, screening, and approving applicants. Or maybe there just aren’t as many well-developed AI-powered solutions for recruiting.
Whatever the reason, we know from the 2023 State of User Research Report that researchers use multiple methods to recruit participants—on average, 3 methods for recruiting customers and 2 methods for recruiting external participants. Perhaps, in the future, AI might emerge as another top recruiting method, but for now, the most popular methods include email, intercept surveys and self-serve recruiting tools like User Interviews.
We also got a handful (0.8% of our audience) of write-ins. Most of these mentioned using AI for writing/editing recruitment assets and different aspects of panel management, such as segmenting audiences and updating participant data.
📚 More reading: 17 Research Recruiting and Panel Management Tools
Effective note-taking during research sessions is tough, especially when you’re trying to be an attentive listener and moderator at the same time. It seems that this is one area where AI has offered a lot of help, with 47.8% of our audience saying they use AI for transcription and 40.8% saying they use it for note-taking.
2.5% of our audience also wrote in other ways they use AI to conduct research, including:
Qualitative coding is one of the most complex and time-consuming types of UX research analysis—so it’s no surprise that a plurality (40.4%) of our audience is using AI to streamline the process.
However, it sounds like AI’s analysis capabilities still leave something to be desired. A handful (0.5%) of write-ins said they’ve experimented with AI for analysis but found it wasn’t very helpful, or that they could do it more quickly and accurately on their own. Although the number of such responses is small, these sentiments align with many of the responses we received about AI’s shortcomings in a later question.
For example, one respondent said:
“I tried qualitative coding and it was a disaster. Worse - it looked right for the first 10 minutes and then turned out to be very, very wrong.”
Given our audience’s high self-reported usage of ChatGPT—a large language model known for its ability to generate smart, textual responses to prompts—it may come as no surprise that a large percentage of our audience (45.5%) reported using AI to help them write reports.
A small percentage of our audience (0.5%) used the “Other” field to tell us they are using AI for editing, data clustering/mapping, and summarization.
📚 More reading: 31 Creative UX Research Presentations and Reports – Templates and Examples
We’ve learned that researchers are using AI in their projects, and we’ve explored the different areas of research that they’re using AI for. Now, let’s uncover why they’re using AI—what benefits does it offer?
In open-response comments describing the benefits of AI, the terms “efficiency,” "time savings," "doing more with less," and related phrases came up the most often (in 28.4% of comments). For example, one person described valuable time savings for their research team of one:
“As the only UX'er in my company, my research projects often span a longer timeline than I'd like (out of necessity). Plugging my session notes into ChatGPT to quickly generate commonalities or using Notion to summarize session transcripts inline gives me a good starting point to dig into my analysis.”
Folks also had a lot to say about AI’s creative support, with one researcher describing themselves as having a “thought partner relationship” with AI. AI seems to help many researchers “just get started” when tackling a blank page:
“If I'm putting off getting started on drafting a research plan, recruitment email, or coding data I'll let AI take a stab at it because it spits out stuff instantly! Then I'm editing instead of generating which feels easier to get started. Most of the time I have to do 95% of the work over, but getting that start is valuable. It also helps me check my initial ideas by [comparing them with] what AI thinks.”
Others said that AI helped them bridge various types of communication gaps, from communicating across cultures and languages to collaborating across teams:
“By far the most valuable part of AI tools for me is being able to translate terminology and requirements from stakeholder language to researcher language, and back again, to help bridge communication gaps.”
This same sentiment was echoed by another researcher, who also said that AI-based insights seem to improve their perceived credibility among stakeholders:
“Stakeholders also tend to believe [my analysis] more if the AI finds the trends over the humans, which is a bit annoying as a researcher, but is gratifying to see the work being accepted by the client.”
Despite these reported benefits, almost everyone seems to agree that AI is not perfect. Many people were careful to include qualifiers about the extent of these benefits and their potentially hidden costs. For example, some researchers mentioned the need to check AI outputs and the dangers of assuming that AI is correct:
“AI tools (LLMs mostly) are useful at processing large volumes of words for things like sentiment analysis and other forms of pattern detection. I've tested a couple of tools that purport to do this with research material but have not had a great deal of success. The tools I've tried have been incomplete and inaccurate to a degree that I question whether I'm saving any time or effort.”
Let’s dive deeper into the limitations of AI in the next section.
The researchers in our audience listed increased efficiency as the top benefit of AI. But does this efficiency come at too great a cost?
In an open-response question about the limitations of AI, many people noted that AI’s capabilities are often very low-quality, spitting out inaccurate, surface-level, or incomplete outputs.
Here’s a handful of the words and phrases that participants used to describe AI’s outputs:
One respondent said:
“Accuracy is the number one issue. AI doesn't know the context and it can only get you 70% of the way there. My fear is researchers will fail to understand the why behind their craft if they lean too heavily on AI to do the work for them. Research is not always black and white and you often have to look beyond the face value of things.”
Many people also said that AI needs so much human review (described by one respondent as “hand holding”) to catch errors that its limitations undermine any efficiency gains. Others noted that, although human review may be considered a limitation, it’s also par for the course; “the tools were never meant to function by themselves.”
Several comments also mentioned the danger of inexperienced researchers (such as junior UX researchers or people who do research outside of the dedicated UX research team) putting too much trust in AI, leading to false insights and misguided decision-making.
For example, one person said:
“It's too easy to accept what the AI has suggested as the right answer — especially working with junior colleagues and interns who may not disclose that they're using AI to generate insights and synthesize data. Everyone has to be more critical, and that ends up creating more friction on the team.”
Ethical concerns such as biased outputs and negative socioeconomic outcomes (e.g. discrimination and unemployment) were also a recurring theme in the responses. One person said:
“The inherent gender and racial bias we've encountered in anything more than summarizations or decisions made with strict parameters are absolutely a deterrent to using [AI] for anything more ‘cutting edge’.”
In general, the researchers in our audience seemed to have very low trust in AI, and many wished AI solution providers were more transparent about their data protection policies (or lack thereof). One researcher said:
“AI tools are not transparent enough about what is happening with the data. We need to know EXACTLY where our data is going (who owns the server that the data is being transferred to, ex AWS, Azure, GCP). This is not immediately transparent when interacting with the feature itself. We do not have the bandwidth to engage reps from each of these companies to gather these answers, and sometimes they don't know themselves [...] Making this transparent would greatly alleviate many of our issues.
Others mentioned a desire for specific guidance on how to use AI effectively without risking user data, describing a “learning curve” that limits their ability to try new tools.
A very small portion (2%) of our audience felt that there were no limitations, or that they hadn't used AI enough to notice any.
It’s safe to say that folks have concerns about AI.
Anecdotally, you don’t have to search for long on the internet to find opinion articles, social media posts, TikTok videos, and other examples of people raising the alarm about the potential implications of AI on how we work, live, and evolve. Indeed, only 0.5% of our audience (3 people!) said they had no concerns regarding AI in UX research.
When we asked researchers to select their primary concern about AI in UX research from a list of options, we found that for a plurality of researchers (29.7%), the #1 concern is the potential for inaccurate and/or incomplete analyses. After inaccuracy, the biggest concerns were a lack of data privacy (19.7%) and the potential for introducing bias into study findings (14.1%).
Folks seem least concerned about the prospect of AI eliminating the need for human researchers (only 6.4% selected this as their biggest concern). This may be due to a lack of trust in AI’s current capabilities; in our open-response questions, many folks said they wouldn’t rely on AI without taking the time to manually review its outputs. As one respondent said:
“AI will never replace human centered research. The 'human' part has to go both ways. I don't think AI can truly catch all the tiny aspects that go into a user experience and synthesize them in a way that makes sense or is helpful/accurate. I see it as a tool to help us save time on tedious tasks, but not to be able to do research on its own.”
Note that in order to get a sense of people’s most pressing concerns regarding AI, we required participants to choose only one response to this question. However, 8 participants couldn’t choose, explaining in write-in responses that they were concerned about all of the above. As one response said:
“I can't pick a single primary concern. I have several. And they all matter.”
Whether you’re worried about it or not, AI is now an intrinsic part of work-life. If researchers are going to use AI at work—regardless of its existing limitations—it stands to reason that there should be guidelines and guardrails in place to ensure the safest, most effective use possible.
So, what kind of steps are researchers taking to mitigate their concerns about AI and its shortcomings?
Nearly half (49%) of our audience said they limit the type of data they're using with AI tools. A large segment (38.8%) also said they ensure human review of all AI outputs, while many others (35.5%) said they collaborate with their legal team to stay compliant with AI laws and regulations.
In some cases, researchers felt that these guardrails may be interfering with AI’s potential to provide true benefit to research teams. As one researcher noted in an open response to a later question about limitations:
“The AI Summaries feature could help our team generate high level insights for informal debriefs... However, we currently can't leverage this technology because of corporate guardrails in place limiting our use of AI.”
Of the write-in responses we received (1.8%), several said they avoid using AI in their research entirely, as it’s “the best guardrail there is.” A handful of other folks said they’re still working to determine the right guardrails with their team.
Of all the folks who participated in our survey, dedicated UX Researchers seem to be the most cautious about AI. This may not come as a surprise, since UX Researchers are typically trained to apply higher levels of rigor to their research than folks who conduct research as an adjacent part of their job.
When we segmented the data by role, Customer Experience/Support professionals were the most likely to say they use AI in almost all of their research projects (40%), followed by ReOps Specialists (19.4%) and Marketers (17%). UXRs were the most likely to say they've tried AI but avoid using it in their research (44.3%).
UX Researchers were also the least likely to use AI in their projects across all aspects of research. For example, 11.5% of UX Researchers said they’re not using AI to conduct research, compared to less than 4% in each of the other segments.
The UXRs who said they are using AI to conduct research are mostly (60.7%) using it for transcription.
The UX Researchers in our audience were also the least likely to have tried AI tools and UXR tools with AI-based features. These results were consistent across all of the tools and use case related questions, which may indicate higher rates of trepidation about AI among folks in the UXR segment.
All in all, our clearest takeaway from this survey is that researchers definitely have mixed feelings about the (mostly) uncharted territory of AI.
These are just some of the complex issues and contradictions that researchers brought up in their survey responses.
At the end of the day, it seems that there’s no quick or clear solution for reaping AI’s benefits without risk. As one respondent said:
“AI that automates decisions can scale both benefits and harm.”
So, while we learned a lot from this survey, the results have also opened up a whole new wave of questions for us, including:
We’ll be chewing on these questions as the UXR industry and AI technology continues to evolve. In the meantime, we’d love to hear from you—what are your thoughts about this report? What questions do you still have? Tag us on Linkedin or Twitter to share your feedback, or email the author directly at lizzy@userinterviews.com.
If you found this report valuable, then you’ll probably like these resources too:
✉️Sign up for our newsletter to be the first to know when we publish new research about the who, what, how, and why of UX research.
Product Education Manager
Marketer, writer, poet. Lizzy likes hiking, people-watching, thrift shopping, learning and sharing ideas. Her happiest memory is sitting on the shore of Lake Champlain in the summer of 2020, eating a clementine.
