It will take years to fully appreciate the way the coronavirus pandemic has disrupted lives and livelihoods. We don’t have the benefit of hindsight yet; we’re still very much in the middle of this thing.
Yet user research has carried on throughout it all. And in many ways, user insights are more relevant and important than ever. Compared to many other fields, UXers and other tech workers have been relatively lucky—according to an August 2020 Glassdoor report:
“Tech job openings increased 13.4 percent month-over-month. This large increase is driven primarily by shifting consumer demand for e-commerce and other online services.”
But that doesn’t mean there haven’t been major changes to the way researchers do research. We wanted to understand exactly how COVID-19 (and all its ripple effects) has impacted UX researchers and the way they work, as those changes stand today.
So, we did what all good researchers do: we talked to people.
Our methodology
In November, we interviewed six UX researchers from the U.S. and U.K. Our participants included:
- A user researcher at a UK-based housing association. Their team was previously fully in-person, and their customers include older, less tech-savvy folks who have struggled with remote research methods.
- The sole UX researcher at an NYC-based fintech startup. They started at the company in January; in October, they also launched their own independent research agency. Some members of their team were remote before COVID.
- The head of research and experience management at a healthcare startup. Their team was predominantly in-person before, with offices in Boston and India, but they have been working remotely for years.
- The lead UX researcher on the market research insights team of a large, CA-based company that makes digital and physical products. They joined the company in August of this year.
- The VP of user experience research at a multinational investment banking company headquartered in NYC. They lead a team of six working on digital customer experiences for the company’s customer-facing digital properties.
- The sole UX researcher on the safety team at a CA-based ridesharing company. Their company has around 30 researchers, spread across different teams. Even before COVID, most of their research was remote in an effort to get participant perspectives outside the SFO area.
We spoke with each researcher for 20 to 30 minutes (over Zoom, of course), and asked them questions like:
- How has your work environment changed as a result of COVID-19 and the economy?
- How (if at all) have you had to rethink your research in light of COVID-19 and the economy?
- What has been your single biggest research pain this year?
- Are there certain changes that you’ve made in response to COVID-19 that you foresee retaining in the future?
Here’s what they had to say. (Responses have been edited for clarity.)
How has your day-to-day work changed as a result of COVID-19?
All of the researchers we spoke with said that their company had gone remote as a result of COVID-19. Some of the researchers we spoke to were working (mostly) remotely prior to the pandemic; but for many, going remote marked a huge shift in their work environment.
The shift to remote work was more painful for some teams than others
Unsurprisingly, researchers who regularly worked remotely themselves prior to COVID—or who had teammates that did—found the shift to remote less jarring than folks who were thrown into a home office environment for the first time.
One researcher, who frequently collaborated with a remote colleague even before the pandemic, said that it took a while to get used to having work around the house. For the first three to four months, they were “super super super busy,” working more than normal. They went from never bringing their computer home before COVID, to checking Slack at 7pm, email at 9pm, etc.
“For the first few months, team productivity was very high. But it’s just becoming harder and harder to keep the energy up.”
Several researchers cited sharing a space with roommates and significant others as a major pain point. Many felt the line between work and home life has become nonexistent. “There’s no escaping it in a small apartment,” said one researcher. “I’m working in my bedroom, which is NOT where you should be working.”
“It’s hard to separate a ‘work’ mindset from a non-work mindset,” said another who is living with their partner. This means work hours often bleed into evenings, and this researcher feels more guilt around taking breaks from work than they did back at the office.
Some companies were making an effort to make remote life more enjoyable. One person we spoke to said their company was picking up the bill for home wifi (up to $50/month) and covering the cost of occasional (socially distanced) team meetups.
The way UX researchers collaborate and communicate has changed
Almost everyone we spoke to said that team collaboration and communication has changed over the past year, and rarely for the better.
Again, teams with prior remote experience had fewer growing pains, since collaboration tools were already widely used before the pandemic—”we mostly just had to add more people to the paid Zoom plan,” said the researcher who works on an international team distributed between the US and India.
But others, like our UK-based participant, felt that “collaboration has been an absolute nightmare” and “it’s really easy for someone to dominate a meeting.” This researcher’s company had been fully in-office before the pandemic, and they shared that while the digital team was comfortable collaborating and communicating in tools like Miro, stakeholders in upper management and sales hadn’t taken to the shift in technology as naturally.
On the other end of the spectrum, some teams had almost too many collaboration tools: One researcher, whose company and its subsidiaries include thousands of employees, said that the lack of a single, centralized communication platform meant that they had to be in many places at once in order to stay on top of conversations. At their company, UXers and PMs mostly used Slack, while the personal market research team uses Teams predominantly. Other people in the company mostly collaborate in Google. “I’m on 3 different tools at any given time.”
Despite all those tools (or, more likely, because of them), clear and effective communication has been challenging. Researchers said they were more aware of the way they communicate, both with each other and with research participants. “We’re trying to create communication standards so we all communicate with each other [more effectively]. If someone says something, we say it back in their own words, as a team.”
📖 Read more: 6 Unexpected Ways COVID-19 Has Changed UX
Meeting fatigue is real
Aside from one lucky researcher who said their team was actually having fewer meetings since going remote, most of the people we spoke to felt that they had far too many Zoom links on their calendars these days. We have “crazy amounts of meetings, unfortunately, to make sure the sync up is there” lamented the researcher juggling Teams/Slack/Google.
The head of user research on the Boston/India team said:
“[I’ve] noticed lots of meeting overload, where every little thing gets [turned into] a meeting instead of just chatting in person or grabbing a coffee with someone. All those things now need explicitly meetings—and then there are more [meetings] on top of that to make sure everyone is included.”
Even as meetings increased, many researchers felt that people had grown more siloed and isolated from the broader team as the pandemic dragged on. One managing researcher we spoke to said that to maintain relationships and keep communication flowing, they’ve had to be more deliberate about scheduling catch ups with their team.
Budget cuts, hiring freezes, and furloughs
Half of the researchers we spoke with told us that budgets, salaries, and hiring had been impacted as a result of COVID-19 and the economy.
For some, this impact was acute. One researcher shared that everyone at their startup had taken pay cuts early in the pandemic—and that their salaries had yet to be restored. The same company has greatly reduced hiring. “We’re lucky that we’re funded for four more years, but we’ve had to put revenue goals on hold to keep everything afloat.”
Another researcher, who joined their company in August, said that while the budgets and hiring on their own team were unaffected (the money was already set aside at the beginning of the year) other teams at their company had experienced significant cuts. To keep on as many employees as possible, everyone in the company was taking a one-week furlough.
Less dramatically, a third researcher said that while their “pretty robust” research budget is technically unchanged, it’s become more difficult to access. Now, they need approval from the top for every little project.
Not everyone felt these pains. “We’re in housing, so there’s plenty of need,” said one researcher. Instead of budgetary changes, their company was experiencing a shift in focus as user behaviors evolved (fewer people are interested in buying in the city these days, they explained).
Uncertain timelines makes planning difficult
Many researchers also said they felt uncertainty about the future of their research. For some, the pain was light—“we are still following our roadmap … but some timelines have shifted”—while other researchers felt like everything was up in the air, thanks to tentative schedules that are continuously pushed back.
The ride share safety researcher we spoke with shared that while their own roadmap was relatively unchanged, other UX teams at their company had needed to make a hard pivot. Those teams are now working on something entirely different, because their original work (shared rides) was simply no longer relevant in the COVID era.
Others said that the pandemic-induced uncertainty had already resolved itself. When asked if their work was impacted by increased uncertainty, the researcher at the NYC fintech startup replied:
“Not really. But if you’d asked me this in March or April my answer would’ve been yes, because no one really knew what was going on. From March through July, the company was kind of frozen… but that is phasing out.”
How has your research been impacted by COVID-19?
We wanted to know how UX researchers were adapting their methods to remote-only research, how their users had changed, and how teams felt about it all.
Remote research has its pros and cons
One thing all our researchers had in common: they did lots of remote research in 2020.
Some teams had a hard time making the shift. The researcher who works at a housing association explained that getting older people involved in remote research has been really challenging. Generally “in-person research is easier than anything involving screen sharing.”
Early in the pandemic they tried lots of different tools—Google Meet, Join.me, etc—trying to find a good tool that doesn’t require a mobile app download. In the end, Teams and Zoom won out; still, “we spend over half the [research session] time teaching people how to use tools.”
Those technical challenges can skew participant pools toward younger, more digitally inclined customers.
“Because of [the technology barrier] the input tends to be biased toward those who are technical, and sometimes excludes those who aren't.”
Since COVID, this researcher has been talking to a younger crowd overall, which isn’t fully representative of their core customer base.
For others, the shift was less painful. Teams that already did a mix of remote and in-person research had a much easier time adapting.
Although they had to make some adjustments—like padding timelines in case something goes wrong in the remote process, and tweaking screener questions to avoid misinterpretation—the UX researcher on the market research insights team said:
“The transition into the actual user research that we do actively has been pretty seamless, because everyone's pretty familiar with the remote testing tools by now.”
In fact, this person’s team is expected to stick with remote testing in 2021. In planning for the year ahead, they picked up additional Zoom licenses and expanded their use of platforms like Userlytics and UserTesting (our “quick and dirty go-to”) for moderated and unmoderated research.
Other researchers talked about ramping up remote recruiting in 2020, relying more heavily on platforms like User Interviews to find participants.
The investment banking research VP we spoke with talked about how this shift had sped up their research, and increased their access to qualitative insights:
“When we were doing in-person qual work, we were dealing with legacy people recruiters to recruit people with ultra high net worth. And those recruits would take anywhere from two to four weeks to assemble before we could kick off our work, so the lead time for a given study was huge. I would potentially have an intake meeting with product and design teams five to six weeks out from the research. Now that we’re using platforms that can put together recruits digitally in a day, I no longer need to maintain such long lead times.”
They were also excited about the diversity of perspectives remote recruiting made available:
“The other thing I really like about remote recruiting is that the base that recruiters is pulling from is a lot broader. That helps us meet our new diversity and inclusion mandate much more easily… We had previously tried doing this with a legacy recruiter. They were unable to find [investors with diverse gender identities] and barely able to meet our ethnic and racial diversity mandates. Now with some of our remote recruit vendors, I can fill a recruit of, say, 10 trans/non-binary investors investing at over 500k in real estate in 10 hours. It’s just mind-blowing how available [different] perspectives are. It’s terrible to say, but [in this respect] the pandemic has been a net benefit.”
User behavior has changed
User needs and behaviors have changed. This was consistent feedback, even among researchers who felt their own work habits were relatively unaffected by the pandemic.
Some cited disruption to supply chains, the shift to online life, and even the migration to less urbanized areas as new considerations that have arisen or been made more pressing due to COVID.
One researcher, who just launched an independent research agency in October, observed that business customers are being more conservative with their research investments—many potential clients are waiting to close a round of funding before they commit to a research budget.
For the researcher who works in healthtech, COVID has dramatically changed who they serve. “Elective surgery has been put on hold, so no one is getting the surgeries” they would normally be following up about. Their company has shifted focus to COVID patients in recovery, helping patients in India recover and escalating their care to hospitals if need be.
Similarly, the ride sharing UX researcher said the pandemic has changed who they can talk to. The pool of people who took a ride a week ago has shrunk significantly. And people’s reasons for riding have changed, often not positively.
“Normally when I speak to people, I ask them about their typical habits. But right now “typical” is just out the window… [there’s also the] weirdness of talking about things that happened pre-COVID. I don’t know what’s relevant anymore to a certain extent, because the world has changed.”
This researcher was trying to be hyper-aware of context. They were cautious about sending too many emails or bombarding people with too much communication. When they do send emails, they’ve had to be super clear about how they communicate. Since they work on the ride safety team, their research involves asking people about safety. But because COVID is so top of mind, many people automatically assume they mean health safety. The virus is an inescapable part of the conversation.
What has been your single biggest pain point this year?
When we asked researchers what their single biggest pain point has been this year, many had a hard time picking just one thing. For many, pain points overlapped and compounded each other, making things that might have been manageable in a pre-COVID world feel more challenging.
Recruiting the right participants is an ongoing challenge
Research recruiting was a common pain point, although not one unique to COVID. Researchers talked about reduced participant pools, and the challenge of getting participants to commit to a session.
“Recruitment is always the single biggest pain point. Getting the right people for the research. It’s unreliable, you never know if people will show—there’s lots of hours put into something and you can’t always have something to show for it. [COVID] has kind of reduced possible different options for that. We had ideas for doing in-person testing with other orgs, which isn’t possible anymore.”
For some products, in-person testing is irreplaceable
The most frequent response to this question was that, although remote research has a lot to offer, in some cases there’s just no substitute for in-person moderated testing.
When it comes to physical products, getting quality qualitative insights is hard to do remotely. One researcher explained that, currently, they ship physical products to participants and have them fill out a survey. “It’s super generic; I don’t think that is the best way to solicit that type of feedback.”
Their team has tried to get creative. Recently, their colleague tried to replicate a popular in-person event online. It required a lot of logistics and planning—including arranging breakout rooms for friends who come together and then having everyone come back into the main room for demos. Not everything went smoothly: The team planned to give promo codes to participants as a “thank you,” but those codes got sent out early. Many people didn’t show up to the event because they’d already gotten a discount.
Other researchers talked about the challenges of running quality mobile research remotely. Available tools make it possible to test for flow, but with no moderator physically around to course correct, testing for interaction (especially micro interactions) is “really troublesome” and it’s difficult to push the boundaries of cutting edge interactions from afar.
For others, the pain was less about technical challenges and more about missing the human connection that so many researchers value. The researcher in healthcare hated that she can’t travel to her users right now to talk to them about their recent experiences.
Another UX researcher said:
“Traditionally, user research has a personal touch... I know a lot of places are struggling because they had products that relied very heavily on seeing people in their own environment. But I think [since COVID], a lot of user testing companies have had to look long and hard at how their platforms present remote options. And I think, going forward… remote testing will be much more powerful and a necessity for businesses across the board.”
Researchers miss their teams
Researchers miss talking to participants in person. They also miss their teams.
“[The thing I’m most looking forward to getting back to] is having that very interesting personal collaboration, where everyone's at a whiteboard, everyone can look at all of the post-its that you've put up from the user research. They can actually get involved physically in sorting them and seeing what that process is like. I think being able to … see exactly what someone's looking at and talking about is much more helpful in person than it is virtually.”
Other people simply miss socializing with colleagues outside of work. “After a day of “faking it” in interviews and meetings, you want to bond outside the office,” said one researcher. The lack of in-person interaction has been especially hard on new hires, who were onboarded during the pandemic. As a manager of six, this researcher was particularly worried about one of his employees; they felt she wasn’t connecting with people in meetings, and had struggled to create opportunities for connection.
“As her manager, this is my biggest struggle this year, to make people feel like they’re on a team.”
There are just too many meetings
Even researchers who said they missed their teammates felt that they were seeing them on Zoom too often. There are just “an insane amount” of meetings, and people are having a hard time getting work done in between conference calls, we were told. The meeting fatigue we mentioned earlier came up again when we asked this question.
Still, many of these meetings feel necessary to keep the team aligned.
“People aren't really talking to each other, and I see that there are breaks in some of the ties between different teams... So I've been trying to make sure that every project that I come with, even though it might be brought on by a UX designer who has talked to PS, I'll make sure that I go back and include those PMs on another meeting to ensure that we're all aligned.”
What did you start doing this year that you’d like to keep doing?
A lot of the answers we received during our interviews were expected: remote life took some adjusting too, uncertainty made planning difficult (especially early on), people were spending too much time in Zoom calls, etc.
We had a pretty good sense of what kinds of difficulties researchers had experienced, and what they were looking forward to putting behind them in 2021. What we were less sure about was how (if at all) COVID had changed UXR for the better. Was there anything researchers started doing this year out of necessity that they wanted to keep doing once things went back to “normal”? Was there a tool or process that they adopted in response to the pandemic that made them think “you know what? This is actually better than the old way of doing things.”?
Turns out, there was.
UXRs adopted new tools for remote and async collaboration
Some researchers waltzed into 2020 with a fully baked tech stack. “I was doing remote testing anyway,” said one researcher, who said all the tools and processes they have in place now were already in use before COVID started.
But others researchers made changes this year that they’re eager to stick with. One person said that tools like Figma have been a game changer. Because they work in such a heavily regulated industry (investment banking), they can’t use most cloud tools unless they’re hosted locally. But Figma lets their team work asynchronously in a single document, which was something they weren’t doing regularly before COVID.
Another researcher was happy with their team’s switch from product review decks to asynchronous presentations and docs. This change means the reviews are less focused on the presentation, and more focused on the actual content. They like that the information is available to dig into when people want, and that the team can come together for more meaningful discussions later.
Remote research was a keeper
Four of the researchers we spoke to called out remote testing as something they’d like to stick with—with a few caveats.
One researcher said that while in-person has its place and time (especially for physical products) they liked how quickly you can reach out to a wide audience with remote recruiting tools. But they also felt they needed to refine some of their communication, especially their screener surveys, in order for the process to be truly successful in the long term.
The VP who said that they loved how the ability to recruit remote participants made diverse perspectives more available also admitted that they weren’t sure if they would continue recruiting remotely after the pandemic. Their company recently invested a huge amount of money in a physical research space, and they are eager to show that they were making use of the space to justify the investment.
Researchers liked working remotely (some of the time)
User Interviews is a fully remote company; everyone here has chosen to work in a remote environment because we love the flexibility and autonomy it provides. Sometimes, we take it for granted that other companies don’t have the same culture or way of working. For many researchers, 2020 was a crash course in remote work.
It hasn’t all been smooth sailing, but many people found that they really enjoyed remote work, at least some of the time.
“Schedules are more open now, especially for research sessions; if I’m home anyway, why not get some research in?” said our UK-based participant, who indicated they’d probably continue this post-COVID if their company allows.
Another researcher, who recently made the move from San Francisco to Portland, OR amid the pandemic, said:
“Remote seems possible. A lot of people are moving, including myself. It’s really made it clear that our team can work remotely and can still communicate and tackle complex things… We don’t need to be in the same office. You know, everyone can work remotely and take breaks.”
Still,
“The official policy is that we’ll all come back to the office. But I just don’t see that happening.”
How has COVID impacted your work?
Yes, you.
We’ve begun to distribute our annual State of User Research survey (you can read the 2020 report here).
This year, we included several questions about how the pandemic has impacted user research, including versions of the questions we asked our interview participants. If you’re a UX researcher or someone who regularly does research as part of your work, we’d love to hear from you.
Special thanks to: JP Allen, Carrie Boyd, Erin May, Jamison Ordway, and Rachel Roppolo for their excellent note taking.
.png)
