Include decision makers from the start

Holly’s main goal when establishing research plans for teams is to create research plans that work for everyone. Naturally, this means that everyone needs to have access to them. Holly has a great formula for creating the guest list for kickoff meetings. You’ll need to invite…

1. The people who work on the product
2. The leadership for the project
3. Anyone who has the power to stop the research
   

Getting all of those people in one room at one time may be a monumental task for some. But it’s important to include all of these people in your kickoff or brainstorming meeting if possible. It means that later on, when you need to implement your research, there will be more people invested in it.

If you can’t bring your stakeholders in at the very beginning, you should try to bring them in as soon as possible. This allows them to have some agency over how the research is carried out, and creates an environment of collaboration. Need more tips on creating great research teams? Check out our field guide module all about effective research teams.

Put the power in stakeholder’s hands

When you do present your research, you can set yourself (and your research) up for success by giving your stakeholders insight. This means going into a meeting or presentation with quotes, snapshots, or videos that help illustrate your user’s point of view.

Holly loves the “snapshot” format, promoted by Teresa Torres in her courses for product people and researchers alike. This involves gathering quotes and videos from your research sessions, and giving context for where that particular user sits within your audience. Ideally you can sketch these quick snapshots directly after your interview or session while the information is still fresh.

These snapshots make it easier for your stakeholders to empathize with the user on the other side of the table. Adding context, like demographic, relevance to your market, and behavioral data, helps put qualitative data into perspective. Adding objective details like direct quotes also helps to take the bias out of your research and the decision making process. Adding things that are inarguable, like quotes also helps to take the bias out of your research and the decision making process. If you really like what someone who represents only 1% of your target market is saying, snapshots make it easier to take a step back and say, “hold on a second, that may not be the best choice for the majority of our users.”

No one formula for success

As with most things, there’s no set way to get research right. There are some wonderful guidelines we can follow and some theories we can implement, but how it works in your organization is entirely dependant on how your organization works in the first place.

Holly works with companies who are B2B, B2C, all shapes and sizes, and at all stages of the journey towards embracing research. If there’s one thing she’s learned, it’s that there’s not really a “magic wand” world in which all of the circumstances are exactly as you would like them.

Involving everyone in research from the start, including quotes and videos, and giving stakeholders the power to make great decisions with your research are all great starts. At the very least, creating snapshots and more thoughtful research presentations will help your team to empathize with users and embrace research more often. Holly’s final words of advice for anyone trying to implement better research practices? “Be patient, change takes time.”

About Our Guest

Holly Hester-Reilly is a die-hard New Yorker and user advocate. She worked in the NYC startup scene for ten years before starting H2R Product Science, which helps companies of all shapes and sizes embrace user insights.

Transcript

Erin: This is Erin May.

‍
JH: I'm John Henry Forster and this is Awkward. Silences.

‍
Erin: Silences. Ah shit.

‍
Erin: Hi. Welcome to another episode of Awkward Silences. We're here with guest Holly Hester-Reilly today. She has a really interesting and awesome background that spans product, product coaching, consulting and she's going to talk to us today about kind of bridging the gaps across departments with decision makers and user researchers. So we're really excited to kick this off. So welcome Holly.

‍
Holly: Thanks Erin, I'm excited to be here.

‍
Erin: Fantastic. So Holly I did a little bit of a spoiler alert just kind of introducing your background just a little bit. But if you could dive a little deeper into how you got where you are and what you're doing now that would be fantastic.

‍
Holly: Yeah absolutely. So I have been in the New York City tech startup scene for about a decade now. I got into it after grad school. I actually started ... I did my academic work, my research in chemical engineering and learned a lot about designing experiments and systems and how systems work and then went to look for a job in chemical engineering and had a rude awakening that those jobs are not in New York City and I love New York City. So I went and said hey where there are a lot of jobs, what industries are fit for me and I just fell in love with startups.

‍
Holly: So I started to work in startups and doing that I was always very user focused. I was really excited about what technology can do for the end user. And my first startup I was part of was a pre-funding bootstrapped startup and then I had another bootstrapped startup and then about five years into doing that kind of work I moved over to a rapidly growing venture funded startup called MediaMath which is a behemoth in ad technology space. And that's the place where I got to really sink my teeth into what is it like to do product at scale, how do we innovate and iterate and communicate across departments, make things good for the customers and people working on the teams. And I was there for three years. I managed the UX design team while I was there, we pushed a lot of continuous discovery, did early design system work. It was a really fantastic learning experience.

‍
Holly: After three years there I moved over to Shutterstock which was already a public company in New York but was one of Silicon Valley's first startup success stories, had been founded by Jon Oringer, still the CEO, and grown to a 650 person company at the time that I joined. They had a small UX research department when I joined, a much bigger design team and a much bigger product team and so I got to work on a bigger team with more people and get exposure to more different ways of working. I was there for about two years and then I went and founded H2R Product Science. I've always been passionate about teaching other people and I want to help as many people as I can learn the things that I've had the privilege of learning in these ten years that I've been doing this work.

‍
JH: Awesome, thanks for that overview. The question I have just to kind of jump right into it is why this topic? So the distance between user research and decision making. Is that something you've had like horror stories with or you've seen done really well or comes up a lot in your consulting, what brings this top of mind?

‍
Holly: Yeah. So I would say all of the above. I think what I see happen again and again is that a small team will be really close to their user. They'll understand their user and have all these deep conversations with their user and then they build a great product and they start to get successful and they start to get traction. They hire more people, their team grows. It's all exciting good things. But as that happens the distance between that user understanding and the decisions that are made grows and all of a sudden we have entire departments instead of just the person sitting next to me as we work together day in and day out and it becomes much more challenging for the people who are making the decisions to understand the users the way that you need to if you want to keep building those amazing products.

‍
Holly: So I've seen that time and again, definitely was the case with I would say some of the high growth startups that I worked at that had already started to happen but also I work with early stage startups and I see that with them as well. And then when I go into an enterprise where some of my consulting clients have been enterprises, for them usually that happened long ago and it's so long ago that they've sort of forgotten it's kind of we have to teach the people there who were are now how do we bring this to the way that we develop our products.

‍
JH: That makes sense.

‍
Erin: It sounds like you're talking about like size as the company grows, the sharing of knowledge becomes more complicated in the organization overall. Are there particular kind of organizational structures that make that problem better or worse?

‍
Holly: Yes, that is a fantastic question. So it's actually a passion of mine, a passion topic of mine, is how you structure your organization in order to make sure that the user knowledge is evangelized and shared everywhere. In a really highly performing organization, one with a lot of communication you can get across it no matter what the structure is but let's be honest, not that many companies are actually that highly performing at communication. So I think the better you structure it the better you are.

‍
Holly: What I have seen work best is when you have more of an embedded model where every team is developing product, has support from user research and from design, from market research right on that team rather than it being a sort of a centralized agency within the company that people have to go to, get projects approved, have those projects executed and then delivered to the other team. When companies have that sort of centralized agency that's not really embedded on the teams usually the communication and the understanding decreases.

‍
Erin: Right. The expertise is sort of outsourced to a central center of excellence.

‍
Holly: Yes.

‍
JH: Yeah. If I'm a user a researcher in a largish company and I don't necessarily have the influence to change my team structure or the org structure are there kind of like smaller tips of ways to start breaking down the walls that you've seen be successful in terms of improving that closeness and that collaboration?

‍
Holly: Yeah absolutely. And that's a great question too because I've certainly been in that position where I don't have the power to change the structure and so I love to figure out how can you help regardless. There's a couple of things that I focus on.

‍
Holly: So the first one, I think the sort of lowest hanging fruit is how do you communicate the results to the other people. So if you don't have approval or permission to bring the whole team with you on the research or they think they don't have the time and they don't want to come, for whatever reason if they're not there with you, how you communicate those findings is the most important thing. The mistake that I see a lot of user research teams make is taking the position that they are supposed to draw all the conclusions and deliver a report that says "Because we learned this from the customers you should do this."

‍
Holly: Or maybe they don't take it quite that far but they still say "We did 15 user interviews and ran a survey with a few hundred people and from that we learned people in our customer market have certain traits like they are willing to pay more and they are excited about this particular aspect" but they don't actually bring out quotes and videos and stories that other people in their company can really hang on to as ways to empathize with those people. And it's not that making that synthesis isn't important, it's good to help people come to conclusions based on what you've learned, but the problem is that if you actually make the synthesis and don't bring the stories, don't bring the videos, the pictures, the quotes, then it's just human nature that the people on the receiving side won't empathize as strongly with the customer that you're trying to make them feel deeply empathized with.

‍
JH: Totally. There's probably some interpersonal stuff there too of if you're starting to reach conclusions and recommend decisions you're almost like stepping on the decision makers toes a little bit and playing in their space. Whereas if you bring in a lot of assets and clips and quotes you're almost like setting them up for an alley oop to do what they do which is make decisions. There's probably some like actual interpersonal office stuff that plays in there too.

‍
Holly: Absolutely. One of my favorite ways to talk about it is actually ... So I'm a mom, I'm a parent, I have two kids, one is four and one is two. And so I've been deep in the toddler days in my past several years and I often think of it the way that they teach you how to parent toddlers is you may have something you want them to do but you can't just tell them to do that because they want to feel they're independent. So you have to sort of set some structure around it and give them some choices. But you're guiding them because in reality all the choices in the world might be there but you say here's three choices. These are the three things, you pick one.

‍
Erin: In my house it's two, never more than two.

‍
Holly: OK that's true. Yeah. No it's usually two when you're dealing with toddlers, you can have milk or water and that's it. In the work world I find three works better. People love to really feel like they have a choice and grownups are more sophisticated than toddlers so they know if you're trying to pull one over but when you give them three choices usually that's a good amount for them to be like OK here's some things to think through and pick something.

‍
Erin: Right. And more than three you get into the paradox of choice and all of that.

‍
Holly: Exactly.

‍
Erin: I tease but three is the magic number.

‍
Holly: Yeah.

‍
Erin: Awesome. Yeah. People talk a lot I think and it's exciting to hear people talking more about how research is shared because that is kind of, that's the work, and that's how the insights get shared and understood and it's so so important, visuals and videos and quotes and kind of these really human artifacts feel like a hugely important part of that. Are there any methods that you've seen work really well, is there a certain number, you hear five all the time for the number of interviews you might want to do for a qualitative study. Is there a certain amount of evidence or the strength of the evidence or how you present the evidence, what works well with those videos and those quotes to kind of drive the insight home to the people who need to know about it?

‍
Holly: Yeah that's also a good question. I can't say that I've hit on like the secret sauce that like this is perfect and I always do this, it definitely depends on how complex the situation is, how big is the sort of space you're exploring. But I would say I think one thing that stands out to me is the idea of having a snapshot from every interview, it's something that I've come across in a couple of places, my favorite place that I've come across it is Teresa Torres.

‍
Holly: She teaches a course called Continuous Interviewing. She teaches the participants of that course how to create an interview snapshot. And the idea is that at the end of the each interview you really quickly pull away like what are some key takeaways from that interview and you generate this artifact from that interview really right away and it has something, some picture or video where you can feel the connection to the user and then it has maybe a quote, two quotes, three quotes, however many quotes it takes to really express whatever is critical about it and then I like to add ... So a lot of times in my work we'll do a mix of qualitative and quantitative. So sometimes I'll come back and I'll also add sort of something that gives you a sense of where this person sits in the quantitative mix and by that what I mean is maybe we are interviewing people in a really tight market segment but that market segment is only 5 percent of our user base. We care about it for this project for a critical reason but it's not a huge segment. It might be something where I would pull out in that snapshot from the interview "Mary is from segment X. It's 12 percent of our user base and here's why it's important to us" or something like that.

‍
JH: Just since we threw a little shade towards decision makers and compared them to toddlers, I'm going to do the same for researchers. Have you ever seen people kind of take the editing or the narration too far? So when they're creating these snapshots whether they're conscious or unconscious biases they have around the outcomes and what they thought they learned where they almost edit it too heavily and present takeaways from the users that are not totally representative of everything that happened in the session or does that tend to not be an issue and people are pretty good about presenting a fair recap of how the session went?

‍
Holly: Great question. So I will say it's really hard to do that if what you're doing is creating clips. You certainly can but you're really sort of picking those clips with a fine assessment of what's important. But I definitely where I often see that is when people are creating decks with words. So they decide that this is ... and I guess I shouldn't even say decide, it's probably not a conscious decision but they're biased toward seeing a certain thing, they want that theme, they say they see that theme and they pull the quotes that match and then you realize that you could have more or less created that deck before you even interviewed anybody.

‍
JH: Okay. So yeah. So when you start getting into the paraphrase or like kind of typing stuff out yourself part of the equation people should be more mindful of that but like when you're just pulling clips and kind of more verbatim stuff it usually is less of an issue?

‍
Holly: Yeah exactly. It definitely can still happen when you're pulling those clips but that is one of the reasons why I as both a researcher and a consumer of research I always look for the verbatim because that's how I can ... essentially it's your source material, it's going to the raw notes.

‍
Erin: I love how you talked about also bringing in that quant evidence if you have it to kind of say how many people does this person represent cause something I've seen is when people are getting newly excited about qualitative research one person says something that's like oh my god that was it, now we know, we figured it all out and that could be true or it could just be one person. So who do they represent and what are they really helping us to kind of figure out that we already know is an issue or an opportunity and tying those things together also seems like a good way to kind of control some of that potential bias on both sides.

‍
Holly: Yeah. One of the ways that I think about it is I kind of think that there are levels of continuous discovery, a skill, on teams. And so at the beginning you're not doing any. And then there's the team that's doing some but they're still not making optimal decisions based on that yet. And there's a couple of ways that that can manifest and one of the key ones is while they're doing interviews but they treat them all as equally important, they don't put them in context, they don't ladder it up to a strategy and thus they often end up kind of chasing their tails every time a different interview comes back with different feedback. I think that that's something that in a healthy learning focused organization is a phase that it's OK to pass through but you should be aiming to get to the other side. And I believe that teams all can do that but it's good to be aware of it if you're if you're experiencing that and start to ask yourself those questions of like well why did I talk to this user and not that user or how do I put their feedback in context.

‍
JH: How far do you get just by having the decision makers involved upfront in terms of agreeing on the problem or agreeing on the thing that needs to be uncovered even if you have silos, even if you don't have like best in class communication, does that go a long way in terms of getting them closer to the same page at the end when there is like these outputs or does that sort of get watered down through all the other challenges and it doesn't make that big a difference?

‍
Holly: I think it makes a huge difference. I really think it does because again it gives those decision makers that agency, gives them the inclusion and the feeling of collaboration and it gives them the space to raise concerns if they have them. So any time that you can get decision makers in early on it's the best way to go. The way that I usually like to do it if we're embarking on sort of a discovery research project is to have a kickoff workshop where we do exercises together. And when I work with a client and they say well who should we bring to this workshop what I tell them is there's three main groups of people you want to have here. You want to have first of all anybody who's actually gonna be working on the product whether they're designers or engineers or product managers but people who are going to be making day to day decisions based off of this.

‍
Holly: Then you want to have the leadership of the projects so the people who are gonna be influencing decisions that are made, maybe making calls on investment into it, who are going to be setting the vision and helping to synthesize the results.

‍
Holly: And then you also want to have anybody who could basically stop what's going on. So anybody who has enough accountability for what's happening here and enough influence that if they didn't believe what you were doing was worthwhile they could make it all change. Whenever possible you want to get all three of those groups of people to come and be a part of what you're going to do, what problem you're solving, why you're doing it, what's important, what are the concerns and questions you might have right from the start. But it's not easy. I've definitely worked in organizations where that may be the ideal but it's really hard to get those people to all come together.

‍
JH: To expand on that really quickly like what is your rapid fire like idealized org setup for this type of process? Like if you were just going to kind of rattle through like process roles, maybe it's fictional but like if you were able to kind of design it yourself what would ... the magic wand question I guess from User Research, what would that look like in your mind?

‍
Holly: What would the magic wand look like? So that's a really good question. So first let me say I definitely find that organizations often look different if they're B2B versus B2C, if they're SAS or not SAS, just because of the changing amounts of outside decision makers, marketing and sales. I tend to work with a lot of B2B so I kind of think in that space which means it's more sales and less marketing generally speaking. So in an organization like that I usually advocate for there being some high level product leader who has a tech partner. So maybe that's your your V.P. of Product and your V.P. of Technology and then they oversee large product areas together and then they have a head of UX and there's somebody in the UX Department who is an expert in research.

‍
Holly: The way that I love to work when I see teams that work this way is that there is also sort of like a head of product data. And those two roles, the head of user research and the head of product data, are really kind of like chief evangelists or chief educators and what they do is they evangelize like why are these two things really important, what do the best practices look like and they have sort of an army of coaches under them which are the people who embed on the teams or coach the teams and help each of the teams understand what does it look like when we do this well and partner with the product manager, the designer, the lead engineer on the product team to do the right research at the right time with the right communication. Does that sort of answer ... I mean obviously there's all sorts of opinion around it.

‍
JH: No I really like how you answered it. Right after I said it I was like bemoaning myself for kind of asking the best practices question because I'm a big believer of context matters and I wish we called them like helpful heuristics or something instead that was like less "Do it this way or it's not the best." So I'm glad that you kind of added all the context of what type of organization are you and what kind of what kind of business are you in and then kind of expanded from there because I think it really is important to factor all that stuff in.

‍
Erin: Yeah and to bring it back to kind of the real world without the magic wand. We talked earlier about how to build reports so that they're useful and less biased. And ideally you would get the decision makers involved as early as possible, in the real world maybe you didn't do that or maybe they can't stick with the project or maybe you didn't exactly know who the right stakeholders were in the beginning, all sorts of things that could happen. Any just kind of top thoughts on how to navigate some of that to get those reports seen. Is it to have a live or an on demand experience? How do you get people to consume and remember this stuff on an ongoing basis?

‍
Holly: Yeah. So I think the first thing that comes to mind for me with some of the scenarios you just laid out aside from those are very real is the value of one on one communication. I think that in the workplace we are bombarded by information in today's workplace. In most of them there's constant emails and wiki pages and articles to read and Slack messages and it feels like this huge endless list and then on top of that we have meeting after meeting after meeting and when are we supposed to actually do the reading or God forbid actually write and create something that we're going to share with others. So I think creating something that can be consumed asynchronously has a value but also is at risk of people just feeling like it's too much and they're not actually watching or consuming it.

‍
Holly: So I would actually start with if you are in a situation where you're down the path and those decision makers or influencers weren't a part of it in the beginning is what can you do to get on their radar now and have a conversation with them and find out what's important to them about what you're doing and what concerns they have and then come back with OK here's what I think you should know from my research or here's how I should share these results with you.

‍
Erin: Great. It's kind of like research right? Ideally you start early but it's never too late and get in there as soon as you can. Same with bringing in stakeholders I guess right.

‍
Holly: It very much is. It's exactly actually what I try to apply the same continuous discovery practices to how we work within our companies.

‍
JH: Yeah everything is so meta. And just to continue the metaness. Like I think it is important for user researchers to have some empathy for the decision makers. Maybe you did learn something that's very obvious or very clearly a customer pain point but maybe in the decision maker's eyes it's not a priority for whatever reason. So they agree with you that you've learned that and that's a truth that needs to be addressed someday but is not a priority now for reasons X Y and Z. So there is just a lot of different factors that way into this stuff like you can learn things definitively but still not act on them for other reasons. So there's a lot of factors at play.

‍
Holly: Absolutely. And keep in mind that in a typical org the people higher up will have a lot more access and information on the company's strategy, the resources that are available, what is the company's competitive advantage. And there may be cases where you hear something in research that seems like a great opportunity but just doesn't actually fit with those elements. And if you can't get a conversation with those people to hear that directly at least keep in mind when you share research results if things aren't acted on that there may be reasons you don't have insight into.

‍
JH: Yeah. When I was working in a larger organization at Vista Print a kind of thing that was common within the organization which I thought was really healthy was this notion of like your ladder of inference. We both climbed up a bunch of steps when you're having a conversation with somebody else in the business based on what we know and assumptions we've made and other things. And so when we start to talk we're both pretty high up our ladders and we might not know the assumptions or the data points that went into the conclusion that we're reaching. And so a thing people would actually say in meetings often was like "Hey why don't we like walk a couple steps down our ladders and you would kind of like unearth like 'Well I think this because of this and I think this because of this" and the other person would do the same. And you'd often find the blind spots that asymmetric information and it's super helpful to find common ground and that might be a strategy that is applicable here as well.

‍
Holly: Yeah I love that. It's great also when you can have visuals that attach to what you're doing. Just helps people understand and picture it in their heads and remember it. So I like the idea of the ladder taking a few steps down it.

‍
Erin: Holly I have one more question and then I'm going to do I'm sure the thing that you do when you say you have one more question. We'll see what happens. But so just to kind of attempt to tie it all together a little bit and we've talked a lot about why and how these silos happen as teams grow in particular. Maybe it's obvious but how can you tell if you're making headway toward breaking down some of those silos between decision making and user research? Are there signs that it's happening? Will everyone start to feel good and know more? Like what are some tangible signs that you're kind of working in a more harmonious and understood across kind of teams fashion when it comes to user research?

‍
Holly: Yeah. So I do think that people start to feel better but the way that you can tell that is the mood of the things they say and especially the things they say about each other. So if you have a one on one conversation with people on your product team and maybe early on in the process of trying to bridge this communication gap there's a lot of complaints and frustrations and people saying "I brought this research back and I don't understand why it's not being used." And then on the other side you have a conversation with people outside of that and they're saying "I couldn't use that research, it didn't tell me anything I didn't already know." That's one thing I hear a lot that especially happens when you sort of synthesize it into platitudes. And then maybe a couple months later and I do mean a couple months, like it takes some time to build that trust and the habit of communication and so you may see early wins where someone understands but on the whole it's going to take some time.

‍
Holly: But if you work at it, a couple months later you may find and I've seen this happen that those teams are now starting to say "Oh I understand the other side. I understand why that research that this team brought me was important and tells me a lot about the customer." And the people on the research side or the design side or the product side are saying "I understand why we're acting on this and not that, why this decision is being made."

‍
Holly: And then I think also if you just start to hear language where anybody who's making decisions or building or working on the product more often references the user and specifics about the user that they got from the research then you know that you're making an impact.

‍
JH: Yeah you're getting me to bring out all the management tropes. But trust and communication are usually inverse related, inversely related. So if you're in a low trust environment usually you need to over communicate to kind of combat that. And if you're in a high trust environment you can usually get away with less communication. So that's like a broad kind of team statement but I think it's probably something that's worth keeping in mind as people kind of go through this process of trying to make positive changes here that it might look different in the beginning than it does a few months down the line.

‍
Holly: Yeah absolutely.

‍
JH: Cool. Just to recap to make sure we got all the main points here. I'm going to kind of do my best to summarize some of the things you've shared with us. Sounds like one good tip for folks is to pull people into the process as early as possible. So make sure upfront there's some alignment and people have awareness as to what's going on.

‍
JH: I think another one you mentioned was when you're sharing what you've learned include great clips or verbatim quotes and maybe present them as a handful of options rather than like a firm recommendation and let people kind of make their own interpretations and conclusions.

‍
JH: You mentioned team structure wise if you can get somebody embedded in the research process or in the decision making process that can go a long way.

‍
JH: Were there any other kind of key takeaway points for people to write down on their cheat sheet to keep in mind?

‍
Holly: Be patient. Change takes time.

‍
JH: For sure.

‍
Holly: I think that covered most of it.

‍
JH: Awesome.

‍
Erin: Well-

‍
JH: We never know how to do the endings so that's like ...

‍
Erin: It's baked into the name, we're covered.

‍
JH: Yeah that's true, yeah we can say that. Let it fade.

‍
Erin: Let it fade.

‍
Erin: Thanks for listening to Awkward Silences brought to you by User Interviews.

‍
JH: Theme music by Fragile Gang.

‍
Erin: Editing and sound production by Carrie Boyd.

‍