Highlights:

[00:04:11] From 0 researchers to 3 UXRs and 50 product people

[00:13:30] The art of observation and note-taking during moderated interviews

[00:18:59] Generative research? Bring in a pro researcher

[00:24:34] Helen’s biggest wins in instructing others on how to do research better

[0:31:07] Researchers develop the craft, but they don’t own it.

[00:33:54] Actionable tips on setting guardrails for non-researchers’ research

Sources mentioned in the episode:

• userinterviews.com/awkward
• userinterviews.com/awkwardsurvey

About Our Guest

Helen Devine, UX Research Manager at The Economist, is a user-focused research expert with experience in insight, ethnography, design, and user research across commercial, government, and non-profit sectors. Her current role involves telling user stories that uncover opportunities and inspire action across the organization’s portfolio of digital properties. Before joining The Economist, Helen has had an illustrious career involving positions at The Guardian, a housing and homeless charity shelter, Lloyds Banking Group, and Asda. Recently she spoke at the Insight Innovation Exchange 2023.

Transcript

Helen - 00:00:00: I think the biggest win is, like, me and my team not needing to do quite as much of the usability basic stuff, which is fine, you know, and obviously it's super valuable. It's just not always quite as rewarding, as a researcher, to do that stuff because it sometimes feels a bit straightforward, so I think like a big win is that we can do plenty of that and we do actually end up testing a lot of stuff, which we probably wouldn't be able to. So I think there is a big win in terms of overall what we're able to do.

‍

Erin - 00:00:31: This is Erin May.

JH - 00:00:33: I'm John-Henry Forster, and this is Awkward Silences.

Erin - 00:00:39: Hello everybody and welcome back to Awkward Silences. Today we're here with Helen Devine, who is a UX Research Manager at The Economist. Thank you so much for joining us, we're very excited to have you here today and to talk about upskilling product folks to do great research, a subject near and dear to our hearts.

Helen - 00:01:04: Great, I'm looking forward to talking about it.

Erin - 00:01:07: Yeah, and we got JH here too.

JH - 00:01:09: Yeah, it's funny that the timing of this recording with somebody from The Economist is right after all the bank run stuff. It feels like we should be talking about that.

Erin - 00:01:16: Can we just talk about Silicon Valley Bank instead?

JH - 00:01:18: Yeah, what’s the user research of that? But no, I think the topic we have is a very important one and evergreen, people are always talking about this, and very important. So, excited to get into it.

‍

Erin - 00:01:26: Yeah, so as I mentioned, this is a topic that we're thinking about a lot: user interviews. There are, it turns out, a lot more product people, or as we call them, powders, people who do research, doing research than researchers. And so this is a fact that doesn't seem to be going away, nor is it a bad thing, but it creates some challenges and opportunities. So, why are you interested in this? I know why we are, but why is this a topic that's important to you?

Helen - 00:01:51: Yeah, I mean, I think it's the same really, in that it's really pertinent where I work, because there's three of us, user researchers, and over 50 people in product. And so there's loads of opportunities because people are really interested in it, really want to do research, there's loads of demand. And I also think for most product folks, such a core part of their skill set and development is being user-centered. So yeah, there's loads of opportunities. There are a few cons sometimes as a researcher as well, so I think it just feels like the industry's kind of grappling with it a bit, doesn't it? And that whole. like democratization by researchers or of research or what does it look like to enable and empower people while also really still highlighting that research is a skill and a discipline in its own right?

JH - 00:02:44: Yeah, there seems to be a lot of tension there. If you, kind of, help people improve at it, you're maybe devaluing the craft and the skill, but if you help people improve, there's utility because they're probably going to talk to users in the course of their work and so now they're doing it in a better way. So it's an interesting push and pull there. When we talk about these product roles, who are you referring to? So you mentioned, like, a lot of them in your organization, is that just product managers, product designers, who's in that umbrella?

Helen - 00:03:07: Yeah, so it's product designers in the main. So they're the ones, I guess, probably that do the most research that I work with, and yeah, there's about 17 of them here at The Economist, but also product managers. And then I’m trying to involve more and more people, so I want everyone to get involved and research is a team sport and all that. But it's probably the product designers and the product managers who are the ones who've most been involved in this so far with me.

Erin - 00:03:34: How many researchers did you say that you had? You said you were about 50, I think, product managers and product designers. So I'm just getting that ratio in my head.

Helen - 00:03:43: Yeah, there's about 50 of them, and it's not a them and us, I've just said it like that, but it's not, there's three of us. So yeah, I sort of manage the team and then I've got two user researchers that work with me. I've been at The Economist for just over a year and I've set this up from scratch, really. So we didn't really have UX research as a discipline before that. I think there were like contractors or freelancers, you know, that came in and helped out and there were product people doing their own research. Yeah, I've come in and sort of set up more processes and lots of templates and I’m doing lots of training and coaching. So yeah, I mean, it's really stepped up, but the people who are sort of professional user researchers are still relatively limited in our number.

JH - 00:04:27: If these people, product folks, these powders, people who do research, want to do research, I'm sure they're very well-intentioned. They want to be informed by the users and user-centric, and they're smart people, they obviously are very talented in their own areas. Where are some of the gaps that you see most commonly for product people when they do research? Are there consistent themes that you see?

‍

Helen - 00:04:46: Yeah, it's interesting actually, because often what they ask for is understanding methods, qual and quant methods, and knowing which method to use when, and you know, some of those things, which are fine, and I have done quite a bit with them there. But to be fair, actually, the product folks that I work with here are great, they're really nice, and we work really nicely together. And they're really open to when I'm suggesting things that maybe they should want, because I think there's a lot around just that marking your own homework, kind of, neutrality bit that's so difficult when, as a product designer, you spent a long time building a journey, like a prototype or whatever, and then you've got to let users critique it. So I think you know, there's quite a skill there, isn't there, of opening your mind up to it, and making sure that you really feel like it's all positive, and it's just going to help it get better. And the people here are really good at that, but I think, yeah, there's some of that more nuanced, maybe mindset and approach, actually, that is the bit that I'm trying to push a bit more. I feel like when it comes to the methods and the basics, you can probably read about that a little bit, and then as you choose your approach, then we can work together to make sure that you feel able to do it. and we've got tools and things. But yeah, there's a bit of that. And then there's also been some things I've been teaching people about like observing and note taking and some of that, which people think is straightforward. But I did find that when I came in and started making a Miro board for people to take notes, you'd go back and look at it after a moderated interview, and it would just be overwhelming because there's just so much on there. And so, yeah, I did a workshop, which was brilliant actually, where we just sort of said, let's focus on the questions, focus on the objectives., we can have a car park for that extra stuff. One of my colleagues actually has done a really good job of that, of just showing the funnel. If you have your requirements that are great, then that'll feed into a really great plan and then that'll make your note taking really easy and then you'll have your analysis and then your playback is really easy, you know, that kind of thing. I think it's helping people think about it all the way through and just helping them make their own lives easy actually. So some of that stuff, maybe people weren't so aware of, as if they approach it a bit differently, it'll actually make their whole doing of the research and finding out what the research is telling them a little bit easier.

Erin - 00:07:10: When you work with product designers, are they running research through that full funnel from start to finish or are they jumping in and out and have some enablement here and there from your team? How does that typically work?

Helen - 00:07:22: We're kind of quite flexible, nimble and we'll work in different ways, depending on what it is and who's got resources and how quickly we need to turn it around. So sometimes product designers will do the whole thing and sometimes we'll work on it together. Generally they'll have input, so we have office hours, we call it a research surgery over here but it's office hours where people can come in. I have said that I would prefer that everyone brings everything to it and we're never going to block things but it just helps us make sure we've got visibility of things and if there's going to be a bit of duplication across squads then we can work with that. So we always have an opportunity to talk through approach and timings and what the sample is and all that kind of thing. But people are doing the full and so that's nice because it's quite rewarding isn't it when you get to see it all the way through.

JH - 00:08:12: You mentioned you’re not going to block things, and you also mentioned workshops. So I'm curious, I think we've heard from other people of how they maybe do some of this enablement and training for folks, there's like, you know, you do it upfront, education and enablement and resources and stuff internally, and then people can draw on it as needed. You can co-pilot a study with them and be there as a buddy and help them, you can do more of an oversight model where it's like, no, we need to see everything come through and we're going to prove it before you move forward. How would you say that maybe you're dividing your time across some of those different options or other categories that I might be missing entirely?

Helen - 00:08:43: I guess, because we've got the office hours, so that's an hour a week where there's always time to talk things through, so I suppose that's where I'm double checking if there's anything that we feel like we've already got the answer to, or it doesn't make any sense, or it's duplicating, or we should do later. And then also an opportunity to say, let's amend the approach a little bit. We're sort of trialing different models internally, actually. So one of the researchers has been working across different things, but next quarter he's just going to focus on one product that's quite good to focus internally. The other researcher, we're trialing having her embedded in one of the squads, so in the apps squad here, because I really want us to be able to spot opportunities more, and you see where research can add things, and without being embedded to a certain extent, it's really difficult. So she's doing that. And then I'm taking a bit of a holistic view, and then also working on some of the big things that cover lots of areas of the organization. So timewise, yeah, I do let the product designers get on with it if they want to. So there's probably not all that much time having oversight, I think it's more jumping in at the points where it helps. Yeah, the other two researchers probably work alongside people a little bit more than I do. We've all set it up that way, so it works quite nicely.

JH - 00:10:04: How do you manage to do this kind of support and enablement and upskilling alongside doing research yourself? Because I think something we've heard from other researchers is, you know, I'm here to do research and I have a skill and I want to be hands-on doing research and now I'm spending a ton of my time supporting non-researchers. Have you been able to find a balance there that lets you still do your thing when it comes to research and still lead?

Helen - 00:10:24: Yeah, that is a difficult one actually, because also I feel like, in all honesty, you don't always get all the credit when you're coming along to your appraisal or anything like that with your manager. When you do spend a lot of time coaching, I feel like you've got to push that quite a lot because obviously people don't necessarily see that. I want other people to get the glory. I don't want to have to shout about it. It's only when you've got to fill in your form at the end of the year that you have to shout about it. So I feel like it's a balance. So there's certain points where one will take up more time than the other, so I think when I was coming in, I probably spent actually more of my time enabling and empowering people because I was building a lot of the templates and tools and processes and stuff, and so I was trying to collaborate whilst doing that to make sure that it actually met the needs of people internally. Now, because more of those are there, and it is a constant process to keep building that library and constant coaching, but that's a little bit less now. Everyone's working in agile, but they're also working on quarterly roadmaps, so there is a little bit of planning there. So I also let people know when I've got big things on, so then we can just sort of be mindful of each other's time a little bit. But you're absolutely right, I want to do the research, that's my favorite bit. I like managing and I like coaching and all that, but my favorite bit is research. So yeah, it's important to have both. I think they can also sit alongside each other, so quite often I'll bring people into stuff I'm doing. So if I'm running a workshop to gather requirements, or if I'm pulling together a topic guide, or doing an analysis and synthesis session, or they can sit in and watch my interviews, all of that stuff actually achieves both, so I think that's showing rather than telling is always a nice thing anyway, isn't it? So I think that's often the approach that works well for everybody.

Erin - 00:12:15: It sounds like there are a few areas that the product folks need to upskill specifically you found. It sounds like, like you were saying, you can Google methods and learn about them and of course, practice makes perfect. But maybe some common areas are an awareness of biases, note-taking, and the need to sort of upskill there, and you mentioned that workshops were a helpful way to teach some of this. Are there other tangible skills that folks who are maybe new to research or product designers, product managers should be aware of that might be blind spots or things to try to upskill in proactively before jumping into research?

Helen - 00:12:51: Yeah, I mean, I think the moderated interviews are often one that are a bit more difficult for people because, you know, as a qualitative researcher, it does take quite a number of years to get comfortable with that. And I also feel like one of the interesting things is when you're talking with people about upskilling them, it makes you reflect yourself as a researcher on, actually, what some of those softer skills are and some of the bits that are part of it are. In all honesty, the product people do more of the usability and the unmoderated testing, but they do do moderated as well, as and when it's relevant, and I think, yeah, one of the things that we've talked quite a lot about is, you know, how you make the conversation feel natural. So they'll often say when they're watching me that I'll feel like we're just going all over the place, but actually we have ticked everything off, so that's nice for them that I am doing that. I mean, it's just to make it feel a bit more comfortable and to make sure the participants have the chance to express themselves. Sometimes you need to let participants just talk a bit about something else because it's going to build that rapport or actually it's demonstrating a point they're trying to make. And I have found that less experienced people sometimes just go through the topic guide without that, and you just miss some bits. There's that skill of having what feels like a natural conversation, but also, you know, being in charge of it and making sure you stick to time. I think there's also some of that analysis and synthesis. I've also found that people who are a bit less familiar can just over analyze things and spend a very long time going through like, you know, full transcripts and, you know, all the notes. Whereas me and my researchers that I work with often will sort of analyze as we go a lot of the time and you sort of figure out what the findings are and build hypotheses as we go. So again, it means you can turn things around very quickly at the end, can't you, and you know, play back your findings because you're pretty much there by the time you've finished your last interview. So I think there's a bit around, yeah, some of those things that people are doing and they're not doing a bad job, they could just probably take it up a level by thinking about it a bit differently. Yeah, I think there's just a bit as well about being in the moment, isn't there? You know, well, ideally, I know you've got your topic guide there on your other screen, but just actually trying to be in the moment and sort of listen to the person and observe the person and, you know, empathize with the person and all that sort of stuff rather than, being so concerned with what your next question is. A lot of that just comes through confidence, through experience as well, though, doesn't it? I think there's quite a lot about just making people feel that, as long as they've got the right consents, you know, in our line of work, we're generally not asking people about super sensitive, super confidential stuff, you know, I like I used to work for a homeless charity and like those interviews were totally different, this one is only about people consuming digital media and like listening to podcasts, it's fine. So they don't have to worry too much. It's not that risky. Yeah, we use like user testing, so again, like if they mess up one interview, it doesn't really matter, it's okay. So I think there's a bit about making people feel comfortable as well that the worst is not going to happen here.

Erin - 00:16:06: And do you find that you're able to teach a lot of that through those workshops or do you tend to shadow people? I'm thinking of things like, sales folks will use Gong or record videos and do training sessions or like role playing, you could just imagine teaching some of this in a variety of different ways, so I'm curious, you know, what's worked well for you?

Helen - 00:16:25: Yeah, when we run the workshops, they're generally shorter, you know, like 45 minutes, 55 minutes, and they're more to sort of collate needs and gaps and those kinds of things, and then maybe just talk about the basics. So actually getting people into it is a lot more of the shadowing, so it's a bit more often of having somebody observe me or one of the other researchers conducting an interview and then have a conversation afterwards about it. I do sometimes observe others, however, I don't want to put them under loads of pressure and I do feel like it's not that nice sometimes if you don't feel that comfortable knowing there's an observer there. Actually, I try not to do that very much and rather let people self-reflect a little bit. And I also think like all the product folks here, they're all really good and I feel very confident with them to be honest, so I don't feel like I need to helicopter that much.

JH - 00:17:24: I was curious about that, I'm just gonna make an arbitrary scale here, right? Of like one to ten and ten is like an amazing perfect researcher. So research done by, you know, professional user researchers is probably going to be, you know, eight, nine, ten, depending on the methodology and whatever else, right? In your experience, like having had a lot of exposure to product people doing research, where are they on that scale typically? Are they starting at a two? Is it a five? Like how capable do they tend to be just out of the gate?

‍

Helen - 00:17:51: There’s a massive difference. So some of them have never really done it before so they start off lower. Some have a lot of experience because I think that's the thing, isn't it, with everyone in product and our kind of roles, people come with all these different backgrounds. So I think that one of the things actually that's useful when someone starts is to have a conversation with them and sort of see where their base level is. I think it depends on the method here though. If you're doing something like maybe a card sort or tree test or, you know, like sort of an unmoderated going through a user journey, I don't think there's all that much difference. There's maybe a bit of difference in the analysis, but it's a bit more logistics, practical, especially when you're sort of looking at like what point did somebody drop out of the journey or, you know, how long did this take? Those things that are much more about like the literal UX or even the UI, I feel like whatever the number was would be very similar. I think it's when you're getting into that more generative stuff where you're trying to find out what you don't know, you're exploring the wider context, I think that's where the professional researcher adds a lot. I think it's not my fault if I work in product, but my role has a smaller scope, doesn't it, because I'm probably there to work on one product, so it's hard to step back and it's even like not in your job description in a way to step back because it's difficult to ask questions or, you know, let the person talk about all kinds of things. Yeah, so then I guess that's where I think the number would diverge a bit more and the person who has research in their job title would be higher up the scale because, again, I think a lot of it's about mindset and approach and sort of the context you're coming to it with.

JH - 00:19:37: I was asking because I think I get the sense that people are afraid that when a non-researcher does research, it's going to be done so poorly that the results are useless or you're actually making the wrong decisions because you picked up the wrong signal. And I don't have a feel for how often that worst case happens or is a real risk versus it's okay, it's not as done as well as it could have been done, but like we're moving in the right direction and we learned some things and maybe, maybe that's a better way of trying to get at it. Have you seen the worst case where it's like, oh, this was done so poorly, we shouldn't have done it at all or we actually made a misstep as a result?

Helen - 00:20:09: I think I've seen it more when there's a bit too many conclusions jumped to, because I think as a researcher you might only start with five or six interviews, but if you haven't found what you needed, you do some more and you generally do that and then iterate and do some more. And I think I have seen it where people have just done that first stage and gone, right, we know everything, one person said this, it's done. So I think there is a bit about potentially researchers could actually do better there to educate, because I think we maybe don't always explain our thought processes when we've done one round, we might not explain properly as to why we've made the decisions. Yes, we know this, but we don't quite know that yet. Yes, we feel like we've got an answer on this one. We need to explore this one further. We might not always explain that right. So I think that what I have seen is people drawing conclusions too early, or again, sort of reporting back that something is an insight or found and it is a very low number of people who've said it, you know, like one or two people have said it. And I think there's always this risk in research, isn't there, that if one person says something that feels quite meaningful, then it's a big deal. Whereas I think there's a job for us researchers to do, you know, a lot of the time, we'll show one video, because that person is articulating themselves really well, and so we show one video, but that represents lots of other things we've heard. But I think maybe if you're not so familiar with it, you'll think, well, that's okay, that one person said it, so it's reported back. So, I'm not saying it's all my fault, but it's also maybe not all their fault. There's a balance to be had. But I'd say that's the major one. I don't think that happens quite so much here, probably because we do collaborate a lot. A lot of the time people will send me stuff to sense check and so I will have a quick look through, so if anything like that looks like it's going to happen we'll catch it quite early. And also because there are three researchers, so it's not loads but it's not like none, so we sort of have a little bit of an eye on what's happening so, again, hopefully we usually catch things before they get too much like that.

JH - 00:22:21: All right, a quick awkward interruption here. It's fun to talk about user research, but you know what's really fun? Is doing user research. And we want to help you with that.

Erin - 00:22:30: We want to help you so much that we have created a special place, it's called userinterviews.com/awkward for you to get your first three participants free.

‍

JH - 00:22:41: We all know we should be talking to users more, so we went ahead and removed as many barriers as possible. It's gonna be easy, it's gonna be quick, you're gonna love it, so get over there and check it out.

Erin - 00:22:50: And then when you're done with that, go on over to your favorite podcasting app and leave us a review, please. Have you found yourself in a situation where you need to help upskill product folks in multiple companies in multiple roles or is this a new one?

Helen - 00:23:09: Yeah, I'm just trying to think. This is the one where I've done it the most, where it's sort of been more of like a strategy and a conscious decision that this is the sort of the model we're going to follow. In my last two roles, I did it a little bit, but it was probably a bit more like shadowing, where UX designers or service designers actually would sort of work alongside me, and then maybe, you know, if we got into something, then they might say, oh, well, can I take this interview now? Or do you mind if I take this bit of analysis? And so it'd be a bit more, you know, of a sort of a handover type thing, or I haven't done so much of the actual training and, you know, setting up rules and things. So that hasn't been the model quite as much. I also think this feels like maybe it is evolving a bit more generally as well. So like it makes sense it's become a bit more formalized, that you're making a conscious decision, you've had like a conversation to say this is the way we're going to work rather than it just ending up being like that because of the way the resourcing or the headcount and structures ended up.

Erin - 00:24:15: Well, that changes the follow-up question, which is, in your time really proactively trying to lead some of this instruction on how to do research better, where have you found the biggest wins? Or have you tried anything you maybe wouldn't recommend other people try? Or where are you getting the best bang for your buck in terms of helping upscale product folks?

Helen - 00:24:35: I think the biggest win is, like, me and my team not needing to do quite as much of the usability basic stuff, which is fine, you know, and obviously it's super valuable. It's just not always quite as rewarding, as a researcher, to do that stuff because it sometimes feels a bit straightforward, so I think like a big win is that we can do plenty of that and we do actually end up testing a lot of stuff, which we probably wouldn't be able to. So I think there is a big win in terms of overall what we're able to do. And I think the other big win is just that culture of research being an option at different stages. I think that's really good. One of the things I'd say is less successful is sort of me and the other researchers being involved in all stages. So I'd really like to be invited to more meetings and workshops as a bit of an advocate for the user and a voice of the user and somebody who just knows a lot about our user base and can just add something there. But I think because so many people are doing research, they don't necessarily see the value of that.

Erin - 00:25:42: Well, that's really interesting because you talked about that a little bit before with, in most organizations, if you have lots of product folks, they probably have a somewhat limited surface area they're focused on, so the discovery they're going to do is going to be either less specific than it might be that you're doing or less informed. And so you have that horizontal knowledge of what's going on with the business that you could imagine being very useful in lots of conversations, and I think, yeah, how do people know when to pull you in? To your point, you're one person or three people as an entire team, so obviously you can't be brought in to spend all your time in meetings instead of doing research. So I think that's a really interesting one I bet a lot of people can relate to.

Helen - 00:26:21: Yeah, because there's a lot of people who say, no, I didn't think you'd have the time to do it. I didn't want to ask you because I know you're really busy and I'm always saying, just ask me, it's fine, like I can manage my time, I can say no if I can't come, but I'd really like to get the invites. And I do think that’s a bit of a fine balance as well, because people are nice and they don't want to fill your calendar up. However, I have been trying to say, just ask me, please, because I feel like I can add something. and it's valuable, it'll be useful. I think the other bit is, you know, that always happens with this is again, when more and more people are doing research and more and more people are doing really good research, it is very, very hard to justify more researchers. And whenever I'm asking for more headcount or saying, oh, we could really benefit from this and trying to show like actually how much more value there is when one of the researchers is semi-embedded in the squad, or when we've got someone who can focus on one product when it's a big priority for the organization, yeah, that makes it really difficult. Yeah, I don't know how you get around that one really. That's the challenge.

Erin - 00:27:32: It's like you want to put yourself out of a job but not like literally.

Helen - 00:27:36: Yeah, exactly.

Erin - 00:27:38: Yeah, and so you said there's three of you, how do you break up across, you know, what you focus on? Do you have research ops or are you all UX researchers?

Helen - 00:27:45: No, so we all do research ops, yeah, we don't have that. Yeah, so I'm more holistic, I'll pick up sort of the business priority pieces usually. One of the researchers is focused on apps and then another one's focused on podcasts and those products around there. I think it always works better if you have two of you on something, even if one's leading it and one's involved a bit, ‘cause it's always nice to have someone to bounce ideas off, isn't it? When you're doing your analysis and things. So we try to get involved a bit, but that's mostly how it works. It's really moveable. We are still a new team and you know, there's other people who are new and other changes in ways of working so I'm not sure we've found the ideal model yet. We're still working that out, I think.

JH - 00:28:34: What's come through in a lot of the way you talk about this is just being very collaborative and sort of open and friendly, it feels like there's a real reaching out across the different departments and teams and building that up a mentality of sort of doing it together. I feel like sometimes you can hear, I think from a place of very good intentions, it becomes a little bit more territorial or, you know, combative in terms of approvals and this and that. And I do just wonder if that is helping here because sometimes if I know I have to go get this thing approved and I'm not sure if it's going to, but I want to get it done, then maybe just kind of keep it hidden and then nobody gets to see it at all, and so has that like been a conscious thing that you all have done or has it sort of just happened because of how you approach things?

Helen - 00:29:13: It is very conscious actually, because my background was in market research and sort of customer insight, and that's what I used to work in. And then I moved over to UX research and user research about five, six years ago, and the main draw for me was the sort of collaboration, everything's really transparent and open, like that's what I really like about it. Yeah, I feel like for me, that's sort of the number one most important thing to do actually, and yeah, not having massive guardrails around it, and even if we researchers are maybe the experts and the ones who've developed our craft the most, it doesn't mean we own things. I do think that enablement, though enablement is a bit of a strange word, isn't it? But, you know, that helping the organization to do the best research regardless of the individual who might make it happen, yeah, is super important. And like, it's a bit funny, the structure where I am, because we actually report in through the data rather than through the product or engineering, which is unusual, I think. It's not where I've necessarily sat in a structure elsewhere. I think again that's got its pros and cons. Pro is probably that it's neutral, it's impartial. It does mean that we've got that holistic view. It does mean that we are somewhat separate from product though and I think that's not always ideal. I've sort of had to do a bit of work to be involved in things. Again, like I'm lucky because the product people here are great, so they do invite me to loads of stuff. You know, I wouldn't be like number one on the invite list, and when there's sort of strategy and planning, I wouldn't necessarily be at the table straight off, which for me, it would be better if I could be. 

JH - 00:31:01: Yeah. You also mentioned, you know, not having a ton of guardrails and stuff, but like, are there certain things that are less of like an upscale and more of like a binary? Like you all just have to do this from a compliance and best practice perspective of make sure you get this consent form or deal with PII this way, like, are there things in that category that you all have also like made sure that the non researchers are doing correctly?

Helen - 00:31:19: Yeah, and it's a really good point that there is stuff around consent forms, what we do for incentives, like how much and how we pay them. There's quite a lot about communications even, you know, how people email potential participants, calendar invites, the wording on those. So I don't specify exactly what the wording is, they've sort of done templates on a lot of these and say this is what people should use. Actually, some of these, the designers have made themselves and said to me, oh, does this work? And I'm like, yeah, great. So, you know, it's not like they're all written by me, but they are approved by me. There's also something that if we're recruiting sort of participants from our user base, again, I am a bit more cautious about that one because I'm aware that anything we do, we're talking on behalf of our brand, aren't we? We've got to be careful, so I feel like when we're deciding who's going to take, you know, whether the product people are going to do it themselves or whether the research team, if it is directly with our own users that we've recruited through a pop-up or whatever, we would be more likely to take those ones just because, again, I feel like there's a little bit more risk there. So we don't always take them, but I suppose that would be part of the decision-making process.

JH - 00:32:33: How did you get everyone to be aware of those? Like, so everyone knows like, oh, incentives, we have to do it this way or consent form, I have to make sure, like, how do you get that baseline in place?

Helen - 00:32:43: Yeah, so we've got a Confluence page with all this stuff, mine is part of the product design section on Confluence, the product designers are the ones who do the most, so it means they can't miss it. We've got a Slack channel as well. So I've just put some on Slack and I do quite often. I'll say, oh, remember this, remember that. On user testing, I didn't before, but I have implemented now that actually no one can launch a test without me or one of the researchers approving it. That's more actually to keep track of our session units and things like that, because otherwise they just feel like it could go out of control and we'll have no budget left and that kind of thing. So I have put a couple of smaller things like that in as well. But to be honest, stuff might have got missed. I'm not saying it's all perfect. I think the product design team is relatively established as much as any of our kind of roles are. Most people there have been in the role for a year or more. So again, I'm relatively comfortable that they all understand this. I think when new people come in, we always meet anyway. Yeah, so actually this makes me think I should probably put together a bit of a checklist actually for new people, because I probably assume that they've found everything and they might not find it quite as easy as all that. That'll be my to-do after this.

JH - 00:34:06: Sounds good.

Erin - 00:34:07: Yeah, I'm getting so many ideas from this on content that we can create to help people, including this episode, but I think there's a lot of need for the checklists and the templates and, of course, some of the training is going to be ongoing and learning by doing, of course, but having some of those just get started kind of guides and where do you need to upskill to get started and these sorts of things I think are super useful to folks. I know one area you mentioned that it's helpful to do some training is around the analysis part and the note taking part, data in, data out. I'm curious if you track or have seen any changes in terms of how product designers and people who do research are able to find the best insights from the research they're doing and then use them. I know that's a hard thing for people to track everywhere in research, but probably particularly when you're not maybe the one doing the research or the team doing the research. So are you seeing any movement in terms of the research actually being used to make good decisions, for example?

Helen - 00:35:10: Yeah, I mean, I think actually the pace has picked up a bit. So I think from what my observations are, there were probably good decisions being made on the back of research, but it was probably not actually being done that efficiently sometimes. So that, you know, people were having to wade through a lot of notes and probably spending a long time on something that was a relatively small decision to make. So I feel like potentially some of the things we've all done together have made people's jobs a bit easier and made them also feel a bit more confident that they don't have to go trawl through absolutely everything, you know, to discover the insights that are in there. So I'd say it's maybe that a little bit more that we can probably achieve more, feel a bit more confident with it, and then move on to the next thing a bit better. But, like you say, it's really quite difficult to look at the actual quality of it, I feel like sometimes. Yeah, it would be an interesting exercise to think about how exactly you measure the quality of that stuff. I mean, because we report into data that, you know, this is the kind of thing that they're always asking about, like, what was the impact of that? What happened? What changed? And the good thing in UX research is the point is this is to change something, and generally, it does lead to a change, a next iteration of a prototype, or it means that we've made a change on this page. It is quite good for that in terms of it's a bit more definable. But as far as like, could this have actually been better? Could we have got better insights from this? That's quite difficult to answer, isn't it?

Erin - 00:36:56: Yeah, hard to scale that one. What are you excited to do next? What's next for you in your quest to upskill product folks?

Helen - 00:37:05: I don't know actually because I was talking about this with someone internally because I was saying it needs to stay on the agenda, you know, like we've done loads in the last year and we need to keep doing it. But yeah, what's next? I mean I did want to do a bit of a mentoring circle actually, this was something that I talked about with the team when I first started and I just haven't had a chance to do it. But I think that would be quite nice, you know like, not necessarily a best practice but more just for people who do research, a nice safe space to just talk through things and work through things and, like you were saying earlier, you know like role play and that kind of thing. Just somewhere to do a bit more of that. And I think as well ideally that safe space where others can come in who want to do research but haven't had a chance yet, just to maybe be part of that conversation. And I think just you know to work through things, so a lot of the time you know when you want to work out your hypotheses or think about a question or think about whether research is appropriate in this context, it's good to just have that time to just bounce it around with other people and just get it done rather than sitting by yourself, or even just having a conversation. So I'm just thinking yeah that space to do that, and I think actually in terms of everyone's development being part of that conversation with others is actually a big part of that. So you know if I come and say, oh this has come up, I can't decide whether it's worth researching, I'm trying to figure out exactly what the problem is, can people help me form the hypotheses? To do that with others is building your research and questioning skills, so I think something like that is what I would like to do. I also want to get doing a bit more in-person stuff because I've started doing some customer closeness, you know, so I've got some readers and listeners into the office and done like sessions with colleagues and them, and done some little breakouts, and they're brilliant, I absolutely love them, and people that come say they're really energized by them. So yeah I want to do a bit more of that as well. I've had a lot of product folks come into those but I'd like to maybe get a few more people into those ones as well because, yeah, they're really nice.

Erin - 00:39:20: Awesome.

JH - 00:39:21: Awesome.

Erin - 00:39:21: This has been great. Thank you so much for joining us.

Helen - 00:39:24: No, thank you very much.

JH - 00:39:26: Yeah, this has been awesome.

‍

Erin - 00:39:28: Hey there, it's me, Erin.

‍

JH - 00:39:29: And me, JH.

Erin - 00:39:31: We are the hosts of Awkward Silences, and today we would love to hear from you, our listeners.

JH - 00:39:36: So we're running a quick survey to find out what you like about the show, which episodes you like best, which subjects you'd like to hear more about, which stuff you're sick of, and more just about you, the fans that have kept us on the air for the past four years.

Erin - 00:39:47: Filling out the survey is just going to take you a couple of minutes. And despite what we say about surveys almost always sucking, this one's going to be fantastic. So userinterviews.com/awkwardsurvey and thanks so much for doing that.

JH - 00:39:59: Thanks for listening.

Erin - 00:40:02: Thanks for listening to Awkward Silences, brought to you by User Interviews.

JH - 00:40:07: Theme music by Fragile Gang.