Plan Ahead: Foresee Problems and Prevent Them

Test the Test

Testing the test is what it sounds like. Once you’ve completed your test design, walk through your test before showing it to real participants. Best case scenario is you can get a co-worker or someone less familiar with your test to try it out. You’ll almost always find something you’d like more information on, a question that is worded in a confusing way, or a broken link in your prototype. There’s nothing worse than being mid-study and realizing your test is flawed in some way you could have easily changed ahead of time. Measure twice, cut once.

Have backup plans for things you can foresee going wrong

One benefit of testing the test is it can reveal weak spots or potential hurdles. Having a backup plan can help you stay calm when things go south and continue your research with minimal speed bumps. Jaclyn, for instance, ran into a testing situation where she didn’t have internet or data to run her live usability tests in the field. She ended up pulling up some Sketch files and showing her prototypes that way, but left the research a little sad that she couldn't show her files the way she had wanted to. If you’re doing field testing that requires an internet connection, consider getting a hotspot device that can keep you connected wherever you are. You could also consider hosting the files you need locally, or creating paper printouts in case of emergency.

If you’re working with electronics of any kind, pack some extra chargers—if you’re doing in-person testing, bring extra for participants. If possible, having a backup device you could do your test on in case of an emergency is a great safety net too.

Count on a few no-shows or less than perfect participants by creating a small backup list of participants. Ideally you won’t need them, but it’s certainly possible that you will.

Don’t Put Unrealistic Fake Data In Your Test

Nothing throws research participants off more than distracting, unbelievable data. Take a look at this contact card.

First off, the picture makes it pretty hard to focus on anything else. While it’s good for a quick laugh, it makes it difficult for your participant to focus on the task at hand. When testing with a prototype that requires written content, it’s best to take some time and try to make things as realistic as possible. This gives you an opportunity to gain some insight on information you may be missing and how the final product actually will work for the user.

Jaclyn also says Lorem Ipsum is a major no-no in prototypes It’s not realistic and it won’t give you a good idea of spacing, or of how the customer would react to the content you’re thinking about putting on the page.

Improv Time: When Research Sessions Don’t Go Perfectly

Don’t Let Interviews Stress You Out

If it’s your first time conducting user interviews, you’re probably pretty nervous about messing it up. The good news is, the people you’re interviewing are also human, and there’s really not much to be afraid of. It’s just you, a person, talking to another person.

If you ask questions that are leading, or violate other best practices, just note it, try to get the insight you need another way if it’s really important, or simply move on.

Get comfortable with awkward silences 😅. You’ll run into them a lot when doing user interviews and, if you wait them out, they can usually offer you some good insights! Don’t feel rushed to give examples after you ask a question to get a quicker answer or fill the silence faster. Letting your participant come up with something on their own will get you where you’re going faster.  

Use your script, but keep it loose, more like an outline. This allows you to improvise a little and make the interview more natural. It also leaves room for you to learn things you may not have with a more formal script and  you’ll stress less about getting through a set number of questions, even if they don’t fit in the interview. Focus on hitting your main learning objectives, and keep it flexible beyond that.

Don’t Make Assumptions

Check your ego at the door. Conducting research is for learning about your users. Don’t be afraid to look (a little) stupid or ask stupid questions. Your participant may have a completely unique way of doing something you think is fairly standard.

Research is for learning and finding new ways to grow your product and business. Keep that humility at the front of your mind and you’ll be less stressed by minor mishaps and more focused on what really matters.

About our Guest

Jaclyn Perrone is a Design Director at thoughtbot. She’s also an artist, musician, and all-around awesome person. She hosts a podcast at thoughtbot centered around digital product design.

Transcript

Erin May: This is Erin May.

JH Forster: I'm John Henry Forster. And this is Awkward...

JH: Silences.

Erin: Silences.

Jaclyn Perrone: This is episode one, huh?

Erin: Yeah episode one, with a guest, where we don't-

Jaclyn: Cool.

Erin: Just sort of, talk to each other. So, thanks for being our guinea pig.

Jaclyn: No problem. I'm happy to.

Erin: So, Jaclyn. Who are you? What's going on with you?

Jaclyn: Okay. Yes. The big question.

Erin: Yeah, Erin starts deep.

Jaclyn: What is my essence?

Jaclyn: So, my name is Jaclyn Perrone. I am the design director of thoughtbot's Boston studio. So, that means, I'm also I'm still a designer. So, I'm working on, client projects, more of an advisory role, but I have some designers report to me. So, I'm doing the management part of that through check-ins, team check-ins and individual check-ins. Also I'm working on our engagement in the community. So, trying to find things we can do, how to meet other designers in the community. But also how to give back.

Jaclyn: So, working on, doing workshops, teaching students about design, things of that nature. And finally working on things our processes like hiring, trying to really just, kind of, hone in on a good process that works for everyone. So, I've been at thoughtbot for three years. I've been in this world for about 10. I started out as an account manager who wanted to learn how to code. So, I taught myself front-end development, did that for a few years, and then I wanted to be more abstract with it. So, I moved into UX design and then product design. And here I am. So, it's been quite the journey. I can't can't believe it. It's like I just started yesterday.

Erin: Fantastic. I always love hearing people's, kind of, career trajectories, in the user research space, in particular, because I find that careers these days in general or not, sort of, linear and predetermined but, I think, especially in the user research space, there are all these really interesting paths into how did you end up here? And no two paths are the same. So, it's really cool to hear about that.

Jaclyn: Totally. Yeah. It's been helping to just, kind of, follow my interests, and that's that's advice I give to the designers just starting out. Everything's up for grabs really. It's just a matter of what do you want to learn. And just, kind of, do what you like and there will be things to do.

JH: For sure. And in terms of how you found your way here to this podcast. I was going to give you the Chris Farley Show style intro in terms of, remember that time I saw you speak at Unbox it was really great and I asked you to come talk with us? That was awesome.

Jaclyn: Like Paul McCartney?

JH: Yeah, exactly. Yeah. And a big part of your talk was around, what happens when things don't go as planned and user research. And what really resonated for me is, I feel like in the hyper productivity, best practice world that we, kind of, live in, people always give you the perfect plan of, "Here's how you do this. And here's how you make your style guide. And here's how you do research." And they don't always go into the stuff that things happen and it doesn't go perfectly smoothly every time. Any time you're doing stuff with other humans unexpected things happen. So, I thought those are really cool angle, and seemed a cool thing to share with folks in terms of ... it's not uncommon and there are strategies to dealing with those things.

Jaclyn: Oh, yeah. Absolutely. I mean, you really don't ... you just don't know until you start doing it. And you can prepare as much as you want. But sometimes things just do go wrong, there is a ...

JH: Yeah, what was an example one? And then we can, kind of, just dance around and go wherever.

Jaclyn: Yeah sure.

Jaclyn: So, one of the things that would happen quite often is just you get to talk to someone and you get to show them your prototype, and it just doesn't work. Maybe you weren't expecting there to be no internet. One time, we were we were actually going to convenience stores and showing shop owners a tablet, a prototype, but we didn't account for the fact that there was no internet, and we were in a different country. So, we didn't have any data either.

Jaclyn: So, basically nothing works. So, what we did instead, we had our laptops with us. So, we had to, kind of, quickly pull up some Sketch files and just, kind of, walk through screens that way. So, it was, kind of, a bummer, because we went to go show this thing that we spent a lot of time on, and just didn't didn't happen for us.

Jaclyn: So, there's a couple of things, just in that experience. And sometimes too, actually a lot of times, maybe you're rushing through the prototype and you don't necessarily link everything up. Either, link them up, at all, or link them up correctly. So, a button, kind of, leads to a different place that you didn't expect.

Jaclyn: So, just, kind of, the first one, for us, It would have been really helpful to have printouts of our screen, if we're going to be in an area where we weren't sure if we had connectivity. Another thing to which would have been helpful is if we hosted all of our files locally. We were showing HTML prototypes. So, we were pulling in some libraries, like jQuery, from a CDN. So, if we actually had those files on our computer, everything would have worked just fine.

Jaclyn: So, just, kind of, making sure, just going offline, turning off your Wi-Fi, making sure whatever you're going to show, works without Wi-Fi. And also too, another thing ... hmm. No. I was on a roll, and I forgot the other thing.

Erin: I have a question while you think about that thing, and then you can cut me off if you think of the thing. So, I always remember the thing as soon as the other person starts talking. So, it's going to work trust me.

Erin: So, things don't go according to plan. This happens when human beings interact with human beings and when technology is involved, even more so. What's the more important skill? Is it just be prepared, be the boy scout, the Girl Scout and anticipate every, possible failure and be ready for it? Or the, sort of, improv skills of, when the thing you didn't foresee happens deal with it?

Jaclyn: Yeah. I think, it's really just making sure you're always focusing on the person you're interviewing, and being really receptive to them and listening to them, because a lot of these things, another example was people getting stuck on fake data. Really instead of, the first instinct is just, kind of, assure them that it's okay, that's we're not testing the data. But if they really keep getting stuck on something to, kind of, respond to that.

Jaclyn: And maybe they're getting stuck on something, because it's just not the right, kind of, content, at all. Maybe you have a fake event name in there or something. Maybe there shouldn't be event there, at all. You know what I mean?. So, really just kind of, I think, you're right. Just, kind of, responding to being adaptable in a situation and it takes a little while to get there. Because it can be really nerve-wracking at first. The first few ones I ever did I was more nervous than the person who is in the room with me. Because you don't know what to expect, but you have to just, kind of, remind yourself that they're just people. We're all people here. What's the worst that could happen?

JH: Yeah, just to piggyback off that a little. For me, when the prototype breaks, that's always the one I take the hardest or most personal, because it feels the most, "This is my fault. I did this." Whereas, if the person is not being as expressive as I would like or they getting confused, I don't own that so personally, and I'm a little bit more better at the improv side of trying to get them out of that. Whereas, when it's my fault, I owe that one harder. So, I think, the prototype one is a good one for people to be aware of and just, it happens to everyone and don't, kind of, bear the burden and instead, just try to troubleshoot.

JH: The other piece that, I think, is tough on the prototype one is, the more complex the prototype when it breaks, it's harder to get it back. When the simple ones break you're like, "oh give me two seconds. I'll just figure it out." So, it's, kind of, a double-edged sword of when it breaks it tends to really break. So, that's one where it's definitely a skilled to get, comfortable with that and figure it out.

Jaclyn: And sometimes if it's so unsalvageable, like what you're saying. I mean, the fact too, that if it was really that complicated, that's another smell too. What are you trying to test? Should you be showing someone's something that's. So, complicated? Are your test focused enough? Should you really be focusing on one workflow, or just just to, kind of, think about that too. Because you also don't want to ... there there's been a couple times in the past where let's say a client was really keen on just, let's just keep adding stuff to the interview.

Jaclyn: And then I always push back after the add another thing. Okay, let's just test one flow for now. We have plenty of people in the world we can talk to. Let's not overburden one person just to try and get some workflows in. You know what I mean? So, that's that too is important to is important to focus on. Focus on what really, kind of ... chat before you talk to someone. "What are we trying to achieve here? What do we want to learn? What are the assumptions that we have going in that we want to validate through to user testing?"

JH: Yeah. Absolutely. Anything you can do to tip the scales in your favor, is always a good idea, right? So, if you can just simplify or focus on a smaller set of problems for now, you're just making your life easier and improving the odds of success. So, I think, that's great advice.

JH: A guy I used to work with, UX researcher at Vistaprint, had this phrase. He would always say, "Test the test." Let's just sit down, me and you, and just run through it. And just let's test it ourselves, and make sure it works. There's something about that phrase, that always sticks in my head. So, simple, but it's just, let's just test it quickly ourselves, and then we'll flesh a couple things out. You almost always find something, and then go smoother from there.

Jaclyn: Yeah, that's the thing. You always find something. Totally. Yeah, it's always something. Yeah. it always helps just click through it yourself. And even that too. And sometimes we do a lot of design sprint's here. So, there's expectation that we build a prototype in one day. So, there's a lot of just, kind of, getting something done and out there. So, sometimes we don't make the time sometimes to actually go through it ourselves. So, we've got into the habit now, of just taking some time at the end of the day, just take out time and look through it, and actually go through and pretend that you're the user, and you just find little things. It's really helpful.

Erin: What are some of the other user interviews mishaps that you've had?

Jaclyn: So, yes. We mentioned the fake data part. That sometimes can really slow down an interview. If someone is new to this whole process. Actually, wrote a blog post too, that was for the user. I wanted to write something that we can maybe give out to someone, if we were going to interview them to, kind of, help give them some tips on what you expect. Just in terms of "We're going to ask you to think out loud. And we're going to show you some stuff that's probably not fully baked. And stuff's not going to be real." Things like that.

Jaclyn: It's just it's easy, because people want to be helpful. So, it is helpful for them to point out. "This thing doesn't make any sense. These numbers would never happen." But sometimes it really just takes a minute to just, kind of, have to explain to them that "Okay, absolutely. We are aware of this and it's just, kind of ... let's ignore it for now. Unless it's really just completely off by the content." But that too, that was something I would kick myself as well, because I would show something, and just be like, "Oh man. I really shouldn't have put that in there. Those fake names or whatever." I was showing a subscription. I was showing something to a customer service rep, and I had fake names of subscriptions, and the way I named it though, had meaning to them. So, they were just really stuck on, "Well, this this would never happen."

Jaclyn: So, that's, kind of, a thing too. Just, kind of, have to really either spend some time and get the content right. One thing too, I never use lorem ipsum. It's just not real it's. So not real. You don't really get a sense of character count with that. You're just, kind of, throwing in a paragraph and seeing just what text looks like. But any, kind of, context you can you can give, it's not in Latin would be helpful. And a prototype, just to also get a little bit of user feedback on whether you're going the right direction.

Jaclyn: But that's just one thing and I find that in an intro, having an intro where you are explaining, the nature of what a prototype is, is really helpful too, because we take for granted sometimes that people know what that means, in terms of tech and apps and stuff.

JH: Yeah. for sure. I think, there's a way to spin that one too. When it's fake data that somebody points out, is almost compliment them of being like, "Wow, you're, you're really good at this. You're. so observant. This is exactly what we want. I've noted that issue. let's keep moving. You're probably gonna find a bunch more stuff. So, we're excited to hear what else you notice or find." I mean, you can almost build some rapport and pump them up a little bit, by telling them that they fixated on the wrong thing, but you can, kind of, paint it as a good thing of their, observation skills, and that's what we're looking for, and just try to us positively a direct them somewhere else. You know what I mean?

Jaclyn: That's such a great point, seriously. Any opportunity you can have to boost their confidence in that, that's just such a great ... Yeah. That's a great point. For sure.

JH: The one that was on your list from the from the talk that I thought about a lot was when the facilitator is leading the witness or maybe doing something that you disagree with as the note taker or observer. That one is tough for me, because when I'm the note-taker usually somehow want to get involved, but it's I know it's gonna be disruptive.

JH: So, I remember you having a couple tips of how do more subtly give some feedback to the facilitator when needed.

Jaclyn: Right. And just to take a step back. The reason why this is important is, when you're doing an interview, with an interview with it with a user. They're really on your side. There should be the person talking to the user, do the interview, and another person taking notes. So, we want to again, to keep it keep the user focused on one person. So, we're having a conversation and there's someone in the background who's taking taking notes, right? And then when that person is talking to the user, and they're actually saying .... they're doing some bad habits, right? What you mentioned, the the leading questions. That's happened to me before and it was, oh man. I mean, really leading. "Oh wouldn't it be useful if this did this? Or wouldn't it be great? Or would you even it if it had this in here? Questions like that, that don't really give you any answer, really.

Jaclyn: So, I had take a minute. Plus, I was remote. I was a remote note taker. So, I couldn't really do anything. I was completely powerless.

Erin: like Slacking in the background, like I'm doing with JH right now.

Jaclyn: Yeah, totally. So, if you have a rapport with someone, and you feel comfortable. If during, there is also this thing you can do is a note taker is, you can have some post-its with you and start taking some notes. Even if you have a question that you want to ask, before you go into the interview just set up where you're as note taker, I know that you're going to talk to the user. But if I have any questions, I want to ask, I'm just going to write them on a Post-It and you can read them, and decide if you want to ask kind of a thing.

Jaclyn: That's worked out really well. And if someone's, kind of, taken a turn for the worse, feel free to write a note. Just, kind of, just a gentle reminder of, you're saying yes or no, or leading, something lie little hints. That's more of if you have that rapport beforehand, and they can read a thing that says, "leading" and know what you're talking about. But it's taking notes during the interview as well, is really helpful, grabbing examples of when they are writing down the questions that they're asking, because he probably don't even know they're doing it. It's just, kind of, in the moment, talking to someone.

Jaclyn: So, if you can, kind of, record what they're saying, just writing them down, afterwards debrief just, kind of, level-set a little bit and be like, "Hey, that was a obviously, it was a really great job. It's no easy task. Here, according to our scripts, we have more these kinds of questions, these questions in our script are more, open-ended, and you asked this, then asked this, and asked this. They were really leading. Maybe we can, kind of, just be mindful of that."

Jaclyn: It's all just a coaching opportunity.

JH: The leading questions stuff for me is, I'm going to talk out of both sides of my mouth a little bit, in the sense of very dichotomous person. Is that, I absolutely agree that, you got to avoid them if you can. It's definitely a skill to know how to do it, and it's something that is worth practicing and we're putting a lot of attention on. And the flip side to me is that I worry sometimes that it can be something that intimidates people from doing an interview, because they don't want to be leading, and they don't know how. When the reality is, even if you do lead someone, you probably don't do it all the time, first off. And then when you do it, there's a note taker, who can note down the time of that question and maybe disregard the insights that come after, because they were led.

Jaclyn: Oh yeah, good point.

JH: I mean, there's opportunity cost. Maybe you missed a chance to learn something with that person, because you, kind of, flubbed it. But the risk that people will paint right is, you lead the person and you make a conclusion that you shouldn't have based on that leading question, then you go build all this stuff. And that is certainly a real risk, but there are layers. The note taker might flag, "That was leading. Let's not use that information." Or if you're recording other people on the team might notice it and pointed out.

JH: So, I'm not that worried about some leading insight going all the way to production. And, I think, it can sometimes scare people away from doing the interviews, because they don't have that skill in their toolbox. And, I think, it's of those things that there's just a balance there. You absolutely should practice it, you should definitely try to avoid it. It's really important, but the odds that you're going to do it so, profoundly and so frequently that it skews the results, I think, are just a little overblown and, I think, there's strategies you can do to kind of, avoid those traps.

Jaclyn: Yeah, absolutely. And the more you do it, the better you are at it. And just get into the mindset, just get in the habit of starting with who, what, where, when, why, if anything. And as note taker, there's no harm in just throwing in a quick "why" if someone says "Wouldn't it be great if x?" And the person answers, "Yeah." Then you can just be like, "Why?" And then maybe they can elaborate a little bit.

Erin: That's a great tip. It's like, put a "Why" on top of it. So many times I've said, and I realize I'm asking that leading question as I'm asking it, right?

Jaclyn: Yes. Yes.

Erin: You acknowledge it, because I'm usually interviewing user researchers, right? So, they get what's going on. right? But then you just throw a "Why" on top of it and it really it, sort of, saves the situation.

Jaclyn: Absolutely.

Erin: So, if leading questions are, kind of, in this middle ground of not the best thing, but not necessarily going to destroy the interview. Do you have any, kind of, top always and nevers for avoiding a user interview disaster, for saving one that isn't going well, any, kind of, top things newbies should know? Or that even kind of, experts don't get right?

Jaclyn: Yeah. I think, one thing you'll have to be comfortable with, first of all, is silence, lots of awkward silence.

Jaclyn: I'm sorry.

Erin: I said, Let's take a moment.

Jaclyn: Yeah, right. Let's just reflect on that.

Jaclyn: Yeah being comfortable with silence. Not doing the thing where you ask a question and give an example right after. Like, "how do you share research? Is it through email? Do you text it?" Don't do that, because you're just giving them examples that they're probably going to spit it right back at you.

JH: Hand up. I do that one more than I should. I'm guilty.

Jaclyn: It's just ... it's fine I'll do it. But the thing that we do in conversation, right? But if you're trying to be an objective ... trying to interview someone and get them to fill your the skeleton you have of questions, and fill them up themselves. You're going to have to just, kind of, let them, kind of, make up the answers. But with that, also too, just checking your ego at the door is really huge. Don't be afraid to ask stupid questions. Maybe you know the answer, to something. Maybe they say "blah blah. Yeah, then I checked what the weather was like and I went to work."

Jaclyn: Like don't assume that they use the weather app on their phone. They might have a crazy answer. About how they check the weather. "Oh how did you know it was going to rain today?" That's a totally ... you might feel a, kind of, stupid for asking that, but it's not a stupid question. And sometimes you'd be surprised when you ask a stupid question that you think is stupid, the answers you get are just so enlightening.

Jaclyn: So, yeah.

Erin: That's a good tip too. I think, where it's not about you. It's like, you can look stupid, but the job is not to, sort of, impress the person being interviewed, you're trying to learn stuff, and, I think, a little humility, on the part of the moderator can go a long way to making that person feel more comfortable. But also to recognizing, sort of, the point of the whole thing.

JH: A word that, I think, plays into that and contributes a little bit is, validation. That gets used a lot in design or product right of ... let's go validate these ideas. And there's this, kind of, assumption in there of, we're right, we just need somebody to tell us we're right. When the goal really is to have a conversation and to learn something. And hopefully what you learn does support your thinking and idea, but it's not a bad word, but, I think, it has implications that, I think, do, kind of, subtly contribute to that of, all right, we know this weather thing, so let's just go confirm it. And that's ... you're, kind of, getting a step ahead of yourselves in a way that, I think, can be detrimental.

Jaclyn: Yeah. That's a really good point.

Jaclyn: Yeah and what you're saying too. Have a conversation. That's huge too. Just, kind of, go with the flow, asking follow-up questions. Yeah, don't be afraid to ask a stray, random question about something that they just said. And having a script is great, for sure, but just, kind of, knowing that you're not going to probably ask all those questions.

Jaclyn: And you won't because you'll be just having a conversation and, kind of, seeing where it goes. I think, it's helpful to note, what are the questions you definitely, definitely, definitely want to answer. Things you definitely want to ask them. And then what are the rest of the stuff that if you maybe have an extra time extra time, you can, kind of, just fall back on. But having a looser script is really good. Starting out, for me, starting out, I wrote questions for sure. But as I've done more and more and more, I get way looser. I more have topics too sometimes. I used to go in interviews with not even any questions, at all. But just have topics that I wanted to cover. And then just have them, kind of, have them lead it, really.

Jaclyn: I mean, you're supposed to really do maybe two percent of the talking. So, I really do, kind of like, going into with a very loose guide. Because it, kind of, helps me focus on them more than on what this next thing I'm going to ask in my next line of questioning.

JH: I'm a big believer in the loose script as well. And, I think, even if you are going to have descriptive questions, you're going to stick to it. One of those important things with that approach even, is that you're not going to hit them in order. Someone's gonna say something in question two that's a natural segue to question six, and just go with it. And you can ask question 3 when you get there. If you do want to go through all your questions, that's fine, but at least be a little bit on your feet, and move around as needed.

Jaclyn: Yeah, that's that's a really good point. And just being responsive. And what we were saying before, being adaptive to the situation. Jumping on a chance to, kind of ...if you stray a little bit, that's okay, but you might make a whole new learning that you didn't even think would happen, by just, kind of, asking a one-off question. And with that too, making sure you're being an active listener. You hear them. They're saying a thing, and you nod and you say, "Yes, that's interesting, tell me more about that."

Jaclyn: Your job is to make this person, who you just met three minutes ago, feel so comfortable that they're telling you how they feel, they're telling you stories, they're telling you why they like a thing why they don't like a thing without any fear of judgment. And you do that by just making them feel comfortable. Kind of, check where you're sitting. Sit next to them, if you're going to show them a thing.

JH: Cool. Should we just wrap up? I don't know if there's any closing words or one final piece of sage advice you want to throw out or?

Jaclyn: I guess, when all else fails, just focus on the user. Just focus on the person in front of you. Yeah, be respectful, show that you're listening. If you just focus on them, you can't go wrong really. So, just relax, and you're just talking to someone, that's going to be okay. Everything will be okay.

Jaclyn: Let's let's keep in touch.

Erin: For sure.

JH: Cool. We'll do that.

Erin: On the e-channels.

Jaclyn: Oh Yes. The electronic channels.

JH: Yes. Stay in e-touch.

Jaclyn: I definitely will.

JH: Alright, see ya.

Jaclyn: Alright, see ya.

Erin: Thanks for listening to Awkward Silences, brought to you by User Interviews.

JH: Theme music by Fragile Gang.

Erin: Editing and sound production by Carrie Boyd.

‍