Resources Mentioned in the Episode

Expiwell - Diary study tool

Adobe XD - Customer journey mapping tool

About our Guest

Tony Turner is the Lead UX Researcher at Progressive Insurance. There, he leads all kinds of user research, like usability testing, contextual inquiries, card sorts, tree studies, first click studies, surveys and interviews. He has previous worked on UXR teams at Phillips and Hyland. He's interested in HCI and studied Cognitive Science during undergrad.

Transcript

Erin: [00:00:00] Hello everybody. And welcome back to awkward silences. We're here today with Tony Turner. He's the lead UX researcher at progressive insurance. Today. We're going to be talking about diary studies, which are one of our favorite methods. And particularly in these. Remote times can be a great alternative to some in person, methods, talking about diary studies and how you can use them to evolve your products as your users evolve. So we're so happy to have you here Tony. Thanks for joining us.

Tony: [00:00:30] No problem. I appreciate being here as well.

Erin: Got JH here too.

JH: [00:00:35] Yeah. I actually don't know much about how to run diary studies. I know them at like high level, so I'm super curious to learn how to pull those off and do them in a really thoughtful way.

Erin: [00:00:43] No. All right. So Tony, I imagine some of our listeners are very familiar with diary studies and others aren't. So let's just start from the beginning. What is a diary study?

Tony: [00:00:55] so a diary study is a longitudinal study, which means that it's happening over time. so it's self reporting, behaviors, frustrations, opinions, desires, things like that. From the participant. so they'll, be shown some questions and as they interact with whatever your product is over time, they'll respond to those survey questions. So like I said, it's self reports, so you're not actually interviewing or observing anything. but it is in context. and as you said in these times, everything's remote.  It's just the, a study technique, for getting at, how someone's interacting with stuff over time.

So it may run like two weeks. It could run a month, it could run longer. and you want to get, responses to your prompts, in intervals. so maybe every day it could be everyday. It could be every week, depending on the product. Like I said, the goal is to get those in context, momentary insights, from the participant.

JH: [00:01:53] Is this our diary study is like a big part of your toolkit or are you running them pretty often or is it something that like only in certain circumstances, do you find it to be a, to be worth it?

Tony: [00:02:03] Yeah, we don't run them too often. we're ramping up now especially now, as you mentioned, Erin, in this time, we want to ramp up all our remote methodologies. So we're definitely doing more of those.

JH: [00:02:15] what were some of the previous considerations in the past that would, cause you to think like, Oh, this is a good case for a diary study.

Tony: [00:02:22] so anything that, is happening over time. so there are some products where for example, a claims rep is dealing with different scenarios and things like that. And we can't really get at everything, all of their experiences from. just an interview or an observation because they, these things happen at different times.

So it's really beneficial for stuff like that. And like with agents as well, as they interact with customers, they may have these unique scenarios that come up. so we want to learn over time with those experiences are.

Erin: [00:02:54] Got it. so one way to do user research, the kind of, I guess not longitudinal more common way we hear about is let's say you wanted to talk to somebody, about their experience, filing a claim. you could find people who had just filed a claim and talk to them. But, by contrast in a diary study, we can meet the person at some other point in their customer journey. And then presumably at some point in that journey, they might file a claim. And so what's the advantage of talking to someone, over that larger period of time, as opposed to, let's just talk to someone right after they, did that behavior that we want to learn about.

Tony: [00:03:32] Yes. So it's really getting the sort of context, that the participant is living in. so then they file a claim at one moment. but before that they may be doing something else. They may be crackling with different apps. They may be focused on some other task, especially even in the insurance context.

And after that they may be doing something else. So getting a broader yeah. View of that context is really helpful. so we can say, for the claims example, Give us your feedback in the moment as you're completing this claim or after you're done completing the claim.

And that's helpful because if we're interviewing them, they may have forgotten something that happened in that moment. and, we may not be able to get those insights, but we can also say let us know what's happening at these specific intervals of time. they may have some kind of feedback on something that's happening before they actually complete the claim, so yeah, it's just beneficial to get that broader context, of use.

JH: [00:04:31] Like something to where people probably just aren't great at like recalling negative experiences, even if they were somewhat recent. I'm just thinking of myself. And recently went through the process of refinancing our house and there's a bunch of headaches in that whole process. And now I'm a few weeks later and I can, I don't even really like with any sort of like Vivian detail recall for you, like some of the hiccups or parts where I got stuck, in that moment, I remember being frustrated at him full times and I like right now, I'm like a little bit like at a blink for what it was.

So is it some of it just like the vivid yeah. That people are able to provide in the moment versus trying to recall it.

Tony: [00:05:05] Yeah, exactly. That's the goal we're trying to get at. so part of that is they can capture, images and videos and things like that, in the moment as they're experiencing those things. yeah. And there may be good things or maybe bad things that they're experiencing, but, but we want to capture as much as possible and then produce a journey map out of that and try to understand that actual context that they're in.

Erin: [00:05:27] Yeah. You talked about there being, you might do a two week or, a two month or even a two year diary study, and that it's, your sort of product that would determine what you might want to do. Obviously a longer study is going to be. more budget, harder to find a participant to, want to participate in a two year study.

But how do you find that right balance of how long and how frequent should those checkins be within that length of this?

Tony: [00:05:54] Yeah, it just depends on the initial understanding of the use cases. so we went investigate those before we begin the study. so if someone is Interacting with the solution over a long period of time. So like a customer that may want to go in and service a policy, things like that.

We may want to do a longer term diary study and with those we don't typically ping them. We don't ping them based on, time. So we don't ping them like every day. But we say, when you go in and complete this specific component of your insurance journey, try to let us know, and we can, use analytic data to say, okay, we know that they went and did this, so now we can ping them and see what their experience was.

So that's a longer term thing, but the shorter ones where we can actually get insights, based on, their experiences in a short amount of time would be like with agents where they're interacting with different customers all day long. so if we ping them a little bit, like every day or twice a day or something like that, we can definitely get some insights and we don't really have to do that for more than a couple of weeks or a month or so. So those are the differences.

JH: [00:07:03] And like specifically, like, how are you doing the pings? Is it different study to study or depending on the tool or I guess for me, I just imagine that a lot of people probably forget to log an entry someday or during some event. And so what are the things you do on that? Like pinging front or notification front to try to make sure that people are complying and actually keeping up with it.

Tony: [00:07:22] Yeah. So it's typically emails and we can automate those emails with XB. the solution that we have, so they send out an initial email kind of thing. this is the study and do you opt into it? and then from there they'll get the regular, Emails, either based on their interaction with the system or based on, the timing.

So could schedule it to be every day. And that just goes out, or we could say, like I said, we could, have some sort of analytics and then manually ping people based on their interaction with the system. So it kinda just depends on how we set it up.

Erin: [00:07:58] Gotcha. And what was the name of the platform you said you were using?

Tony: [00:08:02] Uh, Expiwell, E X P I W E L L. Yeah.

Erin: [00:08:06] Gotcha. Cool.

JH: [00:08:08] Yeah, it's a new one to me. Cool.

Erin: [00:08:10] Great. Okay.  you mentioned using analytics to see when someone has, hit a particular milestone or behavior in the app. Do you, I'm curious, do you combine the self reported videos or notes that people are sharing about their experiences with, data you can actually, passively get from them and their behavior in the app, or how do you think about the kind of self-reported versus inferred and the quantitative and the qualitative together?

Tony: [00:08:39] For sure. Yeah, definitely. We, we combine the two, self report. Isn't always reliable in terms of this specific thing that they did in the system, but we can mirror those things together. And get a bigger picture of it. so yeah, we definitely do that. if we're looking at it from a sort of a use case basis and an interaction basis, so when we're doing it based on just pinging at a certain time period, we don't really typically have the analytic data because we're not opted into that. So it's typically just the self report that we have in that case. but it's still valuable in terms of kind of understanding their perspective, their emotions, and things like that around the system.

JH: [00:09:22] Kind of unique challenge with like long diary studies, let's call it like a month. Is that it something up in the prompts or the cadence or whatever it is you might not know for a while. Whereas if you're doing like one-on-one moderated stuff, maybe it's a half hour, maybe it's 60 minutes and you kinda ask a leading question or whatever, you can adjust really quickly and make sure that for all subsequent participants, you correct that, like what are the big mistakes to avoid with diary studies so that you don't get to the end of a month long study and have in have nothing to use cause you messed up some stuff earlier in the process and just didn't know about it.

Tony: [00:09:56] Yeah, so you want to use the right kinds of questions. so it's a combination of open ended and closed questions. You don't want to focus on just one. Cause then you limit yourself in terms of the type of feedback you get. but as far as the open ended questions, just asking them what did they want to try to accomplish?

And, did they accomplish that? what did they do with the tool today? Things like that, just to get them to think about specific interactions that they're having. I think that's the key. And also to just do a dry run, with a couple of participants over a little bit of time, just to make sure that some of those responses are what you're looking for.

So that's really important as for any kind of study, but it's super important for a diary study. Cause as you mentioned, if it's something that's happening over time, you don't want to find out later that, That you had some problems with your question. So I think the main response to that question is just, having a combination of open ended questions and then like likert scale type questions, like how, like on a scale of one to five, how do you feel about the solution today? things like that, just to get some statistics together.

Erin: [00:11:03] I know we wanted to talk about like temporality in design, which I'm really interested in when, the anti. Diary study of, we are working on this product launch, and we're going to talk to people who are doing that thing and get some feedback, and then we'll launch the thing as opposed to something more temporal that changes over time, same as the user changes.

And I'm curious, just to start, how do you find good participants to participate in that kind of research where, you know, who's going to have the kinds of insights that are going to be meaningful. To you over time, for instance, how do we know that they're going to reach the milestones that we might be interested in?

And then on the other side of it, how do we make sure we're built to be able to update our product so that it changes with users over time? so that's a lot of stuff, but I'm curious how you think about bringing all of that together.

Tony: [00:12:02] Yeah. So I think the key thing is to, focus on recruiting early. so having screeners that, get at how, interactive the participant is in terms of answering questions and things like that. So having questions in the screener that give them an opportunity to be verbose, or not, learning about them that way.

And then learning about their previous experiences and how long they've been using certain applications that may be related and things like that to understand, You know whether or not they're going to use the application over time. so we never really are sure about that. it's just hopeful in terms the screener responses.

But we do try to focus in on our participants that are likely to actually use the system over time. as far as like making sure the application responds to the user over time, with the diary study, we're learning that as well. We develop the solution, So they may, the solution may not actually be adaptive in the beginning, but as we learn about their context changes and what, how, what things they interact with the most in the system, then we can start to adapt the system to, to the users for the future case. So that's kinda how that works typically. 

JH: [00:13:16] Jumping back a tiny bit. Is there like a standard rule of thumb of how many completed diary studies do you want? Like you hear five a lot, for one on one interviews. do you have a number that you try to work off of for diary studies?

Tony: [00:13:27] Yeah. So we try to get like 10 or so participants in a diary study. and, ideally they're all responding. Sometimes we don't get, that many people, to actually respond to all of the, all of the pings, but we usually get most, most. People to respond to enough pings to actually get some insights out of it.

So from our end, we typically try to go with 10 or so. So we'll recruit over that. just to have a bit of a buffer, but we try to get 10 or sober to support then to go through it. and ideally we actually get a number of responses from that

JH: [00:14:02] Cool. And is it like with those 10, you get what you get, maybe, a few of them do every prompt all the way through, maybe some start off strong, fall off in the middle, but resurrect and answer some of the prompts at the end and that's okay. Or do you like people who, have that down period in the middle, do you exclude them and, and you only take the people who made it all the way through.

Tony: [00:14:24] since it's momentary anyway, they may be in a case where they don't really have anything to say. so if we're pinging them based on like in every day, timing or something like that. Like every day at 12 on Tuesday, they may not have anything to say. So they may not say much, or they may not say anything at all.

And that's okay. as long as we're capturing when they are experiencing something of significance to them, and as far as getting pains, when they, are actually interacting with the system that too can just, be something depending on whether or not they're actually experiencing something critical or something that's interesting enough to talk about.

So if they only experienced those things toward the end of the study, that's okay. We'll still keep those results.

Erin: [00:15:07] It feels like, you know, w we work on recruiting. That's what we do, but this feels like one where recruiting is so important. And I know you talked about, you want to get the screening process going early and make sure folks are, Answering questions with you. No more than yes or no answers.

And have something to say, are there other things that you look for to make sure that someone is going to make a good, participant in your diary city

Tony: [00:15:33] Yeah, I think the main thing is just their experience with, with the system, that we're looking at. So we may want to get participants that have no experience at all, that are at the beginning stages of interacting. And we want some participants that are experts. So in the case of like agents and folks like that, I mentioned, we don't want someone that's necessarily brand new to the system, always.

Because they may not, they may be struggling with things that are understood to be problematic. Whereas someone that's an expert may have more nuanced feedback in terms of things that we may not have understood to be an issue, in the beginning. and we want it from both. We want both perspectives.

So we have to recruit for both, but it just kinda depends on the study in terms of what we're looking for, but that's another key component. and then, yeah, and then, like I said, just focusing in, on, focusing in on whether or not they, been able to respond to things appropriately in the screener.

JH: [00:17:13] have you seen, or I don't know if you've done it yourself or seen other researchers take advantage of other tools that maybe give you similar insights. And I guess what's coming to mind for me is, Apps and sites increasingly have those little chat widgets, in the experience.

And I'd imagine that when some people decide to write into those things, unprompted, it is very like momentary feedback. And it's I can't figure out how to do this. Like I'm going insane. and it feels maybe similar to what they might write in a, in a diary prompt on that day or in that moment.

are there things like that you can get some of those similar insights from, or is it like totally different and not a good idea to try to extract stuff from places like that?

Tony: [00:17:48] No, it's definitely a good method. So we use that too, in terms of intercepting folks like on the website and, try to get feedback that way. it tends to be negative if they not opting into a study. so when people respond to that issue, usually, I hate this because I couldn't get this done.

And it's a problem, which is good feedback. But we also with diary studies get at general experiences and things that went well, for them, So unless it's something that was just extremely, amazing for a person, they don't typically respond to that with the intercepts. so that's the difference there.

JH: [00:18:24] Gotcha. Yeah, so it's just not, it's not this, it's not the full picture. The way you can get through a proper diary study.

Erin: [00:18:31] I think, people are a little afraid of diary studies and  a lot of the battle is figuring out. The right method or the right combination of methods for the right research question. And so I don't think you, or anyone would say, go use diary studies all the time for everything. but it sounds like having the right tool  can really help to make them work better for you. What are some of your other thoughts on. getting folks to maybe try out diary studies in a way that's approachable and also active to what they're ultimately trying to accomplish.

Tony: [00:19:07] Yeah, I would say, just start small. you don't have to have a ton of participants in the beginning to get insights, similar to like you were talking about with usability tests. even if you have just like five folks, you can really get a lot of insight from that. We try to go with at least 10 because, There tends to be a lot of variation in terms of those particular experiences agents will have.

But, but yeah, we, we kind of work in, in a way that you can learn a lot from diary studies without having a ton of participants. so I would say that just don't worry about having a ton of participants in the beginning. and then also don't worry about pinging people very often. So if you're just starting out, try to, maybe have it, just have a couple in a week, and see what you get from that. And that way you can use your participants into it to as much, you're more likely to get, longer responses with those few ping. so that's another sort of, Tip, I would say if you're just beginning, of course, as you develop your methodology more, so you want to ping more often, and get more data to analyze.

Just because things happen more often if, things happen more often than just a couple of times a week. So you want to try to get at those events as much as possible. but in the beginning, I would say just go easy and get as, get as much feedback as you can from a couple of, from a couple of engagement.

JH: [00:20:32] Jumping back to the analysis piece real quick, a question I was wondering about is, with moderated or unmoderated sessions, People typically like to try to like pull quotes or sound or, recordings to let the users in their own words, talk about, Whatever it is, they're sharing it.

What seems unique about diary studies is that over the course of time, you get to know each participant to some degree. And you might realize that this person's skews really bubbly or positive. And so like on message one, I thought they were just super excited to buy like message five.

I realized it's just like how they communicate. Do you try to like normalize or put that in context at all since you have that richer perception of like how somebody might communicate or is that just like too much and you just take the parts you need.

Tony: [00:21:14] No, we definitely do that. we try to get an understanding of the participants as they interact with the thing over time. So when we do the journey map, we map out their emotions, And we may see that, one participant, like you said, is always positive and always giving these glowing, responses and things like that.

Whereas, five of the other participants have issues throughout, so we can weigh those that way. so each one has an individual journey map, but there's also the combined perspective and those things balance out in terms of that overall sort of. feedback that people have with the system.

JH: [00:21:54] Artifact, look like, like, what are you creating? What tools you're using.

Tony: [00:21:58] so we just use Adobe XD. we have a template that we use, and, it has, like I said, the emotions that people experience, all those critical decisions and moments that happened throughout the study, And then he insights that we got from those moments, things that we learned or things that we can interpret from the study. And we also show which persona is associated with that particular journey map. so we mapped that back to a persona. And if it's one study, typically it's one. A persona, but that can vary as well. depending on the use cases and things like that. but yeah, those are the main components we just want to get a map of those experiences over time and just plot those critical moments of those experiences.

Erin: [00:22:45] I'm curious, one of the things I think about with journey maps, right? Is. If you're working with one, one persona and one diary study, like you said, you often are, do you build a journey map and then come back to it in other research and kind of iterate on that over time? Or do you have, a million different customer journey maps for every study you've ever done?

Or how do you go about creating, I guess the right number of journey maps that are usable and keeping them up to date, or do you do that?

Tony: [00:23:18] Yeah. So we have a sort of high level, experience for the persona. we like to have a journey map for each participant, just to understand the differences and the nuances and things like that. but we have a higher level journey map that we create based on that particular persona.

And the feedback we've gotten and yeah, we iterate not just based on it's on, additional research we do. so we may go out and do observations or interviews, things like that, or additional diaries that is, but also just insights from the product team. and Their perspective on the issues that came about, and what they were going for when they design the solution or develop the solution and kind of how that relates to those responses, that we've gotten. And we also take those diary studies and connect them together in terms of that longer term process. So we may focus on, like you were talking about the claims process, kind of someone submitting a claim. We may focus on that and then connect that to, the beginning of the process when they first signed up for insurance.

And then, later on in the process when they're doing servicing and things like that. those are the things we do with the diary studies.

JH: [00:24:28] it almost seems like a case where it be fun to do like a premortem one before diary study, like have the team draw out what they think the journey map looks like, and then you can overlay it with the actual one and realize Oh, we were way off in this step of the process. People perceive it very differently or something like that is that you were playing around something like that.

Or is it more focused on the output you get from the participants?

Tony: [00:24:48] Yeah. We don't do a journey map beforehand, but we do get Assumptions and hypotheses and things like that from the product team. and use those to create point of view questions for the diary studies. And, yeah, like I said, we use other methodologies too. We'll do longer term interviews and things like that. So we keep the diary studies pretty high level in terms, those are the questions. but we still reflect on those assumptions, that the product team has.

JH: [00:25:14] what do the participants or the users think about at the end of the diary study? Or are they usually, that was a fun journey and they were like, happy to have done it. Do they feel tired? Cause they had to answer all these prompts. do you have any sense of, How much users enjoy participating in this like longer term kind of longitudinal thing like this.

Tony: [00:25:28] Yeah, I think people generally, and this is with all the research that we've done. all the different methodologies, people generally are happy to have gotten the opportunity to share their perspective. Like I said, in this case, they may be a little tired from getting all the prompts and things like that.

But once it's over,, we do do a survey to see, what their perspective was on it. and they typically say, it was nice to be able to share, share my experiences and things like that. So that's typically the general, response to it.

JH: [00:25:57] How do you think about incentives for longer studies like this?

Tony: [00:26:01] Yeah. we typically, do a higher incentive depending on the length of the study. so we give gift cards for external participants. for internal participants, we don't typically give an incentive. So those, these tend to be shorter. and like I said, with the, like with agents and claims people, we can usually get a lot of insights in a shorter amount of time anyway. But yeah, we tend to give, larger incentives for the external participants, for diary studies. and we used to give out physical, we used to send physical gift cards, but, we've been doing like electronic gift cards and stuff now, lately.

Erin: [00:26:38] Do you ever double down on your, we're talking about you get to know or participant over time with a diary study. Do you ever. Reinvite past diary participants for, round two round three or different studies or

Tony: [00:26:50] Yeah, it's something I would consider so far. I haven't done that. because they've, they may be, burnt out like you were mentioning before and they may, they may be biased in terms of having experienced that solution in the past. But if it's something new, definitely it would be nice to, if it was a really good participant to have them participate again.

And that goes back to incentives and things like that. But so far, I haven't really reengaged with, With previous participants in diary studies, we have with other types of, with usability tests and stuff, but not so much with diary studies.

Erin: [00:27:24] Yeah, fantastic.  What do you love most about user research?

Tony: [00:27:29] I think just getting to help people. it's nice to be able to learn things from people, and apply that to a solution, provide recommendations and things like that. so I think a bit similar to like counseling, but we're not actually, counseling people, but we're also still helping them get through particular, things in their life that are important, like work or, different aspects of insurance and things like that. So I think that's the main thing for me, just being able to help people.

Erin: [00:27:58] Well, thanks for helping us out today.