This chapter explores diary-based study methods, including a review of multiple forms of diaries and several basic study designs for employing diary-based research.
In a diary study, participants keep a log of thoughts and experiences either about a product, a process, or some sort of activity in their real-world environment. It’s a record of their behavior that researchers later parse and analyze to better understand the quality of the user’s experience. It comes with a fun term that describes its timeline: longitudinal! This means over a period of time, usually from a few days up to a few weeks.
Diary studies get into the nitty gritty of what motivates us to take certain actions, why we engage in certain behaviors, and our attitudes about things. You get to see habits and patterns as they’re exposed over time. And they’re great at answering those “why” questions about user behavior.
Diary studies are unique from other research methods in that they’re a relatively hands-off and economical way to gather loads of useful qualitative data (and sometimes quantitative data, too).
There are lots of avenues researchers can take to gather contextual understandings about users in their natural habitats, but some solutions are more costly than others (in the case of extended field studies, for example), or somewhat unnatural (in the case of asking users to repeat real-life behaviors in a lab environment), or unreliable (as can sometimes be the case for trying to understand context, retroactively through a survey, say.)
Diary studies are best applied in cases when you’re looking for an affordable way to observe user behavior in a natural real-life environment as users are living through their experiences. They’re great for gathering data over time as opposed to one-instance-only type studies, like observing users in a lab on a single visit. They’re also appropriate when you’re looking to generate new qualitative and quantitative insights in the earliest stages of the development cycle.
Diary studies have a place in many different types of research, but they are especially useful in UX studies because of the fine-grained and varied information they can provide. It’s one of the few ways to really get a peek into how users interact with your product in a real world setting over a period of days or weeks.
You might find yourself using multiple methods to gather user experience data over the course of the development process, because each method has its own rewards. Diary studies are a great choice when you want to examine the user experience in real-world settings over time.
Diary studies give you insight into where and how in a user’s day they reach for your product, or when life interrupts use of your product. You can see how they’re feeling as they’re interacting with your product and begin to infer patterns in their behavior that might guide you toward decision-making. A diary study brings you as close to your customer’s true experience as possible in the least-invasive possible way.
Diary studies work best in the discovery phase or the very first stages of product development.
You can use diary studies to look at products you want to create or replace, or problems you want your product to solve. Diaries can help you better understand the precise layers of the pain points you seek to solve.
They’re also useful for testing new products in order to identify any necessary changes in the early stages, when changing the course of a build is still not too onerous.
Diary studies can also make sense at the very end of the development cycle. At these later stages, you can dig into the user experience to see if your products are being handled in the anticipated ways in order to fine-tune details.
Actually conducting a diary study has five main phases: planning, onboarding, monitoring, debriefing, and analysis. It’s important to put roughly equal energy and effort into all five phases to get useful and usable results.
Planning begins with identifying your research questions and ensuring that a diary study is truly the most appropriate method for your goals. There are also many different ways to conduct a diary study—which one you choose depends on the kind of information you want, the needs of your participants, and the resources you have. Here are some of the choices you’ll want to consider along the way.
The most basic diary is a notebook and a pen or pencil, a low-cost and straight-forward option. Unfortunately, not all possible participants have legible handwriting, and handwritten diaries must later be digitized, adding another step to the processes of analysis and archiving.
Smartphones have become so ubiquitous that using an app for the diary may be more comfortable for many users, though typing a lot of text into a phone might be uncomfortable for some. Electronic diaries can also take on other forms beyond the written word: entries can be logged by voice, video, or photos.
There are several well-rated diary apps already available in app stores (we cover a handful below in the tools and logistics section), or you can create your own. Using an app does mean you will have to train participants in the use of the app—and there are people who don't have smartphones. You'll have to decide whether excluding phone holdouts is likely to hurt your study by excluding participants who could be good representative users of your product. You can also combine methods: like creating a daily or regular-use questionnaire via mobile that can be paired with typed or handwritten notes. Just be sure not to over complicate things, either for yourself or your participants.
A free-form diary is similar to the personal diaries or journals some people already keep. Participants just write. Even with the most free-form style study, you’ll still want to give your subjects some initial guidance. But by and large it’s quite loose, and the subjects determine how and when to record feedback (when they’re already using a product) or to record their experience as they accomplish certain tasks or go about their day.
A structured diary is more like a survey, with closed-ended questions on pre-set forms.
Mixed approaches of various kinds are also an option.
Free-form diaries require less training and encourage the user to speculate and to offer information you didn’t ask for and might not think to ask for. The risk is that the participant might not include the information you need—or might include way too much information. Free-form diaries are also less straight-forward to analyze.
How much structure you use depends on where your study falls on a continuum between the need for open-ended free communication on the one hand, and clear, concise answers to specific questions on the other.
Interval-contingent diaries have to be filled out at specific intervals, such as one entry per day, one entry per week, one entry per two hours, and so on.
With signal-contingent diaries, you arrange to tell the participant when to make an entry with alerts or calls or communication of some sort.
For event-contingent diaries, the participant makes an entry whenever something happens—for example, after using your product, or when it begins to rain, or whenever they receive a text message, or when the baby needs their diaper changed—all depending on what you need, of course.
The methods you choose depends almost entirely on your research questions.
You may have to conduct a pilot study prior to conducting a full-scale diary study, to make sure your instructions are understandable, and to check on the feasibility of other aspects of your design. Of course, once you have established a particular study design, you can re-use it for multiple products without having to repeat the pilot study stage. If you do a pilot study, the time, money, and other costs of the pilot must be included in your overall plan, together with the costs of redesigning your study should the pilot reveal that you have to.
Depending on the nature of your product, your diary study could collect sensitive or personal information about your participants. You will then have to take steps to either safeguard or destroy all such information. Your recruitment and onboarding processes should include advising participants that you’re collecting such information, what the risks are, why collecting this information is important, and what you’re doing to minimize risk.
Recruitment and onboarding includes both signing participants up and training them. Training can be remote, via video, or in person, by inviting participants to a training session. You’ll want to be sure to recruit the right subjects for your study, and then you’ll want to be sure to train them properly, covering in detail exactly what you want them to do. You’ll want to arrange a get-together of some sort, either in person or on the phone, to share details of the study ahead of time. It’s a good idea to set expectations by going over timeline, schedule, and expectations. If there’s training, in the way of apps or products or tools, walk them through what they need to know. Also ask them if they have any questions for you.
After training, once the study begins, it’s best to check in on your participants early and often. You may need to remind them to make their diary entries. They might have questions for you about the process, and you’ll want to be available to address these as they come up. How closely you monitor participants throughout the study depends on your study design and perhaps also on the needs of individual participants. The longer a study is the harder it will be for participants to stick with it. A few weeks is a good maximum length for most diary studies.
After you collect all the completed diaries, take the time to speak with your participants about the study to gather further insight on your product and on your study design. To prepare for the debriefing interview, review each participant's diary entries. You might also want to begin the analysis process to prepare your debrief. Then, ask your participants to expand and elaborate or clarify where it’s needed. The results of your debriefing interviews should be incorporated into all your other study results and analyzed together.
Analysis transforms your raw data (the diaries, debriefing interview transcripts, and whatever other information you have gathered) into insights you can use. There are a handful of methods of analysis available, from the relatively casual and straight-forward, to the formal and statistically rigorous. It will help to figure out what type of analysis will best serve the study before you begin, since analysis methods must be matched to both your research questions and your study design. Above all, avoid spending the time and money to collect data that you will not be able to analyze.
This stage includes transcribing the diaries if they were hand-written or voice-recorded.
Documenting your results means getting your data into a manageable format that can be accessed and used by anyone who needs the information and insights you gathered.
For the study itself, the main items you need to provide are your product (if there is one) and the diaries (or the app or service that functions as the diary). There are a host of off the shelf options already available for recording and logging data, and you’ll want to spend a little time doing your research to find the best solution for your study, and for your researchers and analysis team.
Below are a few recommendations for apps and tools. This is not an exhaustive list by a long shot, but they do represent a variety of the options and styles currently available:
For training events and debriefing interviews, logistics may be a little more complex, especially if these events take place in person.
The main logistical difficulty often involves scheduling, since once your study goes live—once you have participants out there actually using their diaries—you (or someone you assign) must be available to monitor the study until the diaries come back. Then the initial analysis must be completed rapidly so you can review the diaries and schedule debriefing interviews while the experience is still fresh in the participants' minds.
If you are collecting and retaining sensitive personal information (such as where participants go during the day, how they use social media, or any medical or banking information) you will need some way to keep that information secure until it is no longer needed and can be destroyed.
Since the company’s founding days, Intuit has stressed the importance of studying customers in their normal everyday environments. Intuit oversees a line of business and consumer-oriented financial tools that more than 37 million customers use each year to track expenses, manage payroll, and pay taxes. Diary studies keep the company close to customer behavior. Teams that participate in these studies are cross-functional, consisting of research, engineering, product, and design leaders. These observational research studies uncover points of feedback that may otherwise remain undetected.
By observing someone in their natural habitat we can determine how often they get interrupted when they are trying to do taxes, payroll, or perform some other task. Many people are probably not even mindful of how many interruptions they get or how many things distract them as they work on finances.
This perspective helps Intuit’s product leaders understand how often customers are switching devices, for example. Research teams publicly share the findings that they observe—by posting information around the office, for colleagues to see.
Diary studies can be combined with other methods in order to expand the reach of the study while keeping costs manageable. As we’ve discussed, diary studies are best applied to observe natural use of a product. So, it’s handy to combine diary studies with other methods that might fill in any results-blanks and support this same discovery stage of research.
Surveys are most commonly paired with diary studies. For example, you may use a widely-distributed survey to collect the information you’ll use to develop the diary study. (Example: a survey might tell you how many times a day people drop their phones on the floor, and in a diary study, you get to learn more about the circumstances under which they drop their phones.)
Or, you could use the results of a diary study to develop further research questions to explore in a survey. (Example: a diary study might show that people are super diligent about flossing their teeth in the few weeks after a cleaning, and less so over time. You could follow this up with a survey to ask what might motivate them to continue flossing consistently.)
Every study variety has its challenges, but the good news is: the challenges you might encounter with diary studies are manageable.
Problem: Keeping a diary is a lot of work and many potential participants might lose steam or simply lose the will to put in the effort. Not everyone who agrees to begin the study will record diary entries correctly and diligently, and some will drop out before the end.
Solution: Consistent communication. You may wish to set up alerts, or reach out to participants directly to be sure they’re still engaged and committed and keeping up the momentum. You may wish to reward users for seeing your study all the way through, either periodically throughout, or at the very end. This constant contact can be one of the more time- and resource-consuming aspects of diary studies.
Problem: Data analytics can be heavy with all different types of research, but even more so with a diary study. Results are all fairly subjective. The process of analyzing qualitative data is a whole different ball game from analyzing quantitative data: qualitative data is just more cumbersome, and is more challenging to automate. Diary studies can be even more challenging than other types of qualitative data because you’ll be dealing with the meanderings of people’s minds.
Solution: Create a format, if you can, where users fill in diaries digitally. This at least reduces the need for transcription, which will save time later. It can also enable you to create sortable fields, like time of day of each log, or to have users answer some boilerplate questions each time. For example, in the case of a health app, it might help to know if a user was experiencing physical discomfort while they were using the app. In the case of a parenting product, it might help to know what the kids were doing while the parent was using the app. You can create these fields in a digital diary form. Even if you don’t create custom fields, making diaries digital will allow you to search keywords and more easily identify trends when it comes time for analysis.
We recently conducted a study where we wanted to combine technology with a more traditional diary approach to examine the thoroughness of self-reported participant data. In the study, participants carried a lifelogging camera and a Fitbit, automatically collecting data. At the end of each day, the participants were to write a chronological list of what they had been doing that day.
From conducting the study, the key takeaways are that people tend to leave out details if not asked specifically about them. When comparing the self-reported data to the photo data it became clear that a response like “going downtown with a friend” contains so much more. The results have given me a reason to really think about the questions I ask in a research context to draw out those details. This saves a ton of time examining the data together with participants after the study.
User-Centered Business Designer
Diary studies give you access to information of a type and quality that no other research method can match—though the same can be said of most other research methods. Each is a unique and important tool, a way to add one more piece to the puzzle you want to solve. Effective research is seldom dependent on choosing "the best" research method. It’s usually more often an issue of choosing a series of well-phrased questions, and matching those questions with the most appropriate research method, so that all of the studies together will give you something close to the complete picture.