SUBSCRIBE TO OUR NEWSLETTER
Marie shares how she successfully pivoted from a failed product to FYI, along with which questions give the most insightful takeaways.
Marie and Hiten’s first collaboration was Dogo, a tool that lets users send and receive feedback on pitch decks. People could add comments directly to the draft, then send it back for changes. Once people sent it to investors, they could track who opened the pitch deck and the period of time they looked at it.
They didn’t do any real research beyond reflecting on their own experience to validate the idea. And it didn’t take long before the duo realized that their target market — entrepreneurs looking for funding — comprised an uncomfortably small market. And the pain point solved was just too small.
Still hoping to do something with documents, they pivoted to a new product in 2017: Draftsend. It lets you leave audio feedback on PDFs and see how long your colleagues spent looking at the document.
“With Draftsend, we did do research, but it was so flawed,” she laughed. “We took what we learned on Dogo and then we said to ourselves, ‘Oh, audio is a big trend. People want audio, too.’ So when we did customer interviews, we’d ask questions like, ‘Hey, check out this idea, what do you think?’ And that obviously resulted in hugely biased results.”
It’s worth repeating: you’ll get skewed results by asking questions like, “What do you think of this idea?” People want to reaffirm what you’re doing. But just because they think the idea is cool doesn’t mean it’s something they’ll put down money for, nor does it mean that there’s enough of a market for your solution.
Draftsend generated a lot of excitement on Product Hunt’s upcoming product list. But Marie and Hiten immediately discovered that retention would be a problem. People weren’t regularly creating new content that they wanted to add audio to and the product required more effort than people had time for. “We realized we hadn't done the proper research on Draftsend. We haven't seen if there's actually a problem here to solve. We were just building something because we had a gut feeling,” she said.
Their solution was to open up Draftsend for early access (their preferred term for beta testing). Their survey for early access users got 700 responses. And the pain point the majority identified was not related to leaving feedback on and adding audio to presentations. It wasn’t even related to presentations or PDFs at all. The real problem they could solve for users? Finding and sharing their documents quickly and easily, across all the document tools they use.
They followed up and confirmed their learnings from the survey with 52 customer development interviews, far more than many startups bother to do. The insights they gained were what they needed to release FYI just six months after Draftsend.
So how do you get from Draftsend to FYI? The key involved asking the right question in the right way. Rather than asking leading questions and encouraging biased feedback, they kept the survey questions open-ended and more general than before.
The survey Marie distributed included basic information gathering questions about the respondent’s job role, company size, and so forth. They also asked which apps their customers used for creating and sharing documents.
But the main question that led them to launch FYI was this one:
What’s your single biggest challenge when it comes to creating and sharing documents?
The overwhelming winner was difficulty finding docs across multiple apps. Marie then included one more question to encourage future help:
How can you participate in helping us build this software?
That’s how they found 52 people to interview. It also offered future opportunities for user research.
Note: Struggling to find enough people to talk to? Tell us who you want, and we’ll get them on your calendar. Sign up for free. You’ll only be charged for participants who actually take part in your study.
After Marie and Hiten did the survey and conducted the interviews, they made an assumption that they wanted to test. The fastest way to test their assumption was to build a very simple MVP. It took them just five days to build and get it into people’s hands.
“It was super scaled-down,” Marie recalled, “because there was one core thing we wanted to test. The MVP let you connect to five different apps and search for documents across those apps. That was it. It was just a search box. And what we really wanted to understand was, is a search box going to solve this problem?”
As they suspected, the answer was no. But unlike with their previous products, they had evidence to back up that hunch. It freed them up to begin iterating on better solutions. Every week, they talked to about 10 people who used their MVP and discussed the experience these people were having with it. They learned that many people don’t remember how the docs are named, which app they are in, or what was written inside. A search box alone doesn’t solve the problem of finding documents.
Meanwhile, they researched competitors extensively. Who were they? How successful had they been? When one of them failed, what made them go under?
What they found was that a number of similar companies had either died or been acquired. If they wanted FYI to thrive, they needed to create the best way for people to find documents — and that had to go beyond just a search box. They needed a solution that could work for even the busiest and most forgetful businesspeople.
After putting out the MVP and learning that it wasn’t what people needed, they shut it down and started from scratch by doing user testing on prototypes of the user experience. “We just kept doing research and iterating an interface until we got to what it looks like today,” she said. Then, the real early access launch of FYI began.
Marie does the research for FYI herself. While she loves testing all kinds of methods, these three are the cornerstones of her efforts:
She uses those methods not only to test FYI’s progression but also to test and learn more about how real people respond to competitors’ UI and marketing messages.
“In the early access survey, we always ask a question about competitors. And then in the interviews, we always ask people what apps they're using and how they are solving the problem,” she said.
“In user tests, we actually like to test our competitors. Whether it's their marketing or the actual product, that lets us learn a ton about competitors. But more importantly, it lets us learn about the customers, because we start to understand what value props resonate with customers, what do customers care about, what our customer's problems are... So whenever we look at a competitor, it's always with the lens of the customer,” she continued.
Marie finds surveys to be particularly helpful in doing generative research at the beginning of early access. The goal is to make sure you’re addressing the right problems in a way that will inspire user retention. It’s also a great way to screen for good interview candidates for further study. She also sneaks in some competitor research. In one of the sections below, we’ll dive into the questions she asks to make that happen.
Marie loves to use interviews to follow up on survey answers. They’re valuable for more than one reason, but her primary reason for conducting them is to understand the context and motivations behind what her customers say.
“I would say digging into customer stories is the most essential part of early access,” she emphasized. “It’s the stories that always lead to the most surprising insights.”
For example, when they were testing their MVP, they’d ask people to tell them about a specific time when they couldn’t find a document. The stories often looked a lot like this:
The takeaway? “We learned that, when people can’t find documents, they’ll do a quick search sometimes, but oftentimes they go right into Slack or tap someone on the shoulder and ask them where the document is. We wouldn't have learned that had we not really dug into the stories,” she said.
That insight led to one of FYI’s features: the ability to find documents based on the people who created or shared them.
What would a good early access (beta) period be without some prototype testing? Marie likes to collect feedback on clickable prototypes of competitors’ offers and content as much as the actual features under development for FYI.
Her favorite question to ask during a user testing session is, “Is there anything confusing on the screen?” It forces people to take an extra moment to reflect on what’s before them. If they don’t know what a button does, or why they can’t find something they’d expect to see, then they’ll usually bring that up.
Note: Have enough users, but don’t want to spam the same people over and over with survey and interview requests? Keep track of them with Research Hub: Bring Your Own. You can recruit, schedule, and pay your users all in one place.
Marie runs several surveys throughout the course of a beta testing period. During early access as a whole, they’re hoping to learn the following:
Keep in mind that they can only get the more granular information and use it in a helpful way because they’ve already done the idea-validating research described earlier in this post (their research around Draftsend and testing the MVP of FYI).
These questions are a good start, but it’s important to customize them and add more based on what you’re doing and what you hope to learn.
“When you write the questions, remove bias as much as possible. For example, if you’re asking what their biggest challenge is, you want to make sure that you're not getting really specific in the way you ask that question,” she said.
“For us in Draftsend, instead of asking, ‘What's your biggest problem with sharing and creating documents,’ we could've said, ‘What's your biggest problem when you're sharing documents and adding audio?’ It would have completely skewed the results. We would have missed the actual problem. We’d never have built FYI. So it's important to look at your survey after you've written it and make sure that you're not introducing any bias with the questions that you ask,” she said.
With that in mind, here are the questions that Marie uses:
Here are a few questions that Marie likes to ask the users who agree to an interview.
So once you’ve done the legwork — collected your findings and pored over the results — how do you know it’s time to exit early access and formally release your product?
Marie cares about three main signals:
Launching the tool after early access doesn’t have to mean you’ve iterated to your final product. It just means the early access (beta) version improved enough for users to embrace it on the market. It’s wise to continue research after closing early access.
As for FYI, Marie and Hiten just launched early access for their next phase: team-based access to FYI. If you’re curious, you can request access here.
Marie can’t wait to begin interacting with people, interpreting research, and planning FYI’s next steps.
“I think early access is one of the most exciting parts of product development,” she said. “It's a time where you can hone in on the customer, their problems, and how you solve them the best way possible. You get to interact with customers so much.”
“Just have fun with it instead of making it a stressful undertaking. It can be a more fun, lighthearted undertaking, and I would encourage people to make it like that,” she added.
About to start beta testing? User Interviews offers a complete platform for finding and managing user research participants. Tell us who you want, and we’ll get them on your calendar. Sign up for free. You’ll only be charged for people who actually take part in your study.
Olivia is a content strategist at Grow & Convert who loves science, cats, and swing dancing. She enjoys a mix of writing, editing, and strategy in every work week.
Leadership & Strategy
July 26, 2021
Should you become a UX manager or grow as an individual contributor? Amber Davis, UX Research Director at Audible, shared advice on making big career decisions and knowing your own strengths.