Research Leader Lindsey DeWitt Prat introduces the research risk cascade: a model for understanding how errors don’t just accumulate in qualitative workflows—they compound across a multi-step pipeline. To make the idea concrete, she shares an experiment (“Eval Squared”) that tests how meaning drift branches across transcription and LLM-based synthesis/analysis, and outlines practical steps for navigating the cascade.
AI can make research teams faster, but it can also introduce inconsistencies in ways that are hard to detect. A transcript that’s slightly off, a summary that removes uncertainty, or an analysis that reframes an observation as a recommendation can quietly shift what gets presented as “the insight.”
“It’s literally never been cognitively harder… to notice and catch when things break or when they’re wrong,” says Lindsey DeWitt Prat, a research leader and director at Bold Insight.
Lindsey was recently joined by research leader and AI in research expert Kaleb Loosbrock to discuss the “research risk cascade”—the idea that errors don’t just happen at single steps in an AI-assisted workflow, but rather that they can compound across the pipeline from ground truth to transcription, synthesis, analysis, and the final deliverable. Highlights from the conversation are below.
Ready to jump right into the virtual event recording? Watch it on demand any time.
The Research Risk Cascade: Where Meaning Drifts Across the Pipeline
With AI being increasingly integrated into research workflows, there's a growing risk that AI outputs are accurate even when they suspect the answer is wrong—especially under more delivery pressure from stakeholders. That difficulty matters—modern research teams are being pushed toward speed at the same time the tooling environment increases the odds of subtle drift.
That led Lindsey to the concept of the research risk cascade—a way to map where meaning can change as research moves from what was said to what gets presented.
A typical qualitative pipeline looks like: [Input] Ground truth → Transcription → Translation → Synthesis → Analysis [Output - Insight]
Each box is a transformation point where signal can be lost, changed, or invented. Lindsey uses the telephone game metaphor to make this intuitive: what starts as a whisper can become something else entirely after just a few handoffs.
Learn how to incorporate AI Insights into your research participant recruitment processes
Eval Squared: A Practical Experiment That Shows the Cascade in Action
To make the cascade visible, Lindsey used episodes of Lenny's podcast to build an experiment she refers to as Eval Squared—a simplified, repeatable test designed to show how divergences branch across common AI-assisted steps.
Setting up the Experiment
Lindsey set up her experiment as the following:
- Three five-minute podcast clips on AI evaluation were selected.
- The clips included different varieties of English (standard American, India-accented, Vietnamese-accented)
- Lindsey created a “golden” ground-truth transcript to compare against tool outputs
- Multiple transcript variants were produced and compared
- Multiple LLMs summarized each transcript variant using the same prompt
- Themes and one-sentence recommendations were extracted from the summaries to represent the analysis step
The goal was to surface where drift happens—and how quickly it compounds once the pipeline branches.
Identifying Divergences
Lindsey tracks “divergences”—places where two or more outputs disagree, either with each other or with the ground truth.
1. Transcription: small errors that flip meaning
At the transcription stage, she observed:
- Meaning flips (“wouldn’t” becoming “would”)
- Key terms miscaptured
- Names garbled
- Speakers misattributed
Even in a single five-minute clip, she counted dozens of places where different transcript variants diverged from what was actually said.
2. Synthesis: Uncertainty gets erased
Synthesis is where the cascade can become harder to detect because the output often reads cleaner than the source:
- Hedging and qualifiers are removed
- Tone and register shift
- Words the speaker never used get substituted in
3. Analysis: Observations can become recommendations
By the analysis step, Lindsey describes a pattern where a speaker’s observation about what others do gets reframed as what the speaker is endorsing.
She gives a simple analogy: a participant says “sometimes I skip breakfast,” and the AI-powered report turns it into “users recommend flexible meal scheduling.” It might sound reasonable, she says, but it’s not what the participant actually said.
Need to sharpen your AI skills? Take our free AI Context Engineering in Research workshop.
Navigating the Cascade: What Teams Can Do Right Now
So how can researchers navigate this new risk cascade? Lindsey and Kaleb offered practical steps for reducing risk when utilizing AI tools:
1. Define what must not be lost
Not all research has the same risk profile. A straightforward usability test is different from multilingual, culturally nuanced, or emotionally complex conversations. Lindsey emphasizes starting with context: what matters most to preserve in this work?
2. Evaluate stage-by-stage, not just end-to-end
Instead of asking “Is this tool good?” the more useful question becomes: Is it reliable at this stage, with this kind of input?
3. Calibrate with comparisons
A simple method Lindsey describes: run the same material through multiple tools or models and compare outputs side-by-side to surface divergences early.
4. Treat evaluation as maintenance
Because tools and models change quickly, Lindsey frames evaluation as continuous practice—not a one-time audit.
Additional Resources
- Lindsey's article on The Research Risk Cascade
- The "Double Jeopardy" concept here and here
- Lenny’s podcast shows featured in the experiment:
- See more AI tools in the 2026 UX Research Tools Map
- Check out free AI x research courses from User Interviews
.avif)
