Watch the episode: 

Highlights: 

[00:05:03] Tools Linus is excited to see come to market

[00:06:50] Generative AI and its function at Notion

[00:10:22] Recognizing AI is useful versus finding the use cases for it

[00:13:58] Lessons from the Notion design process

[00:22:09] Breadth versus depth in workflows

[00:26:10] Learning about models in the wild

[00:28:52] Security considerations with AI products

[00:32:13] Understanding the difference between AI generated and human written text

[00:36:34] Should we be scared of AI?

‍

Sources mentioned in the episode:

• userinterviews.com/awkward
• notion.so

‍

About Our Guest

Linus Lee is a Research Engineer at Notion, prototyping new software interfaces for augmenting our collaborative work and creativity with AI. He has spent the last few years experimenting with AI-augmented tools for thinking, like a canvas for exploring the latent space of neural networks and writing tools where ideas connect themselves. Before Notion, Linus spent a year as an independent researcher in New York City.

‍

Transcript

Linus - 00:00:00: It's a hard balance, especially in AI, because, more than other kinds of technologies, user feedback matters a lot. It matters at this sort of base level that it matters everywhere, which is when you watch people use your product, you always learn more about how they're using it and what could be better.

Erin - 00:00:25: Hello, everybody, and welcome back to Awkward Silences. Today, we're here with Linus Lee. Linus is a Research Engineer at Notion and has done a lot of work on AI. And we're really excited to chat about it. Thanks for joining us, Linus.

Linus - 00:00:48: Thanks for having me. Super excited.

Erin - 00:00:50: We've got Carol here too. Carol Co-hosting. JH is no longer with us. He is on the planet, but not on the podcast. Carol, thanks for joining. So excited to have you.

Carol - 00:01:00: Absolutely excited to be stepping in for JH this time.

Erin - 00:01:03: Alright, so, Linus, really excited to dig into this. There's obviously tons to cover. You know, in the early days, it's like, oh, it's the AI episode, but I have a feeling every episode is gonna be the AI episode to some extent in the future. But maybe give us a little bit of background just to get started. We're going to cover the topic of designing interfaces for generative AI tools. You have a really interesting and diverse background, so maybe you could just tell us a little bit about what led you to be thinking about AI right now?

Linus - 00:01:31: Sure, I'll do a brief kind of how I got here spiel. So I'm a Research Engineer at Notion. What that means is I spend some of my time as a kind of a product engineer working on, within the team, more of the interface side of how people should interact with AI inside Notion. What is chat good for? What are possibly other kinds of interfaces good for, which I'm sure we'll talk about a lot. So that's some of my time and the other half of my time, or third of my time, goes towards more exploratory work. A lot of unsolved, unanswered questions in this space, and so some of that I think requires a little more unguided, undirected experimentation, exploration, prototyping, thinking. And so that's the other half of my time. That is more the research-y side of my work. I've been at Notion since the start of the year, so about almost half a year now. Before that, I spent a full year sort of independently prototyping or researching, whatever label you kind of want to put on it. But I spent that year chasing initially one kind of big question, which was so much of knowledge work is about people reading walls of text and producing more walls of text, bullet points of text, transcripts. So much is dominated by text, but text is kind of opaque. It can feel like such an opaque black box if you have piles of research docs for reading notes, or like RFCs to read through. The more you think about it, the more I thought about it, the more I felt like they were so in-ergonomic. When you look at a picture, you don't look at the picture from the top left pixel, row by row to the bottom right. But that's kind of how we read text. And so I wanted to tackle this question of, now that we have computers that can get much closer to understanding the meaning of text rather than just working with them at a character level, what interfaces can we build to make it easier to interact with text? So I spent a year prototyping lots of different things, some of which are feeding into, obviously, the work that we're doing at Notion. And then before that, I came from a kind of web developer, web engineer background. So within that year, I spent a lot of that time upscaling on AI and learning about everything that was going on across the stack.

Erin - 00:03:17: Fantastic. The kind of cold start problem with AI, like I imagine a lot of people listening or working on research on design, again, are at different stages, but they're thinking, we've got to do something with this, I think, like, what should we do with it? And so starting from that large question of, there's a lot of text here. We knowledge workers are just staring at text, creating text all the time. This is ripe for opportunity. How did you go from there to then what?

Linus - 00:03:44: So a lot of the experiments that I did in prototyping I did last year was about augmenting text in various ways. So some of my favorites were around adding annotations and highlights, and adding stuff to the interface of text itself. So if you're looking at a page, one way you could make it easier to parse is by just generating a summary. But a summary is still kind of hard to parse. There's still some reading involved. If you want to dig deeper into a particular topic, you still have to read the original document. So maybe we should just make reading itself easier. And one way you could do that is use AI to find the most important sentences, the most salient keywords, and highlight them. And then your eyes are more attracted to them. And so you're augmenting or annotating the text interface itself. I've done some similar work around automatically connecting keywords and highlighting keywords in notes, so that, again, eliminating manual labor from this process of working with text and making documents themselves a little bit smarter. And then the way I ended up at Notion was I was doing a lot of this work, especially with language models towards the end of the year. I had seen that Notion was also working on AI efforts. And so I had a chance to chat with Ivan, our CEO, about what they were doing and what they were excited for. And it seemed like they were focused on AI as a huge part of our strategy of what we think the future of knowledge work is going to look like in terms of the information we work with, the workflows that we can get AI's help from. And so it seemed exciting. It seemed like the right time to try.

Carol - 00:05:03: Before we jump into Notion, which I think we'll get into in just a second here, I'm curious, I know you built a series of tools in your year of exploration that you used yourself, and I'm curious if there are any of those tools that you're most excited or most hopeful come to market, maybe not at Notion or maybe just on your own.

Linus - 00:05:20: That's a good question. There are a couple that I think people were especially excited about. One was a kind of universal search engine for yourself. And so, like most people, I have a lot of my personal data in a lot of different places. I have journals in my own computer just in random files. I have bookmarks of websites that I read. I have Notion pages. I have contacts, all these things. And sometimes someone asks me, hey, do you know John? And I'm like, oh, I probably know this person, but I don't know exactly where I saw them. I don't know where I could have put that information. Maybe it's in my journal. Maybe it's in my contacts. Who knows? Maybe I just read something that he wrote on the Internet. And so, I built a tool that aggregated all that information and just made it searchable in a couple of different ways. You could do keyword searches over it. You could do more kind of semantic AI searches over it. So, you get similar kinds of content. And that has been useful for exactly those situations where someone mentions something or someone's like, hey, do you know a good blog post about how to understand transformers? And I'm like, I think so. I've read a couple, but I don't remember exactly what they're called. Let me look them up. And this idea of aggregating information across all these different places where you do work and putting it in all kind of one funnel, I think appeals to a lot of people. The second interesting thing that a lot of people comment on is what I alluded to earlier. I built this kind of outliner note taking system where ideas would connect themselves around keywords. And so, you write stuff down and as you're writing, the keywords kind of light up in these highlights. And if you click on one of the highlights, it tells you the previous kind of times you've referenced them. And again, that kind of takes the manual labor out of remembering what you've talked about before that might be relevant to what you're thinking about now or talking about now. And it made your old notes feel a lot more alive. And I think a lot of people like that as well.

Erin - 00:06:50: So again, in the theme of sort of backing up before you get really, really into the weeds with AI and Notion, assuming that most people listening have a rough sense of what generative AI is, given that we've been talking about it for many months now. But let's just not make that assumption and just quickly talk about what is generative AI and how Notion started to explore how they could use it.

Linus - 00:07:12: Generative AI, at a very technical level, is some machine learning system that can generate data that looks statistically like what it was trained on. And so, you have a neural network and you give it, you show it lots of examples of what realistic human written text looks like. Maybe it's text in general. Maybe it's text that's like, you know, how to write fiction stories. Maybe it's about how to write essays, how to write emails. And the model learns to generate sort of a simulacra, what those examples look like in a way that ideally is indistinguishable to humans or hard to tell apart. So that's the base technology. And then I think the technology, at least in the text realm, there's obviously analogs in the image and video realms. But in the text realm, I think it really took off once we had these models not just emulating the sort of raw statistical properties of what the text was, but having the models be a lot more steerable with human input. And so all the models that we're seeing now being commercialized have gone through this additional step of fine tuning or reinforcement learning with human feedback, where we tell the model, hey, don't just emulate any old text that we've trained your model on that we script from the Internet. But here are some examples of how you should follow human instructions. Here are some examples of good instruction following, some examples of bad instruction following. Here's how to have a meaningful conversation about a topic so it can be helpful. And then the model has learned to be helpful in exactly the sort of ways that we demonstrated to the model. And so now we have these models that are trained to first sort of have a general model of the world and the knowledge that we have trained the model on. And then second, know how to, or at least try to be maximally helpful given some user query. And that, I think, is when the technology really hit their stride. And since then, I think we've been figuring out what are all the different kinds of problems that we can solve with this basic technology, where we can instruct the model with some information and get some output out in the text format. And it turns out there's actually a kind of a we've been figuring it out for a long time in academia and the industry, and I think we're going to be doing that for a while.

Erin - 00:08:58: It feels like a really interesting research question, where on the one hand, you've got the faster horse, right? It's not like there's necessarily tons of people banging on the door, like, I need AI to solve this problem. It's like figuring out what those problems are and what those queries are going to be. There's some of that. As people start adopting this, figuring, kind of co-creating what these experiences are going to look like, which feels just like a very interesting place to be doing research.

Linus - 00:09:25: Exactly. I think even within Notion, there are already a few use cases that we see people using Notion for a lot. And so there are, for us, good starting places for thinking about how AI can be useful, right? So people use Notion a lot for documentations and notes. A lot of companies take their meeting notes as their source of truth inside Notion. Many companies that use Notion have Notion as their company knowledge base, their wiki, their HR information, their engineering, documentation, what have you, and then things like project management kind of workflows I've seen a lot of companies and teams use. And given those workflows, we can say, hey, what can AI help with that? But then there's also this whole other side where we see the technology kind of marching forward in capabilities. And then we are also having to ask, okay, given these capabilities, what are potential new use cases or adjacent use cases where this would be useful, where people maybe have not been using Notion for those use cases before? Or there might not even be an existing tool that you could use to do that before, but now we can. And so we're having to straddle both sides, which is an interesting technical problem and a design problem and a research problem.

Carol - 00:10:22: One of the things I'm seeing that it reminds me almost of when maybe mobile came out 10 years ago, which is then we're in a world of people asking, like, oh, we need a mobile app, and we don't know what it's for yet, but we just know we need a mobile app. And I feel like we're in a similar place with AI, where I see a lot of people asking, we need to use AI, but we don't know how. And I wonder how you think about that balance between just recognizing it's valuable and we need to shape it versus more actively trying to find the problems that AI can solve?

Linus - 00:10:51: I think this is an evolving topic, because even what we can use AI to do at a basic technical capabilities level is shifting a lot. I think, right now, my sort of bucketing in my head is that early on, a lot of experiments and building was focused on content generation. So if you are a marketer or a salesperson or a recruiter or a writer, you could use AI to accomplish a task that was mostly about generating text, generating content, and maybe polishing style and iterating on that. And then figured out this technology called Retrieval-based Generation, Retrieval Augmented Generation, where, when you ask the AI for some information, you could also give it a bunch of context about the question that you were asking it for, or you could have the AI be able to search across some database of knowledge. And then the AI would use that information to give sort of an answer that was equipped with knowledge that it had retrieved from some source. And after that, we've, in the ecosystem, seen a lot of companies like Perplexity and OpenAI ship more knowledge-based, knowledge retrieval-focused use cases that look a lot more like search or answering questions. And then now we're at a phase where we see companies exploring ways to get AIs to take action in the world. You sometimes hear the phrase agents around this word, where you ask the AI to accomplish some real action, and you give it a set of tools. And then it can perform some tasks. And who knows what will come later. And so I think with these sorts of buckets of capabilities, there are potentially different kinds of use cases that you can explore. Notion, when we were sort of one of the first products with this out in public, I think because so many people use Notion initially to write and think through writing, and to write things like marketing copy and emails and templates, we were pretty focused on that initial capability of content. But, obviously, as the spheres of capabilities have expanded, we're looking at lots of different other kinds of things as well.

Carol - 00:12:34: That's really interesting bucketing that I haven't heard before. Can you go through those three cases again?

 Linus - 00:12:39: These are all kind of just my words, but content generation. So things that involve writing. If your final work output is writing, AI can help you with that. And then I would say knowledge-based things, information retrieval kinds of things that look like search or answering questions. And then what people have been calling agents or tool use, I think, is the slightly more academic term, where you give AI things it can use and then ask it to perform some actual action in the world of software. 

Erin - 00:13:03: And is there an implied sort of how advanced each of those are or can be from the perspective of the user or where the AI technology is and how well it can serve these use cases?

Linus - 00:13:14: I'm not sure if there is perceived sophistication on the user side. I think there's certainly technical sophistication. I think good enough content generation is probably technically easier than good enough knowledge retrieval, which is probably easier than good enough tool use. But I think the ceiling for all of these is really high. So the ceiling for amazing content generation, if you had the best AI in the world for writing recruiting emails or writing really great stories or taking a kind of boring draft and turning it into a really compelling kind of narrative, that would be just as valuable to me as something that can help automate your calendar or something. And so I think the ceiling for all of these is really high. But in terms of web viability, where you can start really seeing value, it seems like content generation was sort of the easiest place to start that was in front of us early on, I think.

Erin - 00:13:58: Great. So maybe walk us through a little bit about the research that you've done in the design process of some of the products you've been building at Notion. And yeah, curious to see what you've learned and what's worked, what's not worked, what's surprised you.

Linus - 00:14:12: Definitely. So Notion has a couple of things under the umbrella now. The first thing that we came out with, at the time we just called Notion AI, was what we've internally called the writer also because it's for generating content in your documents. But it's a little bar that pops up anywhere in your Notion page and you can ask it to do all kinds of writing. So you can ask it to summarize, to pull out information and extract bullet points or action items. You can ask it to rephrase things. A lot of folks have used Notion AI to improve their writing so you can kind of reflect back on some content and try to fix information or add information again. I also sometimes just use it to get definitions. If I see a word that I don't exactly know what the definition is of or if I'm curious if I'm using the word correctly, I'll just highlight the word and say, hey, is this the right way to use it? And that feels really natural in a way that using a dictionary isn't because there's all this context that the model has access to on what I've written on the page. And people use it to, I think, also build other kinds of recurring flows. And so one of the places where you can use Notion AI is something called an AI block, which is a kind of AI flow that's embedded into a part of a page with a button next to it so you can refresh it. So within our AI team at Notion on every sort of meeting node or meeting dock, at the top we have this AI block that summarizes the rest of the page. And so once we're done with the meeting, we can click the summarize button and it'll generate a summary of the meeting. And that helps with recurring repeated kinds of workflows that you have every day or every week. And then most recently, we launched AI AutoFill, which is placing the power of Notion AI inside Notion databases. And so if you have, say, a database of user interviews, and you want to pull out what are the most important pain points that this customer has experienced from your database of interviews, you can make a new column in your database and just ask the AI, hey, for every page, can you just pull out the most important pain points? And it'll do that and fill out a table. And you can have many columns and databases and have them automatically update any time you update any of these pages. And so we've gone from just having a localized kind of AI looking at where exactly you are to slowly more and more doing things across the workspace and potentially automatically behind the scenes as well.

Erin - 00:16:08: A lot in a short period. Yeah, I'd be curious for any of those you'd be down to talk about just what the research was that led to them being created, how they were created, and then also ongoing research that might be happening in terms of passively or actively watching or talking with folks as they use them and explore those experiences. 

Linus - 00:16:28: Definitely. One of the early learnings that have been reiterated as we talk to folks using Notion AI is that instructing these things to get exactly the value, the result you want out of them is pretty hard. If you want a kind of basic summary, maybe you can ask the AI, hey, can you write a summary of this? But then maybe it's not the right length. Maybe it sounds too formal. Maybe you actually wanted a meeting summary as bullet points. And there's all these kinds of possibilities that you don't really consider when you just ask a human for a summary. Because in a real dialogue, there's all this context that you have with another human. And maybe there's past work that you can reference. But when you're instructing this kind of blank page AI, you have to pack in so much information to get exactly the output you want out of them. What often helps is using examples. And so over time, we've gotten really good at crafting our prompts for AI. So our prompts are now quite lengthy, they contain all kinds of examples. They contain examples in different languages for all the different languages that we support. And we put a lot of effort into making our demonstrations to the AI really good so that when you actually use Notion AI, the summary is the best summary that we can provide. But we've found this theme over and over again across all of our AI products and the upcoming ones that we've investigated with users. Writing really good prompts and instructions to the AI continues to be a big problem. Obviously there’s the “I can't get the result I want out of it” question. But there's also a lot of times just kind of a blank page syndrome, which is, OK, you told me about Notion AI. It can automate some things. I get that in the abstract. But I don't know which part of my specific workflow the AI is good for. If I'm an engineer, I actually live most of my life in the code editor. I occasionally write some engineering docs. And so maybe there is a way to use Notion AI to make that easier. But concretely, what are the actual things that I can use Notion AI to do? Maybe I need summaries. And that's kind of an obvious one, because summaries are kind of everywhere these days. The AI generated summaries. But if you think about it, there's all these things. You can use it to rephrase technical terms to make it easier to understand. Maybe you can use it to generate diagrams. So Notion now supports mermaid diagrams, which are these kinds of diagrams that use describing code and text format that can be automatically rendered. And we've used Notion AI. Some of us have used Notion AI to automatically ask the AI, hey, here's this natural language description of a system, can you generate a diagram for me? But you don't think of these things, and you just look at an empty prompt box. And so one thing that I think we iterated on and ended up doing a pretty good job at early on was rather than just having the prompt box, we initially have a bunch of suggestions when you enter it. So we give you all these possible sort of pre-built prompts that we polished, like summaries and action items and other kinds of examples to inspire you, here are the things that you can do inside this document. And then one of the key changes we made as we were going from a kind of beta to Notion AI to the final release was to make iteration much easier. So maybe you generate a summary, and it's too long. Maybe then you can follow up and just say, hey, can you make it shorter? And then it'll try to iterate. And over time, across a kind of conversation between you and the AI, you try to arrive at something better. There is always more room to improve. Like one way you could improve that is maybe you one time work with the AI to get a kind of perfect version of what you think your particular summary format would be. And then maybe you want to reuse that later. And so you want to somehow tell the AI, hey, can you remember what the summary feels like? And that would be kind of cool to be able to do. There's all kinds of ways to improve on that workflow. And maybe perhaps we'll do some of them. But that's one example of, I think, a hard problem. We as an industry are, I think, still contending with, and not even have fully realized how bad this prompting problem is. It's really hard. Prompting humans is also hard. If you hire an intern to fully describe the task that you want them to do in one shot, it's impossible.

‍Carol - 00:19:47: Yeah, I have an example of that. I think the first time I was playing around with Notion AI, probably earlier this year, I was spinning up some conversation notes, like a one-on-one note with someone on my team. And so I just titled it, Luke and Carol 2023. And just for fun, we were like, let's see what AI generates. Maybe it'll be a to-do list, or maybe it'll be actions above and details below. And you might be able to guess what happened where it basically wrote sort of like a wedding page summary about how we knew each other in high school, and then we fell in love, there was a beach proposal. And it was somewhat embarrassing, right? Because this is someone on your team in a professional context. I was just actually trying to recreate that before the call and did notice that there's a lot more structure now to save me from myself in a case like that. 

Linus - 00:20:31: That reminds me of another thing that we've been focusing more and more on, which is especially in a collaborative setting. Over the last sort of 2022 year of me working independently, a lot of the tools that I built and experiments that I did were focused on individual users, which is mostly me and other people that I occasionally interviewed and tested with. And one of the big design context shifts that I experienced when I joined Notion is that Notion is an extremely collaboration-first tool. Everything is built with the assumption that at some point you're gonna be able to collaborate on it with somebody else, so it exists in the shared environment. And when you're in that, there are a lot of things that you need to consider from the get-go: permissions, security, expectations of privacy, who can see what and all these things need to be front and center. And I think we've done an increasingly better job at thinking about the implications of privacy in a fundamentally collaborative setting. So an example is when you use Notion AI to generate that accidental wedding document, even though it looks like the result is on the page until you confirm that answer and actually put it on the page and escape this AI mode, the output won't be committed. So it won't leave traces in your activity log. Your workspace admin won't necessarily be able to see it because it's sort of local to your computer. And even when AIs make edits, we make sure to track, even in the background, attribute those changes to the AI and always be able to look back at who did this, was this AI making this change; when you give the AI context about information on your page, making sure that only the people who have access to that information can ultimately see what you've given the AI. We're not treading on anyone's data. We're not sharing models between people. And I think, especially in the kind of enterprise B2B setting, I think those considerations have been very upfront and center and I've been thinking a lot about those.

Erin - 00:22:09: Yeah, I was just gonna say, when you think about it, you've already built quite a lot in terms of AI use cases and it sounds like there are many more planned on the roadmap. How do you think about breadth versus depth in terms of really noodling on and trying to perfect some of these workflows, some of the tricky stuff with this nascent technology versus like putting a lot out there and see what sticks and just optimizing for shipping and learning and seeing what use cases maybe people apply with these technologies and then, you know, iterating from there?

Linus - 00:22:40: It's a hard balance, especially in AI, because, more than other kinds of technologies, user feedback matters a lot. It matters at this sort of base level that it matters everywhere, which is when you watch people use your product, you always learn more about how they're using it and what could be better. But it also matters at a technical level, which is that if you want to improve your models and get them to perform tasks better, you want, at some level, you want data, perhaps from them, perhaps synthetic data that it's shaped like how people use your product, that sort of reflects reality. And to get data or make data that reflects reality, you fundamentally need to go to real users. But you have to balance that with-, for such a new technology, one thing that I've been thinking about is what are the actual right primitives and sort of conceptual models. So one thing that Notion's really known for is that we have this very kind of Lego building block method to build up all of our products. Notion's pretty unique in that given our kind of scale and value, most of Notion, the app, is one screen, if you think about it, which is kind of uncommon. We have the settings screen, we have the profiles, but most of the time, if you're in Notion, you're on a dock, which is, I think, somewhat unusual. And the dock is really just a collection of building blocks. So really, most of the product is, here's a pile of paragraphs and titles and these block formats, and you kind of assemble your Notion out of them. And I think that works really well because the blocks kind of look pretty when you put them together. They're visually aesthetic, but they are also pretty functional in that you don't have to worry about choosing fonts and all the sort of minutiae of style. You can snap these things together to build really beautiful looking docks. And the thing that makes that possible is that we chose the right primitive of what is the actual unit of composition in the UI. Internally, we've been talking a lot about what is that kind of building block for composing all these kinds of use cases and utility out of AI. I think that's a tough question that we'll still need to answer. But if you want to take the time to really investigate and choose those primitives, that's kind of opposed to the goal of shipping early, throwing things against the wall, trying experiments. That's been an interesting balance, I think. So far, I've done a pretty good job of delivering things that are actually practically valuable. So lots of people would use Writer to write their emails and write their copy and write their blogs. And same with AI inside databases. But as we add more complexity, I think that this importance of, like, what is the right building block becomes a lot more important. It's even kind of a weird question I'm realizing to talk about, because I don't even know what the building block looks like. Like, is it a building block of an instruction? Is it like a model? Like, are you composing many models together? I don't know. We're internally, obviously, trying lots of experiments. But I think when we do finally arrive at the right model, it needs to be important that we're not going to have to rewrite the book completely on everything that we've done with AI to make it fit that model. And so there's a balance of, like, we do experiments, sometimes internally, sometimes externally, to validate use cases and figure out where the real value is and what these building blocks should look like. But then when we actually realize what the right building blocks are, I think we'll try to hopefully not rewrite too much what the holistic mental model of AI should be, if that makes sense.

JH - 00:25:34: All right, a quick awkward interruption here. It's fun to talk about user research, but you know what's really fun? Is doing user research. And we want to help you with that. 

Erin - 00:25:42: We want to help you so much that we have created a special place. It's called userinterviews.com/awkward for you to get your first three participants free. 

JH - 00:25:54: We all know we should be talking to users more, so we went ahead and removed as many barriers as possible. It's going to be easy, it's going to be quick, you're going to love it. So get over there and check it out.

Erin - 00:26:02: And then when you're done with that, go on over to your favorite podcasting app and leave us a review, please.

Carol - 00:26:10: Going back to something you said just a little bit ago, you were saying that in order to understand how people will interact with these models, you need to see them in the wild. And I'd love to hear more about how you actually learn about these models in the wild. What type of research are you doing? Are you looking at data on the back end? I'd just love to hear about that process.

Linus - 00:26:27: I don't think it looks too dramatically different from the kinds of research you would do for any other product feature. It's just more kind of unpredictable, stochastic technology. So we spent a lot of time talking to real Notion AI users and what they're using it for. I think a lot of our learnings come from just talking to people, as always. So we got some of our most prolific users. One of the benefits of working at Notion is that there's also this huge ecosystem of ambassadors and experts and people that love to talk about what they're using Notion for and how to curate your Notion. We have a large student user base, and so a lot of them are really fanatical about, here's exactly the way I use it to organize my med-school journal or my courses. And a lot of them also contain information about how people want to use Notion AI, because you can build AI blocks and AI autofill into these templates. We actually launched a new template gallery, which is this collection of if you're doing meetings or if you're running a company, if you're a student, here are all the different ways you can use Notion. And so we can go to those sources as well for a lot of inspiration on how people in the real world are actually getting value from AI. And then in addition to that, we have some amount of aggregate quantitative data we can look at. And so one of the other benefits of providing pre-built workflows or templates, like summaries and action items, is that we can look at aggregate information around what are the most commonly used kinds of prompts, what kinds of people use them most often. And maybe if we see a use case outside of the US, because Notion has a large international user base outside of the US, translations are a really popular feature, which we historically haven't focused too much on. But now that we're seeing it, maybe we're thinking about, maybe we can improve the translation experience a little bit more and make it a first class AI thing. And so there's that kind of data that we can look at as well.

Carol - 00:28:05: I see, so you're talking to users, sounds like the ambassadors and the people who create examples are really strong. And then also just actually as you create structure, then you can go look at how often the different prompts are being used. I imagine one of the challenges with so many different types of responses, then how do you actually go through and understand what is most common when it's all just text? Or maybe you're using AI for that.

Linus - 00:28:28: Again, there are expectations of security privacy. We can't look at any sensitive information that people could have in their real Notions. But people have conversations about how they're using AI on social media and templates and all these things. And so that's it. When we first launched, actually, I used to follow this band called Against the Current. And I saw one of the people who is in that band talk about Notion when we launched the thing. And I was like, oh my god, that's crazy. They're using this product that I’m working on. That was a delight. 

Erin - 00:28:52: Yeah, I was going to ask about you, because you mentioned the security with the collaboration. When you're talking to folks who are using these new AI use cases, are there any new kinds of things to be considered about in terms of security or in terms of an awkwardness or discomfort with like, I don't know how to use this, I don't want you to watch me use it, or is that just the same you always experience with any new thing people aren't used to using?

Linus - 00:29:17: It is definitely different with AI, I think. That probably comes down often to it's such an amorphous kind of technology, and it's so new that I think we collectively don't have the best crisp mental models to think about how to predict how things are going to work yet. There's also such a variety in implementation. So some other products out there, when you use them, they collect that data and use it to train their models. Some of them, they're sort of behind the scenes feeding information into these models to make them a little smarter. When you use Google's ‎Bard, for example, they're performing a Google search behind the scenes to inform the model of this. And one kind of opinion you could have is that all of these things are good. Things should implicitly sort of happen as long as the user consents at some point to sharing the information, making the model smarter and proactive is good. And I think it's also reasonable to make an argument that users sort of always know what data is flowing in and out of these models because they're so unpredictable and kind of a black box otherwise. I think so far we've been more on the sort of latter side of anytime you're using the AI, you should have a pretty clear idea of what the AI is looking at and what it's focused on. And so far that's been anything that you see on the page. If you're using Notion AI, it can look at, or if you're using Autofill, things in the particular database row that the AI is looking at, it has access to. But these concerns around what is the AI looking at? What is it learning from? What can it affect? Because it's such a black box and so immensely generally powerful, I think that's a new kind of problem. For me, that's an interface problem too. Because when you look at an interface and you're interacting with the AI, it would be cool if you could see exactly the sort of highlight on the page or maybe in some other way know what the AI has access to? Is it somehow reading information that I feel like it shouldn't? Or maybe you're using the AI to write an email and you want to make sure that only information that you want the respondent to see in the email makes it into the AI so it's not accidentally leaking information. There's all this valid concerns around information provenance where things flow that I think is valid and needs to be designed around.

Erin - 00:31:09: Because you can imagine there's a balance there in terms of the appeal of AI. It's this magic tool that just, I don't know, it exponentially makes everything faster and better and all these things. And then of course, there's lots of concerns about it too. What is happening in this black box and giving folks just enough information to feel comfortable without just like, oh, here's everything happening in the model. It's like, whoa, TMI, right? So finding that balance.

Linus - 00:31:32: I think especially right now, because there's such a diversity in implementation and how people are building these models and what kind of capabilities people are building into them, there's always gonna be an element of education for nNotion communicating to our users, hey, we're not trading on your data. We have all these sort of like ways to help you predict what the AI has access to. But also it's hard to, I think, reach everybody with just kind of outward facing, here are the policies and here are the rules. And I think at some level, you just need to have an interface that communicates and builds that trust with people. Even if you're not paying attention to what's happening in this AI world, at some point, just be able to look at the page and say, hey, this is, I guess, what the AI is reading. I don't think we're quite there yet, but I think there's always more to do on building more trustable AI systems.

Carol - 00:32:13: What do you think about the flip side of understanding, and I think this is going to be increasingly a blurry line, but understanding what AI wrote versus a human wrote? I remember the very first time I used AI in Notion specifically, I felt like I had to write a little caveat. The bots wrote this. I didn't write it because I felt like authorship. It was important to identify who wrote it. Maybe that becomes less important in the future, but I'm curious if you hear about that as you're talking to users.

Linus - 00:32:36: Yeah. Within the team, we have something similar as well, where any time we use, there's a kind of humorous element to it as well. But a lot of times when we use AI to generate a Slack message that's funny or a summary at the top, we say, hey, this is by Notion AI. And some of our AI features, like AI blocks that are sort of recurrently generatable or AI autofill, they all have this kind of purple border or purple icon next to them, which is kind of neat. Because then you can see at a glance, OK, these are the sort of AI-generated ones where the eyes touch this. These are probably not AI-generated. I think it's pretty likely that in the future, we're going to move more into a world where there's a blurrier line between what the human and the AI wrote. Like if you are using AI to source information on what you should write about and collect knowledge, but then you're phrasing it yourself. How should you caveat that? Or like I was writing a kind of internal memo yesterday around how we should think about AI, and I asked the AI for the corrections or word suggestions or to expand this list of examples. I think there's all these blurred lines, and eventually, hopefully, we'll get a lot more comfortable with different modes of collaborating with these systems. But right now, given where we are in such an early place in this technology, it feels comforting to draw a kind of soft fence around these areas in the documents and say this is where the AI touched. This is where this is like auto-generated, auto-updated by the AI, and then other parts have it be more and more kind of human.

Erin - 00:33:53: So now that some of these experiences have been out in the wild for a little while, what's been kind of the coolest stuff you've seen or you're most excited that's coming soon that you're allowed to talk about? 

Linus - 00:34:04: In general in the kind of AI world or at Notion?

Erin - 00:34:07: I meant Notion, but I'd love to hear the rest of the world as well.

Linus - 00:34:10: I'll start with the rest of the world while I try to think about what I can talk about. I think this is a bit outside Notion's world specifically, but I think code generation is really interesting. Language models that write programs. And the base level of that is if you're a developer, you can use language models to write a program. But programs are interesting because they're a kind of expression of meaning or action or workflow that you can also statically verify. Over the decades that humans have been programming, we've built up all these tools to try to check and sort of statically, deterministically validate programs against the rules that they should follow. And so if you write in certain languages, if you write a program to change something's permissions or make a bank transfer or all these things, you can verify that a program does the thing that follows the properties that we wanted to follow and doesn't accidentally do the wrong things against some specification. And programs are also deterministic. So if you write a program once, that program will always do the thing that it was meant to do, which is sort of different than a language model. The reason that I think code generation is interesting is because once we use the language model to translate some natural language requests into code to accomplish some action, if you just get the language model to perform the action once, then you have to kind of call the language model, the AI, every single time. And it might not always do the right thing because that's the nature of the technology. But if you use the AI to write a program instead and you know that the program worked the first time and you can check that program against all these guarantees and invariants that you want the program to follow, then you understand the program and you can look at it and you can check it and you can trust it and you can run it every time. So it straddles this interesting balance between wanting something that we can speak to and understand using natural language, but it's also now this concrete artifact that you can check and validate and trust a lot more. And I think there's a lot that you can do with that by using language models as a kind of bridge between these more opaque deterministic systems versus the natural language that we want to communicate with computers at. That code generation has been a topic in AI for so long. And I think language models have kind of injected a lot of spark and electricity into that world. I'm excited for what's coming there. I'll pause there and then I'll talk about Notion.

Erin - 00:36:03: Yeah, code generation is definitely fascinating. I guess it's because a program scales much more than a snippet of text. I guess that maybe dovetails a little bit into the really zoomed out question on a lot of people's minds of like, is this all going to go sideways? If something scales and is good, then that's exponentially good. If something scales and it's bad, that's exponentially bad. Are you thinking a lot about the implications of AI and how scared are you? How scared should we be?

Linus - 00:36:34: I tend to be generally a pretty optimistic person about a lot of these things. I think there's obviously lots of smart people working on all the different aspects of trust. I think one way that I've learned to think about trust with AI systems is that it's actually kind of a product of two things and you need kind of both aspects. So the first aspect is what I would call the integrity of the system. And the second is control. So integrity is like if you have products or some kind of a system that uses AI, can you trust that? Can you trust when you put in some data where it goes, how it's being used, where it's drawing its training data from? All these aspects around data provenance and security and privacy and just the general trust in the integrity of this mechanical system that the companies are using to run AI models. That's fundamental because that forms the sort of outer boundaries of how much you can trust the system. And then I think a lot of trust also comes down to controllability or steerability, which I hear people talk about less in the context of how much we should be afraid or optimistic about these things. But control is like if I ask the AI to do a thing, let's say I have a report that I write every week at the end of the week and I have a particular format that I follow and I tell the AI, hey, can you write a report summarizing these things and having these sections and it generates the thing. But maybe I wasn't specific enough. Sometimes it generates bullet points. Sometimes the sections are a paragraph long. Sometimes it's like a whole chapter per topic. I would probably trust that thing less because it feels more out of control. But the lack of control isn't actually anything negative or malevolent. It's just the AI didn't have enough context to understand exactly what you wanted. If you had examples or if the AI was able to draw on your past actions to learn exactly what you meant and just have more shared context, then as you got more answers that fit your predictions and your kind of mental sense of what the AI should have produced, maybe you'd be able to develop more trust. And so I think that controllability, steerability, how much do you feel like you can mold the AI output to be what exactly you had in mind? I think that's another critical aspect of trust. That's, again, more of an interface thing, too, I think.

Erin - 00:38:28: I share your tech optimism and also just the knowledge that we're all figuring out this AI stuff together. And I just have so many questions. Most of them are sort of questions of wonder, right? Because if you think about it, we used to think Google was a game changer and you could search something and it would give you information. It's amazing. This is just like orders of magnitude, more potential than that. And so I feel optimistic, too. But I also worry I don't know enough to have the proper view. But I took us on a tangent. Let's go back to where we were. You were talking about code as one of the most exciting and promising applications of AI. And then I think you're about to tell us some of your favorite either existing or soon to come applications at Notion as well 

Linus - 00:39:13: I think one of the first projects I got to work on when I joined the team was the Autofill project, Autofill and databases. And the thing that was unique about that compared to what we had previously was that we started venturing into what we could do with AI more in the background or more automatically, perhaps. The specific way that worked for Autofill was that you sort of turned it on for a particular column in your database or property in your database. And any time the page updates based on some other integration or automation or any time you add more context or notes to that page, the AI will go back and sort of update the property if it needs to be updated. So internally at Notion, we use Notion to keep track of all of our tasks and tickets and things we have to do engineering wise. And if someone goes in and edits, adding more context for how some particular task was implemented, the summary for that particular task will get updated by the AI automatically. And I think that's super interesting because it starts to guide us into questions around like while retaining the kind of good predictability and containedness of what AI touches and what AI can do in this Notion workspace, how can we also venture more into not just using the AI in our flow in real time as we're writing, but perhaps have it be responsible for more pieces of a workflow. And so maybe your workflow is that you have everyone fill out kind of like a database for what all of the teams did and then you want to generate a report. Maybe you can automate a part of that workflow by having an AI generated kind of column automatically update in the background. And this is again a balance between how much do you want the AI doing versus how contained and predictable you want the output to be. Currently in Autofill, the output is sort of restricted to one column, which is nice and trustable, but that pushing AI more into kind of a background thing that's always sort of working for you in different parts of your workspace, I think is an interesting direction. And I'm excited to explore more of what that points us to.

Carol - 00:40:59: I'm curious, maybe drawing from your experience at Notion, but I imagine many people in our audience are thinking of exploring uses of AI at their company, and they're wondering how to even approach that exploration. We touched on this briefly, but do you start by thinking about what your user problems are and then think about which of those are well solved by AI? Do you start by thinking about what AI can do? I'm just curious if, as you've been thinking about the use cases at Notion and brainstorming which felt like the right place to start, if there's anything you can share about what worked in that process. 

Linus - 00:41:30: The first thing that comes to mind is to ground yourself in the use cases that your users are already trying to address with your products. Is it particularly helpful for AI? Because as a first step, there is so much, such a flurry of activity in the AI world that sometimes it's hard to figure out which ones are actually valuable and which ones are just technically cool, which ones are demos that look cool in videos but don't actually work. And I think that's something that I personally experienced too, because when I was more in an experimental, researchy world, I would see a new kind of announcement or capability every day and every week. Oh my god, I have to try this. I have to try this other model. There's a new sort of fancy model every week. But then as I joined Notion, now there's real concrete problems to solve. And some of these problems are like, the AI can't do this thing that a user really wants to do. Some of it is like, hey, we need millions of people to be able to use this. And that places real constraints on the kinds of technologies you can look into. And so grounding yourself in real problems, I think, has ironically a kind of calming effect. Yeah, you don't have to pay attention to everything. It helps you filter what you need to focus on down to the things that are most relevant for the particular problems you need to solve. And I think that effect is helpful. I'm also trying to think of the bucketing I talked about earlier of what are the groups of capabilities that AIs have? Is your product or your use cases more around generated content? Is it more around helping people find information? Is it around, I have a friend that's working on using AI to help people stay in touch, manage other relationships? You could frame that as a kind of content generation problem, because a lot of relationship work is emails and texts and just keeping up with information. You could also frame it as a knowledge problem. Maybe you have sort of a knowledge base of all the people you need to keep in touch with, and it helps to surface the right ones at the right time. And so I think that starting from that bucketing of capabilities, I think, is also helpful.

Erin - 00:43:11: Is a good question, like sort of these are core use cases and capabilities jobs to be done, whatever it might be, how might AI make them faster, easier? Is that like a good starting point for folks or is that to, I don't know, down to earth, on the nose? Like, is that closing us off to more imaginative questions? 

Linus - 00:43:31: I think our approach has been to tackle these in order. And so the first few things that we worked on were, I think, a little more constrained around helping people generate content from the beginning. That is technically a bit more tractable than some of the harder problems. And then once you have the more tractable, well-understood ways of using AI out of the way, perhaps you can use that to-, we learned a lot about the concerns that folks had around AI and how the user problems are like, how difficult it is to get these prompts right from early, more tractable AI launches in the beginning. And then now we're using all of that knowledge to make sure that our more complicated AI products feel right and they don't suffer from these same problems that the folks have had. And so I think starting small with these, let's make existing workflows faster, simpler, cheaper, easier, whatever. And then taking those learnings in your particular domain to then work on potentially harder problems, I think, is one possible good path. 

Erin - 00:44:27: Well, we are right at the hour. So I think this is a good place to stop, but obviously so much more to talk about. And it's been really fun to scratch the surface of your experiences building and designing for AI and doing it at Notion, a product I think a lot of us like to geek out on. So it's been fun. Thanks so much for joining us. 

Linus - 00:44:44: Thank you. It's been a pleasure.

Carol - 00:44:45: Super interesting. Good to have you