Understanding is the new bottleneck — Geoffrey Litt, Notion

AI Engineer · 19 min · 299 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01[music]
2. 00:12What's up?
3. 00:13Yeah, thank you for coming to the design engineering track at AI.
4. 00:16Is everyone having fun?
5. 00:18Yeah, I think this is going to be a great track, so get excited.
6. 00:22All right, let's get going.
7. 00:24Um, my name's Jeffrey Litt.
8. 00:26I'm a design engineer at Notion currently.
9. 00:28And I'm here to drop a hot take for this room, maybe.
10. 00:34I think it is still important for people to understand how code works.
11. 00:39[applause and cheering]
12. 00:42Now, some of you might agree, some of you might disagree.
13. 00:44Let's actually Let's do Let's try a poll.
14. 00:45I'm curious for this room.
15. 00:46Raise your hand if you agree with that opinion.
16. 00:50Okay, maybe some selection bias.
17. 00:51Any brave Okay, raise your hand if you disagree with this opinion.
18. 00:55Wow, okay.
19. 00:56We have
20. 00:57[laughter]
21. 00:57Maybe we'll do a debate later, yeah.
22. 00:59I was hoping we'd be at the AI engineer conference, so we'd have more bull Okay.
23. 01:03I might be preaching to the choir here.
24. 01:06You know, I think we The reality is though we are entering an era where this is a legitimate question that people are debating, right?
25. 01:13Agents are writing tons of code for us.
26. 01:15They're landing 50,000 line PRs.
27. 01:18And it is getting harder to keep up.
28. 01:20We all feel this.
29. 01:22Now, I think the good news is there are lots of ways to understand.
30. 01:27Be, you know, the days of just reading code line by line and that's not the only way anymore.
31. 01:32And what the point of this talk is about is I want to share with you a bunch of the practices that I use to understand the code that my agents are writing for me.
32. 01:41This includes things like explainer docs, teaching me about how my code works.
33. 01:47My agents write quizzes for me to to test my understanding.
34. 01:51Am I still really in the loop?
35. 01:52Am I keeping up?
36. 01:55I have agents build me micro worlds that I can inhabit to get this intuitive sense of how my code works that's deeper and richer than just a written document.
37. 02:05And I think all of these are really exciting new possibilities that are opening opening up for AI to help us understand better, not worse.
38. 02:13And so that's what the point of this talk is going to be about and I hope I can leave you with some techniques that you can take home and use yourself.
39. 02:21By the way, my timer isn't running.
40. 02:23If you could get that running, that'd be great so I know blabbering.
41. 02:28Okay, but let's start let's back up for a sec.
42. 02:29Before we talk about how, let's talk about why.
43. 02:33Why bother understanding?
44. 02:35This is again, it's a question now, right?
45. 02:37And I think a lot of people get this subtly wrong.
46. 02:42So what a lot of people think of why do humans still have to understand, they think we understand to verify.
47. 02:48The agents do dumb stuff, we've all seen it, and your job as the human is to keep them in line, right?
48. 02:53Make sure they don't screw up.
49. 02:56When people say things like code review is the new bottleneck, I think that's the first thing that pops into people's heads is correctness checking.
50. 03:05There's this mental model that's like, "Hey, the agent's going to give you something, and what's your job?
51. 03:10It's to ask, is this correct?"
52. 03:13Now, correctness can have lots of definitions.
53. 03:15Does it match the spec doc you gave it?
54. 03:17Does it take down production?
55. 03:19Is it well architected?
56. 03:21But fundamentally, those are all kind of thumbs up, thumbs down decisions, right?
57. 03:26And the thing is, over time, we've all seen it, the agents are also able to ask these questions and they're getting better at it.
58. 03:33You give it the right verification loop, and over time, this is the reality.
59. 03:38The the role of humans in correctness checking is decreasing.
60. 03:43And you know what?
61. 03:44I actually don't hate that.
62. 03:45If I have a clear idea of what I want to do, and the agent does it correctly, instead of coming back to me with an incorrect thing, that's great.
63. 03:52I'm I'm into it.
64. 03:54So then I think people extend this and say, "You know what?
65. 03:57That means as the agents get smarter and smarter and smarter, we we don't have to understand it all, right?
66. 04:02Get out of the loop, man.
67. 04:03Run the loop."
68. 04:04And that's where I think people miss something really important.
69. 04:08There is a deeper reason to understand what's going on, and that's understanding to participate.
70. 04:14Because here's the thing, it's not just one loop.
71. 04:19When you review what's happening and get in the loop, you come away changed.
72. 04:24You understand something, and that understanding is what you take to the next loop, and the next, and the next.
73. 04:32Your understanding of what's going on is the foundation for you having that next idea and being an active creative participant in a project.
74. 04:42I think probably you've all you've all felt even before AI, you know, the difference of the kinds of ideas someone can have when they really understand what's going on versus when they're a few layers removed are different.
75. 04:51Because when you have rich conceptual structures in your head that you can fluently recombine really fast without going out to like ask some some agent or some human how it works,
76. 05:02that gives you the ability to fluidly take creative leaps.
77. 05:05And that's the human part of the work, coming up with the next idea and the next idea.
78. 05:10So this is actually the real reason I think understanding matters, and this is not something that we can just wash away with better agents.
79. 05:17Because if we want to be active participants, you still got to do this.
80. 05:23There's a great term maybe some of you have heard called cognitive debt that I think really captures the spirit well.
81. 05:28It's an analogy to technical debt, popularized by the scholar Margaret Stories.
82. 05:31Simon Willison also blogged about it.
83. 05:33And I love this idea because similarly to tech debt, you might get away with it for a little bit, but at some point you get burned if your understanding degrades.
84. 05:42And maybe you felt this.
85. 05:43I know I felt it.
86. 05:44You're vibe coding, things are going well, and then at some point you realize, wait, I've no idea what's going on.
87. 05:49I basically can't participate anymore, right?
88. 05:52You've built up too much cognitive debt.
89. 05:56Okay, so maybe it sounds like all of you were already convinced.
90. 05:59We agree.
91. 06:00We need to understand.
92. 06:02But how?
93. 06:04Right?
94. 06:05We don't want to live in 2023.
95. 06:08We are using agents to move fast, and it is harder and harder to keep up.
96. 06:11How do we do it?
97. 06:13I think to answer this question, we should actually take a step back and ask a more fundamental question, which is how do we understand stuff in general?
98. 06:25Plot twist, this is not the first time that any human has asked this question.
99. 06:28There is a field.
100. 06:30It's called education.
101. 06:32Now, when you think education, you might think of bad memories from sitting in lectures or whatever, but I think we can do better.
102. 06:37We can take inspiration from the best ideas that have ever been invented in education and use them to stay in the loop and understand.
103. 06:44So, that's what this talk is about.
104. 06:46We're going to talk about three techniques.
105. 06:50First, explanations.
106. 06:53So, when an agent writes some code, it's an opportunity for it to explain the work to you, right?
107. 06:57And the most naive explanation is, hey, here's the code diff.
108. 07:00That's the raw change, the material of what happened.
109. 07:03But we can do, I think, much, much better.
110. 07:06What would the best possible explanation be?
111. 07:09Like if you sent a team away for a year to come up with a personalized curriculum just to explain this one code change to you, what would that look like?
112. 07:16I think this is a very generative question to ask.
113. 07:20So, I've done a bunch of attempts at this.
114. 07:23One is this skill I wrote called explainedif, which I use every day, and a lot of my co-workers do as well, and I want to walk you through it.
115. 07:30So, we're going to go through a little example here.
116. 07:31I'm working on a video game where you draw Zen gardens, kind of de-stress, you know.
117. 07:37We can all use that these days.
118. 07:39And we made a code change to change the perspective of the game from top-down to isometric.
119. 07:45And I when I run my skill, it produces a code explainer doc.
120. 07:48This can be an HTML file, it can be markdown.
121. 07:50I like to put them in Notion because I work there, but also because it's then collaborative, so my team can comment on it and talk about it.
122. 07:56And here's how it looks.
123. 08:00We start with background.
124. 08:01We do not start with what happened in this change.
125. 08:04It starts by teaching me, "Hey, here's how the system works.
126. 08:07Here's the game engine we're using.
127. 08:08Here's the coordinate system, right?
128. 08:09Here's the subsystems."
129. 08:11It makes sure that I'm sort of being led up to the point where I can even begin to understand what's going on here.
130. 08:17Obviously, you can skip this if you already know.
131. 08:18You can personalize it to what you already know.
132. 08:22Second important principle is intuition before details.
133. 08:26So, before we start, you know, looking at code and stuff, it says, "Hey, the goal of this commit is to make the garden feel three-dimensional using only 2D drawing tricks."
134. 08:36You can think of this sort of as like a well-written commit message, a little deeper.
135. 08:39Give me examples.
136. 08:40Give me a feel for the essence before, you know, you throw a bunch of code at me, right?
137. 08:45This, by the way, this is good teaching.
138. 08:47This is what like good math teachers do.
139. 08:51Third, interactive figures.
140. 08:52So, where it makes sense, give me things to fiddle with and try.
141. 08:56So, with this change, it was like changing how we draw rocks.
142. 08:59So, I can drag around rocks in this little simulation, and it shows me the coordinates that are happening, how the Z layers of the painting are changing.
143. 09:07This, by the way, is actually using a new feature that Notion literally launched this morning of HTML blocks in Notion pages.
144. 09:13So, agents can put interactive simulations into your Notion pages.
145. 09:16Pretty cool.
146. 09:18I think you have to be careful with interactivity.
147. 09:20It can just be a crutch, and it can be kind of slop, to be honest.
148. 09:23But used tastefully, it can provide understanding that's hard to achieve with just static pictures.
149. 09:30Okay, then we finally get to the code, right?
150. 09:32Show me the code.
151. 09:33But we don't just throw a list of files in order.
152. 09:37We do what we what I call literate code diffs.
153. 09:39Give me prose.
154. 09:40Explain it to me in the right order.
155. 09:42Tell me before each file what's going on.
156. 09:44And when you accumulate all this stuff, it's much much easier to follow than just a raw diff.
157. 09:50Oops.
158. 09:51In fact, I print these out and take them to the coffee shop sometimes and just read them.
159. 09:55I find it really beautifully ironic that AI has actually taken this process where I was used to be like glued to my computer my IDE, and now I can go to the cafe and it's like I'm reading a textbook
160. 10:04about this PR.
161. 10:05It's really cool.
162. 10:08Okay, so there is one problem, which is that reading is hard.
163. 10:11And I am lazy.
164. 10:13People are lazy.
165. 10:15You know, there's this one time when I sent a PR to my coworker that I thought I had read the thing.
166. 10:18I thought I understood, and she asked me the most basic question.
167. 10:22And I was like, "Oh, no. I don't know."
168. 10:25I clearly hadn't understood, right?
169. 10:27I had fooled myself.
170. 10:28So, I thought, "How can I create a system where that never happens again?"
171. 10:32For inspiration, I look to the work of the researcher Andy Matuschak, who has this great line, "Books don't work."
172. 10:38What he means by that is it's really easy to read a book and not realize you didn't understand it.
173. 10:43So, that So, he and his collaborator Michael Nielsen tried this thing where in an essay, there are interactive spaced repetition quizzes that test whether you actually remember what you just read.
174. 10:52And this is cool.
175. 10:53It actually keep emailing you the quiz to make sure you remember it forever.
176. 10:57But, this is nice because you cannot get through this essay without understanding it, or at least without remembering it.
177. 11:03That's what I do with my code explainers.
178. 11:04At the very bottom, there's a quiz.
179. 11:06Five questions, medium difficulty.
180. 11:09And my rule is I don't send code to uh others on my team to review unless I can pass the quiz about what my agents wrote.
181. 11:18And it might sound kind of silly, but you should try it.
182. 11:20It really is shocking the number of times this has caught me and made me and made me realize I didn't understand.
183. 11:26I think of it as sort of a speed regulator.
184. 11:28Everything AI is speed up, speed up, speed up.
185. 11:30There's all these incentives to go faster.
186. 11:32How do we make sure we're not just moving at the speed of correctness, but also of understanding?
187. 11:36And the quiz is that speed regulator, that's a system I can use for that.
188. 11:41I did uh just put the scale on the internet.
189. 11:43So, yeah, photo moment.
190. 11:45If you want the Explain Diff scale, uh take that QR code, try it out, make it your own.
191. 11:49It's really simple, actually.
192. 11:51There's two versions at that QR code, one that outputs HTML, one that outputs Notion.
193. 11:57Okay.
194. 11:59Second technique, microworlds.
195. 12:00What does that mean?
196. 12:02So, this takes inspiration from the educator Seymour Papert, real visionary, who had this idea of living in Mathland.
197. 12:10And what that meant was, "Hey, kids learn French from living in France, where do they go to learn math?
198. 12:15Is there a Mathland where you can learn intuitively math just by being there?"
199. 12:19So, he did these great things with this is a a robot called the turtle that kids program to draw stuff.
200. 12:25But the point isn't making robots, the point is they actually learn math by doing that programming.
201. 12:29But the point isn't the robot, it's the kids that are changed.
202. 12:34So, how could we apply this to understanding code?
203. 12:38Here's one example.
204. 12:38Last year, I was trying to implement um for my own learning this interpreter for a programming language Prolog, which is think of it a little bit like a database query language.
205. 12:47And there's all these parts of it that look like this, where when you read them on Wikipedia, they seem really complicated.
206. 12:52And then when you actually get what's going on, it's like, "Wait a second, that wasn't that hard to understand, it just felt hard when I read it that way, right?"
207. 12:59How could we make it click more for my brain?
208. 13:02So, I had Claude make me a microworld.
209. 13:05This is a debugger, ephemeral UI that was built specifically to visualize the internal implementation of my programming language.
210. 13:13What's happening here is that I'm scrubbing through a timeline that's running step-by-step what's my interpreter doing.
211. 13:18It's visualizing all the state at every step.
212. 13:20So, I can kind of open the hood and see what's going on and start feeling it, you know?
213. 13:25And yes, I can use I used this to fix bugs.
214. 13:27I even had a little hard to see here, but there's a commenting feature where I can leave comments for myself on the timeline so I remember what I was thinking.
215. 13:36And I use this to fix fix narrow bugs, but also as I was fixing the bugs, I was getting a feel for the machine, right?
216. 13:42That's something that if you just have an agent go fix the bug, you don't get that peripheral vision.
217. 13:46If you live in a micro world, you do.
218. 13:50Another example, um I was migrating my personal website from one framework to another.
219. 13:55And first thing I did, I had I said, "Claude, write me a script to do."
220. 13:58It did this.
221. 14:00It seemed like it worked, and I read the script and I was like, "Ah, I don't know."
222. 14:03Like I I just don't have a feel for what it's doing.
223. 14:06A bunch of files went a bunch of places.
224. 14:07It seems right.
225. 14:10So, what I did is I said, "Hey Claude, make me essentially a video game where I do the port myself."
226. 14:15And the way this works is old website on the left, new website on the right.
227. 14:18I just click a button, next, next, next.
228. 14:20And at each step, it says, "Here's the commands I'm running.
229. 14:23Your new website's coming to life step by step.
230. 14:24You see it."
231. 14:26There's actually file trees down there where I can see files moving.
232. 14:29And it's the the result is it's kind of like if I did it manually, but I'm just clicking a button.
233. 14:34So, I'm getting some of the benefit of doing it iteratively without the pain.
234. 14:42And I think the big takeaway here is agents can write code to help us understand code.
235. 14:46Where the point isn't building software to ship, it's building these little micro worlds for us.
236. 14:51It's the math land, right?
237. 14:52It's It's a It's a simulation of just this thing.
238. 14:58Okay.
239. 15:00Quickly, the last topic, shared spaces.
240. 15:03So far, we've This has all been about me, solo understanding.
241. 15:07But a lot of the time, I think the challenge is actually you're working on a team.
242. 15:10And your whole team needs to understand together so you can actually jam and have creative ideas together.
243. 15:16We think a ton about this at Notion.
244. 15:19We believe that, you know, the shared understanding that exists between you and someone else is what lets you communicate effectively.
245. 15:25Whether that's names for parts of a system, it could be names for UI elements or concepts, right?
246. 15:33So, we we think a lot of Notion about how do you make tools that enable collective understanding.
247. 15:37Some things we're exploring, can you have multiplayer chat threads between multiple humans and agents together?
248. 15:43So, here, you know, I might ask a product manager on my team, "Hey, you know, what what are users asking for with this feature?"
249. 15:51And she might say, "Hey, I don't know.
250. 15:53Let's ask a different agent."
251. 15:55And that agent comes in and talks to us.
252. 15:57What's happening here is that instead of me and my PM both talking to our own agents, we're in a shared space, we can see each other's communication.
253. 16:04It's kind of like, you know, going from one-on-one conversations to Slack channels, you know?
254. 16:08You see more of the behavior happening together and you understand together.
255. 16:14Also, having documents that you can talk about together is a really powerful primitive.
256. 16:18Here, you know, Claude made us a plan.
257. 16:21What if I have a question about that that I want to discuss with my team?
258. 16:24I can just leave a comment because this isn't a collaborative space, not on my computer locally.
259. 16:29And then I can ask, "Hey, you know, what do you think about this?"
260. 16:32And my teammate can chime in and and we can talk about it right there, right?
261. 16:35I think having these spaces for shared discussion around ideas with our agents is really powerful for building up that collective understanding.
262. 16:44This, by the way, um we just launched last week the ability to bring coding agents into Notion.
263. 16:47So, Claude and Cursor can now live in Notion.
264. 16:50And our team actually builds a lot of our code in Notion itself, mainly because of these benefits, because having it in a shared space is just so valuable.
265. 17:01Okay, so we've talked about these three techniques.
266. 17:03I want to bring it back to the beginning.
267. 17:06You know, I think at the beginning I said, "It's important for humans to still understand how the code works, right?"
268. 17:12But I actually think it's much bigger than that.
269. 17:15I kind of think it's just important for humans to still understand how everything works.
270. 17:22And maybe, you know, you all agree it sounds like.
271. 17:24But this I think is being called into question now and is something we actually have to actively fight for.
272. 17:29The thing is this is not a new battle.
273. 17:32It actually hearkens back to the very origins of our field.
274. 17:36Alan Kay is one of the pioneers of personal computing, co-inventor of the modern GUI.
275. 17:40And literally 50 years ago he wrote this essay that I find very prescient, called a personal computer for children of all ages.
276. 17:48It looks like two kids on iPads watching YouTube or something, right?
277. 17:51It's kind of crazy.
278. 17:52This is 50 years ago his vision.
279. 17:54But that's not YouTube on the iPads.
280. 17:55What he envisioned is, "Hey, these kids, they're playing a video game and they're modifying the code as they play it to learn physics."
281. 18:03So the point isn't the computer, it's the kids.
282. 18:06The point of computers was to level us up as humans, right?
283. 18:09And Alan has talked a lot about how it kind of feels like at some point computers detoured a bit from that vision.
284. 18:16But I think the exciting thing is maybe now's the time to bring that back.
285. 18:20Here's kind of the meme version of that.
286. 18:23I think with AI, a lot of people are waking up to, "Oh my gosh, code is free.
287. 18:28We can make ephemeral UIs, dynamic simulations to understand concepts.
288. 18:31We can make debuggers, playgrounds."
289. 18:34And it's like, "Yes, that's great."
290. 18:36And it's actually not a new idea.
291. 18:38Like this was the goal all along.
292. 18:41And so I think the optimistic thing that I find really exciting is hey, it's still really important to understand how things work.
293. 18:48And with the right tools and the right mindset and the right creativity, we can actually understand better than ever before, not less.
294. 18:55With AI, we can kind of empower ourselves more, not just taking ourselves out of loops, but actually putting ourselves more deeply in loops than we ever have before.
295. 19:05And I think that's a really exciting prospect and I hope it's something that we all, together as an industry, figure out.
296. 19:12That's all I have for you today.
297. 19:13Thank you so much.
298. 19:15[applause]
299. 19:29[music]