WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=WkBPX-oDMnA

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/WkBPX-oDMnA.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.720 --> 00:00:13.320
What's up?

s3
00:00:13.320 --> 00:00:16.160
Yeah, thank you for coming to the design engineering track at AI.

s4
00:00:16.160 --> 00:00:18.520
Is everyone having fun?

s5
00:00:18.520 --> 00:00:22.480
Yeah, I think this is going to be a great track, so get excited.

s6
00:00:22.720 --> 00:00:24.440
All right, let's get going.

s7
00:00:24.440 --> 00:00:26.320
Um, my name's Jeffrey Litt.

s8
00:00:26.320 --> 00:00:28.920
I'm a design engineer at Notion currently.

s9
00:00:28.920 --> 00:00:34.320
And I'm here to drop a hot take for this room, maybe.

s10
00:00:34.960 --> 00:00:39.955
I think it is still important for people to understand how code works.

s11
00:00:39.955 --> 00:00:41.955
[applause and cheering]

s12
00:00:42.800 --> 00:00:44.360
Now, some of you might agree, some of you might disagree.

s13
00:00:44.360 --> 00:00:45.720
Let's actually Let's do Let's try a poll.

s14
00:00:45.720 --> 00:00:46.880
I'm curious for this room.

s15
00:00:46.880 --> 00:00:50.040
Raise your hand if you agree with that opinion.

s16
00:00:50.040 --> 00:00:51.520
Okay, maybe some selection bias.

s17
00:00:51.520 --> 00:00:55.640
Any brave Okay, raise your hand if you disagree with this opinion.

s18
00:00:55.680 --> 00:00:56.640
Wow, okay.

s19
00:00:56.640 --> 00:00:57.640
We have

s20
00:00:57.640 --> 00:00:57.840
[laughter]

s21
00:00:57.840 --> 00:00:59.360
Maybe we'll do a debate later, yeah.

s22
00:00:59.360 --> 00:01:03.560
I was hoping we'd be at the AI engineer conference, so we'd have more bull Okay.

s23
00:01:03.560 --> 00:01:06.000
I might be preaching to the choir here.

s24
00:01:06.000 --> 00:01:13.280
You know, I think we The reality is though we are entering an era where this is a legitimate question that people are debating, right?

s25
00:01:13.280 --> 00:01:15.720
Agents are writing tons of code for us.

s26
00:01:15.720 --> 00:01:18.200
They're landing 50,000 line PRs.

s27
00:01:18.200 --> 00:01:20.040
And it is getting harder to keep up.

s28
00:01:20.040 --> 00:01:22.200
We all feel this.

s29
00:01:22.200 --> 00:01:27.360
Now, I think the good news is there are lots of ways to understand.

s30
00:01:27.360 --> 00:01:32.120
Be, you know, the days of just reading code line by line and that's not the only way anymore.

s31
00:01:32.120 --> 00:01:41.960
And what the point of this talk is about is I want to share with you a bunch of the practices that I use to understand the code that my agents are writing for me.

s32
00:01:41.960 --> 00:01:47.640
This includes things like explainer docs, teaching me about how my code works.

s33
00:01:47.640 --> 00:01:51.440
My agents write quizzes for me to to test my understanding.

s34
00:01:51.440 --> 00:01:52.840
Am I still really in the loop?

s35
00:01:52.840 --> 00:01:55.400
Am I keeping up?

s36
00:01:55.600 --> 00:02:05.720
I have agents build me micro worlds that I can inhabit to get this intuitive sense of how my code works that's deeper and richer than just a written document.

s37
00:02:05.720 --> 00:02:13.480
And I think all of these are really exciting new possibilities that are opening opening up for AI to help us understand better, not worse.

s38
00:02:13.480 --> 00:02:20.680
And so that's what the point of this talk is going to be about and I hope I can leave you with some techniques that you can take home and use yourself.

s39
00:02:21.959 --> 00:02:23.200
By the way, my timer isn't running.

s40
00:02:23.200 --> 00:02:28.000
If you could get that running, that'd be great so I know blabbering.

s41
00:02:28.280 --> 00:02:29.920
Okay, but let's start let's back up for a sec.

s42
00:02:29.920 --> 00:02:33.440
Before we talk about how, let's talk about why.

s43
00:02:33.440 --> 00:02:35.240
Why bother understanding?

s44
00:02:35.240 --> 00:02:37.720
This is again, it's a question now, right?

s45
00:02:37.720 --> 00:02:41.760
And I think a lot of people get this subtly wrong.

s46
00:02:42.640 --> 00:02:48.120
So what a lot of people think of why do humans still have to understand, they think we understand to verify.

s47
00:02:48.120 --> 00:02:53.240
The agents do dumb stuff, we've all seen it, and your job as the human is to keep them in line, right?

s48
00:02:53.240 --> 00:02:56.040
Make sure they don't screw up.

s49
00:02:56.680 --> 00:03:05.760
When people say things like code review is the new bottleneck, I think that's the first thing that pops into people's heads is correctness checking.

s50
00:03:05.760 --> 00:03:10.400
There's this mental model that's like, "Hey, the agent's going to give you something, and what's your job?

s51
00:03:10.400 --> 00:03:13.040
It's to ask, is this correct?"

s52
00:03:13.040 --> 00:03:15.280
Now, correctness can have lots of definitions.

s53
00:03:15.280 --> 00:03:17.120
Does it match the spec doc you gave it?

s54
00:03:17.120 --> 00:03:19.519
Does it take down production?

s55
00:03:19.519 --> 00:03:21.640
Is it well architected?

s56
00:03:21.640 --> 00:03:26.040
But fundamentally, those are all kind of thumbs up, thumbs down decisions, right?

s57
00:03:26.040 --> 00:03:33.760
And the thing is, over time, we've all seen it, the agents are also able to ask these questions and they're getting better at it.

s58
00:03:33.760 --> 00:03:38.519
You give it the right verification loop, and over time, this is the reality.

s59
00:03:38.519 --> 00:03:43.560
The the role of humans in correctness checking is decreasing.

s60
00:03:43.560 --> 00:03:44.160
And you know what?

s61
00:03:44.160 --> 00:03:45.760
I actually don't hate that.

s62
00:03:45.760 --> 00:03:52.160
If I have a clear idea of what I want to do, and the agent does it correctly, instead of coming back to me with an incorrect thing, that's great.

s63
00:03:52.160 --> 00:03:54.680
I'm I'm into it.

s64
00:03:54.680 --> 00:03:57.000
So then I think people extend this and say, "You know what?

s65
00:03:57.000 --> 00:04:02.080
That means as the agents get smarter and smarter and smarter, we we don't have to understand it all, right?

s66
00:04:02.080 --> 00:04:03.160
Get out of the loop, man.

s67
00:04:03.160 --> 00:04:04.880
Run the loop."

s68
00:04:04.880 --> 00:04:08.400
And that's where I think people miss something really important.

s69
00:04:08.760 --> 00:04:14.800
There is a deeper reason to understand what's going on, and that's understanding to participate.

s70
00:04:14.959 --> 00:04:19.799
Because here's the thing, it's not just one loop.

s71
00:04:19.799 --> 00:04:24.160
When you review what's happening and get in the loop, you come away changed.

s72
00:04:24.160 --> 00:04:31.560
You understand something, and that understanding is what you take to the next loop, and the next, and the next.

s73
00:04:32.360 --> 00:04:42.040
Your understanding of what's going on is the foundation for you having that next idea and being an active creative participant in a project.

s74
00:04:42.040 --> 00:04:51.800
I think probably you've all you've all felt even before AI, you know, the difference of the kinds of ideas someone can have when they really understand what's going on versus when they're a few layers removed are different.

s75
00:04:51.800 --> 00:05:02.000
Because when you have rich conceptual structures in your head that you can fluently recombine really fast without going out to like ask some some agent or some human how it works,

s76
00:05:02.000 --> 00:05:05.919
that gives you the ability to fluidly take creative leaps.

s77
00:05:05.919 --> 00:05:10.520
And that's the human part of the work, coming up with the next idea and the next idea.

s78
00:05:10.520 --> 00:05:17.840
So this is actually the real reason I think understanding matters, and this is not something that we can just wash away with better agents.

s79
00:05:17.840 --> 00:05:22.240
Because if we want to be active participants, you still got to do this.

s80
00:05:23.480 --> 00:05:28.440
There's a great term maybe some of you have heard called cognitive debt that I think really captures the spirit well.

s81
00:05:28.440 --> 00:05:31.720
It's an analogy to technical debt, popularized by the scholar Margaret Stories.

s82
00:05:31.720 --> 00:05:33.840
Simon Willison also blogged about it.

s83
00:05:33.840 --> 00:05:42.280
And I love this idea because similarly to tech debt, you might get away with it for a little bit, but at some point you get burned if your understanding degrades.

s84
00:05:42.280 --> 00:05:43.280
And maybe you felt this.

s85
00:05:43.280 --> 00:05:44.280
I know I felt it.

s86
00:05:44.280 --> 00:05:49.760
You're vibe coding, things are going well, and then at some point you realize, wait, I've no idea what's going on.

s87
00:05:49.760 --> 00:05:52.000
I basically can't participate anymore, right?

s88
00:05:52.000 --> 00:05:55.200
You've built up too much cognitive debt.

s89
00:05:56.120 --> 00:05:59.920
Okay, so maybe it sounds like all of you were already convinced.

s90
00:05:59.920 --> 00:06:00.640
We agree.

s91
00:06:00.640 --> 00:06:02.840
We need to understand.

s92
00:06:02.840 --> 00:06:04.240
But how?

s93
00:06:04.240 --> 00:06:05.640
Right?

s94
00:06:05.640 --> 00:06:08.080
We don't want to live in 2023.

s95
00:06:08.080 --> 00:06:11.640
We are using agents to move fast, and it is harder and harder to keep up.

s96
00:06:11.640 --> 00:06:13.480
How do we do it?

s97
00:06:13.480 --> 00:06:23.040
I think to answer this question, we should actually take a step back and ask a more fundamental question, which is how do we understand stuff in general?

s98
00:06:25.040 --> 00:06:28.919
Plot twist, this is not the first time that any human has asked this question.

s99
00:06:28.919 --> 00:06:30.080
There is a field.

s100
00:06:30.080 --> 00:06:32.080
It's called education.

s101
00:06:32.080 --> 00:06:37.320
Now, when you think education, you might think of bad memories from sitting in lectures or whatever, but I think we can do better.

s102
00:06:37.320 --> 00:06:44.200
We can take inspiration from the best ideas that have ever been invented in education and use them to stay in the loop and understand.

s103
00:06:44.200 --> 00:06:46.560
So, that's what this talk is about.

s104
00:06:46.560 --> 00:06:49.240
We're going to talk about three techniques.

s105
00:06:50.120 --> 00:06:52.720
First, explanations.

s106
00:06:53.560 --> 00:06:57.840
So, when an agent writes some code, it's an opportunity for it to explain the work to you, right?

s107
00:06:57.840 --> 00:07:00.480
And the most naive explanation is, hey, here's the code diff.

s108
00:07:00.480 --> 00:07:03.520
That's the raw change, the material of what happened.

s109
00:07:03.520 --> 00:07:06.880
But we can do, I think, much, much better.

s110
00:07:06.880 --> 00:07:09.720
What would the best possible explanation be?

s111
00:07:09.720 --> 00:07:16.520
Like if you sent a team away for a year to come up with a personalized curriculum just to explain this one code change to you, what would that look like?

s112
00:07:16.520 --> 00:07:20.240
I think this is a very generative question to ask.

s113
00:07:20.440 --> 00:07:23.040
So, I've done a bunch of attempts at this.

s114
00:07:23.040 --> 00:07:30.120
One is this skill I wrote called explainedif, which I use every day, and a lot of my co-workers do as well, and I want to walk you through it.

s115
00:07:30.120 --> 00:07:31.960
So, we're going to go through a little example here.

s116
00:07:31.960 --> 00:07:37.480
I'm working on a video game where you draw Zen gardens, kind of de-stress, you know.

s117
00:07:37.480 --> 00:07:39.720
We can all use that these days.

s118
00:07:39.720 --> 00:07:45.000
And we made a code change to change the perspective of the game from top-down to isometric.

s119
00:07:45.000 --> 00:07:48.720
And I when I run my skill, it produces a code explainer doc.

s120
00:07:48.720 --> 00:07:50.480
This can be an HTML file, it can be markdown.

s121
00:07:50.480 --> 00:07:56.960
I like to put them in Notion because I work there, but also because it's then collaborative, so my team can comment on it and talk about it.

s122
00:07:56.960 --> 00:07:59.360
And here's how it looks.

s123
00:08:00.320 --> 00:08:01.360
We start with background.

s124
00:08:01.360 --> 00:08:04.320
We do not start with what happened in this change.

s125
00:08:04.320 --> 00:08:07.160
It starts by teaching me, "Hey, here's how the system works.

s126
00:08:07.160 --> 00:08:08.320
Here's the game engine we're using.

s127
00:08:08.320 --> 00:08:09.720
Here's the coordinate system, right?

s128
00:08:09.720 --> 00:08:11.840
Here's the subsystems."

s129
00:08:11.840 --> 00:08:17.200
It makes sure that I'm sort of being led up to the point where I can even begin to understand what's going on here.

s130
00:08:17.200 --> 00:08:18.760
Obviously, you can skip this if you already know.

s131
00:08:18.760 --> 00:08:22.240
You can personalize it to what you already know.

s132
00:08:22.880 --> 00:08:26.720
Second important principle is intuition before details.

s133
00:08:26.720 --> 00:08:36.400
So, before we start, you know, looking at code and stuff, it says, "Hey, the goal of this commit is to make the garden feel three-dimensional using only 2D drawing tricks."

s134
00:08:36.400 --> 00:08:39.880
You can think of this sort of as like a well-written commit message, a little deeper.

s135
00:08:39.880 --> 00:08:40.760
Give me examples.

s136
00:08:40.760 --> 00:08:45.920
Give me a feel for the essence before, you know, you throw a bunch of code at me, right?

s137
00:08:45.920 --> 00:08:47.480
This, by the way, this is good teaching.

s138
00:08:47.480 --> 00:08:50.839
This is what like good math teachers do.

s139
00:08:51.720 --> 00:08:52.880
Third, interactive figures.

s140
00:08:52.880 --> 00:08:56.880
So, where it makes sense, give me things to fiddle with and try.

s141
00:08:56.880 --> 00:08:59.920
So, with this change, it was like changing how we draw rocks.

s142
00:08:59.920 --> 00:09:07.800
So, I can drag around rocks in this little simulation, and it shows me the coordinates that are happening, how the Z layers of the painting are changing.

s143
00:09:07.800 --> 00:09:13.080
This, by the way, is actually using a new feature that Notion literally launched this morning of HTML blocks in Notion pages.

s144
00:09:13.080 --> 00:09:16.280
So, agents can put interactive simulations into your Notion pages.

s145
00:09:16.280 --> 00:09:18.480
Pretty cool.

s146
00:09:18.920 --> 00:09:20.400
I think you have to be careful with interactivity.

s147
00:09:20.400 --> 00:09:23.400
It can just be a crutch, and it can be kind of slop, to be honest.

s148
00:09:23.400 --> 00:09:29.839
But used tastefully, it can provide understanding that's hard to achieve with just static pictures.

s149
00:09:30.760 --> 00:09:32.640
Okay, then we finally get to the code, right?

s150
00:09:32.640 --> 00:09:33.839
Show me the code.

s151
00:09:33.839 --> 00:09:37.280
But we don't just throw a list of files in order.

s152
00:09:37.280 --> 00:09:39.360
We do what we what I call literate code diffs.

s153
00:09:39.360 --> 00:09:40.040
Give me prose.

s154
00:09:40.040 --> 00:09:42.120
Explain it to me in the right order.

s155
00:09:42.120 --> 00:09:44.600
Tell me before each file what's going on.

s156
00:09:44.600 --> 00:09:50.120
And when you accumulate all this stuff, it's much much easier to follow than just a raw diff.

s157
00:09:50.120 --> 00:09:51.400
Oops.

s158
00:09:51.400 --> 00:09:55.920
In fact, I print these out and take them to the coffee shop sometimes and just read them.

s159
00:09:55.920 --> 00:10:04.600
I find it really beautifully ironic that AI has actually taken this process where I was used to be like glued to my computer my IDE, and now I can go to the cafe and it's like I'm reading a textbook

s160
00:10:04.600 --> 00:10:05.520
about this PR.

s161
00:10:05.520 --> 00:10:07.840
It's really cool.

s162
00:10:08.120 --> 00:10:11.360
Okay, so there is one problem, which is that reading is hard.

s163
00:10:11.360 --> 00:10:13.320
And I am lazy.

s164
00:10:13.320 --> 00:10:15.280
People are lazy.

s165
00:10:15.280 --> 00:10:18.920
You know, there's this one time when I sent a PR to my coworker that I thought I had read the thing.

s166
00:10:18.920 --> 00:10:22.720
I thought I understood, and she asked me the most basic question.

s167
00:10:22.720 --> 00:10:25.800
And I was like, "Oh, no. I don't know."

s168
00:10:25.800 --> 00:10:27.120
I clearly hadn't understood, right?

s169
00:10:27.120 --> 00:10:28.600
I had fooled myself.

s170
00:10:28.600 --> 00:10:32.840
So, I thought, "How can I create a system where that never happens again?"

s171
00:10:32.840 --> 00:10:38.600
For inspiration, I look to the work of the researcher Andy Matuschak, who has this great line, "Books don't work."

s172
00:10:38.600 --> 00:10:43.640
What he means by that is it's really easy to read a book and not realize you didn't understand it.

s173
00:10:43.640 --> 00:10:52.880
So, that So, he and his collaborator Michael Nielsen tried this thing where in an essay, there are interactive spaced repetition quizzes that test whether you actually remember what you just read.

s174
00:10:52.880 --> 00:10:53.320
And this is cool.

s175
00:10:53.320 --> 00:10:57.320
It actually keep emailing you the quiz to make sure you remember it forever.

s176
00:10:57.320 --> 00:11:03.560
But, this is nice because you cannot get through this essay without understanding it, or at least without remembering it.

s177
00:11:03.560 --> 00:11:04.760
That's what I do with my code explainers.

s178
00:11:04.760 --> 00:11:06.520
At the very bottom, there's a quiz.

s179
00:11:06.520 --> 00:11:09.160
Five questions, medium difficulty.

s180
00:11:09.160 --> 00:11:18.320
And my rule is I don't send code to uh others on my team to review unless I can pass the quiz about what my agents wrote.

s181
00:11:18.320 --> 00:11:20.520
And it might sound kind of silly, but you should try it.

s182
00:11:20.520 --> 00:11:26.800
It really is shocking the number of times this has caught me and made me and made me realize I didn't understand.

s183
00:11:26.920 --> 00:11:28.640
I think of it as sort of a speed regulator.

s184
00:11:28.640 --> 00:11:30.200
Everything AI is speed up, speed up, speed up.

s185
00:11:30.200 --> 00:11:32.240
There's all these incentives to go faster.

s186
00:11:32.240 --> 00:11:36.240
How do we make sure we're not just moving at the speed of correctness, but also of understanding?

s187
00:11:36.240 --> 00:11:41.600
And the quiz is that speed regulator, that's a system I can use for that.

s188
00:11:41.920 --> 00:11:43.760
I did uh just put the scale on the internet.

s189
00:11:43.760 --> 00:11:45.000
So, yeah, photo moment.

s190
00:11:45.000 --> 00:11:49.720
If you want the Explain Diff scale, uh take that QR code, try it out, make it your own.

s191
00:11:49.720 --> 00:11:51.520
It's really simple, actually.

s192
00:11:51.520 --> 00:11:56.560
There's two versions at that QR code, one that outputs HTML, one that outputs Notion.

s193
00:11:57.520 --> 00:11:59.000
Okay.

s194
00:11:59.000 --> 00:12:00.760
Second technique, microworlds.

s195
00:12:00.760 --> 00:12:02.320
What does that mean?

s196
00:12:02.320 --> 00:12:10.240
So, this takes inspiration from the educator Seymour Papert, real visionary, who had this idea of living in Mathland.

s197
00:12:10.240 --> 00:12:15.720
And what that meant was, "Hey, kids learn French from living in France, where do they go to learn math?

s198
00:12:15.720 --> 00:12:19.880
Is there a Mathland where you can learn intuitively math just by being there?"

s199
00:12:19.880 --> 00:12:25.400
So, he did these great things with this is a a robot called the turtle that kids program to draw stuff.

s200
00:12:25.400 --> 00:12:29.760
But the point isn't making robots, the point is they actually learn math by doing that programming.

s201
00:12:29.760 --> 00:12:34.080
But the point isn't the robot, it's the kids that are changed.

s202
00:12:34.080 --> 00:12:38.240
So, how could we apply this to understanding code?

s203
00:12:38.240 --> 00:12:38.920
Here's one example.

s204
00:12:38.920 --> 00:12:47.480
Last year, I was trying to implement um for my own learning this interpreter for a programming language Prolog, which is think of it a little bit like a database query language.

s205
00:12:47.480 --> 00:12:52.280
And there's all these parts of it that look like this, where when you read them on Wikipedia, they seem really complicated.

s206
00:12:52.280 --> 00:12:59.080
And then when you actually get what's going on, it's like, "Wait a second, that wasn't that hard to understand, it just felt hard when I read it that way, right?"

s207
00:12:59.080 --> 00:13:02.280
How could we make it click more for my brain?

s208
00:13:02.280 --> 00:13:05.320
So, I had Claude make me a microworld.

s209
00:13:05.320 --> 00:13:13.680
This is a debugger, ephemeral UI that was built specifically to visualize the internal implementation of my programming language.

s210
00:13:13.680 --> 00:13:18.520
What's happening here is that I'm scrubbing through a timeline that's running step-by-step what's my interpreter doing.

s211
00:13:18.520 --> 00:13:20.800
It's visualizing all the state at every step.

s212
00:13:20.800 --> 00:13:25.520
So, I can kind of open the hood and see what's going on and start feeling it, you know?

s213
00:13:25.520 --> 00:13:27.920
And yes, I can use I used this to fix bugs.

s214
00:13:27.920 --> 00:13:36.240
I even had a little hard to see here, but there's a commenting feature where I can leave comments for myself on the timeline so I remember what I was thinking.

s215
00:13:36.240 --> 00:13:42.960
And I use this to fix fix narrow bugs, but also as I was fixing the bugs, I was getting a feel for the machine, right?

s216
00:13:42.960 --> 00:13:46.680
That's something that if you just have an agent go fix the bug, you don't get that peripheral vision.

s217
00:13:46.680 --> 00:13:49.840
If you live in a micro world, you do.

s218
00:13:50.560 --> 00:13:55.840
Another example, um I was migrating my personal website from one framework to another.

s219
00:13:55.840 --> 00:13:58.760
And first thing I did, I had I said, "Claude, write me a script to do."

s220
00:13:58.760 --> 00:14:00.400
It did this.

s221
00:14:00.400 --> 00:14:03.240
It seemed like it worked, and I read the script and I was like, "Ah, I don't know."

s222
00:14:03.240 --> 00:14:06.120
Like I I just don't have a feel for what it's doing.

s223
00:14:06.120 --> 00:14:07.760
A bunch of files went a bunch of places.

s224
00:14:07.760 --> 00:14:10.280
It seems right.

s225
00:14:10.560 --> 00:14:15.560
So, what I did is I said, "Hey Claude, make me essentially a video game where I do the port myself."

s226
00:14:15.560 --> 00:14:18.640
And the way this works is old website on the left, new website on the right.

s227
00:14:18.640 --> 00:14:20.600
I just click a button, next, next, next.

s228
00:14:20.600 --> 00:14:23.120
And at each step, it says, "Here's the commands I'm running.

s229
00:14:23.120 --> 00:14:24.720
Your new website's coming to life step by step.

s230
00:14:24.720 --> 00:14:26.920
You see it."

s231
00:14:26.920 --> 00:14:29.839
There's actually file trees down there where I can see files moving.

s232
00:14:29.839 --> 00:14:34.960
And it's the the result is it's kind of like if I did it manually, but I'm just clicking a button.

s233
00:14:34.960 --> 00:14:39.760
So, I'm getting some of the benefit of doing it iteratively without the pain.

s234
00:14:42.200 --> 00:14:46.280
And I think the big takeaway here is agents can write code to help us understand code.

s235
00:14:46.280 --> 00:14:51.520
Where the point isn't building software to ship, it's building these little micro worlds for us.

s236
00:14:51.520 --> 00:14:52.600
It's the math land, right?

s237
00:14:52.600 --> 00:14:57.160
It's It's a It's a simulation of just this thing.

s238
00:14:58.720 --> 00:15:00.560
Okay.

s239
00:15:00.560 --> 00:15:03.320
Quickly, the last topic, shared spaces.

s240
00:15:03.320 --> 00:15:07.280
So far, we've This has all been about me, solo understanding.

s241
00:15:07.280 --> 00:15:10.120
But a lot of the time, I think the challenge is actually you're working on a team.

s242
00:15:10.120 --> 00:15:16.520
And your whole team needs to understand together so you can actually jam and have creative ideas together.

s243
00:15:16.520 --> 00:15:19.160
We think a ton about this at Notion.

s244
00:15:19.160 --> 00:15:25.640
We believe that, you know, the shared understanding that exists between you and someone else is what lets you communicate effectively.

s245
00:15:25.640 --> 00:15:33.040
Whether that's names for parts of a system, it could be names for UI elements or concepts, right?

s246
00:15:33.040 --> 00:15:37.920
So, we we think a lot of Notion about how do you make tools that enable collective understanding.

s247
00:15:37.920 --> 00:15:43.840
Some things we're exploring, can you have multiplayer chat threads between multiple humans and agents together?

s248
00:15:43.840 --> 00:15:51.880
So, here, you know, I might ask a product manager on my team, "Hey, you know, what what are users asking for with this feature?"

s249
00:15:51.880 --> 00:15:53.960
And she might say, "Hey, I don't know.

s250
00:15:53.960 --> 00:15:55.080
Let's ask a different agent."

s251
00:15:55.080 --> 00:15:57.440
And that agent comes in and talks to us.

s252
00:15:57.440 --> 00:16:04.960
What's happening here is that instead of me and my PM both talking to our own agents, we're in a shared space, we can see each other's communication.

s253
00:16:04.960 --> 00:16:08.880
It's kind of like, you know, going from one-on-one conversations to Slack channels, you know?

s254
00:16:08.880 --> 00:16:14.000
You see more of the behavior happening together and you understand together.

s255
00:16:14.600 --> 00:16:18.760
Also, having documents that you can talk about together is a really powerful primitive.

s256
00:16:18.760 --> 00:16:21.800
Here, you know, Claude made us a plan.

s257
00:16:21.840 --> 00:16:24.520
What if I have a question about that that I want to discuss with my team?

s258
00:16:24.520 --> 00:16:29.320
I can just leave a comment because this isn't a collaborative space, not on my computer locally.

s259
00:16:29.320 --> 00:16:32.000
And then I can ask, "Hey, you know, what do you think about this?"

s260
00:16:32.000 --> 00:16:35.000
And my teammate can chime in and and we can talk about it right there, right?

s261
00:16:35.000 --> 00:16:43.320
I think having these spaces for shared discussion around ideas with our agents is really powerful for building up that collective understanding.

s262
00:16:44.000 --> 00:16:47.839
This, by the way, um we just launched last week the ability to bring coding agents into Notion.

s263
00:16:47.839 --> 00:16:50.160
So, Claude and Cursor can now live in Notion.

s264
00:16:50.160 --> 00:16:59.520
And our team actually builds a lot of our code in Notion itself, mainly because of these benefits, because having it in a shared space is just so valuable.

s265
00:17:01.320 --> 00:17:03.760
Okay, so we've talked about these three techniques.

s266
00:17:03.760 --> 00:17:06.439
I want to bring it back to the beginning.

s267
00:17:06.560 --> 00:17:12.800
You know, I think at the beginning I said, "It's important for humans to still understand how the code works, right?"

s268
00:17:12.800 --> 00:17:15.920
But I actually think it's much bigger than that.

s269
00:17:15.920 --> 00:17:21.480
I kind of think it's just important for humans to still understand how everything works.

s270
00:17:22.000 --> 00:17:24.560
And maybe, you know, you all agree it sounds like.

s271
00:17:24.560 --> 00:17:29.960
But this I think is being called into question now and is something we actually have to actively fight for.

s272
00:17:29.960 --> 00:17:32.080
The thing is this is not a new battle.

s273
00:17:32.080 --> 00:17:35.960
It actually hearkens back to the very origins of our field.

s274
00:17:36.480 --> 00:17:40.600
Alan Kay is one of the pioneers of personal computing, co-inventor of the modern GUI.

s275
00:17:40.600 --> 00:17:48.360
And literally 50 years ago he wrote this essay that I find very prescient, called a personal computer for children of all ages.

s276
00:17:48.360 --> 00:17:51.360
It looks like two kids on iPads watching YouTube or something, right?

s277
00:17:51.360 --> 00:17:52.360
It's kind of crazy.

s278
00:17:52.360 --> 00:17:54.160
This is 50 years ago his vision.

s279
00:17:54.160 --> 00:17:55.920
But that's not YouTube on the iPads.

s280
00:17:55.920 --> 00:18:03.160
What he envisioned is, "Hey, these kids, they're playing a video game and they're modifying the code as they play it to learn physics."

s281
00:18:03.160 --> 00:18:06.240
So the point isn't the computer, it's the kids.

s282
00:18:06.240 --> 00:18:09.480
The point of computers was to level us up as humans, right?

s283
00:18:09.480 --> 00:18:16.000
And Alan has talked a lot about how it kind of feels like at some point computers detoured a bit from that vision.

s284
00:18:16.000 --> 00:18:20.240
But I think the exciting thing is maybe now's the time to bring that back.

s285
00:18:20.840 --> 00:18:23.680
Here's kind of the meme version of that.

s286
00:18:23.680 --> 00:18:28.280
I think with AI, a lot of people are waking up to, "Oh my gosh, code is free.

s287
00:18:28.280 --> 00:18:31.600
We can make ephemeral UIs, dynamic simulations to understand concepts.

s288
00:18:31.600 --> 00:18:34.520
We can make debuggers, playgrounds."

s289
00:18:34.560 --> 00:18:36.440
And it's like, "Yes, that's great."

s290
00:18:36.440 --> 00:18:38.880
And it's actually not a new idea.

s291
00:18:38.880 --> 00:18:41.240
Like this was the goal all along.

s292
00:18:41.240 --> 00:18:48.800
And so I think the optimistic thing that I find really exciting is hey, it's still really important to understand how things work.

s293
00:18:48.800 --> 00:18:55.640
And with the right tools and the right mindset and the right creativity, we can actually understand better than ever before, not less.

s294
00:18:55.640 --> 00:19:05.200
With AI, we can kind of empower ourselves more, not just taking ourselves out of loops, but actually putting ourselves more deeply in loops than we ever have before.

s295
00:19:05.200 --> 00:19:12.200
And I think that's a really exciting prospect and I hope it's something that we all, together as an industry, figure out.

s296
00:19:12.200 --> 00:19:13.240
That's all I have for you today.

s297
00:19:13.240 --> 00:19:15.433
Thank you so much.

s298
00:19:15.433 --> 00:19:17.433
[applause]

s299
00:19:29.843 --> 00:19:31.843
[music]