WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=-I5W5QVAT8E

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/-I5W5QVAT8E.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:21.920 --> 00:00:22.400
Okay.

s3
00:00:22.400 --> 00:00:22.880
Hello.

s4
00:00:22.880 --> 00:00:23.199
Okay.

s5
00:00:23.199 --> 00:00:26.800
Before I get started, you guys, this is a huge keynote room.

s6
00:00:26.800 --> 00:00:32.000
Can everyone like come forward because I'm talking to like four empty rows and disperse people.

s7
00:00:32.000 --> 00:00:32.559
Do me a favor.

s8
00:00:32.559 --> 00:00:35.360
I'm spending 30 minutes telling you all of our secrets.

s9
00:00:35.360 --> 00:00:36.960
I can see you still.

s10
00:00:36.960 --> 00:00:37.360
Thank you.

s11
00:00:37.360 --> 00:00:37.760
Thank you.

s12
00:00:37.760 --> 00:00:38.079
Thank you.

s13
00:00:38.079 --> 00:00:38.399
Thank you.

s14
00:00:38.399 --> 00:00:39.760
We're just going to chat.

s15
00:00:39.760 --> 00:00:42.719
It's a giant room and there's 500 of us.

s16
00:00:42.719 --> 00:00:44.399
This room is way larger than that.

s17
00:00:44.399 --> 00:00:45.440
Thank you.

s18
00:00:45.440 --> 00:00:48.000
Honestly, I knew you guys had it in you.

s19
00:00:48.000 --> 00:00:49.760
It's really not so hard.

s20
00:00:49.760 --> 00:00:50.320
Thank you.

s21
00:00:50.320 --> 00:00:51.760
I also sit in the back.

s22
00:00:51.760 --> 00:00:53.280
I also work during talks.

s23
00:00:53.280 --> 00:00:55.039
I get it.

s24
00:00:55.039 --> 00:00:55.920
I totally get it.

s25
00:00:55.920 --> 00:00:58.160
I did all day, but not for me.

s26
00:00:58.160 --> 00:01:02.960
Okay, I'm gonna start, but I'm gonna still point at you if you're in the back like you.

s27
00:01:02.960 --> 00:01:05.280
Okay, I'm Sarah.

s28
00:01:05.280 --> 00:01:09.200
I'm lead our engineering teams for AI at Notion.

s29
00:01:09.200 --> 00:01:10.799
Um, welcome to my talk.

s30
00:01:10.799 --> 00:01:13.200
Um, it's about token town.

s31
00:01:13.200 --> 00:01:18.080
How do you go from not go from AI pled to AI poor?

s32
00:01:18.080 --> 00:01:19.040
Okay.

s33
00:01:19.040 --> 00:01:22.159
Um, I know that today is all about software factories.

s34
00:01:22.159 --> 00:01:25.920
We're going to talk about that, but we're going to talk about how to do it sustainably.

s35
00:01:25.920 --> 00:01:26.640
This is me.

s36
00:01:26.640 --> 00:01:30.560
This is on my first day at Notion in a very sweaty subway.

s37
00:01:30.560 --> 00:01:33.759
Um, like I said, I lead our AI teams at Notion.

s38
00:01:33.759 --> 00:01:36.479
Um, and I negotiate AI contracts for a living.

s39
00:01:36.479 --> 00:01:38.320
My team jokes I act like Anna Winter.

s40
00:01:38.320 --> 00:01:46.079
So, this is a nice a nice image of me with AI Anna Winter hair after a press article referred to me that externally.

s41
00:01:46.079 --> 00:01:48.640
Um, and that's kind of the idea, right?

s42
00:01:48.640 --> 00:01:52.880
Uh, how do you think about negotiating between different vendors?

s43
00:01:52.880 --> 00:01:56.399
um making sure that you maintain taste for your company.

s44
00:01:56.399 --> 00:01:57.920
I don't do it alone.

s45
00:01:57.920 --> 00:02:00.240
Um this is launch day of one of our recent launches.

s46
00:02:00.240 --> 00:02:01.280
This is just a subset.

s47
00:02:01.280 --> 00:02:05.360
Any good engineering manager points out that we have a whole company of people building this.

s48
00:02:05.360 --> 00:02:08.560
I'm just the one that gets to come talk to you about it.

s49
00:02:08.560 --> 00:02:10.239
So, we've been building a lot.

s50
00:02:10.239 --> 00:02:15.840
Um this is um an example of our AI usage um just in 2026.

s51
00:02:15.840 --> 00:02:19.440
Um and we've been really proud of how we've been able to grow that usage.

s52
00:02:19.440 --> 00:02:24.959
and I'm going to talk to you about how you can build an AI native product and an AI native company.

s53
00:02:24.959 --> 00:02:29.280
Um, but this is just to give me some credit that that we're doing it kind of well.

s54
00:02:29.280 --> 00:02:34.400
Okay, so for those of you that don't know, notion's always been that durable system of record.

s55
00:02:34.400 --> 00:02:37.840
It's always been the place where you can collaborate with your peers.

s56
00:02:37.840 --> 00:02:40.720
Um, but today that point of collaboration is a little bit different.

s57
00:02:40.720 --> 00:02:42.080
It's not just humans.

s58
00:02:42.080 --> 00:02:44.400
Notion's always been the place for collaboration.

s59
00:02:44.400 --> 00:02:47.680
And today that collaboration happens between humans and agents.

s60
00:02:47.680 --> 00:02:51.360
Humans and humans, agents and agents.

s61
00:02:52.400 --> 00:02:56.239
And we like to think about AI transformations going through this journey.

s62
00:02:56.239 --> 00:03:00.160
And I'm sure some of you are looking at the slide and wondering where you are.

s63
00:03:00.160 --> 00:03:03.280
AI as a thought partner is when we all started tinkering.

s64
00:03:03.280 --> 00:03:10.720
We all started just going to the very first version of chat on Thanksgiving when it came out three years ago, four years ago.

s65
00:03:10.720 --> 00:03:21.680
And we started saying like how can I send this email to my landlord to say that I shouldn't pay for repainting right then we'd copy paste it enter it into our email

s66
00:03:21.680 --> 00:03:32.080
eventually we started getting to a place where we could use AI like an assistant AI was able to maybe execute individual tasks that's how notion AI really took off in the beginning

s67
00:03:32.080 --> 00:03:43.200
um and it was able to save employee time but functionally was limited in its capabilities based on what humans asked it to do AI as teammates is what we were really excited to launch almost a year ago now.

s68
00:03:43.200 --> 00:03:51.200
Um, but this is true in many products where we can do repetitive work and think about a process and have AI do that process.

s69
00:03:51.200 --> 00:03:59.680
What I think is really interesting is when AI actually becomes that critical workflow where processes are interfacing with each other and you have entire systems running.

s70
00:03:59.680 --> 00:04:03.599
How many of you guys feel like you have AI as a system down?

s71
00:04:03.599 --> 00:04:06.080
Aren't you sad you came up now?

s72
00:04:06.080 --> 00:04:06.640
Great.

s73
00:04:06.640 --> 00:04:07.360
None of you.

s74
00:04:07.360 --> 00:04:08.400
Exactly.

s75
00:04:08.400 --> 00:04:11.760
We have found that no one has figured out how to do this well.

s76
00:04:11.760 --> 00:04:16.239
88% of people can't even get past AI as an assistant.

s77
00:04:16.239 --> 00:04:17.440
And why is that?

s78
00:04:17.440 --> 00:04:18.799
We have a thesis at notion.

s79
00:04:18.799 --> 00:04:25.199
It's because there's too much siloed data and not a sturable system of record for that point of collaboration.

s80
00:04:25.199 --> 00:04:33.040
And we believe that for your software factory to work, for your company to work, and for your systems to work, you need that durable system of record.

s81
00:04:33.040 --> 00:04:36.080
And that is notion's mission.

s82
00:04:36.479 --> 00:04:38.880
So doing that is expensive.

s83
00:04:38.880 --> 00:04:49.199
Um you see a lot of companies that try and commit themselves to this vision and these are just a series of headlines all within a week of how that's painful.

s84
00:04:49.199 --> 00:04:56.880
So you can put all of your money into a process to try and make a system and you end up feeling like this, right?

s85
00:04:56.880 --> 00:05:00.880
You end up using a blowtorrch to light what is actually a large cigar.

s86
00:05:00.880 --> 00:05:03.840
But you kind of get the idea.

s87
00:05:04.560 --> 00:05:07.440
Cost is a structural barrier to entry.

s88
00:05:07.440 --> 00:05:09.360
It makes it hard for you to serve products.

s89
00:05:09.360 --> 00:05:11.520
It makes it hard for you to build factories.

s90
00:05:11.520 --> 00:05:18.639
And it is ultimately, I would posit, one of the largest reasons why things do not happen at scale successfully today.

s91
00:05:18.639 --> 00:05:27.840
And I would argue for anyone working in an applied AI company, it's something for them to be really familiar with to understand the trade-offs that they're making to build durable

s92
00:05:27.840 --> 00:05:32.160
and exciting and enlightening product for their customers.

s93
00:05:32.320 --> 00:05:35.520
But that's not really how the market is today, right?

s94
00:05:35.520 --> 00:05:39.520
I'm not going to name names here, but you guys have search engines.

s95
00:05:39.520 --> 00:05:40.479
You can figure it out.

s96
00:05:40.479 --> 00:05:43.600
Exhibit A. A reasoning model gets upgraded.

s97
00:05:43.600 --> 00:05:44.080
Amazing.

s98
00:05:44.080 --> 00:05:46.479
The per token pricing is the same.

s99
00:05:46.479 --> 00:05:53.039
What's not to love, you try it out, it uses three times as many output tokens, right?

s100
00:05:53.039 --> 00:05:59.120
Exhibit B. A model gets upgraded, but it has an entire new digit, right?

s101
00:05:59.120 --> 00:06:02.800
Whatever demarcation system that model family likes, it's brand new.

s102
00:06:02.800 --> 00:06:07.680
It's 40% more than its predecessor, which is being deprecated in the next four months.

s103
00:06:07.680 --> 00:06:09.919
These are real scenarios that we face at notion.

s104
00:06:09.919 --> 00:06:15.199
All of you are nodding because these are common pretty much monthly now.

s105
00:06:15.199 --> 00:06:16.479
But here's the problem.

s106
00:06:16.479 --> 00:06:20.000
Are you growing 40% in that time period?

s107
00:06:20.000 --> 00:06:23.120
Are you making 30 3x more revenue?

s108
00:06:23.120 --> 00:06:26.400
No. So, how do you navigate the system?

s109
00:06:26.400 --> 00:06:31.280
If you just auto upgrade your model and everything that you're doing, you're you're giving someone a bad deal.

s110
00:06:31.280 --> 00:06:39.039
Either your customers or your investors, depending on how you charge and where you get your money, neither are good.

s111
00:06:39.759 --> 00:06:43.919
Fortune 5 million companies have the capability to navigate this.

s112
00:06:43.919 --> 00:06:51.199
They can hire large consulting teams, have durable teams on their own, and build expertise on how to navigate these trade-offs.

s113
00:06:51.199 --> 00:06:53.039
Um, most people don't.

s114
00:06:53.039 --> 00:06:59.840
everyone else has no ability to negotiate with leverage and they're stuck in these scenarios, right?

s115
00:06:59.840 --> 00:07:12.080
Part of my job as that Anna Winter joke is to think about advocating for the Fortune 5 million, the non-fortune 500 companies that don't have the mass to have leverage and negotiate, but need to think about how.

s116
00:07:12.080 --> 00:07:21.599
And I'm going to share some of the lessons that I've learned when I have kind of large amounts of traffic behind me that I think scale to those who don't.

s117
00:07:22.080 --> 00:07:30.880
Um, this is probably less of a secret now than it was when I started giving talks like this u maybe four months ago.

s118
00:07:30.880 --> 00:07:33.919
Um, your supplier is your competitor.

s119
00:07:33.919 --> 00:07:38.240
I know very few people who have convinced me that that's not true.

s120
00:07:38.240 --> 00:07:43.680
Um, you will always be getting a bad deal on tokens with someone who builds them natively, right?

s121
00:07:43.680 --> 00:07:46.479
Sometimes the cost of goods served is extremely different.

s122
00:07:46.479 --> 00:07:54.240
you're basically they're serving a first-party product and then you're buying those tokens at a huge search charge and then selling them again at another search charge.

s123
00:07:54.240 --> 00:07:56.400
Um, that's not really value you can defend.

s124
00:07:56.400 --> 00:07:58.160
You're getting a really bad deal.

s125
00:07:58.160 --> 00:08:01.919
And if you tie yourself to one provider, you have no exit.

s126
00:08:01.919 --> 00:08:10.960
If you build an AI product that you're selling with this structure, you are crossing your fingers and hoping that you are a viable business.

s127
00:08:10.960 --> 00:08:13.840
I do not encourage that.

s128
00:08:14.240 --> 00:08:15.280
This is really interesting.

s129
00:08:15.280 --> 00:08:17.360
Dylan in some analysis posted this.

s130
00:08:17.360 --> 00:08:19.120
I think it's it says eight hours ago.

s131
00:08:19.120 --> 00:08:20.000
It wasn't at this point.

s132
00:08:20.000 --> 00:08:21.840
It was probably a month ago.

s133
00:08:21.840 --> 00:08:33.760
Um they purchased a subscription plan and they just highlighted right how different what Frontier Labs charge customers for first-party products are versus what they sell.

s134
00:08:33.760 --> 00:08:34.719
It's a bad deal.

s135
00:08:34.719 --> 00:08:38.800
Don't play this game or try and let me know how you win.

s136
00:08:38.800 --> 00:08:41.120
I don't recommend.

s137
00:08:41.120 --> 00:08:43.680
Think about everyone else.

s138
00:08:43.680 --> 00:08:48.640
Think about what that structure means and where you have expertise.

s139
00:08:48.880 --> 00:08:51.760
I don't think that that's winning on the token economics.

s140
00:08:51.760 --> 00:08:54.160
I think it's about product.

s141
00:08:54.160 --> 00:09:00.160
It's about building data flywheels and understanding your customers better than anyone else.

s142
00:09:00.160 --> 00:09:06.640
Understanding when you need capability, when you need low price, when you need latency improvements.

s143
00:09:06.640 --> 00:09:12.720
I promise you, you don't always need what is usually the slowest but the most capable model out there.

s144
00:09:12.720 --> 00:09:15.519
and then build compelling UI and orchestration.

s145
00:09:15.519 --> 00:09:23.360
And I'll show you some examples of that to justify the cost on the bad deal tokens that you do resell.

s146
00:09:23.920 --> 00:09:25.120
The job is not to train.

s147
00:09:25.120 --> 00:09:31.120
I mean, some of you might be training the best model and I'd love to serve it and come talk to me afterwards, but most of you are not doing that.

s148
00:09:31.120 --> 00:09:36.240
Stop trying to win that game and think about the best product that uses many models.

s149
00:09:36.240 --> 00:09:39.279
Help your customers, help your team.

s150
00:09:39.279 --> 00:09:42.320
Bet on the frontier, not on the lab.

s151
00:09:42.320 --> 00:09:45.760
and we'll talk about what it looks like to do that.

s152
00:09:45.760 --> 00:09:50.320
This cost per capability per second trade-off is actually really intense.

s153
00:09:50.320 --> 00:09:55.040
Um, Citadel came out with this memo um a while ago, maybe two weeks ago.

s154
00:09:55.040 --> 00:09:56.399
I loved it.

s155
00:09:56.399 --> 00:10:04.959
The idea is that for the economy at large, simpler models might be the most cost-effective productivity augmenting pathway.

s156
00:10:04.959 --> 00:10:08.160
They talk about this bifurcation on frontier versus everyday usage.

s157
00:10:08.160 --> 00:10:09.839
I really believe that.

s158
00:10:09.839 --> 00:10:21.600
And for every product, the definition of frontier versus everyday, the definition of saturated capabilities or model capability overhangs depends on your expertise on your product.

s159
00:10:21.600 --> 00:10:24.320
No one can replace that.

s160
00:10:24.320 --> 00:10:26.160
And not all traffic is equal.

s161
00:10:26.160 --> 00:10:30.079
It is a huge miss to send all of these to the latest opus model.

s162
00:10:30.079 --> 00:10:36.720
Some of these absolutely large scale data analysis when you do it on notion will recommend Opus, right?

s163
00:10:36.720 --> 00:10:44.160
When you triage an email inbox, if we're charging you to do that on Opus, we're ripping you off and ourselves.

s164
00:10:44.160 --> 00:10:47.600
Think about where your traffic patterns are.

s165
00:10:47.600 --> 00:10:51.920
And then think about how Frontier Lab model providers are structured today.

s166
00:10:51.920 --> 00:10:54.640
I mean, it's functionally an oligopoly, right?

s167
00:10:54.640 --> 00:10:57.120
And that's fine because they're racing to the top.

s168
00:10:57.120 --> 00:11:00.160
And I think the top is really hard and really important.

s169
00:11:00.160 --> 00:11:04.000
This is not to say that products don't have a place for frontier difficult tasks.

s170
00:11:04.000 --> 00:11:08.320
I want everyone to nod and understand that's not what this talk is about.

s171
00:11:08.320 --> 00:11:11.360
Understand when you need those tasks and it's not everything.

s172
00:11:11.360 --> 00:11:16.640
The problem with those tasks are is keep in mind how pricing is incentivized.

s173
00:11:16.640 --> 00:11:18.959
You can figure out who these players are.

s174
00:11:18.959 --> 00:11:21.360
Either you are the best model.

s175
00:11:21.360 --> 00:11:24.240
Everything above what AI can't do today is your market.

s176
00:11:24.240 --> 00:11:28.000
You can basically price it as high as you kind of want.

s177
00:11:28.000 --> 00:11:36.320
If you're slightly behind that best model, all you need to be is like a dollar per million tokens cheaper and you have the rest of the market.

s178
00:11:36.320 --> 00:11:43.680
You know that economic theory about gas stations where the best gas stations are the ones that are right next to each other because they cover east and west the most.

s179
00:11:43.680 --> 00:11:44.480
Yeah.

s180
00:11:44.480 --> 00:11:52.959
It's the same with model pricing, which means that price does not correlate with capability growth.

s181
00:11:53.360 --> 00:12:01.200
So for this complex task, understand what capabilities you need, but be the expert on what complexity is.

s182
00:12:01.200 --> 00:12:05.120
And keep in mind that who handles complexity changes.

s183
00:12:05.120 --> 00:12:11.600
Um, oftentimes you'll see applied AI companies really be super outspoken on marketing with a specific lab.

s184
00:12:11.600 --> 00:12:20.480
That's always kind of a red flag for me when they're not model agnostic because if you look at this graph, it basically shows that they're behind every month, right?

s185
00:12:20.480 --> 00:12:32.880
the new model and the new model provider of the best Frontier capabilities change and if you hit your ride with one particular provider in exchange for for instance a larger discount

s186
00:12:32.880 --> 00:12:44.959
um you're doing a disservice to your customers like half of the time right so really think about if that discount is worth not actually having a Frontier product and remember that that optionality

s187
00:12:44.959 --> 00:12:51.279
is your leverage if you don't have the capability to walk at any point you are stuck.

s188
00:12:51.279 --> 00:13:00.079
And again, I think that's probably the most expensive decision you'll make regardless of what discount you get or the engineering work to have model interoperability.

s189
00:13:01.600 --> 00:13:05.279
One option to navigate this is stay model agnostic.

s190
00:13:05.279 --> 00:13:15.600
Have different models and capabilities in your system so that at any point if pricing seems unfair or untenable, you are not out of business.

s191
00:13:15.920 --> 00:13:18.160
Notion's auto model does this really well.

s192
00:13:18.160 --> 00:13:21.279
We have state-of-the-art models available always.

s193
00:13:21.279 --> 00:13:27.920
Um, but we also have an auto model there at the top that handles about 75% of our traffic.

s194
00:13:27.920 --> 00:13:29.360
Right?

s195
00:13:29.360 --> 00:13:36.959
We have the ability to switch between models in our product and we also offer it to our customers so that they have access to these models without vendor lockin.

s196
00:13:36.959 --> 00:13:40.639
That's part of our AI Switzerland approach.

s197
00:13:41.440 --> 00:13:43.040
You guys love taking photos of slides.

s198
00:13:43.040 --> 00:13:44.160
This is the slide.

s199
00:13:44.160 --> 00:13:46.079
Okay.

s200
00:13:46.079 --> 00:13:47.519
Model agnostic playbook.

s201
00:13:47.519 --> 00:13:48.880
This is how you do it.

s202
00:13:48.880 --> 00:13:50.079
Build for multimodal.

s203
00:13:50.079 --> 00:13:53.440
It is hard to kill the cache and switch models mid-transcript.

s204
00:13:53.440 --> 00:13:56.720
I understand that we invest in that technology.

s205
00:13:56.720 --> 00:13:58.800
It doesn't even have to be per thread.

s206
00:13:58.800 --> 00:14:02.560
Just think about your harness as model interoperability.

s207
00:14:02.560 --> 00:14:06.399
Think about the cost per capability per second, not just the tokens.

s208
00:14:06.399 --> 00:14:07.839
Here's a great example.

s209
00:14:07.839 --> 00:14:13.600
We posted this review when we um announced our partnership with Parallel as our web search provider.

s210
00:14:13.600 --> 00:14:19.279
If you were to look at just latency of a single call or just cost, parallel might not be the cheapest.

s211
00:14:19.279 --> 00:14:24.480
But if you have expertise in entire web search trajectories, you'll see how it differs.

s212
00:14:24.480 --> 00:14:35.199
The granularity of this eval is what lets us make the best decisions for our customers because we understand all of the trade-offs on entire trajectories, not just single calls.

s213
00:14:35.680 --> 00:14:36.880
Switch fast and often.

s214
00:14:36.880 --> 00:14:38.160
I think we talked about that.

s215
00:14:38.160 --> 00:14:39.360
And give them something back.

s216
00:14:39.360 --> 00:14:43.360
That expertise on use cases is also very valuable to Frontier Labs.

s217
00:14:43.360 --> 00:14:58.399
We find that our eval program partnerships actually help us a lot with Frontier Labs and is something that we can exchange instead of extraordinarily large commits and I don't think the discount is ever worth the loss in optionality.

s218
00:14:58.399 --> 00:15:01.040
That's a perspective you can choose to keep or not.

s219
00:15:01.040 --> 00:15:06.720
The second option is moderate tasks understanding open weights place there.

s220
00:15:06.720 --> 00:15:17.199
um openweight models are really strong enough to handle these tasks and the possibility to RL on top of them has also kind of expanded the upmarket growth that they can cover.

s221
00:15:17.199 --> 00:15:25.440
I view openw weight models as basically lowering the barrier to entry on cost for our customers and they also give you negotiation leverage.

s222
00:15:25.440 --> 00:15:35.680
So it's kind of a credible alternative that's putting that downward pressure on pricing that if there's an igopoly of two or three providers at the top is unavailable right now otherwise.

s223
00:15:35.680 --> 00:15:49.360
I think Kimmy 26 was probably the first time that we really saw a model that outperformed 52, GPT 52, GLM 52 now is another 52 bombshell in the villa that also probably does best here.

s224
00:15:49.360 --> 00:15:54.720
But it's no longer the case where openway models are good for just SFT on small tasks.

s225
00:15:54.720 --> 00:15:59.920
Really think about without RL if they're capable enough for what you need.

s226
00:16:00.320 --> 00:16:06.079
Um, and again, don't just think about external benchmarks.

s227
00:16:06.079 --> 00:16:08.560
Be able to have expertise on your system.

s228
00:16:08.560 --> 00:16:10.399
What are your tool errors?

s229
00:16:10.399 --> 00:16:12.560
What's the actual latency that you need?

s230
00:16:12.560 --> 00:16:13.199
Right?

s231
00:16:13.199 --> 00:16:15.040
Here's an example of a benchmark that we posted.

s232
00:16:15.040 --> 00:16:18.560
It's a little bit stale on purpose, right?

s233
00:16:18.560 --> 00:16:21.199
But you get the idea.

s234
00:16:21.360 --> 00:16:25.519
Philip at BE 10 this showed this slide once and I've stolen it ever since.

s235
00:16:25.519 --> 00:16:28.000
Um, thank you.

s236
00:16:28.000 --> 00:16:29.040
Are you here, buddy?

s237
00:16:29.040 --> 00:16:29.360
Okay.

s238
00:16:29.360 --> 00:16:31.120
Well, chat, hi.

s239
00:16:31.120 --> 00:16:37.839
Um, well, he could come up and say it better, but the idea is that you don't have to be at the top, right?

s240
00:16:37.839 --> 00:16:41.199
I'm not trying to make a case that open weight is the best model out there.

s241
00:16:41.199 --> 00:16:48.079
Um, the case being made, however, is that um the gap gets covered eventually.

s242
00:16:48.079 --> 00:16:54.320
So, if the tasks that you're having today are good enough, then in six months, they're probably covered by open weight.

s243
00:16:54.320 --> 00:16:57.040
So, be prepared now.

s244
00:16:57.040 --> 00:17:00.000
And the last thing is CPUs over GPUs.

s245
00:17:00.000 --> 00:17:03.759
Um, we've we've recently launched something at notion called workers.

s246
00:17:03.759 --> 00:17:07.679
I don't think that the GPU is necessary for every job.

s247
00:17:07.679 --> 00:17:12.640
A lot of the jobs that we have are actually serving um discrete pieces of code.

s248
00:17:12.640 --> 00:17:16.000
Like you don't need an LLM to turn a CSV into a PDF.

s249
00:17:16.000 --> 00:17:19.919
You don't need an LLM to talk to notion tool calls if we have a CLI.

s250
00:17:19.919 --> 00:17:23.760
You definitely don't need an LLM to do deterministic SQL queries.

s251
00:17:23.760 --> 00:17:27.760
This is where people become token poor very quick.

s252
00:17:28.480 --> 00:17:34.720
And I think the last option here besides openw weight CPUs and optionality is actually governance.

s253
00:17:34.720 --> 00:17:36.880
Um there's a lot of AI governance.

s254
00:17:36.880 --> 00:17:38.880
Um one is visibility.

s255
00:17:38.880 --> 00:17:44.000
Um understanding who's using the data, understanding its maintainability and control.

s256
00:17:44.000 --> 00:17:48.720
When you have model optionality, you can offer a lot more to your customers.

s257
00:17:48.880 --> 00:17:52.880
Um here's an example of how that governance works in notion.

s258
00:17:53.440 --> 00:18:00.240
So final tips again, think about architecture, think about open weight, and build value that transcends tokens.

s259
00:18:00.240 --> 00:18:02.640
So we're going to depart Token Town.

s260
00:18:02.640 --> 00:18:04.559
I know I said welcome to Token Town.

s261
00:18:04.559 --> 00:18:09.039
We're going to spend the next 10 minutes really thinking about what to do next.

s262
00:18:09.039 --> 00:18:13.360
So I think the challenge of the next six months doesn't have to do with capabilities.

s263
00:18:13.360 --> 00:18:14.880
I think it has to do with security.

s264
00:18:14.880 --> 00:18:16.320
Let's start there.

s265
00:18:16.320 --> 00:18:18.799
There's this concept called the lethal trifecta.

s266
00:18:18.799 --> 00:18:21.440
Simon Wilson, I think, crafted this.

s267
00:18:21.440 --> 00:18:29.520
If you have access to private data, exposure to untrusted content, whether it be through ingestion, MCP, email, right?

s268
00:18:29.520 --> 00:18:34.559
And the ability to ex to communicate externally, and that can include like payloads in a web search.

s269
00:18:34.559 --> 00:18:37.200
The second you have that system, you're exposing risk.

s270
00:18:37.200 --> 00:18:42.720
And in fact, the more autonomous your system is, the more unsupervised this risk is.

s271
00:18:42.720 --> 00:18:47.919
I think that this is what builds valuable product, not just capability.

s272
00:18:48.240 --> 00:18:50.320
Same with sandboxes and computers.

s273
00:18:50.320 --> 00:19:03.600
We talked about this, but it really is something that builds better determinism in your product and also better token economics for your customers in multi-agent orchestration.

s274
00:19:03.600 --> 00:19:06.960
Understanding what agents see and do and what persists.

s275
00:19:06.960 --> 00:19:13.039
I think persistence of enterprise knowledge is something that's actually really not discussed enough.

s276
00:19:13.039 --> 00:19:17.919
It's starting to be with some recent launches, you know.

s277
00:19:17.919 --> 00:19:19.440
Oh, there is audio.

s278
00:19:19.440 --> 00:19:21.360
So don't have your workflows look like this.

s279
00:19:21.360 --> 00:19:24.640
And I think this is where most software factories are today, right?

s280
00:19:24.640 --> 00:19:30.320
It's like actually your entire engineering time just spends time babysitting the factory, right?

s281
00:19:30.320 --> 00:19:31.440
I mean, I get it.

s282
00:19:31.440 --> 00:19:33.120
Ours started off like this.

s283
00:19:33.120 --> 00:19:38.320
Agent orchestration is one of the most difficult tasks of making factories work.

s284
00:19:38.320 --> 00:19:42.320
So okay, this is me telling Tay to tell people to buy notion AI.

s285
00:19:42.320 --> 00:19:45.520
And the reason I included this slide is I am going to sell notion for a second.

s286
00:19:45.520 --> 00:19:46.320
It's my job.

s287
00:19:46.320 --> 00:19:47.200
Always be closing.

s288
00:19:47.200 --> 00:19:48.000
Always be selling.

s289
00:19:48.000 --> 00:19:48.880
Always be hiring.

s290
00:19:48.880 --> 00:19:49.840
come find me.

s291
00:19:49.840 --> 00:19:54.400
But I'm going to talk for a second about how notion does this today.

s292
00:19:54.400 --> 00:19:57.760
We already have the ability to inspect tasks.

s293
00:19:57.760 --> 00:20:03.280
And you can imagine any task that you look at um in a notion document.

s294
00:20:03.280 --> 00:20:07.200
You can have Claude actually go ahead and scope out what you need.

s295
00:20:07.200 --> 00:20:10.000
We've launched this manage agent capability today.

s296
00:20:10.000 --> 00:20:15.679
So if I go ahead to the top of this task, I can actually ask Claude agent to scope out the task.

s297
00:20:15.679 --> 00:20:17.679
Right?

s298
00:20:18.240 --> 00:20:20.720
Ideally, it's working.

s299
00:20:20.720 --> 00:20:27.200
Um, and you'll see it'll actually populate um an entire spec of what needs to be done.

s300
00:20:27.200 --> 00:20:28.640
In this example, it's not ready.

s301
00:20:28.640 --> 00:20:30.559
It's going to ask me a question.

s302
00:20:30.559 --> 00:20:32.720
Keep in mind, this isn't a markdown file.

s303
00:20:32.720 --> 00:20:34.799
This is an active document.

s304
00:20:34.799 --> 00:20:40.799
Um, let's say I don't actually know the question and I go ahead and I ask my team um what to do.

s305
00:20:40.799 --> 00:20:44.799
Imagine that you can kind of tag in your team into these systems.

s306
00:20:44.799 --> 00:20:47.840
MJ's our PM.

s307
00:20:50.240 --> 00:20:51.760
So in this example, she doesn't know.

s308
00:20:51.760 --> 00:20:55.679
Usually she does, but multi-agent orchestration is important.

s309
00:20:55.679 --> 00:21:01.520
Maybe Claude Code isn't the best at customer voice, but Decagon is, right?

s310
00:21:01.520 --> 00:21:08.000
You can ask Decagon agents, we're proud partners of them as well, to collect the right data that you need.

s311
00:21:08.000 --> 00:21:13.120
Okay, in this example, we think we know enough.

s312
00:21:13.120 --> 00:21:16.960
We're going to go ahead and actually um iterate through some of this flow.

s313
00:21:16.960 --> 00:21:18.799
I'm going to skip ahead a little bit.

s314
00:21:18.799 --> 00:21:20.640
We asked our TL what we needed.

s315
00:21:20.640 --> 00:21:21.520
He replied again.

s316
00:21:21.520 --> 00:21:24.320
It's a collaborative file, not just a markdown.

s317
00:21:24.320 --> 00:21:27.919
And we can have Claude actually go ahead and spin up the PR.

s318
00:21:27.919 --> 00:21:29.840
Hopefully, this is looking a little familiar now.

s319
00:21:29.840 --> 00:21:31.600
This is kind of the vision of software factories.

s320
00:21:31.600 --> 00:21:33.440
It's what we're trying to host.

s321
00:21:33.440 --> 00:21:36.080
Okay, Claude put up a PR.

s322
00:21:36.080 --> 00:21:37.760
Maybe that's not enough.

s323
00:21:37.760 --> 00:21:42.159
Um maybe I want to go ahead and ask Codeex what it thinks.

s324
00:21:48.000 --> 00:21:48.640
Great.

s325
00:21:48.640 --> 00:21:51.120
Found two issues.

s326
00:21:51.120 --> 00:21:54.799
You can think about this scaling in an actual factory.

s327
00:21:54.799 --> 00:22:01.440
So today in notion, you're actually able to orchestrate these agents together and you're not committing to a lab.

s328
00:22:01.440 --> 00:22:06.960
You're committing to the concept that AI is augmenting and automating what you do.

s329
00:22:08.320 --> 00:22:09.280
This is real.

s330
00:22:09.280 --> 00:22:12.000
I asked Rejieve if I could post this.

s331
00:22:12.000 --> 00:22:14.640
This is how it works today internally at notion.

s332
00:22:14.640 --> 00:22:26.799
Almost all of our polish and large feedback like this is actually coordinated um through our software factories both in terms of writing to the right teams and also having coding agents take the first step.

s333
00:22:26.799 --> 00:22:32.960
Verscell does this as well from staging to shipping to closing.

s334
00:22:33.120 --> 00:22:37.200
And we see massive ROI gains from our customers.

s335
00:22:37.200 --> 00:22:39.280
That's over three minutes saved on a given task.

s336
00:22:39.280 --> 00:22:40.880
Imagine that at scale.

s337
00:22:40.880 --> 00:22:44.240
So I think we're trying our hardest to think about the factory lens.

s338
00:22:44.240 --> 00:22:53.600
We cannot do this without optionality and we cannot do this without conviction that we understand what models are required for which tasks.

s339
00:22:53.600 --> 00:22:56.159
It's really wild out there you guys.

s340
00:22:56.159 --> 00:22:57.039
I get it.

s341
00:22:57.039 --> 00:22:58.799
The market is really young.

s342
00:22:58.799 --> 00:23:00.000
It's exceptionally opaque.

s343
00:23:00.000 --> 00:23:00.799
It's moving fast.

s344
00:23:00.799 --> 00:23:08.159
I'm super grateful for communities like AI engineer to bring us together and like talk openly about these things and how we navigate it.

s345
00:23:08.159 --> 00:23:15.120
Um I think we owe it to all of our customers to get it right and to be critical thinkers about how we navigate this together.

s346
00:23:15.120 --> 00:23:18.080
I'm chronically online fortunately.

s347
00:23:18.080 --> 00:23:22.400
Um you can always DM me on Twitter, you can email me, you can find me after this.

s348
00:23:22.400 --> 00:23:28.000
Um but thank you for yapping with me and thinking about this problem and have a good day.

s349
00:23:32.196 --> 00:23:34.196
[music]

s350
00:23:38.091 --> 00:23:40.091
[music]