WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=KHudyx5wW3U

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/KHudyx5wW3U.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.400 --> 00:00:13.120
Hi everyone.

s3
00:00:13.120 --> 00:00:25.840
So originally I thought I was going to give a very technical talk but I saw was in the leadership track which I'm not sure what it means but I was like okay I'm going to do half high level and half technical.

s4
00:00:25.840 --> 00:00:41.520
So it's more a story about what I you know what I did in January because in around December um agents kind of became really good uh you know there was a step function something happened uh I think it was opus 4.6 six

s5
00:00:41.520 --> 00:00:50.800
and that's when I realized and I work in AI where I was like okay this thing is really happening and so I took two weeks out I took two weeks away so I'm the CEO of

s6
00:00:50.800 --> 00:00:58.000
text uh you know which is 15 people company I just told my CTO I was like okay I'm just going to go away for two weeks and I'm just going to dive

s7
00:00:58.000 --> 00:01:10.240
in this thing and try to understand what we can get out of it and how good it is and so the story is you know it is the story of me scratching my own itch for two weeks and trying to figure out how

s8
00:01:10.240 --> 00:01:17.680
we can use actually use agents and what are good primitives to build agents and whether you know it already exists.

s9
00:01:17.680 --> 00:01:24.720
Um this was a really clickbait title but actually it turns out to be a good title even for this talk.

s10
00:01:24.720 --> 00:01:29.759
Um so what you can see here on the left the castle is my office.

s11
00:01:29.759 --> 00:01:30.720
Uh that's true.

s12
00:01:30.720 --> 00:01:42.400
I do rent an office in Castle and the small thing with an arrow uh that you can see is like this robot mower which kind of works unattended all day every day.

s13
00:01:42.400 --> 00:01:51.119
Uh it just does its stuff in the background without anyone having to use a remote control or think about it or anything.

s14
00:01:51.119 --> 00:01:58.240
And I kind of wanted the same thing for my morning because my mornings are always the same thing the first couple hours.

s15
00:01:58.240 --> 00:02:14.640
It's browse market news review of like linear could be Jira my CRM and also I like to walk for an about an hour in the morning and then the next hour is spent trying to process the really long voice note that you know was recorded while walking

s16
00:02:14.640 --> 00:02:26.879
and you know all I wanted was my morning briefing with my coffee and that's kind of what we've been told for a couple of years like what the future would be.

s17
00:02:26.879 --> 00:02:35.920
Um, but then when you really start working with it, even if you're not coding, all you get is a TUI today.

s18
00:02:35.920 --> 00:02:37.200
Uh, so it's amazing.

s19
00:02:37.200 --> 00:02:37.920
You can code.

s20
00:02:37.920 --> 00:02:41.599
You can actually, you know, I started doing things that were not coding in it.

s21
00:02:41.599 --> 00:02:42.640
They're great for this.

s22
00:02:42.640 --> 00:02:52.800
Uh, agents are great for this, but it's kind of the equivalent of having a robot like a tractor mower that you still have to stay on even if it's driving by itself, right?

s23
00:02:52.800 --> 00:03:01.920
It's kind of very frustrating because you have to it can do many things but you still have to be on and so of course the labs didn't stop there

s24
00:03:01.920 --> 00:03:17.760
and they came up with apps uh which I call basically SSH with vibes that's great but in this situation when that came up I was like wow that's awesome I don't have to use like a ter like SSH on my phone anymore codeex is great

s25
00:03:17.760 --> 00:03:27.200
however I noticed I just started you know I was on my walk and I was just instructing the agent to do things while I was walking and so I wasn't thinking you know very clearly anymore.

s26
00:03:27.200 --> 00:03:30.400
I just started running agents on my phone during my morning walk.

s27
00:03:30.400 --> 00:03:37.599
And this is not great because this is the equivalent of this is you're kind of midway uh you know it's not the tractor that you have to stay on.

s28
00:03:37.599 --> 00:03:40.959
It can actually do something without you being right next to it.

s29
00:03:40.959 --> 00:03:45.920
But you still have this remote control that you know you kind of have to change the trajectory every now and then.

s30
00:03:45.920 --> 00:03:47.440
Uh that's useful.

s31
00:03:47.440 --> 00:03:56.480
It's kind of absurd uh when you think about it and actually when you look at people like on their phone all the time just doing this, it's kind of absurd and it's clearly transitional.

s32
00:03:56.480 --> 00:03:59.840
Like surely we're not it's not it's not going to stop there.

s33
00:03:59.840 --> 00:04:12.400
Um and so I did a very dumb thing as a co which is I started coding uh don't tell my board and I started to build the dumbest thing that could possibly work.

s34
00:04:12.400 --> 00:04:16.479
And of course it became a really a crazy rabbit hole.

s35
00:04:16.479 --> 00:04:19.359
Uh the repo is there if you want to take a look at it.

s36
00:04:19.359 --> 00:04:20.880
The code is not amazing.

s37
00:04:20.880 --> 00:04:22.320
Uh but it works.

s38
00:04:22.320 --> 00:04:29.440
Um so the first thing is that you know I started using frameworks.

s39
00:04:29.440 --> 00:04:30.880
Uh I mean there are great frameworks.

s40
00:04:30.880 --> 00:04:36.800
I'm not going to name any frameworks because they're all good in their own way and they all have flows in their own way which is fine.

s41
00:04:36.800 --> 00:04:43.919
Uh but I spent all my time actually editing the prompt within the code and I was like this is actually not very useful.

s42
00:04:43.919 --> 00:04:54.320
So I'm like everyone here, I hate YAML u like the next guy, but I still found that this was actually a lot easier to start implementing agents without code.

s43
00:04:54.320 --> 00:04:58.160
Uh you can version it, you can diff it, uh you can review in the PR.

s44
00:04:58.160 --> 00:05:01.440
Um but it's just and it's just so easy.

s45
00:05:01.440 --> 00:05:09.840
You can just, you know, write your file, you drop it in a folder, and then it just magically appears once you have the runtime and it just magically works.

s46
00:05:09.840 --> 00:05:20.000
Um, and you know, then I needed like my market watch to run every morning while I'm, you know, while I'm walking in the fields.

s47
00:05:20.000 --> 00:05:26.240
And for that we have things that, you know, uh, have been around for a while, which is chron jobs.

s48
00:05:26.240 --> 00:05:40.080
uh and schedules specify you know when the agents need to be run and also we'll see it's very important later uh they publish uh they publish events and you know markdown

s49
00:05:40.080 --> 00:05:53.600
and chron obviously you know it's much more complicated than under the hood but the interface is this you don't write code and that's the whole product so far and honestly just mostly worked at this point I'll I'll come back on mostly

s50
00:05:53.600 --> 00:05:59.759
uh later And so this is actually a real picture of my one of my morning walks.

s51
00:05:59.759 --> 00:06:04.240
And so what I do is I record voice notes while I'm walking.

s52
00:06:04.240 --> 00:06:13.919
Uh but chron you know chron jobs I mean people would use chron jobs for this because that's what's available in codeex today but they're not ideal because they cover when

s53
00:06:13.919 --> 00:06:15.759
but this is just one point in time.

s54
00:06:15.759 --> 00:06:28.720
It doesn't cover because this happened and you know things that happened in our system like automatically when you drop the voice note now in the system it will emit an event

s55
00:06:28.720 --> 00:06:41.520
and an agent will react to that event and it's the same thing when you have a new email a new entry in the CRM I mean anything a new PR that's open a new PR that's merged etc just reacts to events it's not just a chrome job

s56
00:06:41.520 --> 00:06:50.720
and that and that means that you know agents of the Voice note processor is just you know not a chron job but here you have accepts and returns.

s57
00:06:50.720 --> 00:07:01.599
So it just declare what it accepts and what it returns as an event and here it accepts a voice note transcribes it turn it into durable notes on the right

s58
00:07:01.599 --> 00:07:04.960
and it emits a new event and for that it uses structured outputs.

s59
00:07:04.960 --> 00:07:06.479
We'll come back to this.

s60
00:07:06.479 --> 00:07:15.599
And you know, now we finally have the future we're promised because that voice note agent emits voice note processed.

s61
00:07:15.599 --> 00:07:20.800
And then I have my daily brief agent that actually will take the output of the chrome job.

s62
00:07:20.800 --> 00:07:27.759
We'll take the output of the voice note agents and we create my daily brief which is posted as a slack message.

s63
00:07:27.759 --> 00:07:40.800
So the slack message.post post event is actually uh is actually like um a process actually subscribes to this and emits uh and sends a slack message to me.

s64
00:07:40.800 --> 00:07:42.880
It's actually this is a real this is a real thing.

s65
00:07:42.880 --> 00:07:43.360
It's working.

s66
00:07:43.360 --> 00:07:53.680
I can show you after on my phone and you know there are frameworks that are going to sell you the fact that you need graphs for this in code.

s67
00:07:53.680 --> 00:07:56.479
Uh you do not need graph uh in this case.

s68
00:07:56.479 --> 00:07:58.240
All you need is events.

s69
00:07:58.240 --> 00:07:59.919
You have no edges to maintain.

s70
00:07:59.919 --> 00:08:02.960
Agents sub simply subscribe to events.

s71
00:08:02.960 --> 00:08:06.080
Anyone can come in and edit this.

s72
00:08:06.080 --> 00:08:07.120
You don't need to.

s73
00:08:07.120 --> 00:08:08.560
Yeah, you don't need to know how to code.

s74
00:08:08.560 --> 00:08:11.039
You just need to know what events exist in the system.

s75
00:08:11.039 --> 00:08:12.800
Fun and fan out are free.

s76
00:08:12.800 --> 00:08:19.680
No code and it's just drop a file and the topology emerges whatever the log says happened.

s77
00:08:19.680 --> 00:08:24.879
And you know then of course I I tried to run it.

s78
00:08:24.879 --> 00:08:34.880
So the first version took about I mean you know I cheated I cheated I used uh I used codeex and it took about like a day to write like the first thing

s79
00:08:34.880 --> 00:08:47.839
uh out of my week but of course I tried it and it broke uh so these are real examples actually the dates know but it's real examples it's like the first day delay brief was posted to Slack twice

s80
00:08:47.839 --> 00:08:54.959
um on Wednesday one of my voice notes completely vanished and then you know towards the end of the week.

s81
00:08:54.959 --> 00:09:08.399
I I kind of like played with the prompts all week and the market brief was garbage, but I didn't version uh I didn't version my changes and I couldn't remember actually what I changed in the prompt that made the thing

s82
00:09:08.399 --> 00:09:10.000
completely useless.

s83
00:09:10.000 --> 00:09:19.360
Now, if there are distributed or existed engineers in the room, you probably know this shopping list already.

s84
00:09:19.360 --> 00:09:33.519
uh there is nothing new under the sun and you know each failure so each of these failure modes that you found actually led to building one piece of what turned out to be a runtime.

s85
00:09:33.519 --> 00:09:36.880
So the lost note actually turned into a log.

s86
00:09:36.880 --> 00:09:44.080
I just wanted everything to be saved forever so that I could go back to it and look into uh into what happened.

s87
00:09:44.080 --> 00:09:44.959
the duplicates.

s88
00:09:44.959 --> 00:09:49.600
It was because I was not following, you know, it did several attempts and I was not following them.

s89
00:09:49.600 --> 00:09:50.880
I didn't have a proper queue.

s90
00:09:50.880 --> 00:09:58.160
I wasn't counting the attempts, etc., etc. And then probably the most interesting part is the last prompt.

s91
00:09:58.160 --> 00:10:08.560
I got into a really deep rabbit hole in there and I just ended up uh building a content like a content addressed system for this uh content addressed system.

s92
00:10:08.560 --> 00:10:13.120
You can think of git uh you can think of nyx and any other build system.

s93
00:10:13.120 --> 00:10:17.920
And you know that was and I didn't do this because I wanted to design a runtime.

s94
00:10:17.920 --> 00:10:24.720
I mean by that point I still just wanted my agents to work and I also like the distraction.

s95
00:10:24.720 --> 00:10:31.120
Uh and I just paid off debt as it appeared like errors as they appeared.

s96
00:10:31.120 --> 00:10:35.120
I hope my board won't see this talk.

s97
00:10:35.120 --> 00:10:38.800
Uh so the log the log is the systems memory.

s98
00:10:38.800 --> 00:10:42.800
Uh nothing is lost and everything is observed.

s99
00:10:42.800 --> 00:10:53.120
uh you can you know you only have one append only events table on the left it's a real common line uh in like command za events and you get all the events

s100
00:10:53.120 --> 00:11:08.240
they are causally linked as well like you know which event triggered which event which happens to be super useful when you're debugging and you know even with three four agents you start having like major debugging headaches so that was super

s101
00:11:08.240 --> 00:11:16.480
super helpful Um and everything is querable which again for debugging.

s102
00:11:16.480 --> 00:11:31.519
The second thing is you know okay we have a log so we can trace back things etc. But it's still really hard to know what went into the like what went to the model what prompt was sent to the model again

s103
00:11:31.519 --> 00:11:35.120
because what you see when you're using codecs is kind of a lie.

s104
00:11:35.120 --> 00:11:45.600
like you kind of have like a live chat session with the model and so you tend to think that oh that's what the model saw and you know that's exactly so I can understand what happened the truth is

s105
00:11:45.600 --> 00:11:54.160
that's not exactly what the model saw um there are many reasons for that one is I mean compaction obviously is a big part is a big thing but also

s106
00:11:54.160 --> 00:12:06.160
you know there are just quirks also you know openai doesn't share or enthropic for that matter don't share the thinking with you the thinking traces so you have no idea I mean kind of have an idea of what went in but not completely either.

s107
00:12:06.160 --> 00:12:08.160
And so you need something different.

s108
00:12:08.160 --> 00:12:19.600
Uh you need something different and that was the big rabbit hole which is trying to find a way or build a system where you can trace back to what the model saw internally.

s109
00:12:19.600 --> 00:12:24.639
And so what I did was basically built I mean nothing new.

s110
00:12:24.639 --> 00:12:26.639
This is basically how build system works.

s111
00:12:26.639 --> 00:12:28.880
Uh so you have different parts for a prompt.

s112
00:12:28.880 --> 00:12:30.639
You have your system prompt.

s113
00:12:30.639 --> 00:12:33.519
You have a description of your first skill of a second skill.

s114
00:12:33.519 --> 00:12:35.760
Then you have the description of your tools.

s115
00:12:35.760 --> 00:12:39.200
You have your user message which is the question to the model.

s116
00:12:39.200 --> 00:12:46.880
Each one of those is stored and addressed and you know stored somewhere as a identifier which is a hash.

s117
00:12:46.880 --> 00:12:58.480
And so when we build a prompt instead of building a piece of I mean before rendering the text we actually represent the prompt as a list of these u of these hashes.

s118
00:12:58.480 --> 00:13:08.959
And so what that means is that down the line when I have a model answer, which by the way is also stored in the same way, we can trace back to the prompt very easily.

s119
00:13:08.959 --> 00:13:15.360
And then from that prompt, we can know exactly what went into the model's context, which actually matters a lot.

s120
00:13:15.360 --> 00:13:19.440
I mean, it matters a lot for debugging, but it also matters.

s121
00:13:19.440 --> 00:13:21.600
I mean, it makes compaction a lot easier.

s122
00:13:21.600 --> 00:13:23.360
You're just manipulating a graph, right?

s123
00:13:23.360 --> 00:13:25.040
You're not manipulating strings.

s124
00:13:25.040 --> 00:13:26.560
It's just a lot easier.

s125
00:13:26.560 --> 00:13:30.560
And it makes KV cache management a lot easier as well indirectly.

s126
00:13:30.560 --> 00:13:41.360
And but I think that when you know I guess probably the main advantage that's when you use that scale is really auditability.

s127
00:13:41.360 --> 00:13:47.440
It's like you can know exactly what happened with that agent and why it returned what it returned.

s128
00:13:47.440 --> 00:13:52.720
And so you know I'm just going to go pretty pretty quickly over this.

s129
00:13:52.720 --> 00:14:04.560
Uh what you get once you have this graph is you get diffs like you can say okay what changed between these two runs like which com which components changed was it just my message

s130
00:14:04.560 --> 00:14:12.720
did I like give the model a different skill did I give it a different tool so you can just yeah you can just run this function and it will show you

s131
00:14:12.720 --> 00:14:16.749
you know the difference between uh the runs so here you have you

s132
00:14:16.749 --> 00:14:16.880
[snorts]

s133
00:14:16.880 --> 00:14:26.800
know three components that were identical there's one which is you know the user message changed And then you had all these other messages that were actually you know that were continuing.

s134
00:14:26.800 --> 00:14:29.360
It's continuation of a single session.

s135
00:14:29.360 --> 00:14:32.560
Uh then you have another thing for free which is replace.

s136
00:14:32.560 --> 00:14:43.360
Uh replays turned out to be really useful for me because after a while I mean when I saw the cost ramp up like the thing when you have observability

s137
00:14:43.360 --> 00:14:46.560
is that you do realize that cost increase very quickly.

s138
00:14:46.560 --> 00:14:51.839
I wanted to try with open source models and so I wanted to rebuild old request for to eval.

s139
00:14:51.839 --> 00:14:57.199
and see if I got the same thing out, if I got something satisfactory, if I need to change anything.

s140
00:14:57.199 --> 00:15:10.480
And turns out that once you have uh you know this content addressing system, you can rebuild the request from the graph and you can just replay it exactly the same and you can you know resend, you can use a different model.

s141
00:15:10.480 --> 00:15:18.720
Uh you can use a different request if you want, you can uh you can change it and so yeah, you get actually a lot of things I mean for free.

s142
00:15:18.720 --> 00:15:20.399
You need to implement a thing.

s143
00:15:20.399 --> 00:15:30.240
Um and so this is kind of different um from what you find I mean what I found when I started doing this it might be different today because it was a couple of months ago

s144
00:15:30.240 --> 00:15:49.279
is that out there you had a lot of libraries so it's just frameworks and frameworks just call code uh your agents leave inside their abstractions um and I don't like analogies with you know operating system okay everyone has used that analogy but okay let's say a kernel like runs processes

s145
00:15:49.279 --> 00:15:51.040
and your agent kind of is a process.

s146
00:15:51.040 --> 00:15:53.040
It doesn't matter what it does actually.

s147
00:15:53.040 --> 00:15:56.639
Uh but the system can schedule it is built to isolate it.

s148
00:15:56.639 --> 00:16:00.639
It can isolate it and journals it with the log and the agent definition.

s149
00:16:00.639 --> 00:16:07.920
So the markdown is user land like you don't need to use it with that system if you don't want to actually have a front end that doesn't use this markdown

s150
00:16:07.920 --> 00:16:10.240
uh this markdown format at all.

s151
00:16:10.240 --> 00:16:27.120
And okay, here's a very important point and you know that's kind of a takeaway and it's also what justifies me working on this because disclaimer structured outputs is our specialty and we've been working on this for three years and it just ended up being a big dog fooding project and the reason why I did this

s152
00:16:27.120 --> 00:16:32.079
at the beginning was not because I absolutely wanted to use our software.

s153
00:16:32.079 --> 00:16:44.800
I didn't necessarily want to you know forklP to other software etc. It's just because Infropic was terrible at structured outputs and so like 20% of my events were wrong and were rejected by the system.

s154
00:16:44.800 --> 00:16:46.560
So that's why I ended up doing this.

s155
00:16:46.560 --> 00:16:52.880
And the goal, you know, the job of the kernel is actually to make bad actions impossible, not just unlikely.

s156
00:16:52.880 --> 00:16:56.720
And so you have this two boundaries with between agents and the external world.

s157
00:16:56.720 --> 00:16:59.199
The first one is type tool calls.

s158
00:16:59.199 --> 00:17:08.319
The two tool calls uh you don't want, you know, you don't want to call tools that don't exist, etc., etc. and also the boundary with other agents which is typed events

s159
00:17:08.319 --> 00:17:18.720
and this is non-negotiable I found like you can get a lot of errors just from this uh I wrote a really long blog post about this uh it's if you follow the QR code you'll find it

s160
00:17:18.720 --> 00:17:34.960
and yeah and the result of that is I actually deployed it uh within the company after I built this and now today after a month of deploying it we have 20 agents on the left that are not just contributed by technical people by the way which is kind of what markdown

s161
00:17:34.960 --> 00:17:46.160
uh what Maron gives you and then on the right is you know we deploy it's called the internet there's the briefs there's a bunch of I mean there's a bunch of things as you can as you can see

s162
00:17:46.160 --> 00:18:03.600
kind of like a few you know as a conclusion a few lessons uh the first one is that well executed background agents are really magical uh they feel like this you know robot mower that I had at the beginning is I really just sit down and I come back and I have this morning brief that is probably

s163
00:18:03.600 --> 00:18:12.400
even better than what I would have had just doing it manually and it just appears in my inbox every day and processes my you know random thoughts.

s164
00:18:12.400 --> 00:18:17.840
Uh the difficulties that you meet doing this kind of thing is just good old engineering problems.

s165
00:18:17.840 --> 00:18:22.240
I mean there's really nothing new under the sun when it comes to orchestrating these things.

s166
00:18:22.240 --> 00:18:25.039
It's just good old software orchestration.

s167
00:18:25.039 --> 00:18:26.880
Uh open source models are there.

s168
00:18:26.880 --> 00:18:27.919
Uh they're good enough.

s169
00:18:27.919 --> 00:18:31.919
I replaced so I don't have any third party APIs anymore.

s170
00:18:31.919 --> 00:18:37.679
Now I just use open source models and even on my laptop I use a local model.

s171
00:18:37.679 --> 00:18:40.400
Uh so it's good enough for what I do with it.

s172
00:18:40.400 --> 00:18:44.000
For coding I don't know but for what I do with this it's good enough.

s173
00:18:44.000 --> 00:18:46.320
The infro category is definitely unsettled.

s174
00:18:46.320 --> 00:18:50.000
Uh I tried a few things before I started building myself.

s175
00:18:50.000 --> 00:18:55.200
And I would advise that today like definitely start building before you buy.

s176
00:18:55.200 --> 00:19:05.440
Uh so if you're a small company, if you're a tech CEO, it's kind of an advantage because you can just do this without, you know, tasking engineers to do this and get them off track.

s177
00:19:05.440 --> 00:19:13.200
But I would definitely try to build before I buy just to know exactly what I need and you know the limitations of what exist.

s178
00:19:13.200 --> 00:19:21.440
Uh also I will say that to people building uh frameworks for this is please eat your own dog food.

s179
00:19:21.440 --> 00:19:28.160
Sometimes it's pretty clear that people are building you know agent orchestration frameworks etc but not eating their own dog food.

s180
00:19:28.160 --> 00:19:37.039
So please do and the other thing is I'm really glad I took this two weeks off to play with the field because that completely changed I mean that changed the trajectory of the company.

s181
00:19:37.039 --> 00:19:46.320
I know we're an AI company we should be in it etc but you know business is such that you're always thinking about the next thing the next thing the next thing and it's the same everywhere.

s182
00:19:46.320 --> 00:19:55.360
But what I'm urging you to do is to stop and actually immerse yourself in this and try to see how useful it can be for your company.

s183
00:19:55.360 --> 00:19:57.600
Uh so you can steal the code.

s184
00:19:57.600 --> 00:20:01.280
Uh it's not a product that we sell and we don't intend to sell this.

s185
00:20:01.280 --> 00:20:03.200
Uh you can read our blog as well.

s186
00:20:03.200 --> 00:20:07.760
Uh so I haven't explained uh this yet but I will publish something about it.

s187
00:20:07.760 --> 00:20:11.360
And thank you for your attention.

s188
00:20:24.864 --> 00:20:26.864
[music]