Agent Frameworks Considered Harmful — Rémi Louf, .txt

AI Engineer · 20 min · 188 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hi everyone.
00:13So originally I thought I was going to give a very technical talk but I saw was in the leadership track which I'm not sure what it means but I was like okay I'm going to do half high level and half technical.
00:25So it's more a story about what I you know what I did in January because in around December um agents kind of became really good uh you know there was a step function something happened uh I think it was opus 4.6 six
00:41and that's when I realized and I work in AI where I was like okay this thing is really happening and so I took two weeks out I took two weeks away so I'm the CEO of
00:50text uh you know which is 15 people company I just told my CTO I was like okay I'm just going to go away for two weeks and I'm just going to dive
00:58in this thing and try to understand what we can get out of it and how good it is and so the story is you know it is the story of me scratching my own itch for two weeks and trying to figure out how
01:10we can use actually use agents and what are good primitives to build agents and whether you know it already exists.
01:17Um this was a really clickbait title but actually it turns out to be a good title even for this talk.
01:24Um so what you can see here on the left the castle is my office.
01:29Uh that's true.
01:30I do rent an office in Castle and the small thing with an arrow uh that you can see is like this robot mower which kind of works unattended all day every day.
01:42Uh it just does its stuff in the background without anyone having to use a remote control or think about it or anything.
01:51And I kind of wanted the same thing for my morning because my mornings are always the same thing the first couple hours.
01:58It's browse market news review of like linear could be Jira my CRM and also I like to walk for an about an hour in the morning and then the next hour is spent trying to process the really long voice note that you know was recorded while walking
02:14and you know all I wanted was my morning briefing with my coffee and that's kind of what we've been told for a couple of years like what the future would be.
02:26Um, but then when you really start working with it, even if you're not coding, all you get is a TUI today.
02:35Uh, so it's amazing.
02:37You can code.
02:37You can actually, you know, I started doing things that were not coding in it.
02:41They're great for this.
02:42Uh, agents are great for this, but it's kind of the equivalent of having a robot like a tractor mower that you still have to stay on even if it's driving by itself, right?
02:52It's kind of very frustrating because you have to it can do many things but you still have to be on and so of course the labs didn't stop there
03:01and they came up with apps uh which I call basically SSH with vibes that's great but in this situation when that came up I was like wow that's awesome I don't have to use like a ter like SSH on my phone anymore codeex is great
03:17however I noticed I just started you know I was on my walk and I was just instructing the agent to do things while I was walking and so I wasn't thinking you know very clearly anymore.
03:27I just started running agents on my phone during my morning walk.
03:30And this is not great because this is the equivalent of this is you're kind of midway uh you know it's not the tractor that you have to stay on.
03:37It can actually do something without you being right next to it.
03:40But you still have this remote control that you know you kind of have to change the trajectory every now and then.
03:45Uh that's useful.
03:47It's kind of absurd uh when you think about it and actually when you look at people like on their phone all the time just doing this, it's kind of absurd and it's clearly transitional.
03:56Like surely we're not it's not it's not going to stop there.
03:59Um and so I did a very dumb thing as a co which is I started coding uh don't tell my board and I started to build the dumbest thing that could possibly work.
04:12And of course it became a really a crazy rabbit hole.
04:16Uh the repo is there if you want to take a look at it.
04:19The code is not amazing.
04:20Uh but it works.
04:22Um so the first thing is that you know I started using frameworks.
04:29Uh I mean there are great frameworks.
04:30I'm not going to name any frameworks because they're all good in their own way and they all have flows in their own way which is fine.
04:36Uh but I spent all my time actually editing the prompt within the code and I was like this is actually not very useful.
04:43So I'm like everyone here, I hate YAML u like the next guy, but I still found that this was actually a lot easier to start implementing agents without code.
04:54Uh you can version it, you can diff it, uh you can review in the PR.
04:58Um but it's just and it's just so easy.
05:01You can just, you know, write your file, you drop it in a folder, and then it just magically appears once you have the runtime and it just magically works.
05:09Um, and you know, then I needed like my market watch to run every morning while I'm, you know, while I'm walking in the fields.
05:20And for that we have things that, you know, uh, have been around for a while, which is chron jobs.
05:26uh and schedules specify you know when the agents need to be run and also we'll see it's very important later uh they publish uh they publish events and you know markdown
05:40and chron obviously you know it's much more complicated than under the hood but the interface is this you don't write code and that's the whole product so far and honestly just mostly worked at this point I'll I'll come back on mostly
05:53uh later And so this is actually a real picture of my one of my morning walks.
05:59And so what I do is I record voice notes while I'm walking.
06:04Uh but chron you know chron jobs I mean people would use chron jobs for this because that's what's available in codeex today but they're not ideal because they cover when
06:13but this is just one point in time.
06:15It doesn't cover because this happened and you know things that happened in our system like automatically when you drop the voice note now in the system it will emit an event
06:28and an agent will react to that event and it's the same thing when you have a new email a new entry in the CRM I mean anything a new PR that's open a new PR that's merged etc just reacts to events it's not just a chrome job
06:41and that and that means that you know agents of the Voice note processor is just you know not a chron job but here you have accepts and returns.
06:50So it just declare what it accepts and what it returns as an event and here it accepts a voice note transcribes it turn it into durable notes on the right
07:01and it emits a new event and for that it uses structured outputs.
07:04We'll come back to this.
07:06And you know, now we finally have the future we're promised because that voice note agent emits voice note processed.
07:15And then I have my daily brief agent that actually will take the output of the chrome job.
07:20We'll take the output of the voice note agents and we create my daily brief which is posted as a slack message.
07:27So the slack message.post post event is actually uh is actually like um a process actually subscribes to this and emits uh and sends a slack message to me.
07:40It's actually this is a real this is a real thing.
07:42It's working.
07:43I can show you after on my phone and you know there are frameworks that are going to sell you the fact that you need graphs for this in code.
07:53Uh you do not need graph uh in this case.
07:56All you need is events.
07:58You have no edges to maintain.
07:59Agents sub simply subscribe to events.
08:02Anyone can come in and edit this.
08:06You don't need to.
08:07Yeah, you don't need to know how to code.
08:08You just need to know what events exist in the system.
08:11Fun and fan out are free.
08:12No code and it's just drop a file and the topology emerges whatever the log says happened.
08:19And you know then of course I I tried to run it.
08:24So the first version took about I mean you know I cheated I cheated I used uh I used codeex and it took about like a day to write like the first thing
08:34uh out of my week but of course I tried it and it broke uh so these are real examples actually the dates know but it's real examples it's like the first day delay brief was posted to Slack twice
08:47um on Wednesday one of my voice notes completely vanished and then you know towards the end of the week.
08:54I I kind of like played with the prompts all week and the market brief was garbage, but I didn't version uh I didn't version my changes and I couldn't remember actually what I changed in the prompt that made the thing
09:08completely useless.
09:10Now, if there are distributed or existed engineers in the room, you probably know this shopping list already.
09:19uh there is nothing new under the sun and you know each failure so each of these failure modes that you found actually led to building one piece of what turned out to be a runtime.
09:33So the lost note actually turned into a log.
09:36I just wanted everything to be saved forever so that I could go back to it and look into uh into what happened.
09:44the duplicates.
09:44It was because I was not following, you know, it did several attempts and I was not following them.
09:49I didn't have a proper queue.
09:50I wasn't counting the attempts, etc., etc. And then probably the most interesting part is the last prompt.
09:58I got into a really deep rabbit hole in there and I just ended up uh building a content like a content addressed system for this uh content addressed system.
10:08You can think of git uh you can think of nyx and any other build system.
10:13And you know that was and I didn't do this because I wanted to design a runtime.
10:17I mean by that point I still just wanted my agents to work and I also like the distraction.
10:24Uh and I just paid off debt as it appeared like errors as they appeared.
10:31I hope my board won't see this talk.
10:35Uh so the log the log is the systems memory.
10:38Uh nothing is lost and everything is observed.
10:42uh you can you know you only have one append only events table on the left it's a real common line uh in like command za events and you get all the events
10:53they are causally linked as well like you know which event triggered which event which happens to be super useful when you're debugging and you know even with three four agents you start having like major debugging headaches so that was super
11:08super helpful Um and everything is querable which again for debugging.
11:16The second thing is you know okay we have a log so we can trace back things etc. But it's still really hard to know what went into the like what went to the model what prompt was sent to the model again
11:31because what you see when you're using codecs is kind of a lie.
11:35like you kind of have like a live chat session with the model and so you tend to think that oh that's what the model saw and you know that's exactly so I can understand what happened the truth is
11:45that's not exactly what the model saw um there are many reasons for that one is I mean compaction obviously is a big part is a big thing but also
11:54you know there are just quirks also you know openai doesn't share or enthropic for that matter don't share the thinking with you the thinking traces so you have no idea I mean kind of have an idea of what went in but not completely either.
12:06And so you need something different.
12:08Uh you need something different and that was the big rabbit hole which is trying to find a way or build a system where you can trace back to what the model saw internally.
12:19And so what I did was basically built I mean nothing new.
12:24This is basically how build system works.
12:26Uh so you have different parts for a prompt.
12:28You have your system prompt.
12:30You have a description of your first skill of a second skill.
12:33Then you have the description of your tools.
12:35You have your user message which is the question to the model.
12:39Each one of those is stored and addressed and you know stored somewhere as a identifier which is a hash.
12:46And so when we build a prompt instead of building a piece of I mean before rendering the text we actually represent the prompt as a list of these u of these hashes.
12:58And so what that means is that down the line when I have a model answer, which by the way is also stored in the same way, we can trace back to the prompt very easily.
13:08And then from that prompt, we can know exactly what went into the model's context, which actually matters a lot.
13:15I mean, it matters a lot for debugging, but it also matters.
13:19I mean, it makes compaction a lot easier.
13:21You're just manipulating a graph, right?
13:23You're not manipulating strings.
13:25It's just a lot easier.
13:26And it makes KV cache management a lot easier as well indirectly.
13:30And but I think that when you know I guess probably the main advantage that's when you use that scale is really auditability.
13:41It's like you can know exactly what happened with that agent and why it returned what it returned.
13:47And so you know I'm just going to go pretty pretty quickly over this.
13:52Uh what you get once you have this graph is you get diffs like you can say okay what changed between these two runs like which com which components changed was it just my message
14:04did I like give the model a different skill did I give it a different tool so you can just yeah you can just run this function and it will show you
14:12you know the difference between uh the runs so here you have you
14:16[snorts]
14:16know three components that were identical there's one which is you know the user message changed And then you had all these other messages that were actually you know that were continuing.
14:26It's continuation of a single session.
14:29Uh then you have another thing for free which is replace.
14:32Uh replays turned out to be really useful for me because after a while I mean when I saw the cost ramp up like the thing when you have observability
14:43is that you do realize that cost increase very quickly.
14:46I wanted to try with open source models and so I wanted to rebuild old request for to eval.
14:51and see if I got the same thing out, if I got something satisfactory, if I need to change anything.
14:57And turns out that once you have uh you know this content addressing system, you can rebuild the request from the graph and you can just replay it exactly the same and you can you know resend, you can use a different model.
15:10Uh you can use a different request if you want, you can uh you can change it and so yeah, you get actually a lot of things I mean for free.
15:18You need to implement a thing.
15:20Um and so this is kind of different um from what you find I mean what I found when I started doing this it might be different today because it was a couple of months ago
15:30is that out there you had a lot of libraries so it's just frameworks and frameworks just call code uh your agents leave inside their abstractions um and I don't like analogies with you know operating system okay everyone has used that analogy but okay let's say a kernel like runs processes
15:49and your agent kind of is a process.
15:51It doesn't matter what it does actually.
15:53Uh but the system can schedule it is built to isolate it.
15:56It can isolate it and journals it with the log and the agent definition.
16:00So the markdown is user land like you don't need to use it with that system if you don't want to actually have a front end that doesn't use this markdown
16:07uh this markdown format at all.
16:10And okay, here's a very important point and you know that's kind of a takeaway and it's also what justifies me working on this because disclaimer structured outputs is our specialty and we've been working on this for three years and it just ended up being a big dog fooding project and the reason why I did this
16:27at the beginning was not because I absolutely wanted to use our software.
16:32I didn't necessarily want to you know forklP to other software etc. It's just because Infropic was terrible at structured outputs and so like 20% of my events were wrong and were rejected by the system.
16:44So that's why I ended up doing this.
16:46And the goal, you know, the job of the kernel is actually to make bad actions impossible, not just unlikely.
16:52And so you have this two boundaries with between agents and the external world.
16:56The first one is type tool calls.
16:59The two tool calls uh you don't want, you know, you don't want to call tools that don't exist, etc., etc. and also the boundary with other agents which is typed events
17:08and this is non-negotiable I found like you can get a lot of errors just from this uh I wrote a really long blog post about this uh it's if you follow the QR code you'll find it
17:18and yeah and the result of that is I actually deployed it uh within the company after I built this and now today after a month of deploying it we have 20 agents on the left that are not just contributed by technical people by the way which is kind of what markdown
17:34uh what Maron gives you and then on the right is you know we deploy it's called the internet there's the briefs there's a bunch of I mean there's a bunch of things as you can as you can see
17:46kind of like a few you know as a conclusion a few lessons uh the first one is that well executed background agents are really magical uh they feel like this you know robot mower that I had at the beginning is I really just sit down and I come back and I have this morning brief that is probably
18:03even better than what I would have had just doing it manually and it just appears in my inbox every day and processes my you know random thoughts.
18:12Uh the difficulties that you meet doing this kind of thing is just good old engineering problems.
18:17I mean there's really nothing new under the sun when it comes to orchestrating these things.
18:22It's just good old software orchestration.
18:25Uh open source models are there.
18:26Uh they're good enough.
18:27I replaced so I don't have any third party APIs anymore.
18:31Now I just use open source models and even on my laptop I use a local model.
18:37Uh so it's good enough for what I do with it.
18:40For coding I don't know but for what I do with this it's good enough.
18:44The infro category is definitely unsettled.
18:46Uh I tried a few things before I started building myself.
18:50And I would advise that today like definitely start building before you buy.
18:55Uh so if you're a small company, if you're a tech CEO, it's kind of an advantage because you can just do this without, you know, tasking engineers to do this and get them off track.
19:05But I would definitely try to build before I buy just to know exactly what I need and you know the limitations of what exist.
19:13Uh also I will say that to people building uh frameworks for this is please eat your own dog food.
19:21Sometimes it's pretty clear that people are building you know agent orchestration frameworks etc but not eating their own dog food.
19:28So please do and the other thing is I'm really glad I took this two weeks off to play with the field because that completely changed I mean that changed the trajectory of the company.
19:37I know we're an AI company we should be in it etc but you know business is such that you're always thinking about the next thing the next thing the next thing and it's the same everywhere.
19:46But what I'm urging you to do is to stop and actually immerse yourself in this and try to see how useful it can be for your company.
19:55Uh so you can steal the code.
19:57Uh it's not a product that we sell and we don't intend to sell this.
20:01Uh you can read our blog as well.
20:03Uh so I haven't explained uh this yet but I will publish something about it.
20:07And thank you for your attention.
20:24[music]