The Loop Is the Product — Roland Gavrilescu, Introspection

AI Engineer · 18 min · 173 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Hello everyone.
00:14How's everyone doing?
00:17Are you guys ready for some more loops?
00:21Yeah.
00:22My name is Roland.
00:23My co-founder and I were in this mythical place called XAI working hard on agent infra and we realized there's something new that has to be done in a standalone way.
00:35So we left a few months ago to really figure out okay what's the next stage of how we should deploy these always on longunning horizon tasks.
00:45Um and I'm happy to announce we have a few findings that we would like to present you.
00:51Um, and this talk it's all about um, how you should productize these ideas in ways that can scale with your customers.
01:00Um, you've heard a lot about auto research.
01:04Um, we think there's a blueprint for 2026 and beyond on how you should think about auto research and it really comes down to three ideas.
01:14Let's go through the first one.
01:16The loop is the product.
01:21We're all familiar with this.
01:23We've started with everything goes down to RL chief for models and how you should train the model to become better and better reasoning.
01:31We then quickly moved to harnesses and how the model is a commodity and it's all about the harness.
01:37And now we're talking about loops and how you should build these loops uh and not touch code anymore.
01:43But what does it really mean and why is everyone saying that?
01:47Do you guys remember clawbot?
01:50That was the original um original name of what is now now now known as open claw.
01:56And this guy AJ built the first loop around cloudbot.
02:03What he did was to find a way to talk to dealers and talk to Reddit users to get bigger discounts on a car.
02:12He followed these four steps.
02:14Um and is really open call the one that did it.
02:17Go on Reddit, find prices, find inventory, talk to the dealers, put dealers head-to-head, and try to figure out how to make them outbid each other, have a verifiable way to know when the price is right,
02:35and then lock in, get the car, and it worked.
02:39Um, probably this was when all the Mac minis were uh selling off the shelves, but this was the first real example of loop is the product and something that
02:51probably should be a startup at this point.
02:53Um, but we've seen how this became a recipe for everyone to build loops.
02:59But let's take a step back.
03:01Why are we here?
03:02Um, we really think models have been trained with this loop in mind.
03:07And it comes from this idea of udala loops.
03:10It's a terminology coined back in 1970s by the US air force and is the idea of these um jet fighters how to react in fast-paced environments.
03:24If you think of models calling tools and taking observations, it's it's what we've been trained on uh as humans but also as as agents.
03:33Now, now what happens when you put strong signals and verifiable work uh at the other ends?
03:41You get to these workers or cloud code agents.
03:45Um and and what matters here is the quality of the signal determines the uh success rate of the loop and the uh quality of the verifi verifier um um
03:59is able to calibrate if that success is actually correct or not.
04:04But there's another loop here.
04:06Um what happens when you take that and feed it back into the signal?
04:10And this is what looping around is all about is how do you generate these artifacts at the end of the first loop to then run a second loop on and have a way to continuously improve.
04:24And this goes to my second point.
04:26System dillation is the mode and is really the ability to understand what went well and wrong in the first loop and know how to process that in the second one.
04:40So how do we tune these AI systems?
04:42Each loop generates useful information around harnesses, profiles, evals, models, resources, tools, and the environment.
04:52What you really want is to have a way to keep this portable, to have a way to version this and to evolve it over time.
05:01If you think about data recipes in research, this is how RL started to work really well.
05:08you understood the recipes and how to continuously change the recipe to combat some of the behaviors that may happen around hallucinations around reward hacking and then you get to a stack which is your final data recipe.
05:22We don't have that for harnesses.
05:24We don't have that for like AI systems in the general term.
05:28So we thought there's space for something like that.
05:30something that contains the evils and contains the tweaks and the human judgment and all these things that are not predetermined at the beginning but they're defined as you learn more
05:41about your agent acting in in in the environment.
05:47We think recipes can be applied to this and we should use the same name.
05:51So an agent recipe is really something that enables you to create reproducible frontier AI systems.
05:58It's something that allows you to have a mode that keeps getting better over time, which is not tied to any platform or any provider.
06:07It's something that you control lives in your company and is agnostic to the models and providers you use.
06:16And loops should focus on this.
06:17Loops should be the way you distill these systems into recipes.
06:21Failure patterns should become judges and evals.
06:25Repeated behavior should become skills and prompts.
06:27user frustration, extensions and memories to your harness and so on.
06:31You we're all familiar with this, but we didn't have the the the right like terminology of how we should think about it and how we should define it.
06:39And we think recipes is a way to put everything together into a git repo and treat it as your ongoing um strategy for for uh building these self-improving systems.
06:55So we are introspection but you can think of introspection as the way you generate these recipes.
07:00So they're recipes for introspecting on your on your system.
07:05We wanted to build something that is portable and provider agnostic.
07:09So we built our um approach to recipes on the pi harness and on harbor for evals.
07:17We baked it into uh git repos so uh everything could be versioned and agents could have a way to continuously track how this change and why and is meant to be owned by you but managed by your agents.
07:31And this is how products should really be built going forward.
07:34It's something that treats the owner as the um almost like the the the higher taste um personality in the room.
07:44But agents should try to calibrate themselves to to the taste of the of the maker.
07:49So we think recipes should be basically encoding the taste of the makers into how you build these agents.
07:56And if I want to use someone else's recipe, I should be able to also bring that taste.
08:01It's not just the harness, it's not just the model, is how did you arrive at this particular recipe and why?
08:07And that's kind of like what uh what is behind uh reproducible um uh products and services around agents.
08:16Um we have an early release of recipes is called pi.
08:21Recipes.
08:22It's very similar to what skills uh used to be in 2025 but is going a step forward.
08:28And this is what do I need to have a frontier agent is everything about how do I codify paste into evals?
08:35How do I run evals?
08:36How do we have the loops to continuously improve those evals over time?
08:40How do we process signals and know what are the right signals to to use?
08:45Um what are the right tools to work with certain models?
08:48How do I have different profiles of the harness to work with different models?
08:52Um and everything in between.
08:54So have a look at what we've been building here.
08:57It's still early uh but hopefully it's useful enough for you guys to to get going.
09:03And we feel this is going to grow into something that um really allows you to to use uh different um almost like different the to to be able to use the taste of of different makers
09:14as recipes for your agent.
09:18And finally, the last point is valued work per watt.
09:21And why is this the score to really optimize for?
09:23Think of how um cursor and cognition went from building the best product to then building the best evals for the product and finally building the best models based on the previous two artifacts.
09:38We think this is like the recipe for everything going forward.
09:41Um code was the first domain where this um was successful.
09:45Um everything beyond customer support, legal research um everything is going to come down to this idea.
09:52How much value am I getting per watt?
09:54Um, how do I measure the value is the first step and how do I know I'm getting a good deal on that value is the second.
10:04And maybe this makes it a bit more clear.
10:06We've all started from a base harness and a base set of evals and we went to go to the frontier.
10:14Um, and you only go through that by running these systems in prod.
10:17There's no way you you know what frontier is before you uh you start.
10:22Um but the the the last step here which is what is requiring a lot of research um is okay once you've reached frontier how do we make this um
10:33uh economically viable which is how do we not spend more than than uh we need for generating this amount of value um and we think we have the building blocks now to make this accessible
10:45and pretty efficient in the sense of you've seen all these fine-tuning APIs all the infrastructure that has been abstracted away for you to do do this process is just the knowhow that uh is not there yet
10:58and this is what we we we hope we can like push for the knowhow for knowing how to codify taste into evals and how to validate that in experiments
11:08um and you you've you've heard a lot about evals in experiments before but you didn't really think of them of like what are they is it's not just tests is is really what is the taste of the creator
11:19that agents should be able to reproduce and self-improve around.
11:24And no one has thought of how do I make this as portable enough?
11:28How how do I make my taste as an artist or as a software developer um something that anyone can download in their brain and be able to be a onetoone replica to me?
11:39And this is kind of like what RL is is is about now is how do we uh turn these um taste makers into uh environments and evals around them so then we can move them into the weights.
11:53But um there's more than that.
11:55Um you can think of the worker as the inner loop and it generates all these artifacts.
12:00But how you look at the artifacts and know what to change is the taste.
12:05Uh and this is what creates candidates of what you should change and how you should adapt based on that.
12:11And experiments is what how you self-calibrate that okay my taste is actually validated in production with users and we make sure that not only the maker is happy through the
12:23um offline evals but the end users are happy as well and they agree with what we consider good.
12:30Let's go through a practical example of how this works.
12:36Let's take a baseline um agent which could be a talent sourcing agent.
12:41Um and this is a very classical case of everyone is doing recruiting differently and is very much about not what is good recruiting but who is leading that recruiting that considers recruiting as good.
12:57So in this case we're starting with something pretty simple.
12:59um a a bunch of tools, web search, LinkedIn, uh a bunch of sub aents that have been pre-popularized by harnesses like codeex and cloud code and uh system instruction which is about your recruiter.
13:14First step is really understand the signals.
13:18So you can think of patterns as being a way to look at the traces, extract some common um behaviors or common user frustrations and turn them into like a cluster.
13:29So let's say this idea of uh the agent is going uh and reaching out to a lot of big tech employees.
13:36As a recruiter, you don't really want that.
13:38You want to find hidden gems.
13:39You don't want to try to hire John Carmarmac.
13:42But an agent would think that's, oh, John Carmarmac is great.
13:46why would I not reach out to him?
13:47Um, so so this is a behavior that you you'd never think of codifying, but you discover the agent tends to do that.
13:55Um, patterns is how you discover these signals and inform you what you should do next.
14:03um calibration judges and evals is how we used to think about how do we codify these these behaviors into um something that can try to uh apply the same judgment across traces and across uh execution.
14:18So let's say we we build an agent that looks at a trajectory and um identifies exactly that pattern.
14:26Hey, did did this agent reach out to Google employees instead of trying to uh find hidden gems on GitHub?
14:32Um, and the calibration bit and the eval generation bit is not that hard.
14:38It it it should be doable by agents to build.
14:41You just need a human in the loop to say, hey, um, this is the approach we're taking.
14:45Do you agree with this judgment?
14:48Do you really agree that we should look more towards hidden gems rather than reach out to um um big tech employees?
14:57And that's about it.
14:58You don't need the human to actually build the evals.
15:01You need them to calibrate the evals.
15:03And agents should be the ones that really take the the the taste of the maker and and put them in into code.
15:09Once you have this, it's pretty easy to create recipe candidates.
15:13And this should be the the diffs that you really want to taste.
15:17Um, and you can have a pretty good offline evil set around this, but the the the test here is when you go to prod.
15:25So, do the end user agree with your taste of not hitting up um big tech uh employees, right?
15:34And this is kind of like what you want is you build a product that really emphasizes your taste and then you you make sure that your users appreciate and value that taste.
15:44and AB tests have been a way to to to make sure that that's the case.
15:48Um so with a multi-arm banded um scenario for example you you'd be able to do that pretty well.
15:54So once you validate okay I have great taste and my users believe uh I have great taste as well that's when you promote and that's kind of when you go to to the next version of an agent recipe.
16:06The secret is you keep doing this over and over again and you know how to continuously codify your taste and your um what what what good is to you into an agent that can reproduce the same service or product
16:19uh for other people and they also agree you have great taste and you have great execution.
16:24And this is really kind of like the the secret of building good loops is okay can can someone iterate on my um system in a way as uh you know um a good example here is like Miranda
16:36from the Delor product right what would Miranda do uh in certain cases and you kind of want to codify that that thinking into like agents that can do the same stuff at a higher level.
16:48So the takeaways are this.
16:49Um the loop is the product.
16:50You try to automate yourself as the u as a um higher level judge and you want to make sure your second loop agents are able to apply the same judgment to to the agents you're trying to to to push to prod.
17:06Second bit system dissolation is the mode.
17:08So, how do you continuously inject that taste into these uh workers and they how how they continuously self-verify and work together is uh the biggest thing that you should focus on and the faster you do it uh the the the
17:22faster you you build a defensible um approach to to becoming a vertical AI company.
17:28And finally, valued work per watt is how you should measure um am I making progress or not.
17:34So first make sure that uh the the the work you're generating is valuable.
17:40Second make sure that the economics makes sense and the um the the difference in price is is basically what um people would would switch away from cloud code to to something you provide.
17:53We've been thinking a lot about these ideas and we're building some very interesting products around how to deploy this in production.
17:59We'd love to hear from you.
18:00would love to get um to to understand more about how how certain um vertical SAS companies are are looking to go to prod with um or how agent labs have been thinking about this idea of um
18:14um creating these like auto research uh labs around their their own products.
18:20um get in touch.
18:21Uh we're gonna be around the block for for chatting more about this.
18:25And thank you very much.