WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=7taOQBfjDyE NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/7taOQBfjDyE.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.960 --> 00:00:14.240 Hello everyone. s2 00:00:14.240 --> 00:00:16.960 How's everyone doing? s3 00:00:17.840 --> 00:00:21.119 Are you guys ready for some more loops? s4 00:00:21.119 --> 00:00:22.480 Yeah. s5 00:00:22.480 --> 00:00:23.840 My name is Roland. s6 00:00:23.840 --> 00:00:35.200 My co-founder and I were in this mythical place called XAI working hard on agent infra and we realized there's something new that has to be done in a standalone way. s7 00:00:35.200 --> 00:00:45.120 So we left a few months ago to really figure out okay what's the next stage of how we should deploy these always on longunning horizon tasks. s8 00:00:45.120 --> 00:00:51.360 Um and I'm happy to announce we have a few findings that we would like to present you. s9 00:00:51.360 --> 00:01:00.480 Um, and this talk it's all about um, how you should productize these ideas in ways that can scale with your customers. s10 00:01:00.480 --> 00:01:04.559 Um, you've heard a lot about auto research. s11 00:01:04.559 --> 00:01:14.479 Um, we think there's a blueprint for 2026 and beyond on how you should think about auto research and it really comes down to three ideas. s12 00:01:14.479 --> 00:01:16.720 Let's go through the first one. s13 00:01:16.720 --> 00:01:19.840 The loop is the product. s14 00:01:21.280 --> 00:01:23.040 We're all familiar with this. s15 00:01:23.040 --> 00:01:31.600 We've started with everything goes down to RL chief for models and how you should train the model to become better and better reasoning. s16 00:01:31.600 --> 00:01:37.600 We then quickly moved to harnesses and how the model is a commodity and it's all about the harness. s17 00:01:37.600 --> 00:01:43.280 And now we're talking about loops and how you should build these loops uh and not touch code anymore. s18 00:01:43.280 --> 00:01:47.920 But what does it really mean and why is everyone saying that? s19 00:01:47.920 --> 00:01:50.560 Do you guys remember clawbot? s20 00:01:50.560 --> 00:01:56.720 That was the original um original name of what is now now now known as open claw. s21 00:01:56.720 --> 00:02:03.280 And this guy AJ built the first loop around cloudbot. s22 00:02:03.280 --> 00:02:12.080 What he did was to find a way to talk to dealers and talk to Reddit users to get bigger discounts on a car. s23 00:02:12.080 --> 00:02:14.000 He followed these four steps. s24 00:02:14.000 --> 00:02:17.520 Um and is really open call the one that did it. s25 00:02:17.520 --> 00:02:35.280 Go on Reddit, find prices, find inventory, talk to the dealers, put dealers head-to-head, and try to figure out how to make them outbid each other, have a verifiable way to know when the price is right, s26 00:02:35.280 --> 00:02:39.920 and then lock in, get the car, and it worked. s27 00:02:39.920 --> 00:02:51.440 Um, probably this was when all the Mac minis were uh selling off the shelves, but this was the first real example of loop is the product and something that s28 00:02:51.440 --> 00:02:53.519 probably should be a startup at this point. s29 00:02:53.519 --> 00:02:59.519 Um, but we've seen how this became a recipe for everyone to build loops. s30 00:02:59.519 --> 00:03:01.200 But let's take a step back. s31 00:03:01.200 --> 00:03:02.640 Why are we here? s32 00:03:02.640 --> 00:03:07.200 Um, we really think models have been trained with this loop in mind. s33 00:03:07.200 --> 00:03:10.400 And it comes from this idea of udala loops. s34 00:03:10.400 --> 00:03:24.720 It's a terminology coined back in 1970s by the US air force and is the idea of these um jet fighters how to react in fast-paced environments. s35 00:03:24.720 --> 00:03:33.440 If you think of models calling tools and taking observations, it's it's what we've been trained on uh as humans but also as as agents. s36 00:03:33.440 --> 00:03:41.519 Now, now what happens when you put strong signals and verifiable work uh at the other ends? s37 00:03:41.519 --> 00:03:45.360 You get to these workers or cloud code agents. s38 00:03:45.360 --> 00:03:59.360 Um and and what matters here is the quality of the signal determines the uh success rate of the loop and the uh quality of the verifi verifier um um s39 00:03:59.360 --> 00:04:04.560 is able to calibrate if that success is actually correct or not. s40 00:04:04.560 --> 00:04:06.159 But there's another loop here. s41 00:04:06.159 --> 00:04:10.480 Um what happens when you take that and feed it back into the signal? s42 00:04:10.480 --> 00:04:24.080 And this is what looping around is all about is how do you generate these artifacts at the end of the first loop to then run a second loop on and have a way to continuously improve. s43 00:04:24.080 --> 00:04:26.639 And this goes to my second point. s44 00:04:26.639 --> 00:04:40.080 System dillation is the mode and is really the ability to understand what went well and wrong in the first loop and know how to process that in the second one. s45 00:04:40.080 --> 00:04:42.880 So how do we tune these AI systems? s46 00:04:42.880 --> 00:04:52.320 Each loop generates useful information around harnesses, profiles, evals, models, resources, tools, and the environment. s47 00:04:52.320 --> 00:05:01.680 What you really want is to have a way to keep this portable, to have a way to version this and to evolve it over time. s48 00:05:01.680 --> 00:05:08.800 If you think about data recipes in research, this is how RL started to work really well. s49 00:05:08.800 --> 00:05:22.639 you understood the recipes and how to continuously change the recipe to combat some of the behaviors that may happen around hallucinations around reward hacking and then you get to a stack which is your final data recipe. s50 00:05:22.639 --> 00:05:24.080 We don't have that for harnesses. s51 00:05:24.080 --> 00:05:28.000 We don't have that for like AI systems in the general term. s52 00:05:28.000 --> 00:05:30.720 So we thought there's space for something like that. s53 00:05:30.720 --> 00:05:41.600 something that contains the evils and contains the tweaks and the human judgment and all these things that are not predetermined at the beginning but they're defined as you learn more s54 00:05:41.600 --> 00:05:46.479 about your agent acting in in in the environment. s55 00:05:47.039 --> 00:05:51.360 We think recipes can be applied to this and we should use the same name. s56 00:05:51.360 --> 00:05:58.320 So an agent recipe is really something that enables you to create reproducible frontier AI systems. s57 00:05:58.320 --> 00:06:07.520 It's something that allows you to have a mode that keeps getting better over time, which is not tied to any platform or any provider. s58 00:06:07.520 --> 00:06:16.000 It's something that you control lives in your company and is agnostic to the models and providers you use. s59 00:06:16.000 --> 00:06:17.600 And loops should focus on this. s60 00:06:17.600 --> 00:06:21.759 Loops should be the way you distill these systems into recipes. s61 00:06:21.759 --> 00:06:25.120 Failure patterns should become judges and evals. s62 00:06:25.120 --> 00:06:27.520 Repeated behavior should become skills and prompts. s63 00:06:27.520 --> 00:06:31.919 user frustration, extensions and memories to your harness and so on. s64 00:06:31.919 --> 00:06:39.440 You we're all familiar with this, but we didn't have the the the right like terminology of how we should think about it and how we should define it. s65 00:06:39.440 --> 00:06:55.039 And we think recipes is a way to put everything together into a git repo and treat it as your ongoing um strategy for for uh building these self-improving systems. s66 00:06:55.039 --> 00:07:00.560 So we are introspection but you can think of introspection as the way you generate these recipes. s67 00:07:00.560 --> 00:07:05.120 So they're recipes for introspecting on your on your system. s68 00:07:05.120 --> 00:07:09.280 We wanted to build something that is portable and provider agnostic. s69 00:07:09.280 --> 00:07:17.120 So we built our um approach to recipes on the pi harness and on harbor for evals. s70 00:07:17.120 --> 00:07:31.440 We baked it into uh git repos so uh everything could be versioned and agents could have a way to continuously track how this change and why and is meant to be owned by you but managed by your agents. s71 00:07:31.440 --> 00:07:34.639 And this is how products should really be built going forward. s72 00:07:34.639 --> 00:07:44.240 It's something that treats the owner as the um almost like the the the higher taste um personality in the room. s73 00:07:44.240 --> 00:07:49.840 But agents should try to calibrate themselves to to the taste of the of the maker. s74 00:07:49.840 --> 00:07:56.479 So we think recipes should be basically encoding the taste of the makers into how you build these agents. s75 00:07:56.479 --> 00:08:01.520 And if I want to use someone else's recipe, I should be able to also bring that taste. s76 00:08:01.520 --> 00:08:07.759 It's not just the harness, it's not just the model, is how did you arrive at this particular recipe and why? s77 00:08:07.759 --> 00:08:16.960 And that's kind of like what uh what is behind uh reproducible um uh products and services around agents. s78 00:08:16.960 --> 00:08:21.520 Um we have an early release of recipes is called pi. s79 00:08:21.520 --> 00:08:22.319 Recipes. s80 00:08:22.319 --> 00:08:28.240 It's very similar to what skills uh used to be in 2025 but is going a step forward. s81 00:08:28.240 --> 00:08:35.519 And this is what do I need to have a frontier agent is everything about how do I codify paste into evals? s82 00:08:35.519 --> 00:08:36.800 How do I run evals? s83 00:08:36.800 --> 00:08:40.800 How do we have the loops to continuously improve those evals over time? s84 00:08:40.800 --> 00:08:45.040 How do we process signals and know what are the right signals to to use? s85 00:08:45.040 --> 00:08:48.160 Um what are the right tools to work with certain models? s86 00:08:48.160 --> 00:08:52.160 How do I have different profiles of the harness to work with different models? s87 00:08:52.160 --> 00:08:54.720 Um and everything in between. s88 00:08:54.720 --> 00:08:57.600 So have a look at what we've been building here. s89 00:08:57.600 --> 00:09:03.040 It's still early uh but hopefully it's useful enough for you guys to to get going. s90 00:09:03.040 --> 00:09:14.320 And we feel this is going to grow into something that um really allows you to to use uh different um almost like different the to to be able to use the taste of of different makers s91 00:09:14.320 --> 00:09:17.600 as recipes for your agent. s92 00:09:18.160 --> 00:09:21.040 And finally, the last point is valued work per watt. s93 00:09:21.040 --> 00:09:23.920 And why is this the score to really optimize for? s94 00:09:23.920 --> 00:09:38.160 Think of how um cursor and cognition went from building the best product to then building the best evals for the product and finally building the best models based on the previous two artifacts. s95 00:09:38.160 --> 00:09:41.519 We think this is like the recipe for everything going forward. s96 00:09:41.519 --> 00:09:45.519 Um code was the first domain where this um was successful. s97 00:09:45.519 --> 00:09:52.800 Um everything beyond customer support, legal research um everything is going to come down to this idea. s98 00:09:52.800 --> 00:09:54.959 How much value am I getting per watt? s99 00:09:54.959 --> 00:10:04.000 Um, how do I measure the value is the first step and how do I know I'm getting a good deal on that value is the second. s100 00:10:04.000 --> 00:10:06.399 And maybe this makes it a bit more clear. s101 00:10:06.399 --> 00:10:14.240 We've all started from a base harness and a base set of evals and we went to go to the frontier. s102 00:10:14.240 --> 00:10:17.760 Um, and you only go through that by running these systems in prod. s103 00:10:17.760 --> 00:10:22.320 There's no way you you know what frontier is before you uh you start. s104 00:10:22.320 --> 00:10:33.440 Um but the the the last step here which is what is requiring a lot of research um is okay once you've reached frontier how do we make this um s105 00:10:33.440 --> 00:10:45.839 uh economically viable which is how do we not spend more than than uh we need for generating this amount of value um and we think we have the building blocks now to make this accessible s106 00:10:45.839 --> 00:10:58.959 and pretty efficient in the sense of you've seen all these fine-tuning APIs all the infrastructure that has been abstracted away for you to do do this process is just the knowhow that uh is not there yet s107 00:10:58.959 --> 00:11:08.160 and this is what we we we hope we can like push for the knowhow for knowing how to codify taste into evals and how to validate that in experiments s108 00:11:08.160 --> 00:11:19.440 um and you you've you've heard a lot about evals in experiments before but you didn't really think of them of like what are they is it's not just tests is is really what is the taste of the creator s109 00:11:19.440 --> 00:11:24.560 that agents should be able to reproduce and self-improve around. s110 00:11:24.560 --> 00:11:28.640 And no one has thought of how do I make this as portable enough? s111 00:11:28.640 --> 00:11:39.200 How how do I make my taste as an artist or as a software developer um something that anyone can download in their brain and be able to be a onetoone replica to me? s112 00:11:39.200 --> 00:11:53.440 And this is kind of like what RL is is is about now is how do we uh turn these um taste makers into uh environments and evals around them so then we can move them into the weights. s113 00:11:53.440 --> 00:11:55.680 But um there's more than that. s114 00:11:55.680 --> 00:12:00.480 Um you can think of the worker as the inner loop and it generates all these artifacts. s115 00:12:00.480 --> 00:12:05.440 But how you look at the artifacts and know what to change is the taste. s116 00:12:05.440 --> 00:12:11.839 Uh and this is what creates candidates of what you should change and how you should adapt based on that. s117 00:12:11.839 --> 00:12:23.040 And experiments is what how you self-calibrate that okay my taste is actually validated in production with users and we make sure that not only the maker is happy through the s118 00:12:23.040 --> 00:12:30.160 um offline evals but the end users are happy as well and they agree with what we consider good. s119 00:12:30.480 --> 00:12:34.639 Let's go through a practical example of how this works. s120 00:12:36.000 --> 00:12:41.120 Let's take a baseline um agent which could be a talent sourcing agent. s121 00:12:41.120 --> 00:12:57.040 Um and this is a very classical case of everyone is doing recruiting differently and is very much about not what is good recruiting but who is leading that recruiting that considers recruiting as good. s122 00:12:57.040 --> 00:12:59.839 So in this case we're starting with something pretty simple. s123 00:12:59.839 --> 00:13:14.560 um a a bunch of tools, web search, LinkedIn, uh a bunch of sub aents that have been pre-popularized by harnesses like codeex and cloud code and uh system instruction which is about your recruiter. s124 00:13:14.959 --> 00:13:18.000 First step is really understand the signals. s125 00:13:18.000 --> 00:13:29.760 So you can think of patterns as being a way to look at the traces, extract some common um behaviors or common user frustrations and turn them into like a cluster. s126 00:13:29.760 --> 00:13:36.240 So let's say this idea of uh the agent is going uh and reaching out to a lot of big tech employees. s127 00:13:36.240 --> 00:13:38.160 As a recruiter, you don't really want that. s128 00:13:38.160 --> 00:13:39.600 You want to find hidden gems. s129 00:13:39.600 --> 00:13:42.720 You don't want to try to hire John Carmarmac. s130 00:13:42.720 --> 00:13:46.160 But an agent would think that's, oh, John Carmarmac is great. s131 00:13:46.160 --> 00:13:47.920 why would I not reach out to him? s132 00:13:47.920 --> 00:13:55.600 Um, so so this is a behavior that you you'd never think of codifying, but you discover the agent tends to do that. s133 00:13:55.600 --> 00:14:03.440 Um, patterns is how you discover these signals and inform you what you should do next. s134 00:14:03.600 --> 00:14:18.720 um calibration judges and evals is how we used to think about how do we codify these these behaviors into um something that can try to uh apply the same judgment across traces and across uh execution. s135 00:14:18.720 --> 00:14:26.079 So let's say we we build an agent that looks at a trajectory and um identifies exactly that pattern. s136 00:14:26.079 --> 00:14:32.959 Hey, did did this agent reach out to Google employees instead of trying to uh find hidden gems on GitHub? s137 00:14:32.959 --> 00:14:38.639 Um, and the calibration bit and the eval generation bit is not that hard. s138 00:14:38.639 --> 00:14:41.120 It it it should be doable by agents to build. s139 00:14:41.120 --> 00:14:45.440 You just need a human in the loop to say, hey, um, this is the approach we're taking. s140 00:14:45.440 --> 00:14:48.880 Do you agree with this judgment? s141 00:14:48.880 --> 00:14:57.680 Do you really agree that we should look more towards hidden gems rather than reach out to um um big tech employees? s142 00:14:57.680 --> 00:14:58.720 And that's about it. s143 00:14:58.720 --> 00:15:01.440 You don't need the human to actually build the evals. s144 00:15:01.440 --> 00:15:03.920 You need them to calibrate the evals. s145 00:15:03.920 --> 00:15:09.920 And agents should be the ones that really take the the the taste of the maker and and put them in into code. s146 00:15:09.920 --> 00:15:13.519 Once you have this, it's pretty easy to create recipe candidates. s147 00:15:13.519 --> 00:15:17.040 And this should be the the diffs that you really want to taste. s148 00:15:17.040 --> 00:15:25.680 Um, and you can have a pretty good offline evil set around this, but the the the test here is when you go to prod. s149 00:15:25.680 --> 00:15:34.959 So, do the end user agree with your taste of not hitting up um big tech uh employees, right? s150 00:15:34.959 --> 00:15:44.639 And this is kind of like what you want is you build a product that really emphasizes your taste and then you you make sure that your users appreciate and value that taste. s151 00:15:44.639 --> 00:15:48.880 and AB tests have been a way to to to make sure that that's the case. s152 00:15:48.880 --> 00:15:54.639 Um so with a multi-arm banded um scenario for example you you'd be able to do that pretty well. s153 00:15:54.639 --> 00:16:06.399 So once you validate okay I have great taste and my users believe uh I have great taste as well that's when you promote and that's kind of when you go to to the next version of an agent recipe. s154 00:16:06.399 --> 00:16:19.199 The secret is you keep doing this over and over again and you know how to continuously codify your taste and your um what what what good is to you into an agent that can reproduce the same service or product s155 00:16:19.199 --> 00:16:24.240 uh for other people and they also agree you have great taste and you have great execution. s156 00:16:24.240 --> 00:16:36.480 And this is really kind of like the the secret of building good loops is okay can can someone iterate on my um system in a way as uh you know um a good example here is like Miranda s157 00:16:36.480 --> 00:16:48.240 from the Delor product right what would Miranda do uh in certain cases and you kind of want to codify that that thinking into like agents that can do the same stuff at a higher level. s158 00:16:48.320 --> 00:16:49.600 So the takeaways are this. s159 00:16:49.600 --> 00:16:50.959 Um the loop is the product. s160 00:16:50.959 --> 00:17:06.400 You try to automate yourself as the u as a um higher level judge and you want to make sure your second loop agents are able to apply the same judgment to to the agents you're trying to to to push to prod. s161 00:17:06.400 --> 00:17:08.720 Second bit system dissolation is the mode. s162 00:17:08.720 --> 00:17:22.160 So, how do you continuously inject that taste into these uh workers and they how how they continuously self-verify and work together is uh the biggest thing that you should focus on and the faster you do it uh the the the s163 00:17:22.160 --> 00:17:28.960 faster you you build a defensible um approach to to becoming a vertical AI company. s164 00:17:28.960 --> 00:17:34.880 And finally, valued work per watt is how you should measure um am I making progress or not. s165 00:17:34.880 --> 00:17:40.080 So first make sure that uh the the the work you're generating is valuable. s166 00:17:40.080 --> 00:17:52.799 Second make sure that the economics makes sense and the um the the difference in price is is basically what um people would would switch away from cloud code to to something you provide. s167 00:17:53.039 --> 00:17:59.200 We've been thinking a lot about these ideas and we're building some very interesting products around how to deploy this in production. s168 00:17:59.200 --> 00:18:00.480 We'd love to hear from you. s169 00:18:00.480 --> 00:18:14.799 would love to get um to to understand more about how how certain um vertical SAS companies are are looking to go to prod with um or how agent labs have been thinking about this idea of um s170 00:18:14.799 --> 00:18:20.000 um creating these like auto research uh labs around their their own products. s171 00:18:20.000 --> 00:18:21.039 um get in touch. s172 00:18:21.039 --> 00:18:25.039 Uh we're gonna be around the block for for chatting more about this. s173 00:18:25.039 --> 00:18:28.080 And thank you very much.