WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=2e9ANoOEn28 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/2e9ANoOEn28.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.640 --> 00:00:02.280 Hi everyone. s2 00:00:02.280 --> 00:00:05.480 My name is Audit, pronounced Audit like a tax audit. s3 00:00:05.480 --> 00:00:11.600 Um and I'm here to talk to you about what if the harness mattered more than the model? s4 00:00:11.600 --> 00:00:13.080 A little bit about me first. s5 00:00:13.080 --> 00:00:15.160 I'm a staff engineer at Etsy. s6 00:00:15.160 --> 00:00:18.640 I'm also Etsy's IC initiative lead for agent of commerce. s7 00:00:18.640 --> 00:00:22.320 I wrote Grokking Algorithms, which is an illustrated book on algorithms. s8 00:00:22.320 --> 00:00:26.920 Uh and I also do illustrated posts on ducktyped.org. s9 00:00:27.520 --> 00:00:35.320 Um and I want to talk to you about what if the harness mattered more than the model? s10 00:00:35.320 --> 00:00:43.920 By which I mean what if we should be paying more attention to building expertise in harnesses? s11 00:00:43.920 --> 00:00:54.160 Um and my controversial take here is I hear a lot of people say, "Oh, the models are so good that you can just keep the harness simple. s12 00:00:54.160 --> 00:00:57.680 Just give the model a few tools and it'll do the rest." s13 00:00:57.680 --> 00:01:06.440 And I've been hearing this so often that it has now uh started becoming the wisdom of the tech industry. s14 00:01:06.440 --> 00:01:20.920 Uh to which my answer is, "Yes, but that's moving in the wrong direction because that is making us reliant on fancy proprietary models that can't be run locally. s15 00:01:20.920 --> 00:01:23.560 So, what if we instead talked about the other direction? s16 00:01:23.560 --> 00:01:28.080 What if we focused on open-source models that can be run locally? s17 00:01:28.080 --> 00:01:37.120 What if we focused on building a harness that is so good that we can get the performance of a cutting-edge model through a local open-source model? s18 00:01:37.120 --> 00:01:41.080 So, what if the harness mattered more than the model? s19 00:01:41.080 --> 00:01:44.920 What if our focus should be on the harness? s20 00:01:45.240 --> 00:01:47.400 Um how much does a harness matter? s21 00:01:47.400 --> 00:01:47.640 Right? s22 00:01:47.640 --> 00:01:52.120 This is something I have been researching a lot in my spare time. s23 00:01:52.120 --> 00:02:02.600 Um there's a really interesting paper that talks about Harness Bench, which is a benchmark for harnesses, which is something we haven't seen a lot of yet. s24 00:02:02.600 --> 00:02:06.240 Um it's got 106 tasks. s25 00:02:06.240 --> 00:02:10.360 Um it's testing these different models in different harnesses. s26 00:02:10.360 --> 00:02:15.200 Uh but the point is that it's running the same setup. s27 00:02:15.200 --> 00:02:22.560 It's doing the same evaluation, same model, but testing different harnesses. s28 00:02:22.560 --> 00:02:25.040 Um and the results are pretty interesting. s29 00:02:25.040 --> 00:02:30.959 So, scores range from 52.4% to 76.2%. s30 00:02:31.040 --> 00:02:36.400 So, more than a 20-point difference, and only the harness changed. s31 00:02:36.400 --> 00:02:44.320 Um and the really interesting thing is that for weaker models, the harness matters more. s32 00:02:44.320 --> 00:02:47.920 Um if the harness matters more than the model, that's great news. s33 00:02:47.920 --> 00:02:57.920 Uh because if the model matters more, then we're dependent on a handful of companies who are able to build and train these models. s34 00:02:57.920 --> 00:03:09.440 Uh but if the if a good harness can compensate, and can make a weaker model perform better, then you can build our own harnesses, and that's something that any of us can do, s35 00:03:09.440 --> 00:03:13.200 and we don't have to depend on paid models. s36 00:03:13.200 --> 00:03:25.640 Um so, I have been spending a lot of time and doing a lot of research specifically into what would it take to build a really good harness. s37 00:03:25.640 --> 00:03:37.120 Um and I realized early on that none of the existing tools or frameworks of were able to do what I wanted. s38 00:03:37.120 --> 00:03:43.519 And I realized that what was actually needed was a language-level solution. s39 00:03:43.519 --> 00:03:53.000 So, this is my other controversial take in this talk is I think building a good harness actually requires language-level support. s40 00:03:53.000 --> 00:04:04.760 Um and this is kind of where my research has gone in the last 6 months and I've spent a good part of that 6 months building a new language s41 00:04:04.760 --> 00:04:10.360 called Agency, which is a language for building agents. s42 00:04:10.360 --> 00:04:22.200 Um and just to talk about the terminology there real quick because the terms agent and harness aren't super crisp in the industry yet. s43 00:04:22.200 --> 00:04:24.200 They're not super well defined. s44 00:04:24.200 --> 00:04:28.120 Um but in this talk I'll kind of use them interchangeably. s45 00:04:28.120 --> 00:04:33.520 But when I talk about the agent, I really mean the harness and the model. s46 00:04:33.520 --> 00:04:38.560 So if you take Claude code for example, Claude code is an agent. s47 00:04:38.560 --> 00:04:44.200 It uses Opus, which is the model, and then everything else is the harness. s48 00:04:44.200 --> 00:04:55.400 Um but you know, when we talk about Agency, it's really for building agents, but the thing it's providing is the harness. s49 00:04:55.400 --> 00:04:58.400 Um Agency is really good. s50 00:04:58.400 --> 00:05:00.720 Uh you know, it's still new. s51 00:05:00.720 --> 00:05:04.880 Building a language is a lot of work and it's only 6 months in. s52 00:05:04.880 --> 00:05:09.320 Uh but I love this quote by Alan Kay. s53 00:05:09.320 --> 00:05:12.960 Simple things should be simple, complex things should be possible. s54 00:05:12.960 --> 00:05:26.640 And that's something I love about Agency is that I think you know, when people build agents, there are a lot of primitives, a lot of things that every agent s55 00:05:26.640 --> 00:05:32.800 will need to be able to do and those things should be simple to do. s56 00:05:32.800 --> 00:05:42.640 Um and there's a lot of complex things that we want to be able to do with agents and it should be possible to do those things in the framework s57 00:05:42.640 --> 00:05:46.080 of your choice and with Agency it is possible. s58 00:05:46.080 --> 00:05:53.120 Um So now we have a clear goal, which is try to build the best possible harness. s59 00:05:53.120 --> 00:05:56.520 And we have a great tool to do it, which is agency. s60 00:05:56.520 --> 00:06:02.560 Uh so, the rest of this talk I'm going to talk about building a coding agent in agency. s61 00:06:02.560 --> 00:06:07.680 And we'll see the same agent evolve over seven different examples. s62 00:06:07.680 --> 00:06:18.800 We'll see the same model, the same task, but the harness will improve each time, and we'll see the harness improve the agent's performance each time. s63 00:06:18.800 --> 00:06:23.560 Uh So, you know, like I said, I've been doing a ton of research on this. s64 00:06:23.560 --> 00:06:29.120 My goals with this talk are to give you like the basics of building a harness. s65 00:06:29.120 --> 00:06:32.480 When we talk about building a good harness, what does that mean? s66 00:06:32.480 --> 00:06:36.880 You know, you're starting from zero, what are the basics? s67 00:06:36.880 --> 00:06:50.600 Um I'll cover a good bit about agent safety because, you know, when we talk about building agents, what you really want them to be able to do is take action on our behalf. s68 00:06:50.600 --> 00:07:06.880 Um But the big problem with that is how do you get how do you give agent capabilities so they can take actions, but not so much capability that they do something unsafe. s69 00:07:06.880 --> 00:07:14.360 So, when we talk about agent quality and capability, agent safety goes hand in hand with that. s70 00:07:14.360 --> 00:07:21.760 Um At the end, I'll do an introduction to some of the more advanced concepts like sub-agents and self-optimization. s71 00:07:21.760 --> 00:07:35.600 I think that's where, you know, more interesting things come in, and we really start to talk about like what are all the possibilities, you know, because when you talk about building a good harness, there are so many things to explore. s72 00:07:35.600 --> 00:07:44.040 And so, I hope you stick with me till the end there because that's where we get into some really interesting stuff. s73 00:07:45.400 --> 00:07:51.160 Um so this is the code we're going to ask our coding agent to edit. s74 00:07:51.160 --> 00:07:59.480 This is a code to calculate the median of an ordered list of numbers. s75 00:07:59.480 --> 00:08:05.400 Um and if it's an odd length list, you just take the middle element. s76 00:08:05.400 --> 00:08:13.320 If it's an even length list, you need to take the middle two elements and take the mean of those two elements. s77 00:08:13.320 --> 00:08:16.760 But right now this function doesn't do that. s78 00:08:16.760 --> 00:08:18.960 So there is a bug in this function. s79 00:08:18.960 --> 00:08:29.160 There's a test that shows the failure um and we're going to ask our agent to fix this code. s80 00:08:29.919 --> 00:08:31.800 So let's go ahead and start. s81 00:08:31.800 --> 00:08:33.520 Let's just dive in. s82 00:08:33.520 --> 00:08:35.840 Um this is the first example. s83 00:08:35.840 --> 00:08:37.919 This is agency code now. s84 00:08:37.919 --> 00:08:42.440 And you'll notice that agency looks a lot like TypeScript. s85 00:08:42.440 --> 00:08:48.600 It's heavily inspired with by TypeScript with some Python syntax thrown in. s86 00:08:48.600 --> 00:08:52.000 Um so the entry point is this node main. s87 00:08:52.000 --> 00:08:53.680 I have a simple prompt. s88 00:08:53.680 --> 00:08:58.440 The Python function median in demo/median.py has a bug. s89 00:08:58.440 --> 00:09:00.360 Please fix the bug. s90 00:09:00.360 --> 00:09:03.120 And I just passed that prompt into the LLM function. s91 00:09:03.120 --> 00:09:06.360 I get the result and I print the result. s92 00:09:06.360 --> 00:09:12.839 Um And of course this isn't going to work because it can't read or write to this file. s93 00:09:12.839 --> 00:09:17.520 So let's run this example real quick. s94 00:09:25.520 --> 00:09:25.960 Okay. s95 00:09:25.960 --> 00:09:32.360 So of course as expected it says you know, in order to help me fix the bug I'll need to see the existing code. s96 00:09:32.360 --> 00:09:35.280 Um so this is just the model. s97 00:09:35.280 --> 00:09:38.560 Very little harness and of course it can't do anything. s98 00:09:38.560 --> 00:09:45.160 So, the most obvious improvement to any harness is give it some tools. s99 00:09:45.160 --> 00:09:52.600 So, real quick, let's talk about how tools work in the agency cuz it's very simple. s100 00:09:52.600 --> 00:09:55.000 Um I'll define a function. s101 00:09:55.000 --> 00:10:02.040 In this case, just for an example, I'm going to define a greet function that greets a user. s102 00:10:02.040 --> 00:10:07.200 And then I'll make an LLM call that says, "Use your greet tool to greet audit." s103 00:10:07.200 --> 00:10:09.880 And then pass in that tool. s104 00:10:09.880 --> 00:10:15.640 Um and every function can automatically be used as a tool in agency. s105 00:10:15.640 --> 00:10:18.000 So, there is nothing else to do. s106 00:10:18.000 --> 00:10:24.200 Uh agency creates a JSON schema out of this function and sends it to the LLM call. s107 00:10:24.200 --> 00:10:27.920 The doc string here will get used as a tool description. s108 00:10:27.920 --> 00:10:32.200 So, let's go ahead and run the second example. s109 00:10:32.320 --> 00:10:39.040 So, NPX agency Oops, actually. s110 00:10:39.520 --> 00:10:47.680 NPX agency example since actually 01 cuz we're going to talk about tools first. s111 00:10:47.680 --> 00:10:49.400 So, howdy audit? s112 00:10:49.400 --> 00:10:57.200 And how can we know that it ran that tool and then just generate an example, right? s113 00:10:57.200 --> 00:11:00.680 So, I'll just also show real quick. s114 00:11:00.680 --> 00:11:07.120 Um agency writes out logs and has this great log viewer built in. s115 00:11:07.120 --> 00:11:16.480 So, I'll just expand this last run and you can see here it's making this tool call to the greet function with these parameters. s116 00:11:16.480 --> 00:11:23.440 The tool responds by returning howdy audit, and the assistant just passes that straight back to us. s117 00:11:23.440 --> 00:11:30.120 Um So, that's tools. s118 00:11:30.120 --> 00:11:34.640 Now, let's look at using the tools in Agency. s119 00:11:34.640 --> 00:11:41.400 So, again, this is you know, the next step of our agent, which is give it tools. s120 00:11:41.400 --> 00:11:48.000 Um here is an example where we're giving the agent read and write tools. s121 00:11:48.000 --> 00:11:58.000 Again, read and write are just functions that are built into Agency, and because every function can be used as a tool, I can just pass this straight to the LLM. s122 00:11:58.000 --> 00:12:08.200 Uh except giving the agent the ability to read and write arbitrary files on our file system is really unsafe. s123 00:12:08.200 --> 00:12:12.400 So, by default Agency won't allow you to do this. s124 00:12:12.400 --> 00:12:17.440 Agency has a human-in-the-loop feature that would require some sort of approval. s125 00:12:17.440 --> 00:12:22.520 We're not providing it anywhere, so this code will crash and print an error. s126 00:12:22.520 --> 00:12:26.160 So, let me show you what that looks like. s127 00:12:33.320 --> 00:12:33.920 Great. s128 00:12:33.920 --> 00:12:36.880 So, interrupt standard read was not handled. s129 00:12:36.880 --> 00:12:40.240 Your interrupt message, "Are you sure you want to read this file?" s130 00:12:40.240 --> 00:12:49.680 You can see the file name it's trying to read, and it's basically telling us that we didn't approve this interrupt. s131 00:12:49.680 --> 00:12:52.280 And it's giving us a pointer to the guide, right? s132 00:12:52.280 --> 00:12:54.680 Tells us how to use handlers. s133 00:12:54.680 --> 00:13:05.520 Um so we started with just a model, we tried tools, and the next step is to make the tools safe. s134 00:13:05.520 --> 00:13:12.600 So, we're going to do exactly what that message asked us to do and use a handler. s135 00:13:12.600 --> 00:13:21.720 So, now, here's the same code that you know, passing through that read and write uh those read and write functions as tools. s136 00:13:21.720 --> 00:13:25.440 Um but now it's wrapped in a handle block. s137 00:13:25.440 --> 00:13:28.880 Uh and it has a handler function down here. s138 00:13:28.880 --> 00:13:36.680 So, now when those tools call or raise an interrupt, this code is going to execute. s139 00:13:36.680 --> 00:13:42.120 So, it'll print the information about that interrupt. s140 00:13:42.120 --> 00:13:48.320 Um, it'll ask the user for input using the built-in input function. s141 00:13:48.320 --> 00:13:52.480 And if the user approves, the tool call is approved. s142 00:13:52.480 --> 00:13:56.880 And I know I'm saying interrupt, and I'm not really defining what that means. s143 00:13:56.880 --> 00:14:04.760 Um, and I can get to that a little bit later, but it's essentially a way to pause execution at some point. s144 00:14:04.760 --> 00:14:08.680 Um, and ask for human input. s145 00:14:08.680 --> 00:14:25.120 And so, interrupts are a great feature in agency, and all the functions in the agency standard library that mutate code or mutate something, or have some sort of destructive action, or read sensitive data data, s146 00:14:25.120 --> 00:14:29.360 all throw an interrupt first. s147 00:14:29.360 --> 00:14:45.600 So, here's the safe version of our agent where the tools will raise an interrupt by default, and now this code is going to ask the user, do you approve? s148 00:14:45.600 --> 00:14:48.560 Let's run this example. s149 00:14:53.960 --> 00:14:56.320 Okay, are you sure you want to read this file? s150 00:14:56.320 --> 00:14:58.680 I'll just say yes. s151 00:14:58.680 --> 00:15:00.080 You can see it's reading. s152 00:15:00.080 --> 00:15:01.920 Now it's trying to write to that file. s153 00:15:01.920 --> 00:15:05.875 So, you know, that's an example of s154 00:15:05.875 --> 00:15:06.320 [snorts] s155 00:15:06.320 --> 00:15:12.920 us making the agent safer by asking for user input. s156 00:15:12.920 --> 00:15:17.240 And this is better, and it is safe, but it's also very slow. s157 00:15:17.240 --> 00:15:26.400 So, the next thing we want to do is make the agent autonomous, while still keeping it safe, and that's the tricky part, right? s158 00:15:26.400 --> 00:15:41.760 How do you give it just enough capability so that you don't have to manually approve every single action without giving it without giving it so much capability that it's like able to do something s159 00:15:41.760 --> 00:15:44.320 destructive with your file system. s160 00:15:44.320 --> 00:15:48.200 Um so agency has a really cool feature to handle this. s161 00:15:48.200 --> 00:15:51.070 It that's called partial function application. s162 00:15:51.070 --> 00:15:51.360 [clears throat] s163 00:15:51.360 --> 00:15:55.240 Um it's a concept borrowed from functional programming. s164 00:15:55.240 --> 00:16:01.920 It's a really fancy term for a pretty simple concept. s165 00:16:01.920 --> 00:16:05.520 So, here's a signature of the read tool, right? s166 00:16:05.520 --> 00:16:08.200 Takes a file name and a directory. s167 00:16:08.200 --> 00:16:14.800 Um and when you read a file, it's going to look inside that directory for that file name. s168 00:16:14.800 --> 00:16:22.160 And with partial function application, what we do is we take that read function and call the partial on it. s169 00:16:22.160 --> 00:16:24.000 And give it a directory. s170 00:16:24.000 --> 00:16:29.800 And what we're doing here is we're locking the directory argument to demo. s171 00:16:29.800 --> 00:16:34.240 Um the LLM isn't going to be able to change that argument. s172 00:16:34.240 --> 00:16:36.920 It's not even going to know that that argument exists. s173 00:16:36.920 --> 00:16:42.560 When we pass in the read tool now, it's just going to see one parameter, which is file name. s174 00:16:42.560 --> 00:16:44.839 It's going to pass in a file name. s175 00:16:44.839 --> 00:16:49.040 And it will read that file from this directory. s176 00:16:49.040 --> 00:16:51.680 So, we've locked the directory parameter. s177 00:16:51.680 --> 00:16:58.720 Now, the agent can only read files from and write files to this directory. s178 00:16:58.720 --> 00:17:06.880 Uh partial function application is a really great way to constrain the capabilities of your agent. s179 00:17:06.880 --> 00:17:12.040 Now, no human input is needed, but it's still safe. s180 00:17:12.040 --> 00:17:18.480 So, let's see that in So, again, run NPX agency examples. s181 00:17:18.600 --> 00:17:21.839 PFA is partial function application. s182 00:17:21.839 --> 00:17:28.920 So, here it's thinking and you can see it read the file. s183 00:17:28.920 --> 00:17:31.120 Didn't need to ask us for permission. s184 00:17:31.120 --> 00:17:35.280 And now it said, you know, to fix this issue, it's kind of giving us a new code. s185 00:17:35.280 --> 00:17:38.360 But it's not updating the code. s186 00:17:38.360 --> 00:17:41.280 It asks, "Would you like me to update the file with this corrected function?" s187 00:17:41.280 --> 00:17:43.720 But it didn't do that itself. s188 00:17:43.720 --> 00:18:01.040 So, you know, this version of this agent is the best one we've seen so far because it does give us the correct solution without any human input needed, but it hasn't actually fixed the code. s189 00:18:01.040 --> 00:18:04.920 So, the next step is to get it to actually fix the code. s190 00:18:04.920 --> 00:18:08.840 Um and that's where the feedback loop comes in. s191 00:18:08.840 --> 00:18:14.800 Um so, there's a very popular agent pattern called react. s192 00:18:14.800 --> 00:18:18.040 And react stands for reason and act. s193 00:18:18.040 --> 00:18:24.480 And essentially, you're asking the agent to work in this loop. s194 00:18:24.480 --> 00:18:32.600 Where you reason, act, observe the consequences of your action, and then decide what to do next. s195 00:18:32.600 --> 00:18:47.200 So, in this case, reason, read the two files, act, run the tests, observe, if the tests fail, read the error, and then reason again. s196 00:18:47.200 --> 00:18:53.120 So, look at that test failure and think about what you want to do next. s197 00:18:53.120 --> 00:18:59.840 And just do this in a loop until the tests pass. s198 00:18:59.840 --> 00:19:05.760 So, this is a really key pattern to making the agent better. s199 00:19:05.760 --> 00:19:08.920 And so, now let's see the same agent. s200 00:19:08.920 --> 00:19:22.760 The only thing we're changing is the prompt and giving it the ability to run these tests and now the agent should work better. s201 00:19:24.640 --> 00:19:31.960 In this case, I'm going to also print all the tool calls just so you can see what's happening. s202 00:19:31.960 --> 00:19:52.240 Um Well, should maybe All right, hang on. s203 00:19:52.240 --> 00:19:54.760 It's always there. s204 00:20:02.640 --> 00:20:03.280 There we go. s205 00:20:03.280 --> 00:20:04.880 Okay. s206 00:20:04.880 --> 00:20:07.040 Now, let's see it in action. s207 00:20:07.040 --> 00:20:12.120 Um So, now you can see it's calling the tools. s208 00:20:12.120 --> 00:20:14.720 It's reading those two files, like we said. s209 00:20:14.720 --> 00:20:16.720 It's running the tests. s210 00:20:16.720 --> 00:20:18.360 The tests fail. s211 00:20:18.360 --> 00:20:20.360 It's writing to the file. s212 00:20:20.360 --> 00:20:24.200 It's running the test again and confirming success. s213 00:20:24.200 --> 00:20:42.880 So, this is a really important stage in the harness development because now we finally have an agent that successfully figures out the task, checks that the code is failing, s214 00:20:42.880 --> 00:20:48.160 modifies the code, and checks that the code now passes. s215 00:20:48.160 --> 00:20:53.840 Um So, let's talk about how we have improved the harness so far. s216 00:20:53.840 --> 00:20:57.400 So, we started with just the model, couldn't do anything. s217 00:20:57.400 --> 00:21:01.920 Then we added tools, but the agent wasn't safe. s218 00:21:01.920 --> 00:21:08.600 Then we added handlers, which added safety, but needed human input. s219 00:21:08.600 --> 00:21:14.480 Then we added partial function application, which was safety without requiring human input. s220 00:21:14.480 --> 00:21:21.040 And finally, the last one that we just looked at was a feedback loop, which added reasoning. s221 00:21:21.040 --> 00:21:27.360 So, the agent made the change and confirmed that the change worked. s222 00:21:29.080 --> 00:21:31.160 So, I have two more examples. s223 00:21:31.160 --> 00:21:34.600 This one we're going to go in a slightly different direction. s224 00:21:34.600 --> 00:21:38.440 And this is where we're getting into some of the more advanced concepts. s225 00:21:38.440 --> 00:21:40.440 So, this one is sub-agents. s226 00:21:40.440 --> 00:21:43.920 Um everything so far we've done has made the same task better. s227 00:21:43.920 --> 00:21:49.040 But with sub-agents, we're asking, "How do you make the agent do more things?" s228 00:21:49.040 --> 00:21:57.480 Um and let me show you what this code looks like real quick. s229 00:22:02.400 --> 00:22:03.200 Okay. s230 00:22:03.200 --> 00:22:12.080 So, this program, I'll start by just showing you the prompt we're giving it because it's slightly different. s231 00:22:12.080 --> 00:22:15.600 So, fix the failing test in test median.py. s232 00:22:15.600 --> 00:22:22.280 Then research Jensen's inequality for medians with the Wikipedia agent and explain it to me in the network form. s233 00:22:22.280 --> 00:22:29.440 So, here is a case where we're actually asking the agent to do a little more, right? s234 00:22:29.440 --> 00:22:38.720 And this one is interesting because this is our main agent, but for tools, we're not giving it the read and write functions. s235 00:22:38.720 --> 00:22:42.880 Instead, we're giving it two sub-agents as tools. s236 00:22:42.880 --> 00:22:55.640 And I really like this pattern in agency because I've seen a lot of frameworks make sub-agents their own concept and make them kind of hard to grasp. s237 00:22:55.640 --> 00:22:59.440 Um in agency, a sub-agent is just another function. s238 00:22:59.440 --> 00:23:03.800 And you just call it, use it like a tool just like you would anywhere else. s239 00:23:03.800 --> 00:23:06.360 And that consistency has a lot of benefits. s240 00:23:06.360 --> 00:23:09.480 Makes it really easy to write and reason about. s241 00:23:09.480 --> 00:23:13.440 So, here's the coding agent sub agent. s242 00:23:13.440 --> 00:23:16.000 Um same one as before. s243 00:23:16.000 --> 00:23:19.200 You know, there's a system message. s244 00:23:19.200 --> 00:23:24.336 This sub agent has its own LLM call, gets its own set of tools. s245 00:23:24.336 --> 00:23:24.800 [snorts] s246 00:23:24.800 --> 00:23:30.880 Similarly, you have the Wikipedia agent that has its own tool, which is the search tool. s247 00:23:30.880 --> 00:23:36.480 Uh which is a tool built into this agency standard library that lets you search Wikipedia. s248 00:23:36.480 --> 00:23:42.480 Uh so, agency has a big standard library that lets you do all kinds of things, including this Wikipedia search tool. s249 00:23:42.480 --> 00:23:49.040 I'm also using the callback feature to print the tool call information. s250 00:23:49.040 --> 00:23:52.440 And now, let's see this example in action. s251 00:23:52.440 --> 00:23:59.280 And this is an example where we're actually asking the agent to do a little bit more. s252 00:23:59.280 --> 00:24:04.040 And we're giving it two sub agents to accomplish its task. s253 00:24:04.040 --> 00:24:07.480 So, here you can see it's calling those two sub agents. s254 00:24:07.480 --> 00:24:21.120 Uh it's calling them in parallel, so the Wikipedia agent is doing the search, the coding agent is modifying the file, and that fixed the error, and now here's a limerick. s255 00:24:21.120 --> 00:24:33.520 So, sub agents are cool because they let you add new capabilities without bloating context. s256 00:24:33.520 --> 00:24:48.000 Um and this is a good pattern because it allows your agent when it has a lot of different tools at its disposal, it allows your agent to do the right thing more consistently. s257 00:24:48.000 --> 00:25:00.440 Often when agents get confused and fail, it's because they have too much context blow, but also because they have too many unrelated concepts in their context. s258 00:25:00.440 --> 00:25:03.000 And they have too many tools that are unrelated. s259 00:25:03.000 --> 00:25:06.280 And so they have a hard time picking the right tool to use. s260 00:25:06.280 --> 00:25:10.320 In this case, we make that problem a lot cleaner. s261 00:25:10.320 --> 00:25:14.800 For the top-level agent, we just say, "Pick the sub-agent you want to call." s262 00:25:14.800 --> 00:25:21.680 And then the sub-agents have those tools that are kind of grouped in a way that makes sense. s263 00:25:21.680 --> 00:25:28.080 Um So now, let's look at the last example, which is self-optimization. s264 00:25:28.520 --> 00:25:32.280 Uh Agency has a Japa optimizer built in. s265 00:25:32.280 --> 00:25:38.640 It actually has a few optimizers built in, but the Japa one uh is a really interesting one. s266 00:25:38.640 --> 00:25:49.000 And the way the optimization feature works is you mark any variables that you want the optimizer to optimize. s267 00:25:49.000 --> 00:25:51.800 You can just use this optimize modifier. s268 00:25:51.800 --> 00:25:55.960 And then you run the optimizer and give it a goal. s269 00:25:55.960 --> 00:26:02.920 So, you know, in this case, you can see that the two initial prompts I have are very basic. s270 00:26:02.920 --> 00:26:06.040 I'm giving it hardly any information. s271 00:26:06.360 --> 00:26:12.600 And then the goal I give it is, "Fix the bug in median.py using TDD." s272 00:26:12.600 --> 00:26:21.880 And the reason this is such a great step for our harness is we're no longer doing trial and error. s273 00:26:21.880 --> 00:26:24.800 We're no longer just trying stuff to see what works. s274 00:26:24.800 --> 00:26:31.000 Instead, now we have a way to systematically measure and improve performance. s275 00:26:31.000 --> 00:26:35.680 So, I'm just going to go ahead and run this code. s276 00:26:35.680 --> 00:26:39.240 Um I'm actually going to also reset. s277 00:26:42.480 --> 00:26:47.120 Here, I'm going to modify this code. s278 00:26:53.240 --> 00:26:54.360 Cool. s279 00:26:54.360 --> 00:27:15.400 So now I won't show the full run here because the optimizer takes a while, but you can see first it's like checking what prompts it has that it needs to optimize. s280 00:27:15.400 --> 00:27:19.360 It's going to run the agent to establish a baseline. s281 00:27:19.360 --> 00:27:25.360 So the baseline objective is 0.2 and it's going to try to improve upon that. s282 00:27:25.360 --> 00:27:32.160 So in this case I did run the optimizer earlier just so I could show you what a full run looked like. s283 00:27:32.160 --> 00:27:43.080 Um and so here is one where you can see it achieved the objective in the first iteration, so it was actually quite quick in this case. s284 00:27:43.080 --> 00:27:49.120 Um and you can see that it's rewritten this prompt. s285 00:27:49.120 --> 00:27:58.240 Um So the reason this self-optimization is so great is because we're not guessing and checking. s286 00:27:58.240 --> 00:28:04.240 We're systematically measuring and improving, which is a big leap forward. s287 00:28:04.240 --> 00:28:08.120 So this is the ladder we just went through for improving the harness. s288 00:28:08.120 --> 00:28:13.760 Um you know, first step was nothing, just a model can't even read the file. s289 00:28:13.760 --> 00:28:19.840 Then we added tools, but in a way that was unsafe and not allowed in agency. s290 00:28:19.840 --> 00:28:29.960 Next we added handlers, which added safety, but made things slower because they required human input. s291 00:28:29.960 --> 00:28:34.200 Then we added partial function application, which was safe and fast. s292 00:28:34.200 --> 00:28:39.400 Then we added reasoning, which gave us higher confidence in the output of the agent. s293 00:28:39.400 --> 00:28:46.680 Uh and then we switched tracks to subagents, added more capabilities, and finally we did self-optimization. s294 00:28:46.680 --> 00:28:51.160 So we got measured improvement and not guess and check. s295 00:28:51.160 --> 00:29:03.480 So, you know, I hope this gives you an idea of how you can build harnesses and then slowly improve those harnesses. s296 00:29:03.480 --> 00:29:13.560 And I hope, you know, you try doing that and come along on this, you know, experimentation with me to see if we can build a better harness. s297 00:29:13.560 --> 00:29:17.280 Um because what of the harness matters more than the model. s298 00:29:17.280 --> 00:29:29.840 You know, what if we can build a harness that's good enough that we can now start using local models and still get the performance we need for the tasks we want to do. s299 00:29:29.840 --> 00:29:32.480 So, what language level support does Agency provide? s300 00:29:32.480 --> 00:29:34.680 You know, we used Agency for all these examples. s301 00:29:34.680 --> 00:29:36.680 How did it help? s302 00:29:36.680 --> 00:29:39.320 Um simple syntax for defining tools. s303 00:29:39.320 --> 00:29:42.360 Again, you know, the simple thing should be simple. s304 00:29:42.360 --> 00:29:44.480 Every agent is going to need tools. s305 00:29:44.480 --> 00:29:47.360 Let's make it easy to use them. s306 00:29:47.360 --> 00:29:50.120 Tools for safety, very important for agents. s307 00:29:50.120 --> 00:29:52.560 So, interrupts handlers, PFA. s308 00:29:52.560 --> 00:29:55.000 True pause and resume execution. s309 00:29:55.000 --> 00:29:59.400 I didn't cover this a ton cuz I didn't want to make this talk too long. s310 00:29:59.400 --> 00:30:12.480 Um but one of the really killer features of Agency is that when you raise that interrupt, it's actually going to pause execution and return control back to the user. s311 00:30:12.480 --> 00:30:15.680 But in a way where you can resume execution later. s312 00:30:15.680 --> 00:30:21.560 Uh and as far as I know, very few languages allow you to do something like this. s313 00:30:21.560 --> 00:30:29.040 And so, typically most frameworks you'll use, if they have a human in the loop feature, it has a lot of restrictions. s314 00:30:29.040 --> 00:30:32.080 There's a lot of things you need to do to work around that. s315 00:30:32.080 --> 00:30:34.040 Agency is completely different. s316 00:30:34.040 --> 00:30:42.560 If you wanted, you could, you know, raise interrupts inside of a for loop, inside of a tool call, inside of a sub-agent. s317 00:30:42.560 --> 00:30:57.840 Um, and it would correctly pause execution, go back to the user, and then resume execution at that exact point inside that subagent, inside that tool call, at that exact point in that for loop. s318 00:30:57.840 --> 00:31:00.280 Um, which is really cool. s319 00:31:00.280 --> 00:31:08.120 You can serialize the execution, you can come back to it a week later, and you don't need to change anything about all the code we have been looking at. s320 00:31:08.120 --> 00:31:10.800 It just works, which is really cool. s321 00:31:10.800 --> 00:31:19.960 Uh, it also has these built-in optimizers, which are great, you know, which kind of get you to a more analytical, mathematical way of doing the improvements. s322 00:31:19.960 --> 00:31:22.080 So, if you want to try it out, please do. s323 00:31:22.080 --> 00:31:29.840 You can PM install All you need is the package and a API key, and you're set. s324 00:31:29.840 --> 00:31:37.120 Um, main takeaways, you know, an agent is a model plus a harness for this talk, at least. s325 00:31:37.120 --> 00:31:41.640 Uh, better harness equals better performance, especially for weaker models. s326 00:31:41.640 --> 00:31:43.920 And then, how do you build a better harness? s327 00:31:43.920 --> 00:31:52.120 Give it tools, make it safe, make it autonomous, make it reason, give it subagents, and have it self-optimize. s328 00:31:52.120 --> 00:31:52.840 Thank you so much. s329 00:31:52.840 --> 00:32:01.160 Again, my name is Aditya Parikh, check out agency agencylang.com, uh, and then please follow me on Blue Sky, if you wish. s330 00:32:01.160 --> 00:32:01.680 Thank you. s331 00:32:01.680 --> 00:32:03.800 Take care.