WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=dAoOYFx7ZYw NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/dAoOYFx7ZYw.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.520 --> 00:00:14.280 Thanks for everyone for coming. s3 00:00:14.280 --> 00:00:16.360 So, my name is Mikail. s4 00:00:16.360 --> 00:00:29.240 I head up the open source group at Arise AI and we work on a fully open source observability and evaluation framework and we also work on our own agent. s5 00:00:29.240 --> 00:00:36.360 So, today we want I want to kind of walk you through what Code Mode is and our journey embracing Code Mode. s6 00:00:36.360 --> 00:00:44.200 So, if you look back in the last 30 years of software development, you could kind of segment software into two different categories. s7 00:00:44.200 --> 00:00:49.480 You could think of there's software built for software engineers and software for non-engineers. s8 00:00:49.480 --> 00:00:55.880 Take for example, you have you know, you're asked to get all the invoices out of your email inbox. s9 00:00:55.880 --> 00:00:57.640 If you're a programmer, you could write a script, s10 00:00:57.640 --> 00:00:57.960 [snorts] s11 00:00:57.960 --> 00:01:03.360 you can go in there, you can dynamically like see if there's an attachment, download it, put it into a folder. s12 00:01:03.360 --> 00:01:12.320 It might be a you know, 50 lines of code and you get this tool that you can use versus if you were a non-technical, you'd have to go into these inboxes, s13 00:01:12.320 --> 00:01:14.000 you have to manually download it. s14 00:01:14.000 --> 00:01:20.160 You're restricted by the interface of the software and you had to kind of buy that software. s15 00:01:20.160 --> 00:01:26.040 And the thing that's really interesting about things like Cloud Code is that that boundary is completely dissolving, right? s16 00:01:26.040 --> 00:01:32.400 Basically, anything you need, the coding agent can actually build that software for you dynamically. s17 00:01:32.400 --> 00:01:35.520 And the labs and the foundation model providers know this, right? s18 00:01:35.520 --> 00:01:40.920 If you look at all the benchmarks like SWE-bench, that is what these models are being trained to do. s19 00:01:40.920 --> 00:01:52.040 And so, doesn't it make sense basically to align the agents you're building to be able to kind of build the software you need dynamically to dissolve that boundary. s20 00:01:52.200 --> 00:01:58.760 So, before I get to started in Code Mode, let's look at kind of what an agent looks like. s21 00:01:58.760 --> 00:02:04.280 So, hopefully you've built this uh uh kind of agent before. s22 00:02:04.280 --> 00:02:17.080 You give an agent kind of a huge catalog of tools, and then iteratively a user question comes in, it calls the tools that get pushed in the context window, it reasons about it, and then it calls another tool s23 00:02:17.080 --> 00:02:19.160 that get pushed in the context window. s24 00:02:19.160 --> 00:02:21.959 And hopefully most of you have built an agent like this. s25 00:02:21.959 --> 00:02:26.080 Uh certainly if you have a certain set of tools, this can work pretty well. s26 00:02:26.080 --> 00:02:29.520 But, we're now in that age where we want a lot more. s27 00:02:29.520 --> 00:02:31.680 We want much more emergent behavior. s28 00:02:31.680 --> 00:02:34.440 We want to put a lot more context into the agent. s29 00:02:34.440 --> 00:02:36.760 We want to have longer horizons. s30 00:02:36.760 --> 00:02:40.640 And so, this is where you kind of get this bloat inside of the context. s31 00:02:40.640 --> 00:02:46.720 You'll see there's a lot of tools that get added into the context window that might not be relevant to the task. s32 00:02:46.720 --> 00:02:53.520 Um maybe the tool call is not responding with a lot of junk, or maybe needs to paginate through data. s33 00:02:53.520 --> 00:02:56.080 And so, you deal with this kind of context rot. s34 00:02:56.080 --> 00:03:02.800 You also deal with multi-turn, a lot of turns, as a result, and you're going to have to deal with compaction. s35 00:03:02.800 --> 00:03:07.040 So, luckily CodeMon solves a lot of these things. s36 00:03:07.040 --> 00:03:09.959 It's kind of an alternative architecture. s37 00:03:09.959 --> 00:03:23.519 Um instead of, let's say, just giving it a huge index or a menu of tools, you give it the ability to search for the specific tool, and then you also give it the ability to execute the tool. s38 00:03:23.519 --> 00:03:30.640 But, you don't give it just the ability to execute the tool, you actually give it a programming environment to execute those tools in different types of flows. s39 00:03:30.640 --> 00:03:36.600 So, they can do it sequentially, they can do them in parallel, um can put conditionals around it. s40 00:03:36.600 --> 00:03:47.400 Um and the nice thing about this is you can call multiple tools in one turn, and also return only what's necessary to push back into the context window of the model. s41 00:03:47.400 --> 00:03:59.800 And so, to kind of summarize it, put those these two things hypotheticals side by side, traditional tool calling, you know, you accumulate context window blow every time you call a tool. s42 00:03:59.800 --> 00:04:03.360 That's obviously going to, you know, add to latency and cost. s43 00:04:03.360 --> 00:04:05.720 It's also going to increase increase turns. s44 00:04:05.720 --> 00:04:10.320 So, the time your user takes to get to value increases, right? s45 00:04:10.320 --> 00:04:12.560 So, has a lot of overhead. s46 00:04:12.560 --> 00:04:23.480 Um, on on the flip side, code mode potentially, when done well, can unlock a lot of abilities because it can call as many as it wants in a single turn s47 00:04:23.480 --> 00:04:31.680 and it only has to put in exactly what's necessary into the context window to solve its problem and you can also get a reduction in turn. s48 00:04:31.680 --> 00:04:36.600 So, that's sort of the the goal of of code mode. s49 00:04:36.600 --> 00:04:40.840 And so, let's just kind of go over some of the reasons why code mode works. s50 00:04:40.840 --> 00:04:46.640 Code mode works because one, you don't have to give it the giant catalog of tools up front. s51 00:04:46.640 --> 00:04:52.000 It can just pick and choose the tools that it needs and understand those tools and then execute just those. s52 00:04:52.000 --> 00:04:54.520 So, there's a progressive disclosure of tools. s53 00:04:54.520 --> 00:04:59.040 Uh, certainly there's things like tool search now also built into the foundation model providers. s54 00:04:59.040 --> 00:05:00.480 There's also the composability. s55 00:05:00.480 --> 00:05:10.320 There's a huge amount of benefit to not being able to being able to call tools in different sequences, either in sequence or parallel, fan out, chaining them together. s56 00:05:10.320 --> 00:05:13.480 You can create much more complicated and efficient things. s57 00:05:13.480 --> 00:05:20.760 If you've heard of obviously Claude's dynamic workflows is built on this composability but composability of sub agents. s58 00:05:20.760 --> 00:05:23.200 I already mentioned sort of token efficiency. s59 00:05:23.200 --> 00:05:30.720 You don't have to spit out all the tools outputs and worry about the tools outputs being shoved into the context window. s60 00:05:30.720 --> 00:05:38.120 The model can dynamically kind of trim and search for certain areas and only put what it needs back into its context window. s61 00:05:38.120 --> 00:05:45.640 And the fourth one, which is the kind of the one of the major reasons we've embraced code mode, is this idea of emergent behavior, being able to build its own tools, right? s62 00:05:45.640 --> 00:06:01.800 So, for example, traditional with traditional tool calling, if I said for example like take all these records and remove personally identifiable information, it wouldn't be able to do that because it doesn't have a tool, or it might try to do it inside of its its context window, but then you're leaking credit card numbers s63 00:06:01.800 --> 00:06:03.600 in the context window, and that's not great. s64 00:06:03.600 --> 00:06:11.000 So, the emerging behavior of being able to create its own tools has a huge amount of benefit. s65 00:06:11.440 --> 00:06:15.600 So, here are some ways in which we've embraced Code mod. s66 00:06:15.600 --> 00:06:25.640 Uh so, we build our own agent uh and harness inside of Phoenix, our open source uh our observability platform, and we started with Bash. s67 00:06:25.640 --> 00:06:38.800 You know, Bash is a great tool of being able to compose together things like uh you know, said and grep to be able to uh compose together, you know, built-in Linux primitives with also our own CLI s68 00:06:38.800 --> 00:06:42.160 and be able to offload those into files for later use. s69 00:06:42.160 --> 00:06:49.919 We also added um you know, a Python's sandbox, and we hoisted all our rest CRUD endpoints. s70 00:06:49.919 --> 00:07:04.480 Those are like perform getting traces or uh manipulating data sets, and we put that into this coding environment so that it can compose those create, read, update, delete operations, which unlocks like a whole category of things. s71 00:07:04.480 --> 00:07:09.840 And we also added one additional one, which is the ability to write SQL. s72 00:07:09.840 --> 00:07:17.120 And you know, text of SQL, you've obviously like heard like that probably many times before, but this also unlocks a lot of capabilities. s73 00:07:17.120 --> 00:07:22.160 The reason why is not everything fits into the tool category of just creating, updating, and deleting. s74 00:07:22.160 --> 00:07:31.360 Maybe you need to be able to do aggregate queries like tell me all the LM spans that have P95 latency of X, Y, and Z. You don't want to have to paginate s75 00:07:31.360 --> 00:07:33.560 through all your traces to do that. s76 00:07:33.560 --> 00:07:39.800 You want it the LM to be able to pinpoint find exactly what you're looking for. s77 00:07:39.960 --> 00:07:43.400 This is kind of a a neat and creative way we've also used code mode. s78 00:07:43.400 --> 00:07:48.960 So, we've actually also created a sort of a programming paradigm in our UI that's bound to our UI. s79 00:07:48.960 --> 00:07:57.840 So, we've been able to kind of be able to give it the ability to change the time range or be able to read data from the data store inside of the front end. s80 00:07:57.840 --> 00:08:09.520 And what's really cool about this is you kind of almost get this computer loose use like emergent behavior because we're able to put that programming paradigm in a web worker that has a two-way data binding s81 00:08:09.520 --> 00:08:19.000 directly into our browser and it can kind of do complex UI interactions for you and it's really pretty fun to to watch it uh cook. s82 00:08:19.160 --> 00:08:22.720 So, code mode is not uh obviously for free. s83 00:08:22.720 --> 00:08:25.480 Um you're dealing with LLM generated code. s84 00:08:25.480 --> 00:08:30.000 If you've been at to the the Anthropic talks, you certainly probably seen some of these. s85 00:08:30.000 --> 00:08:31.360 You're dealing with resources. s86 00:08:31.360 --> 00:08:39.039 So, if you're working with hosted sandboxes, you have to worry about CPU, memory, how long those sandboxes stay around. s87 00:08:39.039 --> 00:08:40.680 Networking is a huge one. s88 00:08:40.680 --> 00:08:44.880 Maybe you want it to be able to go out to the web, but you never know what's out in the web. s89 00:08:44.880 --> 00:08:48.760 And so, you have to carefully think about network egress. s90 00:08:48.760 --> 00:08:51.400 Uh secrets is a hot topic. s91 00:08:51.400 --> 00:08:56.080 I think every sandbox provider is trying to come up with a different vaulting strategy. s92 00:08:56.080 --> 00:09:07.880 But, let's say you want to use Jev inside of your sandbox like Pi uh durable just shipped, you know, that's a crazy cool idea uh to be able to make fast decisions inside of a sandbox, s93 00:09:07.880 --> 00:09:12.120 but you have to make sure that those secrets don't come back into the context window. s94 00:09:12.120 --> 00:09:13.600 And then the other one is permissions. s95 00:09:13.600 --> 00:09:19.960 Uh there's some very interesting research around this as well, but typically permissions are overbroad. s96 00:09:19.960 --> 00:09:25.640 And so, maybe in the first five sets the permissions are correct, but in the sixth operation it is not. s97 00:09:25.640 --> 00:09:28.720 Or maybe in composition uh the permissions are not. s98 00:09:28.720 --> 00:09:34.960 And so, you have to kind of think of guard railing some of these things if that is necessary. s99 00:09:35.240 --> 00:09:38.080 There's also operational complexity. s100 00:09:38.080 --> 00:09:42.200 Uh certainly sandboxes don't boot uh all the time perfectly. s101 00:09:42.200 --> 00:09:45.800 And so you have to you know be able to deal with failures. s102 00:09:45.800 --> 00:09:51.960 They're harder to debug because now you're debugging code and you're not debug debugging individual tools. s103 00:09:51.960 --> 00:09:57.640 And tool parameter accuracy becomes a much bigger issue because the tool parameter now is code. s104 00:09:57.640 --> 00:09:59.800 Uh so what if something fails in the middle? s105 00:09:59.800 --> 00:10:04.160 Uh tool discovery is also still an issue. s106 00:10:04.160 --> 00:10:12.200 Previously if you did have a tool, the LM would up front know it, but now it might have to go through a handshake to be able to understand which tools s107 00:10:12.200 --> 00:10:15.040 are in its catalog of things to be able to do. s108 00:10:15.040 --> 00:10:24.360 And then this is sort of a big one, you know, you see code mode being injected into MCP, but then you see a lot more code mode being pushed also into harnesses. s109 00:10:24.360 --> 00:10:27.600 And now you have a double sandboxing uh problem. s110 00:10:27.600 --> 00:10:34.480 So sandboxing, you know, is going to return stuff out as serialized JSON and then it needs to be parsed and then put into another code mode. s111 00:10:34.480 --> 00:10:42.360 And so there's this like double duty that ends up happening that also can be on operational complexity. s112 00:10:42.880 --> 00:10:44.880 These are sort of open problems. s113 00:10:44.880 --> 00:10:48.440 Uh they're not like blockers, but durability is a problem. s114 00:10:48.440 --> 00:10:54.160 So let's say you build a giant piece of code that needs to execute and it breaks in the middle. s115 00:10:54.160 --> 00:10:56.640 You can't resume from that step. s116 00:10:56.640 --> 00:11:05.000 Um if determinism and mathematics are important to you, for example, it might actually force you to think about which programming language you're using. s117 00:11:05.000 --> 00:11:10.440 So JavaScript is something LLMs are really good at doing, but maybe not great for math. s118 00:11:10.440 --> 00:11:22.320 Um For example, uh code mode can be great way to do multimodal and manipulating images or creating bytes, but getting that out of the sandbox is quite tricky, right? s119 00:11:22.320 --> 00:11:25.760 Because you don't want to serialize the whole binary, the whole image. s120 00:11:25.760 --> 00:11:27.400 That's going to end up in the the context window. s121 00:11:27.400 --> 00:11:31.360 So you might have to do some sort of signed image handoff. s122 00:11:31.360 --> 00:11:44.360 And then the other one which is like the Frontier models are amazing at code mode, but if you're trying to do cost saving and you want to swap out a smaller model, its capabilities might not just quite um get there. s123 00:11:44.360 --> 00:11:48.440 So, how do we at Arize evaluate code mode? s124 00:11:48.440 --> 00:12:00.360 Um we use our own We dogfood our own platform, which is called Arize Phoenix, um and we run our agents through various permutations through something called the Harbor framework, s125 00:12:00.360 --> 00:12:13.400 which basically puts the agents into uh sandbox environments and runs them on clear instructions with network access turned off, and we run this multiple times in trials, so we eliminate a lot of the flake. s126 00:12:13.400 --> 00:12:20.360 And we build evals in the form of Harbor verifiers to make sure that the agent's doing the correct thing. s127 00:12:20.360 --> 00:12:23.520 And then, Harbor's sort of rooted in this RL framework. s128 00:12:23.520 --> 00:12:32.440 We give it a a one or a zero to indicate whether or not performs the task, and we test it across many different permutations. s129 00:12:32.440 --> 00:12:38.280 So, this is kind of why you want to test like this, because as I mentioned, you can get a lot of flake. s130 00:12:38.280 --> 00:12:42.040 Um sometimes it might work where you're dealing with an indeterminate system. s131 00:12:42.040 --> 00:12:52.600 It might take different trajectories uh depending how you do it, and we uh track pass at K of each individual task that we think it should pass. s132 00:12:52.600 --> 00:12:55.200 So, you know, how does this work? s133 00:12:55.200 --> 00:12:59.120 Uh this is kind of our results from uh running this Harbor run. s134 00:12:59.120 --> 00:13:07.000 You can see pre uh code mode and post code mode, we, you know, decreased the amount of tokens by 43%. s135 00:13:07.000 --> 00:13:12.680 The cost went down by 31%, and you know, there was some overhead with the sandbox, but the latency stayed pretty constant. s136 00:13:12.680 --> 00:13:23.520 So, we knew that across these multiple surfaces, whether or not it was cloud code or MCP or the CLI, we had accomplished something that uh was definitely better than it was before. s137 00:13:23.520 --> 00:13:25.000 How do we use this data? s138 00:13:25.000 --> 00:13:30.400 It's really uh a huge unlock if you have you both capture the trajectory and the evals. s139 00:13:30.400 --> 00:13:40.080 As you've seen with a lot of the other talks, this gives you a very great environment to reinforce and look at the behavior of these agents and be able to tell it how to hill climb. s140 00:13:40.080 --> 00:13:45.720 So, maybe, you know, first time we introduce a new tool, sometimes actually the efficiency goes down. s141 00:13:45.720 --> 00:13:54.600 And so, we can tell a coding agent why in these specific scenarios, what parts of the instruction are are guiding it in the wrong direction and we're able to hill climb these s142 00:13:54.600 --> 00:13:58.520 evals to be able to make our agents better. s143 00:13:58.880 --> 00:14:05.080 So, if you don't take anything other than this away, hopefully this will sum it up. s144 00:14:05.080 --> 00:14:10.800 We really like code mode at RI because of the fewer turns you get out of code mode. s145 00:14:10.800 --> 00:14:13.440 We want you to be able to bring any model for us. s146 00:14:13.440 --> 00:14:20.720 So, we want it to be cheaper and we're really excited about the emergent behavior and the long horizon tasks that our agents can do. s147 00:14:20.720 --> 00:14:29.720 And so, I hope this talk convinces you to give your agent a sandbox and then use one of our tools to evaluate it. s148 00:14:29.720 --> 00:14:30.520 And that's it. s149 00:14:30.520 --> 00:14:32.800 Thank you.