WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=CLttOU7n6sI NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/CLttOU7n6sI.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.480 --> 00:00:01.160 Hi everyone. s2 00:00:01.160 --> 00:00:06.160 My name is Andrew Dumont and I work on AI engineering at Watershed, the sustainability air platform. s3 00:00:06.160 --> 00:00:13.680 At Watershed, I work on AI for building product carbon footprints and in turn measuring the emissions associated with all the things that a company buys and sells. s4 00:00:13.680 --> 00:00:22.680 Sustainability, the vertical that Watershed is in, is one with a ton of expert judgment calls spread throughout it and this has made building and deploying agents both exciting and challenging. s5 00:00:22.680 --> 00:00:35.320 In this talk, I'll go deep on one of the tasks within sustainability that we spend quite a bit of time working on and share learnings from our work deploying coding agents on that task and why do do I doing so has required that we respect the process. s6 00:00:35.480 --> 00:00:46.000 So just to give a bit of context on sustainability as a vertical, we need to answer questions like what are the emissions attributable to one bottle of wine or which method should be used to allocate the emissions to various co-products s7 00:00:46.000 --> 00:00:50.800 when there are multiple sellable co-products that come from the same industrial process. s8 00:00:50.800 --> 00:00:58.560 In these cases, there are many ways to get the right answer the wrong way and there are also many right answers that experts will disagree on. s9 00:00:58.560 --> 00:01:08.000 And so, you have to verify the process in addition to the answer because the answer is really only justified in so far as it the process that produced that answer is correct. s10 00:01:08.000 --> 00:01:24.840 One example of this comes from a study in 2020 where six experts were given the exact same data on the exact same bottle of wine and despite having all access to the exact same things, they came to answers that varied by up to 50%. s11 00:01:24.840 --> 00:01:38.600 Their expert judgment were all correct in a sense, but purely validating the answer here is not sufficient to know that the answer produced by a system trying to mimic those experts is itself correct. s12 00:01:39.120 --> 00:01:47.360 So with that context on sustainability broadly and what kind of things that we need to answer, let's zoom into a specific task that we'll be talking about for the remainder of the talk. s13 00:01:47.360 --> 00:01:50.480 That task is to help users edit complex graphs. s14 00:01:50.480 --> 00:01:54.800 Each of these graphs represents the supply chain of a single product. s15 00:01:54.800 --> 00:01:59.280 For example, on the right here, you can see the graph represents dark wash jeans. s16 00:01:59.280 --> 00:02:11.680 Upstream of that is the assembly of that jeans from the denim thread, labels, uh and zipper, along with the energy, transportation, and packaging that's required to move uh move that through the supply chain, and create s17 00:02:11.680 --> 00:02:14.200 uh and run those industrial processes. s18 00:02:14.200 --> 00:02:25.760 Each graph here represents the entire flow of all of those things moving through the supply chain, and is comprised of thousands of nodes, each with rich metadata describing the materials and processing at each step. s19 00:02:26.520 --> 00:02:31.560 And so, when we first tried to solve this problem a little over a year ago, it worked decently well on one graph. s20 00:02:31.560 --> 00:02:39.120 It did have some problems, but we gave our React agent highly specified tools for exploring and interacting with the graph via function calls. s21 00:02:39.120 --> 00:02:42.840 And even while it worked, a few problems were already apparent. s22 00:02:42.840 --> 00:02:53.720 Um it lacked consistency where it could struggle to explore sufficiently, and those tool calls on a single graph took a lot of context as it read deeply across these nodes with lots of metadata. s23 00:02:53.720 --> 00:03:00.120 But then, when we tried to scale it up to many graphs, or frankly even just a few graphs, it absolutely broke. s24 00:03:00.120 --> 00:03:10.680 The lack of consistency was greatly magnified with the agent taking one approach in one graph, a different approach in a second graph, and completely forgetting about the third graph, or even to handle it at all. s25 00:03:10.680 --> 00:03:18.280 And exploration itself also became a huge bottleneck because it took many tool calls to explore and operate across these graphs at scale. s26 00:03:18.280 --> 00:03:29.640 And it also worsened the context problem because these tool calls really gobbled up context both on the edit side where it actually needed to like make some changes, and on the exploration side where it needed to figure out what changes to make. s27 00:03:29.640 --> 00:03:39.959 And worse, the agent then really started to hallucinate different parts of the schema as those contexts got eaten, and despite those specialized tools, this led to retries and ultimately errors. s28 00:03:39.959 --> 00:03:48.760 In data terms, the task now comprised tens or hundreds of graphs, and tens to hundreds of thousands of nodes that the agent had to operate over. s29 00:03:48.760 --> 00:03:54.080 So, time passed, and coding agents got way better, and we thought these will work great. s30 00:03:54.080 --> 00:03:56.120 So, naturally, we swapped them in. s31 00:03:56.120 --> 00:04:01.800 And swapping in a coding agent gave us like three really important outcomes that we were missing before. s32 00:04:01.800 --> 00:04:04.280 First, it gave the agent the ability to delight users. s33 00:04:04.280 --> 00:04:11.040 The agent could start to find clever ways to solve these under-specified and kind of not fully formed problems. s34 00:04:11.040 --> 00:04:14.680 It could also create fancy visualizations on the fly by writing code to do that. s35 00:04:14.680 --> 00:04:20.680 And it could even answer related questions because it had kind of the full power of the coding agent in this environment. s36 00:04:20.680 --> 00:04:26.280 And what from our end we saw the agent could explore and now edit way more efficiently than before. s37 00:04:26.280 --> 00:04:28.520 It could write loops over graphs and nodes. s38 00:04:28.520 --> 00:04:33.160 It could write scripts to unpack and summarize the node content underneath it all. s39 00:04:33.160 --> 00:04:40.240 Basically, following the same pattern of data exploration that goes on in kind of agentic data science workflows. s40 00:04:40.240 --> 00:04:45.919 And beyond that, it also gave it the flexibility and power to do stuff outside what we even designed it for, which was really exciting. s41 00:04:45.919 --> 00:04:55.520 And we're to this day consistently finding entirely new use cases via new questions users ask and new things that they want to try with this agent. s42 00:04:55.760 --> 00:04:56.919 We were really excited. s43 00:04:56.919 --> 00:04:58.160 We put it out into the world. s44 00:04:58.160 --> 00:05:00.800 We started to write a bunch of evals for it. s45 00:05:00.800 --> 00:05:04.800 And we quickly learned that unconstrained code is quite scary. s46 00:05:04.800 --> 00:05:11.800 If you're watching this, you've probably had that experience with cloud code where it's gone a bit haywire towards achieving whatever goal you set it out to do. s47 00:05:11.800 --> 00:05:19.000 It reached for something you didn't think it could or should reach for or even had access or found a way to access something you didn't think it could. s48 00:05:19.000 --> 00:05:24.040 Um for us, this looks like the agent will find creative ways to pretty much solve any problem. s49 00:05:24.040 --> 00:05:31.520 In our case, we saw it write Python when we expected TypeScript and and instructed it to write TypeScript because it found Python on the virtual machine that we had given it. s50 00:05:31.520 --> 00:05:45.120 Or it would directly edit these graph artifacts underlying the data without leaving any lineage behind by just like directly modifying um parameters or data rather than actually like writing code to effectively do that. s51 00:05:45.120 --> 00:05:50.800 Second, the agent actually started to gaslight users sometimes saying it had made edits when it hadn't. s52 00:05:50.800 --> 00:05:57.160 Um it would write code that did that it thought was going to do make like have the effect that it wanted to. s53 00:05:57.160 --> 00:05:58.840 And then it said, "I'm done. s54 00:05:58.840 --> 00:06:00.240 Everything's Everything's working. s55 00:06:00.240 --> 00:06:01.520 It's exactly what you said." s56 00:06:01.520 --> 00:06:03.320 And the edits were not actually made. s57 00:06:03.320 --> 00:06:10.280 And so it it kind of lied to users or at least let them to believe that the edits were done as they expected it to be done. s58 00:06:10.280 --> 00:06:16.440 And then lastly, once the agent had completed its task, it was now much harder to review its work. s59 00:06:16.440 --> 00:06:19.919 Um manual review of code is not something that are is in our users' wheelhouse. s60 00:06:19.919 --> 00:06:24.120 They are not software engineers or may or may not have had experience with code before. s61 00:06:24.120 --> 00:06:35.240 And when the agent is right for the wrong reasons, which happens uh or can happen frequently in our in sustainability broadly, it's really hard to check its work without going and reading that code directly. s62 00:06:35.880 --> 00:06:39.760 And so this process is not the answer. s63 00:06:39.760 --> 00:06:41.800 Story is really not new. s64 00:06:41.800 --> 00:06:50.080 Um a recent paper uh in from 2026, the Open Proof Corpus, showed that there can be quite a big gap between the correct final answer and and the correct proof. s65 00:06:50.080 --> 00:06:52.040 Though that is, you know, potentially closing. s66 00:06:52.040 --> 00:06:56.320 But And this is in math where there is like a fully verifiable answer. s67 00:06:56.320 --> 00:07:00.480 And so this We view this problem as even worse where you can't fully verify the answer. s68 00:07:00.480 --> 00:07:10.320 And this has been found in the agents what reward hack um even as something as complex as the Erdos problems, um you're going to get many, many pages of of s69 00:07:10.320 --> 00:07:25.880 errors in your proof which have which lead nowhere um to impossible bench to um other kind of areas in math where this can where they can The authors of this paper, Beyond Correctness, conclude that this can pose significant risk for critical applications. s70 00:07:25.880 --> 00:07:28.360 Now, all this is not to say that we shouldn't use coding agents. s71 00:07:28.360 --> 00:07:30.400 We don't want to constrain how the agent reasons. s72 00:07:30.400 --> 00:07:35.960 We get so many benefits from these powerful models, but we also can't perfectly verify the answer in our case. s73 00:07:35.960 --> 00:07:39.320 So we don't even know when the final answer is definitely correct. s74 00:07:39.320 --> 00:07:43.480 Uh because these expert judgment calls are ever present our space. s75 00:07:43.480 --> 00:07:50.280 So, the main lever left is validating the process and making sure that we are following it in a way we expect. s76 00:07:50.280 --> 00:07:55.760 And to do that, we frame it as constraining the effects, not the expression. s77 00:07:55.760 --> 00:07:59.200 So, our task begins with a user request. s78 00:07:59.200 --> 00:08:04.880 And from this, the agent can then go off and freely write code to execute on that user request. s79 00:08:04.880 --> 00:08:16.000 However, we require that all the critical code, really the stuff that edits the graph, goes through a filter of this typed SDK that we've put together where we can lint and check for errors. s80 00:08:16.000 --> 00:08:24.600 And then we own the final execution step to guarantee that we create these typed objects that we can commit as the actual edits to the graph. s81 00:08:24.600 --> 00:08:27.480 And these are then faithful to the process. s82 00:08:27.480 --> 00:08:32.880 And all of this together means that the entire process is valid, traceable, and replayable. s83 00:08:32.880 --> 00:08:40.120 And if it's not, we can reject and retry and send it back to the agent to let it know that something has gone awry. s84 00:08:40.120 --> 00:08:44.720 So, let's go a little deeper on the typed SDK and the deterministic execution. s85 00:08:44.720 --> 00:08:47.480 So, first, our SDK is the only door. s86 00:08:47.480 --> 00:08:57.839 What we give to the agent in our case is a TypeScript SDK with all the edit primitives the agent needs to make changes to graphs, as well as explore and interact with the kind of graph objects. s87 00:08:57.839 --> 00:09:09.440 This for us enforces which fields are editable versus which are derived from other fields, so the agent can't create conflicts with itself where it edits just the target thing that it cares about, but not the thing that creates that target thing. s88 00:09:09.440 --> 00:09:17.480 And it also guarantees that it objects that we expect it to so that we can actually like make use of the outputs of this in a deterministic way. s89 00:09:17.480 --> 00:09:29.600 Now, of course, this does require teaching the agent how to actually make use of this SDK, but this is the same kind of pattern that we see with teaching an agent to like write code in your code base where you tell it how it works. s90 00:09:29.600 --> 00:09:32.360 You can, you know, do the prompt and all of that kind of stuff. s91 00:09:32.360 --> 00:09:38.560 The agent also has full access to the docs and the code underlying the SDK if it really needs to go and read that as well. s92 00:09:38.560 --> 00:09:43.200 Just for a brief example on the right here, you can see an example with a few pieces of the SDK on display. s93 00:09:43.200 --> 00:09:46.200 We import um from our API. s94 00:09:46.200 --> 00:09:51.120 We The agent has to then define kind of a top-level edit function with a well-defined name. s95 00:09:51.120 --> 00:09:56.480 It then can use uh find nodes by exact name to find specific nodes that it's looking for in the graph. s96 00:09:56.480 --> 00:10:02.280 It can write assertions that we give it to make sure that um its code fails early um as well. s97 00:10:02.280 --> 00:10:12.360 And then it can use uh specific mutator functions like set rate and edit node to actually interact with the graph in a way that will produce those kind of typed objects that we care about. s98 00:10:12.360 --> 00:10:20.280 So, that's the SDK, but the actual guarantee here is when we own the deterministic execution. s99 00:10:20.280 --> 00:10:26.000 So, even with the typed SDK as our entry point, that really only guides the agent towards the desired end state. s100 00:10:26.000 --> 00:10:31.080 And the real guarantee comes from the final script that we orchestrate on agent completion. s101 00:10:31.080 --> 00:10:42.600 This run executor script um it is called when the agent has kind of completed running, and we think that we're in a like almost completed state or a ready to be completed state. s102 00:10:42.600 --> 00:10:49.600 And what that script does is it starts by linting the agent code, where we can immediately send it back to the agent if something is wrong. s103 00:10:49.600 --> 00:10:52.920 Better to kind of fail early rather than fail later. s104 00:10:52.920 --> 00:10:54.120 We can also detect conflicts. s105 00:10:54.120 --> 00:11:02.320 This is where sometimes the agent in one part of the code edited something in another part edited the same thing or something that depended on the first one and didn't realize that that happened. s106 00:11:02.320 --> 00:11:10.960 Because we have these guarantees, by giving the agent this well-defined structure, we can detect those conflicts for it so that we can send it back to the agent. s107 00:11:10.960 --> 00:11:15.920 We then run the agent edited code, and that means that we can then validate the output artifacts. s108 00:11:15.920 --> 00:11:20.360 Of course, if the code itself fails to run, we can send that back to the agent, too. s109 00:11:20.360 --> 00:11:32.040 And then finally, once we have the validated output artifacts, we can create a well-structured review artifact that makes it easy to review what the agent did without having to actually ever go read the underlying code that produced it. s110 00:11:32.040 --> 00:11:39.520 To give you a flavor of what that looks like, here's a little snippet from an emissions report where the graph edit function impact analysis ran on 50 graphs. s111 00:11:39.520 --> 00:11:51.200 Um there were two functions that it applied that produced 749 edit actions, and it ultimately in this case reduced the overall emissions by 45.6% and we can even go to s112 00:11:51.200 --> 00:11:54.640 go to see there are two code there are two edits here. s113 00:11:54.640 --> 00:11:58.800 Um one kind of did a big chunk and the other one made a very small change. s114 00:11:58.800 --> 00:12:10.000 Maybe you'd go look at what this actually did, and then you can even go at a graph level and say, "Okay, for this graph, like what was the option one and which nodes did it edit within that graph to get a sense of s115 00:12:10.000 --> 00:12:11.960 um what actually happened under the hood?" s116 00:12:11.960 --> 00:12:17.440 So, you can see all of this without actually having to go read the low-level code. s117 00:12:17.880 --> 00:12:30.200 So, to return to those three problems that uh we started with, now that we have these kind of two broad concepts that are part of the harness, um when the agent will find creative ways to solve every problem, this is now s118 00:12:30.200 --> 00:12:31.960 a great thing, but we've limited it a little bit. s119 00:12:31.960 --> 00:12:41.960 These invalid actions that the agent will take on its route to solve whatever problem we presented are actually prevented with the type dust SDK as the only way to properly edit the graph, s120 00:12:41.960 --> 00:12:50.640 and then we can guarantee that the graph was only edited through the SDK by owning the deterministic execution and the validation of that written code. s121 00:12:50.640 --> 00:13:02.960 The second problem about the agent saying that it made an edit when actually that the edit didn't go through in the way that it or the user expected, these false reports are caught via that deterministic execution so that we can surface those back to the agent s122 00:13:02.960 --> 00:13:05.800 um to make sure that it's doing what it claimed. s123 00:13:05.800 --> 00:13:20.560 And finally, our users never have to look at the code because the deterministic execution system produces artifacts in an expected form, and we can build verification against those to catch both the kind of good outcomes and the errors that are now surfaced clearly s124 00:13:20.560 --> 00:13:23.080 to both the user um and the AI. s125 00:13:23.080 --> 00:13:35.560 So, even when it hasn't maybe gotten exactly the right answer or the one that the user expected, it's easy to follow its logic and kind of get to that um and loop back on itself. s126 00:13:35.880 --> 00:13:45.280 Now, the harness is all well and good as it's kind of adds constraints, but I also want to emphasize that even with those great guarantees, you still need to hill climb to get better at the task itself. s127 00:13:45.280 --> 00:13:54.640 Since we started tracking how well the agent did at these complex edits, where each task here um in our eval set is a collection of graphs and a goal, s128 00:13:54.640 --> 00:14:00.560 um we've been able to improve our outcomes from about 43% to 92% on our set of internal evals. s129 00:14:00.560 --> 00:14:03.720 And to achieve that, all the typical approaches were still effective. s130 00:14:03.720 --> 00:14:09.640 We did a ton of prompt improvements, like rewriting the system prompt and all the kind of skills that the agent had access to. s131 00:14:09.640 --> 00:14:18.640 We've added few shot examples to both teach and coach the agent how to make use of the SDK and work with these different classes of tasks that we expect. s132 00:14:18.640 --> 00:14:28.040 We also had to make the tools, the SDK itself, and the tools the agent had access to better fit for use by improving their kind of ergonomics and how obvious they were for the agent to use. s133 00:14:28.040 --> 00:14:45.800 We also broke the task down through doing this, breaking up the um overall problem to a plan and execute kind of standard loop, as well as teaching the agent some of that expert judgment that is endemic in our space, making that easier for the agent to operate on it and to elicit that expert judgment from the user. s134 00:14:45.800 --> 00:14:57.200 But the important thing here is even though we've kind of improved the accuracy here, which is amazing, and it's great that we'll now kind of get better at that first time a user goes through the full workflow in the task, s135 00:14:57.200 --> 00:15:08.960 um even when the agent errors, or even our ground truth is, you know, one of those points on the possible range of expert judgments, we can still guarantee and validate the um process itself. s136 00:15:09.400 --> 00:15:14.400 So, to bring it all together in a domain full of expert judgments, you must respect the process, too. s137 00:15:14.400 --> 00:15:18.240 And for that, you really need guarantees about the process. s138 00:15:18.240 --> 00:15:24.720 We all know coding agents are necessary for complex tasks, and they will continue to be necessary and extremely useful. s139 00:15:24.720 --> 00:15:30.640 But those power that power comes with many risks that are like good to go and clear eyed about. s140 00:15:30.640 --> 00:15:37.000 For tasks with complex data and judgment calls, we now think that the harness should do a few important things. s141 00:15:37.000 --> 00:15:43.840 First, it should give the agent well scoped primitives to interact with the kind of external system that it's actually supposed to work with. s142 00:15:43.840 --> 00:15:50.120 Um free freely operating code can, you know, lead to a lot of the things that we talked about earlier. s143 00:15:50.120 --> 00:16:00.400 And this really allows the builders of the system to allow and disallow specific things and exercise the full power of these coding agents while still trusting the process that they're working in. s144 00:16:00.400 --> 00:16:04.400 I think the harness should also maintain full control of the final execution. s145 00:16:04.400 --> 00:16:11.040 These very smart agents may declare victory in an unexpected way from what you or your user really want them to declare. s146 00:16:11.040 --> 00:16:17.440 And so it's important to be able to constrain and validate that the agent has actually done what it said it's done. s147 00:16:17.440 --> 00:16:23.160 And finally, you should use that deterministic final outcome to produce outputs that are easy to validate even for non-coders. s148 00:16:23.160 --> 00:16:26.440 The code is kind of just the means to an end. s149 00:16:26.440 --> 00:16:36.160 And then of course, the kind of standard context and prompt engineering will of course remain critical to better scoping and defining that task and making it clear what the agent needs to do. s150 00:16:36.160 --> 00:16:36.880 That's it for me. s151 00:16:36.880 --> 00:16:40.160 Again, I'm Andrew Dumit at Watershed, the sustainability AI platform. s152 00:16:40.160 --> 00:16:43.520 Thank you again for your time and attention.