Respect The Process - Andrew Dumit, Watershed Technology Inc.

AI Engineer · 16 min · 152 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:00Hi everyone.
00:01My name is Andrew Dumont and I work on AI engineering at Watershed, the sustainability air platform.
00:06At Watershed, I work on AI for building product carbon footprints and in turn measuring the emissions associated with all the things that a company buys and sells.
00:13Sustainability, the vertical that Watershed is in, is one with a ton of expert judgment calls spread throughout it and this has made building and deploying agents both exciting and challenging.
00:22In this talk, I'll go deep on one of the tasks within sustainability that we spend quite a bit of time working on and share learnings from our work deploying coding agents on that task and why do do I doing so has required that we respect the process.
00:35So just to give a bit of context on sustainability as a vertical, we need to answer questions like what are the emissions attributable to one bottle of wine or which method should be used to allocate the emissions to various co-products
00:46when there are multiple sellable co-products that come from the same industrial process.
00:50In these cases, there are many ways to get the right answer the wrong way and there are also many right answers that experts will disagree on.
00:58And so, you have to verify the process in addition to the answer because the answer is really only justified in so far as it the process that produced that answer is correct.
01:08One example of this comes from a study in 2020 where six experts were given the exact same data on the exact same bottle of wine and despite having all access to the exact same things, they came to answers that varied by up to 50%.
01:24Their expert judgment were all correct in a sense, but purely validating the answer here is not sufficient to know that the answer produced by a system trying to mimic those experts is itself correct.
01:39So with that context on sustainability broadly and what kind of things that we need to answer, let's zoom into a specific task that we'll be talking about for the remainder of the talk.
01:47That task is to help users edit complex graphs.
01:50Each of these graphs represents the supply chain of a single product.
01:54For example, on the right here, you can see the graph represents dark wash jeans.
01:59Upstream of that is the assembly of that jeans from the denim thread, labels, uh and zipper, along with the energy, transportation, and packaging that's required to move uh move that through the supply chain, and create
02:11uh and run those industrial processes.
02:14Each graph here represents the entire flow of all of those things moving through the supply chain, and is comprised of thousands of nodes, each with rich metadata describing the materials and processing at each step.
02:26And so, when we first tried to solve this problem a little over a year ago, it worked decently well on one graph.
02:31It did have some problems, but we gave our React agent highly specified tools for exploring and interacting with the graph via function calls.
02:39And even while it worked, a few problems were already apparent.
02:42Um it lacked consistency where it could struggle to explore sufficiently, and those tool calls on a single graph took a lot of context as it read deeply across these nodes with lots of metadata.
02:53But then, when we tried to scale it up to many graphs, or frankly even just a few graphs, it absolutely broke.
03:00The lack of consistency was greatly magnified with the agent taking one approach in one graph, a different approach in a second graph, and completely forgetting about the third graph, or even to handle it at all.
03:10And exploration itself also became a huge bottleneck because it took many tool calls to explore and operate across these graphs at scale.
03:18And it also worsened the context problem because these tool calls really gobbled up context both on the edit side where it actually needed to like make some changes, and on the exploration side where it needed to figure out what changes to make.
03:29And worse, the agent then really started to hallucinate different parts of the schema as those contexts got eaten, and despite those specialized tools, this led to retries and ultimately errors.
03:39In data terms, the task now comprised tens or hundreds of graphs, and tens to hundreds of thousands of nodes that the agent had to operate over.
03:48So, time passed, and coding agents got way better, and we thought these will work great.
03:54So, naturally, we swapped them in.
03:56And swapping in a coding agent gave us like three really important outcomes that we were missing before.
04:01First, it gave the agent the ability to delight users.
04:04The agent could start to find clever ways to solve these under-specified and kind of not fully formed problems.
04:11It could also create fancy visualizations on the fly by writing code to do that.
04:14And it could even answer related questions because it had kind of the full power of the coding agent in this environment.
04:20And what from our end we saw the agent could explore and now edit way more efficiently than before.
04:26It could write loops over graphs and nodes.
04:28It could write scripts to unpack and summarize the node content underneath it all.
04:33Basically, following the same pattern of data exploration that goes on in kind of agentic data science workflows.
04:40And beyond that, it also gave it the flexibility and power to do stuff outside what we even designed it for, which was really exciting.
04:45And we're to this day consistently finding entirely new use cases via new questions users ask and new things that they want to try with this agent.
04:55We were really excited.
04:56We put it out into the world.
04:58We started to write a bunch of evals for it.
05:00And we quickly learned that unconstrained code is quite scary.
05:04If you're watching this, you've probably had that experience with cloud code where it's gone a bit haywire towards achieving whatever goal you set it out to do.
05:11It reached for something you didn't think it could or should reach for or even had access or found a way to access something you didn't think it could.
05:19Um for us, this looks like the agent will find creative ways to pretty much solve any problem.
05:24In our case, we saw it write Python when we expected TypeScript and and instructed it to write TypeScript because it found Python on the virtual machine that we had given it.
05:31Or it would directly edit these graph artifacts underlying the data without leaving any lineage behind by just like directly modifying um parameters or data rather than actually like writing code to effectively do that.
05:45Second, the agent actually started to gaslight users sometimes saying it had made edits when it hadn't.
05:50Um it would write code that did that it thought was going to do make like have the effect that it wanted to.
05:57And then it said, "I'm done.
05:58Everything's Everything's working.
06:00It's exactly what you said."
06:01And the edits were not actually made.
06:03And so it it kind of lied to users or at least let them to believe that the edits were done as they expected it to be done.
06:10And then lastly, once the agent had completed its task, it was now much harder to review its work.
06:16Um manual review of code is not something that are is in our users' wheelhouse.
06:19They are not software engineers or may or may not have had experience with code before.
06:24And when the agent is right for the wrong reasons, which happens uh or can happen frequently in our in sustainability broadly, it's really hard to check its work without going and reading that code directly.
06:35And so this process is not the answer.
06:39Story is really not new.
06:41Um a recent paper uh in from 2026, the Open Proof Corpus, showed that there can be quite a big gap between the correct final answer and and the correct proof.
06:50Though that is, you know, potentially closing.
06:52But And this is in math where there is like a fully verifiable answer.
06:56And so this We view this problem as even worse where you can't fully verify the answer.
07:00And this has been found in the agents what reward hack um even as something as complex as the Erdos problems, um you're going to get many, many pages of of
07:10errors in your proof which have which lead nowhere um to impossible bench to um other kind of areas in math where this can where they can The authors of this paper, Beyond Correctness, conclude that this can pose significant risk for critical applications.
07:25Now, all this is not to say that we shouldn't use coding agents.
07:28We don't want to constrain how the agent reasons.
07:30We get so many benefits from these powerful models, but we also can't perfectly verify the answer in our case.
07:35So we don't even know when the final answer is definitely correct.
07:39Uh because these expert judgment calls are ever present our space.
07:43So, the main lever left is validating the process and making sure that we are following it in a way we expect.
07:50And to do that, we frame it as constraining the effects, not the expression.
07:55So, our task begins with a user request.
07:59And from this, the agent can then go off and freely write code to execute on that user request.
08:04However, we require that all the critical code, really the stuff that edits the graph, goes through a filter of this typed SDK that we've put together where we can lint and check for errors.
08:16And then we own the final execution step to guarantee that we create these typed objects that we can commit as the actual edits to the graph.
08:24And these are then faithful to the process.
08:27And all of this together means that the entire process is valid, traceable, and replayable.
08:32And if it's not, we can reject and retry and send it back to the agent to let it know that something has gone awry.
08:40So, let's go a little deeper on the typed SDK and the deterministic execution.
08:44So, first, our SDK is the only door.
08:47What we give to the agent in our case is a TypeScript SDK with all the edit primitives the agent needs to make changes to graphs, as well as explore and interact with the kind of graph objects.
08:57This for us enforces which fields are editable versus which are derived from other fields, so the agent can't create conflicts with itself where it edits just the target thing that it cares about, but not the thing that creates that target thing.
09:09And it also guarantees that it objects that we expect it to so that we can actually like make use of the outputs of this in a deterministic way.
09:17Now, of course, this does require teaching the agent how to actually make use of this SDK, but this is the same kind of pattern that we see with teaching an agent to like write code in your code base where you tell it how it works.
09:29You can, you know, do the prompt and all of that kind of stuff.
09:32The agent also has full access to the docs and the code underlying the SDK if it really needs to go and read that as well.
09:38Just for a brief example on the right here, you can see an example with a few pieces of the SDK on display.
09:43We import um from our API.
09:46We The agent has to then define kind of a top-level edit function with a well-defined name.
09:51It then can use uh find nodes by exact name to find specific nodes that it's looking for in the graph.
09:56It can write assertions that we give it to make sure that um its code fails early um as well.
10:02And then it can use uh specific mutator functions like set rate and edit node to actually interact with the graph in a way that will produce those kind of typed objects that we care about.
10:12So, that's the SDK, but the actual guarantee here is when we own the deterministic execution.
10:20So, even with the typed SDK as our entry point, that really only guides the agent towards the desired end state.
10:26And the real guarantee comes from the final script that we orchestrate on agent completion.
10:31This run executor script um it is called when the agent has kind of completed running, and we think that we're in a like almost completed state or a ready to be completed state.
10:42And what that script does is it starts by linting the agent code, where we can immediately send it back to the agent if something is wrong.
10:49Better to kind of fail early rather than fail later.
10:52We can also detect conflicts.
10:54This is where sometimes the agent in one part of the code edited something in another part edited the same thing or something that depended on the first one and didn't realize that that happened.
11:02Because we have these guarantees, by giving the agent this well-defined structure, we can detect those conflicts for it so that we can send it back to the agent.
11:10We then run the agent edited code, and that means that we can then validate the output artifacts.
11:15Of course, if the code itself fails to run, we can send that back to the agent, too.
11:20And then finally, once we have the validated output artifacts, we can create a well-structured review artifact that makes it easy to review what the agent did without having to actually ever go read the underlying code that produced it.
11:32To give you a flavor of what that looks like, here's a little snippet from an emissions report where the graph edit function impact analysis ran on 50 graphs.
11:39Um there were two functions that it applied that produced 749 edit actions, and it ultimately in this case reduced the overall emissions by 45.6% and we can even go to
11:51go to see there are two code there are two edits here.
11:54Um one kind of did a big chunk and the other one made a very small change.
11:58Maybe you'd go look at what this actually did, and then you can even go at a graph level and say, "Okay, for this graph, like what was the option one and which nodes did it edit within that graph to get a sense of
12:10um what actually happened under the hood?"
12:11So, you can see all of this without actually having to go read the low-level code.
12:17So, to return to those three problems that uh we started with, now that we have these kind of two broad concepts that are part of the harness, um when the agent will find creative ways to solve every problem, this is now
12:30a great thing, but we've limited it a little bit.
12:31These invalid actions that the agent will take on its route to solve whatever problem we presented are actually prevented with the type dust SDK as the only way to properly edit the graph,
12:41and then we can guarantee that the graph was only edited through the SDK by owning the deterministic execution and the validation of that written code.
12:50The second problem about the agent saying that it made an edit when actually that the edit didn't go through in the way that it or the user expected, these false reports are caught via that deterministic execution so that we can surface those back to the agent
13:02um to make sure that it's doing what it claimed.
13:05And finally, our users never have to look at the code because the deterministic execution system produces artifacts in an expected form, and we can build verification against those to catch both the kind of good outcomes and the errors that are now surfaced clearly
13:20to both the user um and the AI.
13:23So, even when it hasn't maybe gotten exactly the right answer or the one that the user expected, it's easy to follow its logic and kind of get to that um and loop back on itself.
13:35Now, the harness is all well and good as it's kind of adds constraints, but I also want to emphasize that even with those great guarantees, you still need to hill climb to get better at the task itself.
13:45Since we started tracking how well the agent did at these complex edits, where each task here um in our eval set is a collection of graphs and a goal,
13:54um we've been able to improve our outcomes from about 43% to 92% on our set of internal evals.
14:00And to achieve that, all the typical approaches were still effective.
14:03We did a ton of prompt improvements, like rewriting the system prompt and all the kind of skills that the agent had access to.
14:09We've added few shot examples to both teach and coach the agent how to make use of the SDK and work with these different classes of tasks that we expect.
14:18We also had to make the tools, the SDK itself, and the tools the agent had access to better fit for use by improving their kind of ergonomics and how obvious they were for the agent to use.
14:28We also broke the task down through doing this, breaking up the um overall problem to a plan and execute kind of standard loop, as well as teaching the agent some of that expert judgment that is endemic in our space, making that easier for the agent to operate on it and to elicit that expert judgment from the user.
14:45But the important thing here is even though we've kind of improved the accuracy here, which is amazing, and it's great that we'll now kind of get better at that first time a user goes through the full workflow in the task,
14:57um even when the agent errors, or even our ground truth is, you know, one of those points on the possible range of expert judgments, we can still guarantee and validate the um process itself.
15:09So, to bring it all together in a domain full of expert judgments, you must respect the process, too.
15:14And for that, you really need guarantees about the process.
15:18We all know coding agents are necessary for complex tasks, and they will continue to be necessary and extremely useful.
15:24But those power that power comes with many risks that are like good to go and clear eyed about.
15:30For tasks with complex data and judgment calls, we now think that the harness should do a few important things.
15:37First, it should give the agent well scoped primitives to interact with the kind of external system that it's actually supposed to work with.
15:43Um free freely operating code can, you know, lead to a lot of the things that we talked about earlier.
15:50And this really allows the builders of the system to allow and disallow specific things and exercise the full power of these coding agents while still trusting the process that they're working in.
16:00I think the harness should also maintain full control of the final execution.
16:04These very smart agents may declare victory in an unexpected way from what you or your user really want them to declare.
16:11And so it's important to be able to constrain and validate that the agent has actually done what it said it's done.
16:17And finally, you should use that deterministic final outcome to produce outputs that are easy to validate even for non-coders.
16:23The code is kind of just the means to an end.
16:26And then of course, the kind of standard context and prompt engineering will of course remain critical to better scoping and defining that task and making it clear what the agent needs to do.
16:36That's it for me.
16:36Again, I'm Andrew Dumit at Watershed, the sustainability AI platform.
16:40Thank you again for your time and attention.