Brains vs Hands: How to Run AI Agents Safely in Production — Viren Baraiya

AI Engineer · 16 min · 194 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Cool.
00:14Thank you all for joining this morning.
00:16I know it's been an exciting week.
00:19A lot of talk about agents, building agents, coding agents and everything.
00:25And what I want to talk in the next 15-20 minutes is is about running agents in production.
00:32Um And things that is kind of on the top of pretty much everybody's mind when it comes to building agents and running agents is you know, harnesses.
00:43Before I get started, like I would like to ask in the audience, right?
00:47How many of our many of you are running agents in production today?
00:53Awesome.
00:53So, a lot of hands up.
00:54Um I was in another conference couple of weeks ago and I asked the same question.
00:59I think I had like one hand, so this is pretty good.
01:02It seems like, you know, things are moving forward.
01:05Um All right.
01:08So, when we think about agents, right?
01:10Like if we go back to kind of the hello world equivalent examples of agents, uh it's always starts with, you know, an agent with a tool call and how well an LLM kind of decides to call the tool,
01:22maybe use some sort of context or a memory and and plans things ahead, right?
01:26Um What What we start to kind of realize is is you know, that's a very small narrow picture if you think about it.
01:35When you think about agents in production, it's not just always about chatbots.
01:39The early examples that we had seen were all about like, you know, hey, we are going to build a customer service chatbot or some sort of chatbot.
01:47Um But when we start thinking about agents in a broader terms and picture, think about background agents, background workers, agents that are running on schedule.
01:58Um I would like to have an agent that runs every hour to check uh what's happening with my schedule and if there is something new that has popped up that I should think about.
02:07Uh event-driven agents, right?
02:09Agents that are listening on various events.
02:11Um to give you an example, I have a production system running where my alerts are firing every now and then.
02:17I have a logs coming through hotel.
02:20I would like my agents to listen to those events and react on it and see what's going on there, right?
02:25Um I could have long-running coordinators and what I mean by that is I have an agent running for a longer period of time.
02:32I would like to have an agent monitoring that agent to see, "Hey, what's going on there?
02:37And should I nudge that agent to kind of move forward, take a different action?"
02:42And things like that, right?
02:43Um also one more thing is about multi-agent systems, right?
02:48Agents talking to other agents.
02:50Uh the last thing you want is to have an agent be responsible for multiple things and risk hallucinations.
02:56Uh So, more we think about agents, right?
03:01Uh the way I would like to think about is the agent starts to feel more like an application, not just a component, right?
03:09Um If you go back to the old school thought of microservices, when we and probably we are still building microservices, right?
03:18Um you are not going to ship a microservice in production that does everything, right?
03:23It kind of stops being a microservice at that point in time.
03:26Uh typically, we are shipping multiple microservices.
03:30They are talking to each other through choreography or orchestration.
03:34And you have an application that is relying on that the set of microservices, uh which essentially are single responsibility principles uh to kind of deliver the business goals.
03:48There is kind of a very clear uh parallel to that when it comes to agent harnesses, right?
03:53If you think about a harness that is responsible for achieving a specific goal.
03:57So, let's say I have an harness for doing my SRE work.
04:02Um, it's operating on multiple agents to see what's going on there, right?
04:06Maybe it's listening on events coming from my logging system, talking to my Kubernetes cluster as an agent uh, to find out, you know, what's going on there.
04:15Uh, my metric systems, my customer service uh, dashboards and and so on and so forth.
04:23Um, and what I want to drive there is that if you think about it, right?
04:26Your harness is the application and vice versa, right?
04:29An agent alone doesn't kind of deliver the whole premise, but rather when you start putting things together and you build an harness that controls the execution of the agent,
04:39delivers on the premise, that's where you start thinking about um, an application, right?
04:44And which is you have systems uh, which are your databases, your internal systems, enterprise systems.
04:52You have humans, human in the loop.
04:54Um, you have tools through either APIs or MCP that is all kind of playing together.
05:00Um, and as I mentioned, right?
05:02Harness runs more than just an agent loop.
05:05And this is an important thing that I kind of learned uh, while trying to build harnesses for real-world production use cases.
05:11It's a combination of both deterministic and non-deterministic part of the application, right?
05:17Non-determinism is delivered by an LLM in terms of the reasoning, in terms of thinking process.
05:24Um, and then there is non-deterministic and there is deterministic part, right?
05:28Which you do not want non-determinism seeping into it, right?
05:32Think about payments.
05:34Um, or let's say if I have an agent harness that is responsible for monitoring my production deployment, how I manage my Kubernetes clusters, I probably want to have a very well-defined workflow in terms of what are the sequence of steps that I execute
05:49when I want to restart my cluster.
05:51And that's a very deterministic set of processes which every time it runs, I know exactly what it does.
05:56So, I want a determinism to be delivered by my harness when it matters.
06:03Um and of course the harness is a long running process, right?
06:06It runs across the time.
06:07It can run um anywhere from few seconds if it's uh something very quick like hey, check what's going on here to all the way running for days, months, uh even longer than that, right?
06:19Think about long running processes, order management systems where you are waiting on third party services to deliver your shipment or waiting on humans to take actions and approve things
06:30and so on and so forth.
06:31Or Harness is just waiting, right?
06:33It's waiting for events to happen.
06:34So, when that event happens you take an action and do something around it.
06:38So, that brings in another important point, right?
06:42That when you think about long running systems, you need durability, right?
06:46You want to be able to recover when things fail because at the end of the day this harnesses are running somewhere in your entire infrastructure stack, right?
06:54Maybe in the cloud, maybe on a sandbox but those things can go up, go down.
06:59There could be network failures, partitioning, anything that could be happening.
07:02So, you need the harnesses to be durable.
07:04And one thing about durability here is it's a table stake thing, right?
07:08At the end of the day durability is a cost of admission.
07:11That's not the feature that you're looking for in a harness.
07:14Um And as I mentioned, right?
07:17The loop that the harness runs spans across agent um and everything else.
07:27So, now let's think about uh how the harnesses kind of operate, right?
07:33Um If you think about a harness, right?
07:36And and a loop, essentially what it is doing it it has a state of the world.
07:40It knows what work been completed.
07:43Um what has been recorded in terms of the side effects, right?
07:47So, like if I did a cluster restart, uh I know I have that recorded.
07:51Uh if I sent an email, I know that has happened.
07:54Um I know what worked or didn't work, and then based on the current state of the world and the goal, I know what needs to happen next, right?
08:02And that's where the reasoning and LLM comes into the picture.
08:08Um and what really happens here is that if you think about a clear distinction, and this is the most important thing.
08:15If If one thing that I would like everyone to take away from here is this slide, which is that the responsibility of a non-deterministic agent is to plan, is to plan what should happen next,
08:26not really to do things.
08:28Um and then Harness is the one who does actually execution.
08:32And And this is important for uh various reasons, right?
08:36One being that Harness is a deterministic piece of code that actually understands what is involved in, you know, actually executing a piece of work.
08:45So, as I mentioned, right?
08:47If I am trying to uh build a Harness that runs my uh DevOps or SRE uh agents, uh when the agent says that, "Hey, this cluster is unhealthy and you should restart."
09:00Harness should decide how to restart the cluster, what involves in restarting cluster, and that has to be, at least in my world, has to be very deterministic set of processes, right?
09:10So, that every cluster restart is exactly same.
09:13There is no other uh category there, right?
09:16Um Also, if it needs to have an approval, for example, if I am trying to restart a production cluster, I probably want to have a human gate, uh probably send a Slack message
09:26to somebody to say, "Hey, I'm going to restart this cluster.
09:29Do you think it's okay to do that or not, right?"
09:31And I do not want this to be left to a hallucination by an LLM that, you know, it doesn't need to do it.
09:36It needs to be guaranteed in terms of execution.
09:39So, it has to be very deterministic when it comes to those kind of things.
09:43Uh ideally, I want this to be idempotent and if not, I want it to be recording that, you know, it is not idempotent and this is what has happened.
09:50So, I can get take care of the side effects or I can handle the side effects separately, right?
09:55Uh So, this is the most important thing, right?
09:58The clear separation between as we would like to call, right?
10:00The brains and the hands.
10:01Harness is the hands, the brain is the LLM.
10:05Um So, yeah, if the agent writes the plan, uh Harness executes the plan, uh and I'll show you in a brief uh a short demo in terms of how all of these things kind of matches together.
10:20Um But, is this a new concept?
10:24Like, if you think about it, this is not necessarily a new concept.
10:26This has been around for a while, right?
10:28If you think about workflows as sagas, which we used to write ourselves and define exactly what happens.
10:34It was a very and it is a very deterministic set of processes.
10:38When you think about agentic harnesses, they are essentially late bound sagas.
10:41And what I mean by that is that they have a very finite and deterministic set of tools uh and the task that they can operate on.
10:50Instead of putting them together upfront, the agent is kind of proposing and building them at runtime.
10:56Um and therefore, they are essentially late bound sagas, right?
10:59Uh but you get all the benefits of a traditional saga in a workflow out of the box in terms of visibility, what's happening, being able to control things, and iterating upon like, you know, how far ahead in the future
11:13uh the agent is able to plan.
11:15You are able to kind of plan one step at a time or multiple steps at a time.
11:21Um And yeah, if you think about it, right?
11:24It's it's more like a branching workflow.
11:26If only you could build a workflow with every possible combination of a branch, um then, you know, it kind of builds that thing for you right versus with agents that kind of simplifies the work.
11:37If you have n number of tools it can do n different combinations of executions which otherwise is going to be almost impossible to you know think ahead of time and and do it.
11:49So let's take a look at it right.
11:51So what I'm going to do is I'll quickly show a completed run.
12:02Of what I talked about earlier right.
12:05Here is one of my example agent that essentially is acting like an SRE agent right.
12:11What its job is to do is understand what's going on in the current system and try to plan what should happen next and execute on those things.
12:21So it's essentially a remediation loop that runs twice.
12:25In the first iteration it tries to understand the root cause of the problem, tries to react to that, observes the output of it and then runs another loop and plans another set of steps to see what should happen next right.
12:41So as an input to my agent let's see what was the input given right here.
12:49So step number one is you know it makes a call to LLM.
12:53So as you can see right like it's an SRE agent.
12:56Of course it's a demo so you know everything is kind of pre-planned and a can demo.
13:02The output of LLM as you can see right are the steps what it should do.
13:09It's talking about so as you can see one more thing here right is that instead of just doing one step at a time essentially it is proposing a sequence of steps to do.
13:19You should gather evidences, you should analyze logs and if required based on the evidence that you gathered you should either roll back a deployment and then verify recovery.
13:30This is then given to a specialized tool called plan and compile.
13:35So, this is the plan that agent gave.
13:39This gets compiled into a very deterministic workflow.
13:42We are relying on a conductor as a workflow execution engine here.
13:45So, the output is a fully runnable workflow.
13:49This gets executed.
13:50So, like you know, what I'm going to show you here very quickly is how that looks like.
13:54Um So, this is the first step of execution, right?
14:02If we look at it here as I said, right?
14:05Like we did analyze the logs, look at the query of the metrics, decided to do a rollback and verify recovery and everything.
14:15So, step number one is completed.
14:17It runs another loop.
14:19Um same thing goes here.
14:22Um In the next iteration, it decides that, okay, looks like, you know, these things have been done.
14:32Let's check the downstream.
14:33And if required, verify the recovery and complete it, right?
14:36Um And then the next one runs and completes the task.
14:40But here is an example of a loop that is self uh kind of uh planning, right?
14:46So, and to show you very clearly what's going on here, here is a loop.
14:50Um At every step of the way, essentially, it is looking at the current state of the world.
14:56So, this is my agentic loop that is kind of understanding the world, planning and executing uh one step instead of one step at a time, it is actually planning multiple steps at a time, executing that,
15:07verifying, and completing the loop.
15:09Um Now, this can run in production as many times as you want.
15:15This can be running on the events.
15:16It can be running on a schedule.
15:18Um As I said, right?
15:19Like this the whole thing could be running for a much longer period of time um as opposed to just running in a short period of time.
15:30All right.
15:30So, that's towards the end of it.
15:34This is kind of overview of what we just did here.
15:38As you can see right in the second iteration, we decided not to do a rollback because it was already done in the first iteration.
15:46But yeah, that's about the harnesses and agents and how they bring determinism to your production application.
15:54The example that I showed you runs on conductor.
15:57Conductor is an open source workflow orchestration platform that supports building agentic loops as well as agentic systems.
16:05It's fully open source.
16:06Orkes, we provide enterprise edition.
16:09But feel free to try it out.
16:11Here's a QR code.
16:13Give it a try.
16:14Join our Slack and you know, if you have questions, happy to help.
16:18We have a booth here at Orkes.
16:21Drop by if you want to see a live demo.
16:23If you want to run your LangChain agents or OpenAI agents or any kind of agents, Conductor can handle all of those things.
16:30Thank you.