Beyond the Harness: A Journey Towards Adaptative Engineering - Rajiv Chandegra, Annicha Labs

AI Engineer · 37 min · 322 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01Hello, I'm Rajiv.
00:03I'm a practicing medical doctor here in London and been doing some AI engineering for the past few years.
00:10Particularly interested in real world application and really the future which is multi-agent multi-human multi-institutional collaboration through a company called Anitcha Labs.
00:22And really the premise of my talk today is to introduce a new design philosophy for the future of AI engineering which is beyond a harness or beyond a fixed
00:33static harness and towards adaptive engineering.
00:41This slide basically summarizes my entire talk and really it's about exploring what the current AI engineering paradigm is which is to use fixed harnesses to steer agents.
00:56So we either use or build an existing harness like Pi, Claude, Code, Cursor, CodeX, whatever it is ahead of run time that remains fairly consistent throughout the engineering process
01:08and is largely constrained with fixed roles and tools and sequencing which again ahead of the process beginning of the process we can tweak and it essentially produces very reliable
01:20and orderly work and it's great for most engineering problems and ideas.
01:25And so your role as an AI engineering right now is to use a harness and steer agents.
01:32But the future's going to be different in at least two ways.
01:36Number one, models are going to become so powerful that existing fixed harnesses are constantly going to become outdated.
01:45And number two is there's going to be a huge exposure to the real world.
01:50Right now a lot of AI engineering is software based.
01:54It's within behind a screen.
01:57But actually, as things get more powerful, as models get more powerful, we're going to have exposure to the real world where I'd argue that this is where the real challenges are.
02:09And the real world is dynamic and messy.
02:12You know, it's full of multi-agent, multi-human, cross-institutional, and really touching the physical world, which means that a fixed harness is quite brittle.
02:23And so that's where we enter what I coin as adaptive engineering, where you actually allow the harness to emerge and adapt mid-engineering to find its most optimal position and structure.
02:40And essentially, the engineer's role is going to be to design some of the constraints, which are kind of the rules of play, and really allowing the harness to emerge,
02:51stabilize, change, and eventually dissolve as you go through the runtime of engineering.
03:03So, let's look firstly at the current paradigm for AI engineering.
03:10And it's essentially where a harness serves as the primary method to guide an LLM, which is typically stateless, and make it into something quite useful.
03:23And there's a couple of aspects of a harness, which I won't go to in detail, but essentially things like system prompts, which is established by the harness vendor, meaning that users can't actually modify it.
03:34And this essentially tells the harness what it can and can't do.
03:40You've also got the agents.md files or the Claude.md if you're using Claude code, which is loaded into every context window at the start of a session, tool calling, and essentially
03:53then you have the genesis of agents, which are specialized entities that have been harnessed uh, granting them unique capabilities um, and and allowing them to differentiate from other agents.
04:08And these capabilities are typically manifest as specific skills when it comes to role typology, their capabilities, the rules um, and that enables them to deliver targeted outputs and specified
04:23outputs, whether it be code documentation issues, handoffs and and most recently we've seen the genesis of kind of loops, loop engineering has become a thing um so that eventually
04:39we reach some sort of outcome uh, whether it's a feature, uh, a problem solved which is eventually reviewed by a human.
04:49Um, so what you see uh, from this is that before running the engineering process the entire harness has been predetermined or pre-engineered.
05:00And that's harness engineering.
05:02And uh, it's been remarkable at getting the most out of the latest models.
05:15Um and just sort of stepping a bit back, um, defining exactly what a harness is.
05:21Essentially the model is the engine and the harness is everything built around to make that engine useful.
05:29Um, there's many different types of harnesses.
05:32I mean, there's hundreds uh, actually, but I guess the most common ones are, um, you know, CLI coding, um, such as Claude Code, Codex, Pi.
05:42You have IDEs like Cursor uh, multi-agent orchestration through LangChain and and things like Hermes as well, Klein and Goose.
05:52And And these are all differentiated based on number one, properties and how they're configured.
05:58But actually more importantly, number two, based on some sort of design and engineering philosophy that they believe in.
06:08And what makes them so great and so distinct is that they are opinionated to a lesser or greater degree.
06:15Um and they allow for, I guess different use cases.
06:21Um and depending on your temperament as an engineer, you'd pick one over the other.
06:30But what we do see in all of these is that everything's predefined.
06:34You know, you can customize them, of course, like for example, the Pi agent.
06:39It's minimalist and it's maximally extensible.
06:44Um but everything is predefined.
06:47You know, the roles, fixed outputs, sequencing protocols, how memory is created.
06:53Uh and so I want to be absolutely clear here.
06:56I don't mean that you can't customize harnesses.
06:58Of course, you can.
07:00But the customization occurs ahead of the engineering runtime.
07:04Uh and not mid-engineering.
07:05And that's largely directed by us as humans.
07:08And for someone like me who started AI engineering about two years ago, it's probably the most powerful or predictive uh predictable method.
07:19And it's got really three payoffs.
07:20It's reliable.
07:22Um you know, the same input leads to a similar set of outputs.
07:25Obviously, there's going to be some variation.
07:28It's auditable.
07:30Um you can inspect exactly what changed and when.
07:33And there's some sort of linear causality where uh you know, if something breaks, you can actually trace it back to its source.
07:41Um this is like a factory line though.
07:43Like think Taylorism for AI.
07:45Just like a factory assembly line, every station is engineered in advance.
07:51Um, each agent has one job, a fixed place, a sequence, and maybe a defined handoff to the next agent.
08:04I mean, there's clearly some incredible uses for this and, you know, factory engineering gives you, you know, accuracy, speed, reproducibility, behavior that you can certify, and it really shines in these kind of closed deterministic
08:19systems, where you have a well-defined product or problem, and you know the kind of features you want to build.
08:25You know the sort of solutions that you need.
08:29Um, but here's the catch.
08:33Um, and and it's a real trade-off that that reliability isn't free.
08:38Um, you buy it by suppressing, um, variance that I'd argue novelty requires.
08:47Um, and determinism and emergence pull kind of in opposite directions.
08:52Um, and this is where we have to pivot to the real world, because in real life situations, in when when AI eventually, uh, when it becomes a norm for AI to
09:04capture real-world scenarios, where you've got multi-agents, multi-humans, multi-institutions interacting, and also touching the physical world, um, there's a hard ceiling on novelty, um, because the real world is messy, and it keeps changing.
09:24And so, fixed harness starts to break down.
09:28Um, so like I mentioned, there's a hard ceiling on novelty, um, because, uh, essentially, its reliability is predicated on suppressing variants.
09:40Um the models are going to keep accelerating.
09:43You know, you can build a careful harness today or you can use one off the shelf.
09:48But it could be irrelevant in the next month.
09:50The model just got so good that it didn't need that kind of scaffolding anymore.
09:57Um and that kind of leads to this whole dilemma of brutalness where every situation you didn't anticipated you didn't anticipate needs a human to patch the harness.
10:10The more real world it needs, the more rules you bolt on and eventually the harness just becomes ever more complicated than the actual problem that you need to solve.
10:20And I guess if there's one takeaway here is that the factory method is the right answer to a fixed problem and the wrong answer to a moving problem.
10:37And so let's really examine in what the real world is about and and I'm going to go down a bit of a philosophical route here.
10:46Um as we've said, you know, agentic systems will eventually collide with the real world and that raises a question.
10:55What is the real world actually made of?
10:58Because there's two ways to answer this actually.
11:00So the first paradigm, the first way to see the world is through what we call a reductionist or analytical view which we're all very used to actually.
11:10It says to understand the whole a big problem or a system or a big thing like an organization we've got to take it apart and study the individual parts.
11:22So essentially the world is made up of these individual parts which are things that are stable and change is just something that happens to those stable things now and then.
11:35The relationships amongst those things are secondary or an afterthought.
11:39And this is really the metaphysics of a factory and of nearly all software that we build.
11:45You know, you make the components.
11:48Uh you wire them together.
11:52And you get a product.
11:55The second paradigm, what we call the systems or relational paradigm, um takes the emphasis away from there being separate things or components and actually puts more what we call ontological
12:10emphasis on the relationships amongst those things.
12:14So, flips the order.
12:15It says that the real world isn't made up of things at all.
12:19It's made up of processes and um relationships which come first.
12:27So, what we call a stable thing really is just a slow pattern in an ongoing flow.
12:32Think an organism.
12:34It's more like a flame than a crystal.
12:37A flame looks like an object, but it isn't.
12:39It's a pattern held together moment by moment by a process.
12:45You stop the process and the thing doesn't exist.
12:50Uh and there's a really good payoff in understanding this paradigm.
12:54Um and once you see the world in this way, you start to notice something remarkable.
12:59Where where you have simple local rules between, let's say, particles or between people or between creatures, that local interaction gives rise to a whole new level of order that can't be predicted
13:17from its constituent parts and that no constituent part designed ahead of time.
13:25For example, water.
13:26Water has a property of being wet.
13:30But it's constituent parts, oxygen and hydrogen, are not wet.
13:36But their coming together, again putting emphasis on the relationships, their coming together is where you had the emergence of a novel new property, i.e. wetness.
13:48Another example is a flock of birds.
13:50Not one bird wakes up every morning intending to make a flock.
13:56Each probably just follows three simple local rules, which is align with your neighbor, don't crash into them, stay close, and that's it.
14:06Local rules, local interactions, and out of that you get a flock, a new pattern alive and coordinated that you could never understand by pulling it apart and studying one bird because no single bird
14:20has a flock in it.
14:21The flock lives in the relationships, not the parts.
14:27So this factory method of engineering assumes a world of parts.
14:32But the non-machine world, and I'd argue that AI is sort of on the cusp of that, the non-machine world is a world of patterns and moving patterns of emergence.
14:48And that sort of reflects what sort of problems we face in this moving messy world.
14:56Um and there's a line from Russell Ackoff that I absolutely love.
14:59He says that managers aren't handed neat separate problems.
15:04They're handed dynamic situations, tangles of problems that keep changing and keep bumping to each other.
15:11And he And he gave that a name.
15:12He called it a mess.
15:14A mess is a isn't just a pile of separate parts, it's actually a system of relationships in motion.
15:22Which is exactly why the factory based on a fixed harness way of engineering will struggle when it collides with real-world live scenarios.
15:34Because you can't decompose a mess into tidy boxes and solve each of them because they're not individual parts, they're a moving pattern of relationships.
15:49We can then look at problems through this this distinction between different types of problems.
15:55So, um engineering is all about solving problems and I'd argue that very simply put there are two kinds of problems.
16:03There are complicated problems um and there are complex problems.
16:08So, a complicated system or complicated problem is like building a jumbo jet or a clock.
16:15There are passive parts um but experts can take them apart, analyze them, plan, predict, document.
16:24It's hard, but it's actually knowable and it works in a very predictable way.
16:31In contrast, when you have a complex system or a complex problem like a flock of birds, a market, a human organization, it's slightly different.
16:42It's parts are constantly interacting and adapting to one another.
16:47So, the whole can't be derived from analyzing the parts.
16:53So, you don't analyze and plan a complex system.
16:56Instead, you probe and sense and respond and this is probably one of the most expensive mistakes we make in the modern world of design and engineering, which is that we treat a complex problem
17:09like a complicated one um and and things fail not due to lack of execution, but essentially because there's a failure in categorizing the problem spaces.
17:24If we go just one step deeper into what a complex system is, um essentially, you know, the ingredients of it are first, you need a diverse set of agents
17:36or actors if it's a human organization.
17:39Um not clones essentially, but different from one another in some sort of way.
17:43So, because the diversity is the fuel there.
17:47Um they need to interact locally uh cuz no agent, no individual actor, no individual component sees the whole.
17:55Um instead each respond to whoever's next to it.
17:59And this is like the bird aligning with the neighbor.
18:04And they learn.
18:05They learn recursively.
18:06They adapt, then they adapt to the results of their own adapting, and it constantly loops, and um you know, with respect to one another and the environment.
18:18Everything is moving in response to everything else, and nothing holds still.
18:22And out of all of that local adaptive looping interaction, you get something called emergence, which is a key concept in complexity science.
18:35Um basically, it's a new novel pattern uh or organization or behavior that uh no single part designed on its own and can't be explained by splitting that whole into its parts and studying them.
18:53Again, think the flock of birds.
18:56No bird has a flock in it.
18:58Think about the wetness of water.
19:00None of the individual oxygen hydrogen molecules have wetness within them.
19:07And yet, it leads to the emergence of something quite novel.
19:12Now, these systems don't fly apart into chaos.
19:16They tend to settle into attractors, which are essentially stable states.
19:21Uh they keep pulling back towards some sort of equilibrium.
19:25So, for example, water is stable at room temperature.
19:28So, you get this sort of beautiful tension.
19:30You've got constant change at a local level, but then you've got recognizable stable patterns at a whole system level.
19:40And crucially, nobody's steering here.
19:43It's self-organizing.
19:44And that's the whole mystery.
19:47And I'd argue that's the whole opportunity for design and engineering.
19:52And this is where really want to introduce this new design and engineering philosophy called adaptive engineering.
20:03Before that, let's just take stock a bit.
20:05So, everything we covered so far lives inside a particular framework, which is this kind of fixed or factory harness.
20:12Uh the human engineer decides it all up front, the sequencing, the roles, capabilities.
20:18That's really the factory model, and it works.
20:20Um and we've seen that um because problem spaces that are complicated yet predictable is exactly right.
20:28And I'd argue for most engineering problems, that is the perfect method.
20:33But the moment AI becomes exponentially better, and it collides with the real world, which is, as we found, messy and complex, not just complicated, we need something genuinely new, a different idea of what optimal design and engineering
20:51even is.
20:52Um And it's obviously it rests on those two assumptions that models have become become exponentially better.
20:59Uh and that AI will uh come outside of a sandbox situation, and actually um engineering will run in real time in contact with the social and physical environment.
21:13Um so, essentially what we're saying is that the harness must adapt mid-flow.
21:20And that's the crux of harness engineering.
21:22And I guess if I were to crystallize um the definition, um adaptive engineering is the discipline of designing constraints to the extent that the harness emerges on its own,
21:37stabilizes, and adapts as needed in response to the changing environment in ways that you could not specify in advance.
21:50Essentially, the harness becomes the ongoing output rather than the input.
21:56And you know, what this means in practice, uh just to make it a bit more concrete, is given that agents have capabilities um to interact with each other, organization
22:10emerges from those interactions in response to the environment or a goal, such that new levels of order appear, ones that you could never have specified up front.
22:23Um so, think bird in a flock, think agent and harness.
22:28So, the agents are actually creating the harness in the same way as the birds are relating to one another so that the flock can emerge.
22:38So, you don't build the harness anymore.
22:40You let the agents form the harness that best fits the environment in that moment, in that context.
22:48[snorts]
22:48The harness then becomes a self-organizing, constantly evolving multi-agent system.
22:57And if we're to just simulate what it may become, because this is when agents become a lot more capable.
23:04Um essentially, you have agents that are isomorphic um but yet undifferentiated.
23:10Um but fully capable of interacting and um learning and changing.
23:14Um you let those agents interact.
23:17At first, there's nothing.
23:18It's a bit of few exchanges here and there.
23:21But the coupling continues and slowly builds until you reach a point where roughly one connection on average um occurs um with each agent.
23:30And suddenly, you start to see this whole emerge.
23:34[snorts]
23:34Um a system, you could say.
23:36Uh and I guess as an engineer, your new lever uh becomes uh how the rate of coupling.
23:43Like, do you dial it up or do you dampen it down?
23:47Um again, the adaptive path focuses on the interactions.
23:52Um until, you know, agents start to actually specialize with respect to one another.
23:59So, let's say you got two agents doing the same thing in the same place.
24:02That's just redundant and the environment knows that.
24:05The pressure from the environment knows that.
24:07So, it rewards anything that essentially breaks the tie.
24:11Um so I guess a tiny difference um you know, who got there first, who is slightly better gets amplified by the feedback until those two agents are no longer interchangeable.
24:25And so, the pool stops becoming something uniform and it actually falls into uh niches.
24:31And really, the key insight here is that the agent's identity isn't something you gave it.
24:39It's actually the position, role, or capability it took relative to the others and its environment.
24:45So, here we see specialization emerges from the interaction.
24:51And if you go on a bit further, uh they stop connecting at random.
24:55Uh in fact, they start to work well together um, in terms of sort of clusters.
25:01Um, and you see the first boundaries appear and crucially the system drew them, you didn't as an engineer ahead of time.
25:09Um, so these are kind of emergent clusters.
25:12Um, until a new convention crystallizes, you know, a division of norms and protocols, and the system produce some level of governance without a governor.
25:23Conventions emerge spontaneously from the local coordination with no central authority because as we've seen, central authority can lead to brittleness in a changing environment.
25:36Um, so there's just enough repeated interaction to tip the group into some sort of shared norm.
25:43And this is crucial really because it means that it can continue to adapt in a decentralized way based on the changing environment or problem space.
25:57And essentially a new um, emergent order um, arises from those tiny little interactions amongst those agents, you have a new level of organization that is fairly stable up until the next iteration, up until the environment changes
26:14and um, the order needs to essentially restructure itself.
26:23So the question is that what is the role of an engineer?
26:26And and really in adaptive engineering, uh, you don't abolish the engineer, you're just relocating the emphasis of engineering.
26:34So if you can exploit the model's capabilities, um, so agents can essentially um, interact, learn, and change.
26:43What you're doing as an engineer is essentially um, exploring the constraints, you're dictating the constraints, the rules of the game.
26:51You're not deciding what should happen or what you want to happen, but you're giving the agent space to explore that field.
27:01[clears throat]
27:01And then, once that emergent harness uh starts to take shape, you are sensing it and responding to that rather than um stopping the engineering process and starting from scratch.
27:15Um And I think this slide sort of um portrays that.
27:21This is not binary, this is not either or.
27:25Uh in fact, it's very much a continuum.
27:29Um if you you there's no purely fixed harness way, and there's no purely uh completely adaptive autonomous way.
27:37Um fixed engineering is essentially in one hand um prescribing the structure, which agents run and you rarely change the structure mid-engineering.
27:48Um whereas on the other end of the scale, you've got adaptive engineering where you allow the agents to freely interact and allow the harness or the structure that guides them
27:57uh to be created by them and change mid-engineering.
28:02Uh so it's not like a swarm of agents loose with no roles and this sort of intelligence that magically appears.
28:10Um and it's not sort of better one one way is not better than the other.
28:15They just have two different use cases.
28:17And um if I could just, you know, bring it back to some examples of existing uh harnesses, so for example, Hermes AI um on on their website, it states that it's a self-improving
28:30AI agent that creates skills from experience and learns from it.
28:34So, obviously, this is a huge step in terms of adaptive direction.
28:38Um but I do want to make a distinction here um because um that adaptation is in the form of vertical intelligence, which is about making individual agents smarter.
28:50What I'm talking about here is horizontal intelligence, which is all about how groups of agents coordinate.
28:57And so, those two directions are a bit orthogonal.
29:01Um And the thesis I'm presenting is that it's horizontal intelligence uh that is the most adaptive method, the most agile method.
29:11Um and and and arguably the higher leverage point.
29:17Uh compared to vertical.
29:20Um Another example is Pi, um you know, a great harness, uh which is uh sort of essentially minimal, um yet it's maximally extensible.
29:34Um so, it's fully customizable to an extent.
29:37Um so, you could class that as adaptive, but again, there's a distinction here.
29:41There's adaptive at the design stage, like when you're preparing for that engineering process.
29:48Um Uh where the engineer can customize and the tool's quite malleable.
29:55But, when it comes to adaptive engineering, that's all about being adaptive during the run time, where the system reorganizes itself whilst running in response to the changing problem space of the environment.
30:09And you know, Pi is a strong example of the first kind, um but it doesn't make any claims um to be anything other.
30:17Uh it's not a self-organizing multi-agent system yet.
30:24I'm just going to skip over some of the uh nitty-gritties of what that could mean.
30:30Um so, for example, um uh when it comes to agent capabilities, you know, obviously, they need to interact, learn, and change.
30:40That's kind of a prerequisite for adaptive engineering and I don't doubt that that will become the case as models get exponentially more powerful.
30:52There's a few things that the designer can actually do in the form of constraints and I'm not going to list out all the different types of constraints cuz they're sort of largely dictated by
31:02some of the properties like the roles and the sequencing and the memory.
31:05These are all sort of categories or properties of harness.
31:09But essentially there are three questions that you could ask yourself as an engineer which is do you want to enable the agents more or do you want to close them in and govern them more
31:22and produce more give them more guardrails.
31:24Are you rewarding them to cohere towards a particular goal or are you costing them if they fall outside a particular container?
31:35And how fast or slow do you want this um to happen?
31:40Some of the properties, you know, what's the speed for example the rate of coupling.
31:44So these are some of the engineering tweaks that you can make when it comes to adaptive engineering.
31:50And then finally like this emergent harness that is constantly shaping and reshaping itself based on the problem space.
31:58You can only really sense and respond to that emergent harness.
32:01You're not going to completely be able to I guess edit it in some sort of hard engineering fashion cuz that's not in the best interest of you as an adaptive engineer.
32:21This slide is just a more detailed comparison of both approaches and again I want to emphasize that one is not better than the other.
32:29It's just two different design engineering philosophies.
32:34Where on one hand in the factory model, um you know, the structure is imposed up front by the engineer, and so the harness becomes the input.
32:43Um whereas in the adaptive path, the harness emerges and is essentially the output.
32:49Um And And one of the crucial things here is that control is decentralized in the [clears throat] adaptive path.
32:59Um in the field continuously um because the problem space eventually when AI um comes or AI engineering comes off a screen into the real world, um the problem space is constantly changing.
33:14Um Now, it's important to understand that, you know, there are particular use cases for this.
33:22Um but there are a huge number of failure modes um in this as well.
33:27Um you know, just to list a few of those failure modes, you know, emergence leans towards uh stability, what we call an attractor, uh where you find a structure
33:39that feels stable and optimal, but that doesn't necessarily mean it's the best.
33:45Um you need some sort of genuine selection pressure.
33:49So, if you imagine in the social world or the physical or the natural world, the environment is um giving us certain uh pressures.
33:59You know, that's what encourages us to adapt.
34:02That is evolution.
34:03You know, there is some sort of selection pressure here.
34:06And you kind of need to find that um in or we as a community need to find that in adaptive engineering, otherwise you just get drift.
34:15There is a risk of monoculture, where you don't get genuine diversity amongst agents because they're all trained on the same data.
34:22Um legibility does collapse.
34:25Um you know, you as adaptability increases, um you can't really pin down or explain things that well.
34:34And again, that's a feature of complex systems.
34:36You can't really pin them down.
34:40And and and obviously goes without saying that there's no more predictability ahead of runtime because things are constantly moving.
34:49Um And and and these are the real failure modes that we must apprehend because I do feel like adaptive engineering is something that we're going to be more inclined towards and it's going to race forward
35:05um from the future.
35:08[clears throat and cough]
35:13I guess a bit of a takeaway then.
35:15Um you know, as models improve exponentially um AI engineering will move into the real world scenarios.
35:26You know, the real world where it touches um the physical space uh on an ongoing basis.
35:33It's uh in contact with multiple agents across different institutions.
35:38It's in contact with multiple human beings.
35:40So So there's a real cost peer where AI is heading towards.
35:46And so the discipline of design and engineering is going to be forced to rethink itself around continual production.
35:54[snorts]
35:55And the limiting factor which is the case now, but probably more so in the future, is not going to be the strength of the model.
36:03It's going to be the adaptability of the harness.
36:07Um and adaptability really here means multi-agent um which is essentially about coordinating amongst agents.
36:16A multi-agent orchestration which is decentralized.
36:22Um and it's not adaptable ahead of runtime.
36:28In fact, it starts becoming adaptable mid run-time or during an entire engineering a process.
36:35And I think that's real intelligence.
36:37That's Well, when I say real intelligence, I mean that's the intelligence when it comes to adaptability of the harness.
36:46Uh, I'd be really interested in um hearing your thoughts on this um and whether this is something that uh other people are also exploring.
36:56Um please do reach out to me.
36:58Thank you.