Harness Engineering: How to Build a Software Factory — Dru Knox, Tessl

AI Engineer · 20 min · 221 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Quick introduction.
00:13My name is Drew Knox.
00:15I'm the head of product and design at Tessle.
00:19Tessle is an agent enablement platform.
00:22We help you go from scaling skills to building your own software factory.
00:27So today I'm going to talk about a few things.
00:30The core is going to focus on harness engineering which we see as sort of the new discipline of agentic development and how you can use it to ladder up to a software factory.
00:42So before we get started a few prerequisites.
00:46This is a bit of an advanced talk in the sense that like if you aren't already using coding agents what I'm about to walk through is probably not where you want to get started.
00:55So, I'm sort of assuming that if you're working with coding agents, you're used to having a couple sessions at a time.
01:04Uh you've gotten to a place where coding agents will for like easy tasks and maybe medium complexity tasks frequently do the right thing.
01:14This you're sort of at the sweet spot for starting to think about harness engineering, starting to think about a software factory.
01:21What I'm going to say here is not exclusive to that, but that's where you'll get your best results.
01:27So, I'm going to walk through a few things.
01:29First, I want to define software factory and talk a little bit about why you might want to build towards one.
01:37Second, we're going to then focus a lot on harness engineering, which is kind of the core practice or skill that you'll use to reach a software factory.
01:45We'll talk about what are going to be the components you work on while you're doing harness engineering.
01:50And finally, I'll walk through how the Tesla product uh briefly can help you along that journey.
01:56But most of the topics I'm going to talk through here today work with any stack.
02:00They're mostly techniques.
02:02Tesla makes them easy, but they're not exclusive to us.
02:07Okay.
02:07Part one, software factories.
02:09What are they?
02:12There's no strict definition.
02:14I'm sure you're all probably as frustrated as I am with how quickly terminology changes and migrates in the in the industry today.
02:22But roughly speaking, a software factory is any agentic system where all of the end product of what you're building and shipping to users is created by agents and your software engineering teams effectively
02:37focus on building the factory.
02:38So they work on making the factory more autonomous, more automated, and raising the quality of what it produces.
02:45I'll go into each of those terms in just a second.
02:48That's kind of the core idea of a factory though is everyone on your team effectively becomes internal tool builders and then you're building a system that produces your product.
03:00So I mentioned three key metrics.
03:02These are sort of like the pillars you need to think about when building towards a software factory and they sort of go in this order.
03:09So autonomy is effectively how much human intervention do you need to get to the right answer.
03:16So how many times you need to correct the code or provide a nudge on how the agent should be building.
03:23Automation uh oh sorry I've gotten them flipped here.
03:26Uh autonomy is the first one.
03:29It's how much do you have to do humans have to correct?
03:31Automation is how much can you let them run, right?
03:35How much do you need to review or build trust in the solution before you accept it?
03:41They sound quite similar, but there is a difference.
03:43You can have high autonomy in the sense that agents frequently are oneshotting the problems you're giving them, but low automation because you don't trust it yet.
03:53And so you're reviewing all of the code directly.
03:55You are manually verifying everything.
03:58So they are distinct things that obviously have a connection, right?
04:01You have to build up autonomy first before you can move to automation.
04:07The final quality, probably the most self-explanatory, how good is the product that you're actually producing for your users?
04:15These metrics are like your usual user analytics, your uh test quality, your test coverage, etc. As you work towards a factory, you're going to go improving autonomy.
04:27then approving automation all while keeping quality constant and then the sort of payoff at the end.
04:34Why would you want a software factory is that ultimately we think you can raise quality once you have a factory.
04:40So talk about this a bit.
04:42I think most folks look at factories as a way to ship faster.
04:48They'll say like oh yeah you'll get a little bit of slop but my god you'll you'll launch so much more features like the cost return the ROI will be worth it.
04:56I think that's a very shortterm phenomenon, right?
05:00As agents are sort of coming online and getting better.
05:04In reality, yes, you will move to higher velocity with a factory.
05:08But every team has backlogs of bug fixes or improvements that they wish they could be making that they just don't have time to do or explorations that they wish they were they were doing.
05:21With a software factory, the idea of a backlog kind of goes away.
05:24And so you actually just have much more capacity to work on test quality improvements, architectural refactors.
05:32So at Tesla, we believe that yes, at first as you transition, you will want to keep quality constant or maybe like a very small dip, but that the ultimate payoff is you should see better quality
05:43uh in the code that you're producing.
05:46Also, it's great for your overall team collaboration styles.
05:49So once you've moved to a software factory, it's much easier for people who aren't in technical roles to contribute ideas.
05:56So it's easier to explore more and to have more perspectives coming in and contributing while your engineering team is much more focused on building the underlying system that everyone is using to push features out to your users.
06:09So I think ultimately it leads to a more dynamic and inclusive uh development experience.
06:16So, how do you get to a software factory?
06:18Harness engineering, uh, as a sign of how fast terms evolve, it's also started to be called loop engineering over the last couple weeks.
06:27Um, this is really the core discipline that you're going to take to get towards a software factory.
06:34At its core, harness engineering is, you know, I mentioned the software factory is building your product.
06:40you are building the loops that automate and improve the quality of your factory.
06:47So what do I mean by loops?
06:49What are the loops that you might be working on?
06:52There are three core concepts each at maps to a certain phase of development.
06:59So you have your inner loop which is as the coding agent is working on a PR before it has put it up.
07:07Right?
07:07So these are very fast iterative loops.
07:09You want them to be cheap.
07:10You're expecting the agent to run them all the time.
07:13Improvements here will drive better autonomy.
07:16The they help catch things and correct the agent without your uh coming in and human intervention.
07:23Once the PR is up, you move to the outer loop.
07:27So this is where you're going to put more expensive exhaustive checks that don't make sense to run over and over again as the agent is iterating on features, but
07:37they do encode things that otherwise a human would have to verify to build confidence in the system.
07:42And so this is because they're more expensive, but they are taking away human review time.
07:47They can be worth it, right?
07:48So this is where you might put something like aentic QA or where you might say like run mutation testing to check the quality of our test suite.
07:58This is for expensive things you want to run once when the PR goes up and then have the agent iterate on on the results.
08:06Finally, you have the metal loop.
08:08This is where you'll drive quality.
08:11So the metal loop sits outside of the development process.
08:15It observes your coding agent logs.
08:17It looks at PRs, your issue tracker, user feedback, and it's basically finding mistakes that are making their way through the pipeline to your users or that you had to correct as a human to stop that from happening
08:33and it feeds back into the inner and the outer loop to make sure that that mistake is not made again.
08:39So the outer loop is what really kind of once you have the infrastructure for each of these loops, the outer loop or sorry the metal loop is where you're going to be actually driving from
08:50a certain percentage of like how AI native am I?
08:53You're going to be driving it up by investing in your metal loop.
08:59Before we get into like what are the components, what do you do to actually accomplish this?
09:04I think full disclosure upfront harness engineering at least uh raw unassisted harness engineering is hard for a few specific reasons that basically come down to human psychology.
09:16So the first hopefully this one will go away over time but it's a new discipline and it's changing very fast and so a lot of teams that get into harness engineering
09:29you basically become an AI researcher just to try and keep up.
09:32you're reading papers, you're watching blogs, you're coming up with best practices that go out of date the next week and then two weeks after that and actually at some point they become
09:42antiatterns and you don't want to do them anymore.
09:45Uh, and so there just like a lot of time and space you'll devote to keeping up with how to do harness engineering.
09:51And so if you want to get into this discipline, you need to have a sort of upfront approach to how you're going to solve this problem.
09:58How are you going to make time and space to keep up with the knowledge that's changing every week?
10:04The second, and this one is more durable in my opinion, harness engineering is fundamentally unplanned work.
10:10So you will start shipping a feature.
10:13You have no way to anticipate will agents fail?
10:16How will they fail?
10:18How long is it going to take me to fix it?
10:20And all of that work that you're going to do to try and make the agent better competes with shipping, right?
10:25So it will slow you down from getting the features out.
10:27And we all know that is just a perpetually hard trade-off to make.
10:32You either don't do it and you get stuck in the local maxima of the agent never improves or you do it and you miss your deadline for the feature and you get in trouble and so no one ever does it.
10:43The final piece is that once you've solved these initial problems to like make time and space for harness engineering, you'll find that a lot of the information you want access to
10:54is not available.
10:55it's hidden away in local coding agent logs or it is on someone's machine somewhere.
10:59It's in someone's head.
11:01A lot of the signals that you want to see to make agents better, you need to do some work to move your workflows onto surfaces where everything is saved and publicly available
11:12for later optimization loops.
11:16Okay, with all that aside, then what are the pieces that you should be working on as you're doing harness engineering?
11:23I've broken it into three layers.
11:25These aren't by any means standard terms, though if you want to help me make them so, I would be forever grateful.
11:32The first layer, and these are sort of like in the order you need to do them as far as Tesla is concerned, is you have to get your control plane right.
11:39And so I mentioned how most of the data that you want to have access to to make agents better and move towards a software factory is not legible.
11:49If you move your workflows into for example at Tessle we have all issue all work starts as an issue on the issue tracker.
11:58It gets sent to a headless agent running in a sandbox.
12:02The agent puts up a PR and then engineers engage with comments on the PR.
12:07So it still has manual correction.
12:08It still has manual interface but now all of your touch points with the agent are legible.
12:15So we can you can then point tools at them to pull that information down and make things better.
12:21Typically the three things that everybody comes to when they're trying to set up a control plane.
12:25So you have some way of tracking issues and kicking off agent work from that.
12:30You're going to have some way to review.
12:32GitHub PR review is probably like the easiest.
12:34Uh and then you're going to have some way to uh standardize and distribute workflows, playbooks, things that make the agents better.
12:42Typically, that ends up being something like a skills registry or a shared GitHub repo where you push skills that improve agent quality.
12:51Next is a massive grab bag that I call agent it, which is uh if anyone's ever tried to move like a project that had a lot of local config into something that someone else can set up and you realize with horror all of the dependencies that you thought were cleanly isolated and are not,
13:08you basically are going to have to go through the same thing with agents.
13:11There's going to be all these things that you thought were easily accessible by an agent, right?
13:16Like CLI, API access, giving a way for the agent to click through your product to build it.
13:23I I promise it's worse than you think.
13:25Like there's no one who has come in thinking like, yeah, yeah, yeah, it's not going to be that hard for us has left saying that at the end.
13:32And so there's just like a lot of pieces you'll work work through.
13:35And this is probably the most unplannable and most like spend a week, spend two weeks and just get it done.
13:43But it will come through things like writing writing down in your company brain how you get work done, what are your workflows, etc. You're going to have to give access to all your internal services, which will require governance and like API access questions.
13:59You're going to want to give a way to access production logs.
14:02Uh, and that's going to have its whole own question of like what's your compliance stance, things like that.
14:07Uh, and then you'll need an environment where the agent can execute the code that it is running.
14:12There's infinite more, but they're generally pretty company specific, and you'll just work through as you try to get agents to put code up without you intervening.
14:19You'll quickly learn what are the problems.
14:23Finally, this is where you'll end up spending most of your time.
14:26These are the improvement loops.
14:28So these are the things I mentioned before the like inner outer metal loop.
14:32These are components that tend to go into your metal loop.
14:35So repo maintenance.
14:36You're going to want things that sweep your codebase every day, every night, every week, something like that to look for problems.
14:44You're going to want playbooks for common development practices.
14:47So how to add a feature to your CLI.
14:51You're going to want to find a way to identify repeated tasks and automate them and ideally as quickly and easily as possible because otherwise people won't do it.
15:00And then you're going to want something that looks at the output quality just on a consistent basis and brings back improvements and learnings to the rest of your codebase.
15:08So say things like, oh, I saw the agent made this mistake.
15:11Let's update a skill on add like the playbook you have for adding features to the CLI.
15:16Let's update that so it doesn't make this mistake again in the future.
15:22So those three control planes Tessle sort of exist to try and help you solve them.
15:27As I mentioned, everything I've said before here are just techniques.
15:30You could go do them all yourself if you wanted.
15:32There's plenty of tools that do that.
15:34So I'm going to talk a little bit through how Tessle will help you if you want to use us.
15:40To start, why would you want to use us?
15:42So we focus really hard on making it relatively iterative and sustainable to get to the cutting edge.
15:50So we try to make it like a bunch of small lifts rather than one massive snake eating the moose all at once.
15:58We're batteries included.
15:59So we solve the knowledge gap by saying we will keep up and you will have an agentic experience that basically knows the best practices of harness engineering is kept up to date on your behalf.
16:10We have a real commitment to being modular and open.
16:13So we don't think every component of a factory is going to be bestin-class with a single company.
16:18We want to make sure you can pick and choose what is the most important piece for your stack.
16:23And then like I said, we focus on making it easy.
16:25So we have automated loops for you that you can install and just react to the changes that are proposed.
16:31And we really try to make sure it's just incremental steps so that you six months later like, oh wow, we're like 40% there to a software factory.
16:39We never had to stop and delay shipping or anything like that.
16:43So the first place that Tesla helps is in setting up your control plane.
16:47We have a skills registry where you can publish and version the workflows and the automations that you're working on with built-in governance things like security reviews, quality reviews, controls for who can publish and update what.
17:02We also have easy connectors for issue trackers to GitHub.
17:06Right now we're focused on linear to GitHub.
17:08So you can basically file a linear ticket and get it to run a GitHub workflow uh with more connectors coming soon.
17:15Uh and then we also have a suite of tools for code review.
17:18So this make it easy to set up your standards for agentic code review as well as more targeted checks.
17:25See here an example of something that like a workflow that has been posted to the registry being scanned for security quality uh and how much it actually improves the agents output.
17:38After this, I'm going to play a video while we talk.
17:40See how this goes.
17:41So, uh, we have an agentic experience called Tesla agent, which will help you with creating and then maintaining your improvement loops over time.
17:52So, the first thing that we do is we will help you with that change management of like how do I actually make the time to find repeated tasks and
18:02set them up in automation.
18:03So the agent will mine through PRs, issues, things like that and find like every week you do a hunt for flaky tests.
18:12Why don't we just set that up as a skill for the workflow and then we'll put it in automation on a GitHub action.
18:18So we help you actually make time to do the automations just like one workflow at a time.
18:23The second is that we come with a lot of out ofthe-box maintenance tasks.
18:27So I mentioned you want to have weekly scans for things like architecture quality, code duplication, uh how good are your test suites, do you have any security vulnerabilities?
18:38So if you use Tesla, you can just sort of like oneclick install a bunch of those things and just get weekly improvements to your codebase without any other effort.
18:47And then finally, we offer a way to turn any skill into an automated workflow.
18:51So it's through a command called Tesla launch where you effectively take a sk a skill that codifies a workflow.
18:58You can then pick a particular coding agent that you want to handle it.
19:02Tesla agent can be one of them.
19:04But for most coding tasks you probably want to use something like codeex, cloud code, gemini.
19:08We have access to all of them and we will run them in a sandbox that has the appropriate permissions.
19:14Can be long running.
19:15Uh we'll have GitHub token so that it can put up a PR and respond to comments as you leave them.
19:20Uh, so it just makes it really easy to automate workflows like that.
19:24And so with these things together, we just make it really easy to get started and just one at a time find an improvement, take a workflow, turn it into a skill,
19:33ship it to everyone, and then get your improvement loops going.
19:39The what you're going to want to focus on as you're doing this work is driving down manual takeovers, driving down human PR comments.
19:47So these are like metrics that you can track to see like how far am I towards software factory.
19:52And then over time you want to see more PRs initiated without human input.
19:55That's a sign that you're automating more.
19:58And then of course you want to first hold quality constant and then aim to drive it up after that.
20:05That's it.
20:05If you want to learn more Tesla booth is just that way.
20:08Please give a scan, come over, see us.
20:10We'd love to chat.
20:11Um thank you all for your time.