WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=TN3mj92oZ8I

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/TN3mj92oZ8I.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.520 --> 00:00:13.680
Um my name is Siraj Gupta.

s2
00:00:13.680 --> 00:00:15.880
I lead Harness development at Warp.

s3
00:00:15.880 --> 00:00:23.240
Um in case you haven't heard of Warp before, um we started off by building a state-of-the-art agentic development environment.

s4
00:00:23.240 --> 00:00:26.120
Um we have about a million active users that use that.

s5
00:00:26.120 --> 00:00:28.720
Um it was born out of a modern-day terminal.

s6
00:00:28.720 --> 00:00:35.360
Um and nowadays we're building a um cloud agent platform to enable teams to build software factories.

s7
00:00:35.360 --> 00:00:46.480
Um today I'm going to talk about um self-improving software factories, which I know might sound kind of buzzy, um but I'm going to break it down and hopefully you can walk out of here with something concrete

s8
00:00:46.480 --> 00:00:48.480
that you can implement yourselves.

s9
00:00:48.480 --> 00:00:57.240
Um you've probably heard of both of these terms, self-improvement and software factories, used throughout this conference, but I'm sure you've heard of them independently.

s10
00:00:57.240 --> 00:01:12.800
Um you've heard about how self-improvement is about agents being able to um improve over time and agents and models and, you know, phasing out humans from that loop, so that way the the improvement loop can be automatic.

s11
00:01:12.800 --> 00:01:25.320
Um and I'm sure you've heard about software factories being um the way that like the software development paradigm is shifting from individual coding agents to building out automations that take work from triage

s12
00:01:25.320 --> 00:01:26.760
up until production.

s13
00:01:26.760 --> 00:01:36.560
Um but less has been said about how software factories themselves can self-improve over time, so that the the factory can get better, more efficient, faster.

s14
00:01:36.560 --> 00:01:46.880
Um and I think this is going to become an extremely important part of our jobs as software engineers as we transition from building products to building factories and and maintaining them.

s15
00:01:46.880 --> 00:01:48.680
Um so let's dive in.

s16
00:01:48.680 --> 00:01:58.080
Um self-improvement is a pretty broad umbrella, but I'm going to focus on three concrete ways that agents can improve over time, agent factories can improve over time.

s17
00:01:58.080 --> 00:02:04.479
I'm going to start with um skills, then we'll talk about persistent memory, and then I'll finish off by talking about model routing.

s18
00:02:04.479 --> 00:02:09.119
Um so skills are a great way to give your agents procedural memory.

s19
00:02:09.119 --> 00:02:11.440
I don't think I have to kind of spell that out for you.

s20
00:02:11.440 --> 00:02:17.120
You describe a a procedure to your agent that you expect it to have to do a task over and over.

s21
00:02:17.120 --> 00:02:23.040
Um for example, let's say you're building a triage agent, which is a pretty common thing you might build in a software factory.

s22
00:02:23.040 --> 00:02:28.000
Um and you give it a skill on how you want it to go about reproducing issues.

s23
00:02:28.000 --> 00:02:30.040
Um those skills get stale over time.

s24
00:02:30.040 --> 00:02:32.760
Um humans give feedback to the agents.

s25
00:02:32.760 --> 00:02:36.840
Agents learn things in their own um runs and trajectories.

s26
00:02:36.840 --> 00:02:41.760
Um and the like the skill that you previously gave it becomes dated.

s27
00:02:41.760 --> 00:02:45.880
So, how do we fix this and let the agent improve over time using that skill?

s28
00:02:45.880 --> 00:02:49.000
Well, one way to do it is to deploy an outer loop agent.

s29
00:02:49.000 --> 00:02:53.720
Um and this is what we've done in our own internal software factory that's been really productive.

s30
00:02:53.720 --> 00:02:59.920
The idea is that your um your inner loop agent is the thing that's actually applying the skill.

s31
00:02:59.920 --> 00:03:01.480
It's like doing the triage.

s32
00:03:01.480 --> 00:03:14.080
And then you have this other outer loop agent that is observing your inner loop agent's runs and improving its skill over time by looking for um for mistakes it made or looking at feedback that the human has given it.

s33
00:03:14.080 --> 00:03:16.800
Um so, I'm going to show an example of that.

s34
00:03:16.800 --> 00:03:19.600
Um let me swap here.

s35
00:03:19.600 --> 00:03:20.239
Cool.

s36
00:03:20.239 --> 00:03:23.480
Um so, this is actually a skill that we've used internally at Warp.

s37
00:03:23.480 --> 00:03:28.200
We've open-sourced our client repository and it's fully run as as a factory.

s38
00:03:28.200 --> 00:03:34.800
And so, this is an example of an of a of a skill that's that's backing a triage agent that we have today.

s39
00:03:34.800 --> 00:03:50.840
The idea is that um you you get an issue that comes in on GitHub, and you want to automatically run an agent to figure out like what sorts of um like it what sort of information did the user not provide that is going to help us figure out if this is something that we should build, an existing

s40
00:03:50.840 --> 00:03:54.440
issue that we might actually just want to dedupe against or something else.

s41
00:03:54.440 --> 00:03:59.440
Um and then we have a Sorry, let me just switch over here.

s42
00:03:59.440 --> 00:04:06.160
And then we have a workflow that actually like runs that that triage um agent.

s43
00:04:06.160 --> 00:04:07.720
Um this is a pretty simple workflow.

s44
00:04:07.720 --> 00:04:13.560
It just Every time an an an issue gets submitted on the repo, this um this workflow runs.

s45
00:04:13.560 --> 00:04:20.720
And then here's the actual outer loop agent, which is about looking at the the work that the triage agent has done and improving it over time.

s46
00:04:20.720 --> 00:04:33.200
So, the idea is that the agent looks at the decisions that the triage the inner loop triage agent made, and it um collects feedback signals, which might be uh you know, thumbs up or thumbs down on the issues.

s47
00:04:33.200 --> 00:04:35.440
Um it might be users commenting on the issue.

s48
00:04:35.440 --> 00:04:42.040
It might be someone from Warp um an employee of Warp commenting on the issue, uh giving the agent feedback on what it did.

s49
00:04:42.040 --> 00:04:49.000
Um and then ultimately that leads to like some synthesis of of the of that um iteration.

s50
00:04:49.000 --> 00:04:54.919
And so, we we end up basically updating our inner loop skill with this outer loop agents.

s51
00:04:54.919 --> 00:04:58.000
Um and that's what leads to this self-improvement loop with skills.

s52
00:04:58.000 --> 00:05:05.360
What's really nice about this is that um the So, in step four, you'll see we we added the triage skill, and then we open a pull request.

s53
00:05:05.360 --> 00:05:10.760
So, that means that um all of the improvements to the inner loop skill are going to be tracked through get.

s54
00:05:10.760 --> 00:05:15.480
So, you get like full observability into like how that skill is transformed over time.

s55
00:05:15.480 --> 00:05:19.720
And that it also means that um a human is actually going to review those updates to the skill.

s56
00:05:19.720 --> 00:05:26.320
So, that way, you know, this this outer loop agent doesn't make a mistake and ultimately cause your triage agent to to actually perform worse.

s57
00:05:26.320 --> 00:05:31.240
Um yeah, so that's that's kind of what what I want to talk about for skills.

s58
00:05:31.240 --> 00:05:35.200
Um let me talk about persistent memory now.

s59
00:05:35.200 --> 00:05:37.360
So, I'm going to go back to my deck.

s60
00:05:37.360 --> 00:05:39.800
Um cool.

s61
00:05:39.800 --> 00:05:43.880
So, skills are great for remembering procedures, but what about everything else?

s62
00:05:43.880 --> 00:06:00.400
So, like what what happens when your Sentry agent um looks for an issue and and finds an issue, does some work to like gather context about it, and fixes it, but then now you have you encounter a similar issue in the future.

s63
00:06:00.400 --> 00:06:08.160
Your century your agent might get lucky and like be able to figure out the same root cause that it did the first time around, but that's not guaranteed.

s64
00:06:08.160 --> 00:06:15.000
And and even if it does, you're you've probably wasted a lot of tokens re-gathering context on an issue that you've already fixed before.

s65
00:06:15.000 --> 00:06:18.720
So, that's where something like persistent memory comes in.

s66
00:06:18.720 --> 00:06:22.200
You can think of this as a fact store scoped to an agent.

s67
00:06:22.200 --> 00:06:30.720
Your agent runs and similar to the skill improvement loop, you have an outer loop agent that's going to extract facts, learnings, outcomes from your from your inner loop agent.

s68
00:06:30.720 --> 00:06:35.160
So, that way future runs of the agent can lean on what's already been done in the past.

s69
00:06:35.160 --> 00:06:37.200
So, I'm going to show an example of this.

s70
00:06:37.200 --> 00:06:39.320
Come back to the deck here.

s71
00:06:39.320 --> 00:06:46.720
So, in warp or in Oz rather, like I mentioned, we built out a a cloud agent platform where you can run agents in the cloud.

s72
00:06:46.720 --> 00:06:51.000
And for all for all these agents, you can attach memory stores to them.

s73
00:06:51.000 --> 00:06:54.200
Again, a memory a memory store is a collection of facts.

s74
00:06:54.200 --> 00:07:03.440
In this case, I have a century agent that has collected some memories over time about root cause analysis that it's done in the past.

s75
00:07:03.440 --> 00:07:13.200
What's cool is like you can actually version this memory, you can update it as a human, create new memories or delete them yourselves, but primarily this this memory creation loop is driven by agents.

s76
00:07:13.200 --> 00:07:16.360
You can also see where these memories were sourced from.

s77
00:07:16.360 --> 00:07:22.400
So, I can actually like open up the run here to see Oh, well.

s78
00:07:22.400 --> 00:07:33.000
I think this my IP address here keeps changing and this my our staging platform is gated to an IP address, but anyways, you should like you can see where those memories were sourced from.

s79
00:07:33.000 --> 00:07:40.400
So, that way you can be like, oh, like maybe that wasn't actually an important memory or that was kind of like a local maxima that I don't want to include in my memory store.

s80
00:07:40.400 --> 00:07:49.400
Um and then when your agents continue to run, so in this case I have the sentry agent that's um that's done a bunch of runs after it's created those memories,

s81
00:07:49.400 --> 00:07:55.080
it actually can lean on those memories in the future um when it's making um when it's triaging new issues.

s82
00:07:55.080 --> 00:08:04.160
So in this case, you'll notice that um this agent ran, it did a bunch of work to do some root cause analysis, and then it ultimately reported back on which memories it used

s83
00:08:04.160 --> 00:08:13.800
um uh in this case it it it it was able to use those um five memories that I had in that memory sort to to speed up the work that I did this time around.

s84
00:08:13.800 --> 00:08:16.080
Um, cool.

s85
00:08:16.080 --> 00:08:19.560
Uh let me go back to the deck.

s86
00:08:20.400 --> 00:08:29.760
So, what's cool in Oz is that persistent memory works across all harnesses, so Warp Zone, proprietary harness, plot code, Kodak, it doesn't really matter what you're running on Oz,

s87
00:08:29.760 --> 00:08:32.200
um memories will automatically be created for you.

s88
00:08:32.200 --> 00:08:39.719
Um, but you also have the control to to review those memories, version them, delete them, or edit them as you please.

s89
00:08:39.719 --> 00:08:42.800
Um, all that being fully traceable.

s90
00:08:42.919 --> 00:08:43.919
Um, cool.

s91
00:08:43.919 --> 00:08:45.880
That's what I wanted to say about memory.

s92
00:08:45.880 --> 00:08:48.839
Um, last thing I want to talk about here is model routing.

s93
00:08:48.839 --> 00:08:54.040
So, um you may be wondering how this plays into self-improvement loops for um factories.

s94
00:08:54.040 --> 00:08:57.640
Well, first I want to touch on why model routing is important in a factory in the first place.

s95
00:08:57.640 --> 00:09:05.880
Um, it can become prohibitively expensive to run all of your agents that are doing simple things like triage or fixing simple CI failures with Opus.

s96
00:09:05.880 --> 00:09:14.040
That's just like going to be really expensive over time, and I'm sure some of you have maybe been burnt by this or have heard stories of of or organizations being burnt by this.

s97
00:09:14.040 --> 00:09:21.520
Um, so model like choosing which models you use for which tasks that is actually very important um and something that we're we've been working on at work.

s98
00:09:21.520 --> 00:09:27.280
Um, so in Warp, you can actually use uh our out-of-the-box model routers, we call them our auto models.

s99
00:09:27.280 --> 00:09:33.839
Um, over like as new models come out, we're all we're always evaluating like what are the best models when it comes to Pareto efficiency.

s100
00:09:33.839 --> 00:09:42.040
And so, like if you're a new user and you just want to get started, it's much better than like pinning your model to like Opus or or or Haiku, for example.

s101
00:09:42.040 --> 00:09:47.920
Um let me show you all um that experience in Warp.

s102
00:09:47.920 --> 00:09:51.640
So, let me go here.

s103
00:09:53.080 --> 00:09:54.640
Give me 1 sec.

s104
00:09:54.640 --> 00:09:56.920
Got to drag this over here.

s105
00:09:56.920 --> 00:10:07.480
So, here I have my Warp desktop um and you can see that um alongside being able to define models um being able to use Warp's auto models is one option,

s106
00:10:07.480 --> 00:10:16.960
like these models here, but you can also define your own model routed um uh models, which uh basically boil down to like a set of rules that you can give

s107
00:10:16.960 --> 00:10:21.200
um to us and we will like make decisions about model routing based on those rules.

s108
00:10:21.200 --> 00:10:31.520
So, here I have an example of a configuration file that I've defined um where I've defined a few different rules about like I want database migrations done with with GLM fire with GLM

s109
00:10:31.520 --> 00:10:36.800
or I want um you know, uh runbooks and API documentation done with Quen.

s110
00:10:36.800 --> 00:10:47.960
Um So, this really allows you to like like to define classes of tasks and be able to say, "Okay, I want this this class of task done by this model because that's kind of what you've seen."

s111
00:10:47.960 --> 00:10:56.440
At this point though, that's kind of more of an art than a science um and so uh what we're working on next is being able to define evals for

s112
00:10:56.440 --> 00:11:11.480
um customer-facing evals where you can actually define what sorts of things you care about and what knobs you want to turn on these configurations and actually see, "Okay, is is Haiku better than fi- than GLM than Opus in a specific class of issues?"

s113
00:11:11.480 --> 00:11:19.080
That's like specific to your workflows and not just some like generic benchmark that um that someone posted online.

s114
00:11:19.120 --> 00:11:20.560
Um let me go back here.

s115
00:11:20.560 --> 00:11:22.760
Give me a sec.

s116
00:11:24.760 --> 00:11:26.760
Cool.

s117
00:11:30.720 --> 00:11:31.120
Cool.

s118
00:11:31.120 --> 00:11:37.960
Um and yeah, so let me uh I'm in my slideshow.

s119
00:11:42.880 --> 00:11:46.400
So, here is kind of like a a high-level overview of how this works in Warp today.

s120
00:11:46.400 --> 00:11:56.600
So, um internally we have our model routing rules, um but then we have like an eval sidecar that's determining um what sorts of tasks are actually better with which models.

s121
00:11:56.600 --> 00:12:04.720
And we do this with a simple like best-at-k um approach where when you type a prompt, we run a bunch of agents in Oz uh across a different set of models.

s122
00:12:04.720 --> 00:12:10.320
Um and we found success in like being able to determine, "Oh, like UI tasks are really well done with GLM.

s123
00:12:10.320 --> 00:12:11.680
Um we don't really need to run those with Opus.

s124
00:12:11.680 --> 00:12:14.000
It's much more efficient to run those with GLM."

s125
00:12:14.000 --> 00:12:22.080
Um so, that's how we've been using it, but um sooner enough we're going to be building this into our product so that way our our customers can actually use it in their own workflows.

s126
00:12:22.080 --> 00:12:25.800
Um Yeah, that's uh that's mostly what I had today.

s127
00:12:25.800 --> 00:12:28.760
Um we're around in a booth around that side.

s128
00:12:28.760 --> 00:12:31.400
So, if you're interested in this kind of stuff, come talk to me.

s129
00:12:31.400 --> 00:12:33.320
I'm always down to talk about this kind of stuff.

s130
00:12:33.320 --> 00:12:34.720
But yeah, thanks for your time today.

s131
00:12:34.720 --> 00:12:36.960
That's all I had.

s132
00:12:49.633 --> 00:12:51.633
[music]