WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=DouK81O8R3o

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/DouK81O8R3o.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.600 --> 00:00:19.800
For the past couple years, feels like all we've been talking about is how to increase the velocity of our SDLC.

s3
00:00:19.800 --> 00:00:21.760
Y'all know what I'm talking about.

s4
00:00:21.760 --> 00:00:25.560
And specifically this big fat build chunk in the middle, okay?

s5
00:00:25.560 --> 00:00:29.240
Where we've been doing a lot of our coding and intellectual work.

s6
00:00:29.240 --> 00:00:35.480
And sure enough, at least where I work, that step has basically been obliterated, right?

s7
00:00:35.480 --> 00:00:38.080
And we've reclaimed a bunch of time.

s8
00:00:38.080 --> 00:00:51.760
But I think and I'm no prophet in imagining some future where agents are doing the work and humans are there in the middle as exec or as executives signing off on each stage.

s9
00:00:51.760 --> 00:01:03.280
But I work at Geico and we're not a tech company, we're a risk management company and risk management is baked into everything that we do, including the way we develop and deliver software.

s10
00:01:03.280 --> 00:01:18.600
So while we've seen this inner loop speed up to use Gene Kim's concept, we find ourselves seemingly stuck within the bounds of this outer loop, which is to say everything else that happens other than the coding and the building.

s11
00:01:18.600 --> 00:01:19.680
[snorts]

s12
00:01:19.680 --> 00:01:26.560
And we find ourselves wondering what life might be like outside of these walls in a world where the outer loop keeps pace with the inner loop.

s13
00:01:26.560 --> 00:01:36.800
So this is a story of how we broke free, as it were, and my intention with this talk is to power you guys to do the same.

s14
00:01:39.160 --> 00:01:53.400
Now when Geico manages America's fleet of automobiles or the risk therein and something happens on the road like an accident, we incur what we call loss, which is typically

s15
00:01:53.400 --> 00:02:02.600
reported by one of our policy holders and it becomes a claim and claims technology is where the policy comes to life and gets the customer back on the road as quickly as possible

s16
00:02:02.600 --> 00:02:05.960
with minimal disruption, which we call a resolution.

s17
00:02:05.960 --> 00:02:06.420
And I'm I'm

s18
00:02:06.420 --> 00:02:06.560
[snorts]

s19
00:02:06.560 --> 00:02:07.440
Ford Prior.

s20
00:02:07.440 --> 00:02:14.360
I'm a senior staff engineer and tech evangelist embedded in the claims tech group at Geico.

s21
00:02:14.360 --> 00:02:26.440
And I think I'm the only tech evangelist at Berkshire Hathaway, but basically what that means is within our 300 engineer group, we have like 32 pizza teams as we used to call them

s22
00:02:26.440 --> 00:02:29.440
with a tech lead and counterpart for each business counterpart for each.

s23
00:02:29.440 --> 00:02:34.560
And my job is to find and implement best practices in our SDLC.

s24
00:02:34.560 --> 00:02:43.440
I run a claims innovation lab, and I share stories like this one to attract the best engineers in the world to our problem space, which we find very interesting,

s25
00:02:43.440 --> 00:02:45.880
not to mention important part of the American economy.

s26
00:02:45.880 --> 00:02:50.240
So, a little bit more context in how we develop software.

s27
00:02:50.240 --> 00:02:55.440
Um as many organizations do, we conceive of our work in terms of semantic versioning.

s28
00:02:55.440 --> 00:03:07.320
And about 6 months ago, we looked at our backlogs and found that our engineers, after they finished their latest patch or chased down the latest minor upgrade, had only 25%

s29
00:03:07.320 --> 00:03:14.200
on a good day or less of their bandwidth to dedicate towards the major changes that keep us competitive, step ahead of the competition.

s30
00:03:14.200 --> 00:03:16.519
That's like V1 to V2 to V3.

s31
00:03:16.519 --> 00:03:31.040
So, we began strategically investing in Geico managed AI agents on a variety of harnesses using a variety of models to offload that work all the way from spec to deployment for our patches and minor work.

s32
00:03:31.040 --> 00:03:42.000
Um with the goal of redirecting or rerouting that work so that our engineers can stay our our brilliant engineers can stay focused on the stuff that matters and that they find most joyful, which is to do the major work.

s33
00:03:42.000 --> 00:03:46.200
Which again, as I mentioned, is key to our competitive edge.

s34
00:03:46.280 --> 00:03:48.239
Okay, so this is current state.

s35
00:03:48.239 --> 00:04:02.560
But as we look at this major work, we find ourselves rate-limited by our ability to collaborate in rooms full of the right individuals summoned at the right time to make the right decisions at every interval in whatever

s36
00:04:02.560 --> 00:04:07.480
life cycle would need to happen in order to get get get the major work to prod.

s37
00:04:08.560 --> 00:04:25.160
And not only are we missing the ability to collaborate, but we're also missing this intelligence, this ability to weave together all the variables relevant to the problem space as we learn it and deepen our understanding of it, including the fact checking, the code crunching, basically everything that needs to happen to serve up the insights

s38
00:04:25.160 --> 00:04:28.840
as quickly as the decisions come up.

s39
00:04:28.960 --> 00:04:32.720
And that's what ultimately gives us the confidence to bring those major changes to market.

s40
00:04:32.720 --> 00:04:34.720
Okay, so it's collaboration and intelligence.

s41
00:04:34.720 --> 00:04:38.080
We're missing both of those at the major change scale.

s42
00:04:38.080 --> 00:04:42.680
So to solve for this, we created a multiplayer AI-native war room.

s43
00:04:42.680 --> 00:04:54.640
And it consists of a roster of key decision makers and subject matter experts relevant to the topic, a driver to wield these expensive AI tools, and an AI-native productivity suite,

s44
00:04:54.640 --> 00:04:59.440
um which includes our comms channels for all the chatter, an interactive design studio.

s45
00:04:59.440 --> 00:05:02.560
I can't mention the vendor name, but y'all can take a wild guess.

s46
00:05:02.560 --> 00:05:08.800
Um to uh for all of our our collaborative elements, and then obviously we have our code repositories.

s47
00:05:09.600 --> 00:05:23.120
And all of these are wired into a powerful combination of workflows, agent skills, MCPs, enterprise MCPs, um and these agent fleets to do the heavy lifting, back checking, synthesis, etc.

s48
00:05:23.120 --> 00:05:31.919
So my question was, how might we accelerate the delivery of agent-callable APIs for our legacy core capabilities?

s49
00:05:32.088 --> 00:05:32.280
[snorts]

s50
00:05:32.280 --> 00:05:36.040
And for context, our legacy claim systems is like millions of lines of code, okay?

s51
00:05:36.040 --> 00:05:43.240
So um and the deep context that we're trying to build here um gets built once and then can be reused.

s52
00:05:43.240 --> 00:05:56.200
So that's just some additional context, but my bet was that with a thousand dollars in hand of AI tokens in one day we could map out one domain end-to-end including like graph representations

s53
00:05:56.200 --> 00:06:02.440
of the business topology, all of the complex code um in our legacy systems and everything in between.

s54
00:06:02.440 --> 00:06:15.960
And we can have the agent find and propose one serious change um which in our case we scoped down to an agent callable API into our legacy claim systems.

s55
00:06:16.120 --> 00:06:32.520
And have the whole thing deployed to our skills agent platform and running for an a a a demo to my boss at 4:00 p.m. So, I put the pressure on myself um but we ran this experiment three different times across three different code modernization

s56
00:06:32.520 --> 00:06:33.560
missions.

s57
00:06:33.560 --> 00:06:40.600
Um and this little taxonomy here uplift, transform, reimagine is from the Anthropic code modernization plugin which I'll touch on in a minute.

s58
00:06:40.600 --> 00:06:51.960
Um but the story is um that we first we looked at our rentals domain which is a really simple one um that is like all the tech behind getting you guys your rental cars when your primary car gets in an accident.

s59
00:06:51.960 --> 00:06:58.960
And we implemented a newer version of the same tech which is again the definition of uplift and we extended an existing API.

s60
00:06:58.960 --> 00:07:12.720
And in this case it was an interesting story because the agent used some uh interesting um uh process to find and distill data to figure out that we could connect a couple dots and save ourselves

s61
00:07:12.720 --> 00:07:18.480
eight and a half thousand calls per month by giving the customer up-to-date rental information.

s62
00:07:18.480 --> 00:07:24.040
Um so that was the first experiment that we ran um and then the second one was transformation of our damages domain.

s63
00:07:24.040 --> 00:07:28.880
And damages is like arguably our most complex No, sorry.

s64
00:07:28.880 --> 00:07:31.720
Damages I'm meaning from notes here.

s65
00:07:31.720 --> 00:07:36.040
Damage domain is how we put dollar signs on a reported loss.

s66
00:07:36.040 --> 00:07:43.090
And so for this case we built in a set of new agent callable APIs to unlock a larger workflow that needed damages data.

s67
00:07:43.090 --> 00:07:43.280
[snorts]

s68
00:07:43.280 --> 00:07:48.520
And then lastly, we reimagined our salvage domain, which is arguably the most complex of the three.

s69
00:07:48.520 --> 00:07:51.280
Um it's basically everything that happens when your car's totaled.

s70
00:07:51.280 --> 00:08:01.360
Um and in this case, we produced a fully vetted migration plan with API specs and design mock-ups, ratified, and this is the most important part, green-lit by product and tech at every level.

s71
00:08:01.360 --> 00:08:09.600
Okay, so it's like a fully vetted project plan that's ready to be implemented, which is like as you guys know is like that's like a a milestone unto itself, getting everybody on the same page.

s72
00:08:09.600 --> 00:08:11.600
Um including the technical details.

s73
00:08:11.600 --> 00:08:16.480
Um and honestly, like in this case, it understood the domains really well.

s74
00:08:16.480 --> 00:08:20.640
And um in this and the the story with the salvage work was that it found a prod bug.

s75
00:08:20.640 --> 00:08:29.080
I can't talk about more, but it was Um so anyways, uh you may be curious about kind of what like a little bit more detail into how this all worked.

s76
00:08:29.080 --> 00:08:34.919
Um really like we had our friends at Anthropic who provided a bunch of stuff, and then we had us that provided a lot of stuff.

s77
00:08:34.919 --> 00:08:38.110
So on the left here, we have um

s78
00:08:38.110 --> 00:08:39.080
[snorts]

s79
00:08:39.080 --> 00:08:46.280
the harness, the model, this thing called FleetKit, which I'll explain in just a moment, and this plug-in that did code modernization tasks.

s80
00:08:46.280 --> 00:08:48.680
That's on the left, brought by the vendor.

s81
00:08:48.680 --> 00:08:55.640
Okay, we could replace any of that, by the way, with another vendor, another model, another framework, etc. And on the right was the stuff that we can't replace.

s82
00:08:55.640 --> 00:08:57.640
This is everything GEICO, and we need a ton of it.

s83
00:08:57.640 --> 00:09:07.880
We had context in our knowledge base, including software standards and all of this past decision archaeology we had to do, list of tools that the agent could access on my machine,

s84
00:09:07.880 --> 00:09:10.520
and instructions for the agents on how to use it.

s85
00:09:10.520 --> 00:09:11.760
We had the methods.

s86
00:09:11.760 --> 00:09:17.760
I'm doing these out of order here, but we had methods, which is essentially like all of our skills and all of the the the kind of the quirky domain-specific,

s87
00:09:17.760 --> 00:09:21.720
highly optimized way that we develop software, probably different from the way you guys do it.

s88
00:09:21.720 --> 00:09:26.640
Um and um and lastly, judgment, which is like who can decide what and at what gate, right?

s89
00:09:26.640 --> 00:09:33.400
So both from the subject matter expert point of view and from the um the uh decision-maker leadership point of view.

s90
00:09:33.400 --> 00:09:36.120
Um and um we grouped them into two repos.

s91
00:09:36.120 --> 00:09:43.000
and why in a minute, but that allowed us to ver- allows us to have the agents add to it and correct it um and also humans to do the same.

s92
00:09:43.000 --> 00:09:48.320
Um but that AI, and this is kind of one of the interesting parts, uh is is expensive, right, to run at scale.

s93
00:09:48.320 --> 00:09:53.680
And so, we needed a fleet manager, like a PhD-level fleet manager.

s94
00:09:53.720 --> 00:10:04.760
And this was a bit of a collaboration with uh with Anthropic, and it isn't open source yet, but I can share how it works today and hopes that you guys can construct your own version cuz it's not rocket science.

s95
00:10:04.760 --> 00:10:11.360
So, FleetKit is this tight structured collection of rules, protocols, and instructions all written in markdown.

s96
00:10:11.360 --> 00:10:17.920
Optimized for the Claude harness, specifically Fable, and designed around the core task of architecture transformation.

s97
00:10:17.920 --> 00:10:24.160
Um and as you guys can kind of guess, like this type of thing doesn't really work with some of the previous iterations of models.

s98
00:10:24.160 --> 00:10:32.280
I think like Opus 48 and beyond I I I would I would guess is like something that could handle this type of like uh steering.

s99
00:10:32.280 --> 00:10:36.040
Um cuz it's like one level above um the actual system prompting, right?

s100
00:10:36.040 --> 00:10:39.800
This is like rules that it has to follow, the agent has to follow.

s101
00:10:39.800 --> 00:10:43.080
Um and um it's pretty simple.

s102
00:10:43.080 --> 00:10:48.640
Takes a goal, breaks it down, dispatches leaders to do all the research, and decide about the approach.

s103
00:10:48.640 --> 00:10:51.640
It puts everything into this like human-in-the-loop queue.

s104
00:10:51.640 --> 00:10:59.120
You could use GitHub issues, for example, something that the humans can comment on and approve or that other agents can add more context to.

s105
00:10:59.120 --> 00:11:04.480
And then when everything looks tight from a approval standpoint, it dispatches builders to do the rest.

s106
00:11:04.480 --> 00:11:05.960
And in this case, we use beads.

s107
00:11:05.960 --> 00:11:12.680
I'm not sure if y'all know about beads, but beads kind of helps the agents keep track everything um within their own agent space.

s108
00:11:12.680 --> 00:11:22.000
Um and um ultimately, the goal is to produce what we call a material change, which is like a a change request into our systems or code change into our code repositories.

s109
00:11:22.000 --> 00:11:30.120
And the ultimate like idea with this though is that like developing software looks less like playing in a canvas and more like working in a factory.

s110
00:11:30.120 --> 00:11:33.360
Um Okay, I need to go faster.

s111
00:11:33.360 --> 00:11:36.760
Uh so ultimately on the left here you see these three repositories.

s112
00:11:36.760 --> 00:11:40.480
They contain everything that Fleet Kit needs to run including the Fleet Kit source.

s113
00:11:40.480 --> 00:11:44.840
Um and by the way, all of these slides are available on the blog post that I'll link at the very end.

s114
00:11:44.840 --> 00:11:50.800
I can't tell with these lights who's taking pictures, but I just want to let you all know that if you want to look at the uh the the visualizations

s115
00:11:50.800 --> 00:11:55.280
that I'm showing here, they're all in a blog post that was released today and I link it at the end of the slides.

s116
00:11:55.280 --> 00:12:02.000
So anyway, um so anyway, all this boots up, gets pulled into a cloud cloud code session with a plugin, gets invoked, and you give it a mission.

s117
00:12:02.000 --> 00:12:08.040
It adopts or creates a Slack channel um or a in our comms platform.

s118
00:12:08.040 --> 00:12:12.120
And um and then that's that that that mission channel is where everyone communicates.

s119
00:12:12.120 --> 00:12:16.240
All the war room members, the people who are relevant to the decisions in the project are invited.

s120
00:12:16.240 --> 00:12:24.920
And um every time the agent reaches a milestone um as it follows its nine-step process, it creates one of two artifacts.

s121
00:12:24.920 --> 00:12:31.280
Um and by the way, on the right there where it says war room, that's where it like it can ping individual people based on the roster.

s122
00:12:31.280 --> 00:12:37.600
So for example, if it's missing missing some product details, it'll figure out who owns that from a product perspective and mention that individual person.

s123
00:12:37.600 --> 00:12:43.680
Um but other than that, it creates these worksheets that are like actually really well-done HTML using cloud artifacts.

s124
00:12:43.680 --> 00:12:44.680
People can fill them out.

s125
00:12:44.680 --> 00:12:46.160
They're automatically saved.

s126
00:12:46.160 --> 00:12:47.960
Um and that's to fill the gaps.

s127
00:12:47.960 --> 00:12:57.600
And then also creates design documents including like these like complex visualizations of our architecture that didn't exist before um that teams were delighted to get cuz they kept them afterwards

s128
00:12:57.600 --> 00:13:11.040
um that we can then edit and then the agent can then reread our edited like uh design document like a draw.io or Figma, and then all that gets that all that gets slurped back up into the the um the session.

s129
00:13:11.040 --> 00:13:11.920
Okay.

s130
00:13:11.920 --> 00:13:16.489
Um and if you had stumbled into our war room in Richmond,

s131
00:13:16.489 --> 00:13:16.560
[snorts]

s132
00:13:16.560 --> 00:13:24.839
Virginia, which is where I'm from, um uh this summer you would have seen either a big cloud code screen um with a bunch of people huddled around it or one of these two screens here.

s133
00:13:24.839 --> 00:13:29.560
On the left is like a product worksheet that we had to fill out.

s134
00:13:29.560 --> 00:13:33.440
And then on the right um there it's like one of our our diagramming tools.

s135
00:13:33.440 --> 00:13:39.520
And so like the big gist with this and the thing that was so delightful for me is that these were really high-quality artifacts.

s136
00:13:39.520 --> 00:13:43.480
So, there wasn't like ever a question of like why is the agent asking me dumb questions?

s137
00:13:43.480 --> 00:13:50.200
It was like, "Wow, that's actually a great question that we've never thought about identifying an edge case either in the business or with the technology."

s138
00:13:50.200 --> 00:14:00.240
Um and then the design artifacts, like I said, the teams could keep them afterwards and oftentimes they were mapping stuff that just no one had taken the time to do cuz as you all as you all know not every engineer loves doing diagrams.

s139
00:14:00.240 --> 00:14:03.560
Okay, I'm skipping stuff here, but we're going to keep moving cuz I got like 3 minutes.

s140
00:14:03.560 --> 00:14:15.360
Um these war room tools um just really simply uh basically there's two types of agents, the code dives that we called them um where it did lots of code crunching and graphing over of our legacy

s141
00:14:15.360 --> 00:14:17.920
legacy domain-specific languages.

s142
00:14:17.920 --> 00:14:30.160
Um that happened in the cloud and then we had local agents running that had access to all of my laptop all the MCPs on my laptop, uh many of which were not in production yet, and enterprise-specific,

s143
00:14:30.160 --> 00:14:37.640
highly optimized for our problem space um and that could surface really viable data including like chunked versions of the code bases that were being dived.

s144
00:14:37.640 --> 00:14:45.440
So, um there's there's a bindings file that I gave the agent to basically describe what's on my machine, how it works, and how to map it to the fleet kit

s145
00:14:45.440 --> 00:14:46.320
work.

s146
00:14:46.320 --> 00:14:51.320
Um and you can see the built-ins, MCP servers.

s147
00:14:51.320 --> 00:14:55.839
The really quick, the code monetization plugin um this is at the heart of the whole thing.

s148
00:14:55.839 --> 00:15:05.320
And what's inside of it uh I think uh you guys will find somewhat interesting um and I would encourage you to to look into it after the fact after this presentation to get more information.

s149
00:15:05.320 --> 00:15:07.200
Um and I'm going to I'm going to keep moving here.

s150
00:15:07.200 --> 00:15:10.480
Um ultimately, the driver's laptop, my laptop, was the runtime, okay?

s151
00:15:10.480 --> 00:15:16.680
So, I had to run nine steps that are that are um started with the booting step, so eight steps after that.

s152
00:15:16.680 --> 00:15:25.200
And what I want you all to take from this super busy slide is that on the left here, those were all steps forked from the code mod plugin.

s153
00:15:25.200 --> 00:15:28.880
Okay, so we adapted it to our needs as I would would encourage anyone to do.

s154
00:15:28.880 --> 00:15:33.760
And on the right, those orange colored elements are worksheets that were created.

s155
00:15:33.760 --> 00:15:35.720
So that was going to require human approval.

s156
00:15:35.720 --> 00:15:46.720
And so as we did this like modernization journey, you can see how there's a lot of human input in the beginning, less than on the on the end, and how the core of it was this code modernization plugin.

s157
00:15:47.280 --> 00:15:58.800
And uh just an interesting detail, and this is something that the code modernization plugin does is it does like a deterministic analysis of like did the previous step create the artifacts that it needed to, including a file.

s158
00:15:58.800 --> 00:16:00.400
So, some interesting lessons.

s159
00:16:00.400 --> 00:16:04.171
Um obviously I burned a lot of uh burned a lot of tokens.

s160
00:16:04.171 --> 00:16:04.640
[laughter]

s161
00:16:04.640 --> 00:16:18.320
Um and um one of just as a little takeaway, do not let uh Claude running Fable set up crons to check on work uh because you'll it'll just launch a Claude uh a Fable session each time and burn your budget immediately.

s162
00:16:18.320 --> 00:16:25.160
Um but um some other interesting takeaways we had were relevant to prompt caching, um which was way higher than I thought it would be.

s163
00:16:25.160 --> 00:16:27.320
And then some quirky subagent behavior.

s164
00:16:27.320 --> 00:16:35.800
But one of the interesting things was is like after we did all the context building and how these agents dispatched to like write our complex legacy code, we ran benchmarks

s165
00:16:35.800 --> 00:16:46.360
um against like using our existing code recently written as the golden data and found out that Fable or uh Sonnet 5.5 wrote it just as good as if not better than Fable.

s166
00:16:46.360 --> 00:16:49.600
So, um model selection for the win.

s167
00:16:49.600 --> 00:16:57.640
Um and then looking forward, um I think you know, uh enhanced gate enforcement is going to be the biggest win that we have um in my mind at least.

s168
00:16:57.640 --> 00:17:05.600
Um we also have some interesting updates to make with decision models within our harness um and within the our model selection harness framework.

s169
00:17:05.600 --> 00:17:10.560
And then um our our like to our context was just like markdown files, y'all.

s170
00:17:10.560 --> 00:17:14.280
Like we didn't really have any complex way to graph things out.

s171
00:17:14.280 --> 00:17:21.280
Like we weren't using anything fancy with a um like um like graphify or um obsidian.

s172
00:17:21.280 --> 00:17:28.000
I mean any of these I mean honestly if you guys find me afterwards, I love your input on this, but like a more optimized way to do search and

s173
00:17:28.000 --> 00:17:32.000
uh and look at all that context that we're using to inform this war room.

s174
00:17:32.000 --> 00:17:35.160
Um and then lastly um audio as a data source.

s175
00:17:35.160 --> 00:17:44.480
Like I want to turn a recorder on and have these conversations we have like automatically transcribed, but I had to like instead sit there and furiously write down all the notes that were happening during these meetings.

s176
00:17:44.480 --> 00:17:48.600
Um and so that was kind of like a distracting thing that I think and I'm sure I lost a bunch.

s177
00:17:48.600 --> 00:17:52.960
Um so a less lossy way to incorporate the um inputs from the war room.

s178
00:17:52.960 --> 00:17:56.600
And then lastly, how can we deploy this thing across the organization?

s179
00:17:56.600 --> 00:18:04.880
Um y'all find some group in your organization that doesn't know how to make decisions, deploy this, drive it, control the cost, make it yours.

s180
00:18:04.880 --> 00:18:12.240
And at GEICO, we're like trying to figure out how as these agents are hill climbing and figuring out better ways to do things, where can we put humans in the loop?

s181
00:18:12.240 --> 00:18:23.160
And right, how can we like put our best brightest engineers who are receiving genius divine intervention as it were and have them pick the next hill as the agents climb the first one.

s182
00:18:23.160 --> 00:18:24.520
All right, and that is it.

s183
00:18:24.520 --> 00:18:27.640
Um we broke free of our outer loop.

s184
00:18:27.640 --> 00:18:28.927
Now it's your turn.

s185
00:18:28.927 --> 00:18:29.800
[laughter]

s186
00:18:29.800 --> 00:18:34.320
Uh and that's the blog that you guys can take a picture of um if you'd like.

s187
00:18:34.320 --> 00:18:36.132
All right, thanks for your time.

s188
00:18:36.132 --> 00:18:38.132
[applause]