WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Usufn8IQJgw

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Usufn8IQJgw.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:00.080 --> 00:00:01.680
So what is context engineering?

s2
00:00:01.680 --> 00:00:08.000
It's kind of like deabstracting the abstractions that have been layered on top of rag memory, agentic history.

s3
00:00:08.000 --> 00:00:11.280
At the end of the day, they're all different ways to pass tokens into a model.

s4
00:00:11.280 --> 00:00:13.519
What is a smart zone and what is a dumb zone?

s5
00:00:13.519 --> 00:00:17.199
The less context window you use, the better outcomes you'll get always.

s6
00:00:17.199 --> 00:00:20.560
A new paradm that is spreading up is loop engineering.

s7
00:00:20.560 --> 00:00:22.480
What do you think is bad about it?

s8
00:00:22.480 --> 00:00:26.720
Problem with loops is like at a certain point, you're going to generate so much code that you can't read it anymore.

s9
00:00:26.720 --> 00:00:32.640
We built a lights off software factory in July of 2025 and by November we had shut it down.

s10
00:00:32.640 --> 00:00:36.719
Can we talk about what you mean by token harder and token smarter?

s11
00:00:36.719 --> 00:00:41.040
I'm in a group chat called hyperengineering and it's all like people trying to max out their cloud subs.

s12
00:00:41.040 --> 00:00:49.140
That's my idea of token harder and the goal is what happens when you let AI agent ship code for

s13
00:00:49.140 --> 00:00:49.360
[music]

s14
00:00:49.360 --> 00:00:52.000
months and no developer reads a single line.

s15
00:00:52.000 --> 00:00:53.600
Today's guest tried exactly that.

s16
00:00:53.600 --> 00:00:58.950
He built a lights off software factory and four months later he had no choice but to shut it down as things just

s17
00:00:58.950 --> 00:00:58.960
[music]

s18
00:00:58.960 --> 00:01:00.239
stopped working.

s19
00:01:00.239 --> 00:01:07.439
Dexory is the founder of human layer and the person who coined the term context engineering days before Andre Carpathy and Tubiluska made it famous.

s20
00:01:07.439 --> 00:01:10.740
He spent the last two years talking to hundreds of AI engineers about what actually works

s21
00:01:10.740 --> 00:01:10.799
[music]

s22
00:01:10.799 --> 00:01:14.479
when you build with LMS and is testing the most extreme ideas with his own team.

s23
00:01:14.479 --> 00:01:15.980
In today's conversation we discuss

s24
00:01:15.980 --> 00:01:16.400
[music]

s25
00:01:16.400 --> 00:01:21.600
context engineering, what it is and the physics of context windows, including what the dump zone is.

s26
00:01:21.600 --> 00:01:21.875
loop

s27
00:01:21.875 --> 00:01:22.000
[music]

s28
00:01:22.000 --> 00:01:27.770
engineering from the Ralph Wim technique to the slow loops that Dex's team runs every night to wake up to code cleanup PRs

s29
00:01:27.770 --> 00:01:28.080
[music]

s30
00:01:28.080 --> 00:01:32.355
the rise of software factories from a NATO conference in 1968 through DevOps

s31
00:01:32.355 --> 00:01:32.640
[music]

s32
00:01:32.640 --> 00:01:39.280
to today's agentic factories specdriven development and why specs always drift from the code itself and many more.

s33
00:01:39.280 --> 00:01:51.600
If you want to understand increasingly important concepts like concept engineering and harness engineering or want to know how far you can push the let agents build everything idea from someone who pushed it further than almost anyone then this episode is for you.

s34
00:01:51.600 --> 00:01:53.520
This episode is presented by antithesis.

s35
00:01:53.520 --> 00:01:55.936
If you work with agents your job is no longer just writing

s36
00:01:55.936 --> 00:01:56.159
[music]

s37
00:01:56.159 --> 00:01:56.640
code.

s38
00:01:56.640 --> 00:02:03.119
It's specifying and testing it and antithesis is the most effective method of verifying agenda code today.

s39
00:02:03.119 --> 00:02:10.160
Today's episode is brought to you by Buildkite, the CIOS platform trusted by OpenAI, Entropic, Cursor, Nvidia, Uber, Canva, and more.

s40
00:02:10.160 --> 00:02:14.400
Today, we're talking about pushing the right context into models so that they write better code.

s41
00:02:14.400 --> 00:02:18.480
Right after that starts working, your agents will write more code, a lot more.

s42
00:02:18.480 --> 00:02:21.599
Trusting that code avalanche is where many teams face a challenge today.

s43
00:02:21.599 --> 00:02:26.800
Every change that an agent makes still has to be built, tested, and proven safe before it ships.

s44
00:02:26.800 --> 00:02:29.200
Worked on my machine is not enough.

s45
00:02:29.200 --> 00:02:30.879
So, you obviously need CI.

s46
00:02:30.879 --> 00:02:37.280
But when agents are pushing five, 10, or 50 times the commit volume into your pipelines, faster CI runners won't save you.

s47
00:02:37.280 --> 00:02:41.440
Shaving 30 seconds off a single build is meaningless when the queue is 100 plus jobs deep.

s48
00:02:41.440 --> 00:02:52.160
What you really want is a CI system that gets faster as the volume grows and CI that offers instant parallelization to give you unlimited concurrency and to intelligently route changes at runtime.

s49
00:02:52.160 --> 00:02:56.319
This is what Bill Kite does and why global software leaders continue to rely on it.

s50
00:02:56.319 --> 00:03:05.040
The same architecture that observed the scale of Shopify and Uber a decade ago now runs about 1.4 billion job minutes a week across cursor meta and snowflake.

s51
00:03:05.040 --> 00:03:10.480
While the rest of the CI world are cracking under the weight of rearchitecting your platform, build kite continues to reliably grow.

s52
00:03:10.480 --> 00:03:12.640
Agents running on your infrastructure or build kites.

s53
00:03:12.640 --> 00:03:16.879
Any cloud, any chip, your secrets, your scale, every artifact and log is captured.

s54
00:03:16.879 --> 00:03:20.560
So when something fails, either you or your agents have immediate insight for why.

s55
00:03:20.560 --> 00:03:24.640
As you're ensuring the context you'll give to your agents, think about how you'll verify what they hand back.

s56
00:03:24.640 --> 00:03:29.120
If your system is buckling under the increased volume, head to buildkite.com/pragmatic.

s57
00:03:29.280 --> 00:03:33.519
30-day all access trial, no credit card, and an actual human engineer on standby.

s58
00:03:33.519 --> 00:03:36.000
His name's Ola, and he's very helpful.

s59
00:03:36.000 --> 00:03:38.000
So, Dex, welcome to the podcast.

s60
00:03:38.000 --> 00:03:39.360
Super stoked to be here, dude.

s61
00:03:39.360 --> 00:03:46.560
Before we get into some of the context engineering and some of some of the the more spicy stuff as well, how did you get into tech?

s62
00:03:46.560 --> 00:03:48.640
How did you fall in love with computers?

s63
00:03:48.640 --> 00:03:49.200
Oh, man.

s64
00:03:49.200 --> 00:03:53.280
So, I uh I was doing undergrad as a as a physics major.

s65
00:03:53.280 --> 00:03:57.120
Um, and I realized that uh I didn't like academia.

s66
00:03:57.120 --> 00:04:08.319
And there's like basically like two or three paths out of physics is basically you go get a PhD or you go into finance or you go do programming.

s67
00:04:08.319 --> 00:04:12.879
At that time, this was, you know, 2012, 2011 when it was like in the middle of undergrad and deciding what to do.

s68
00:04:12.879 --> 00:04:15.840
And I had done an internship when I was in high school.

s69
00:04:15.840 --> 00:04:20.479
I was working with NASA researchers to a jet propulsion lab in California.

s70
00:04:20.479 --> 00:04:33.280
They had just gotten this really highfidelity like the most uh you know fine grain data set of altitudes like the heights of very at very like top topographical map of the south pole of the moon.

s71
00:04:33.280 --> 00:04:38.000
And the south pole of the moon is really interesting because some of the craters there are so deep because of the angle it has.

s72
00:04:38.000 --> 00:04:41.360
It got hit by meteor storms like no other part of the moon.

s73
00:04:41.360 --> 00:04:44.400
So there's very deep craters that have never seen sunlight.

s74
00:04:44.400 --> 00:04:48.960
And so there's frozen liquid water in there from the formation of the moon.

s75
00:04:48.960 --> 00:04:51.759
And so scientists were really interested in getting down there and exploring.

s76
00:04:51.759 --> 00:04:54.639
And uh so we had this really fine grain map and it's like okay cool.

s77
00:04:54.639 --> 00:05:07.440
Let's build software so that I I have point A to point B. I know the limitations of my rover can you know max incline up is this, max incline down is that find a path from point A to point B that doesn't like break those rules of the incline.

s78
00:05:07.440 --> 00:05:09.280
So I was you know 17.

s79
00:05:09.280 --> 00:05:10.960
I had never cracked a CS textbook.

s80
00:05:10.960 --> 00:05:16.960
So I wrote I basically like wrote a really naive bad version of Dystra's algorithm for pathf finding.

s81
00:05:16.960 --> 00:05:18.000
Uh so I was in college.

s82
00:05:18.000 --> 00:05:21.840
I was like, I don't know if I want to do the academics thing, but I really enjoyed programming back in the day.

s83
00:05:21.840 --> 00:05:31.039
And so, uh, so I decided to go I got like half of a CS minor and then started working on a API platform team at a software company in Chicago and

s84
00:05:31.039 --> 00:05:32.080
Sprout Social, right?

s85
00:05:32.080 --> 00:05:32.800
Yes.

s86
00:05:32.800 --> 00:05:35.840
And uh, basically never went back.

s87
00:05:35.840 --> 00:05:36.080
Yeah.

s88
00:05:36.080 --> 00:05:38.880
And then then where did you go from there?

s89
00:05:38.880 --> 00:05:40.800
Where did you pick up like the parts of the trade?

s90
00:05:40.800 --> 00:05:43.840
Because very early on, your first job that's not really common.

s91
00:05:43.840 --> 00:05:47.680
you were doing platform engineering back in you know more than a decade ago.

s92
00:05:47.680 --> 00:06:02.400
From that point, it took me about two or three months to notice that like the most valuable work that was being done in the company was being done by like of course it's obvious like the first couple engineers who know everything and understand where everything was and like you spend a day on a support ticket from a

s93
00:06:02.400 --> 00:06:06.400
customer and they solve it in 5 minutes but like you have to solve it so you learn and whatever.

s94
00:06:06.400 --> 00:06:16.319
And I realized like the most valuable people in the company were the people that were building the developer platform CI/CD sandbox environments preview stuff.

s95
00:06:16.319 --> 00:06:25.759
And so I kind of like that was my first step into the journey and I've basically been obsessed with software factories since that like three or six months into my first job.

s96
00:06:25.759 --> 00:06:29.520
We talk about software factories now but you're you're talking about software factories back then.

s97
00:06:29.520 --> 00:06:36.560
So like you were you're you were starting to already think that this is how we can produce better software inside this is pre AI world right?

s98
00:06:36.560 --> 00:06:40.800
Well and I'm always surprised like there's a huge class of developers that say I don't want to work on CI/CD.

s99
00:06:40.800 --> 00:06:41.759
I hate CI/CD.

s100
00:06:41.759 --> 00:06:48.479
I'm like really because building the thing that builds the thing and building the thing that builds the thing that builds the thing is like as software engineers we're lazy.

s101
00:06:48.479 --> 00:06:51.280
We want to do the most high lever thing that makes our job easier.

s102
00:06:51.280 --> 00:06:58.720
So how do we if we can build a thing that helps us build a thing that helps us move faster then that's the best use of my time as a as a lazy engineer.

s103
00:06:58.720 --> 00:07:03.280
And then you went to another startup uh as aspiration.

s104
00:07:03.280 --> 00:07:03.919
Aspiration.

s105
00:07:03.919 --> 00:07:04.240
Yeah.

s106
00:07:04.240 --> 00:07:06.479
Aspiration also platform engineering.

s107
00:07:06.479 --> 00:07:06.560
Yeah.

s108
00:07:06.560 --> 00:07:12.160
I was brought in and then like three 3 months into the job, the VP of engineering who hired me quit or got fired.

s109
00:07:12.160 --> 00:07:12.479
I don't know.

s110
00:07:12.479 --> 00:07:13.360
There was some drama about it.

s111
00:07:13.360 --> 00:07:14.560
I probably shouldn't talk about it.

s112
00:07:14.560 --> 00:07:22.880
And then I was there for about a year uh and was kind of like acting CTO for a while like hired a couple people, helped hire the new VP of engineering, but I was out of there.

s113
00:07:22.880 --> 00:07:24.560
I don't think I'll ever do consumer again.

s114
00:07:24.560 --> 00:07:26.560
I think I'm actually a B2B guy.

s115
00:07:26.560 --> 00:07:27.360
Good to know.

s116
00:07:27.360 --> 00:07:34.800
And then you went to replicator where you spent like a good like like solid like four years and went from engineer for deployed engineer to product manager.

s117
00:07:34.800 --> 00:07:36.800
Yeah, I did core engineering for like two years.

s118
00:07:36.800 --> 00:07:41.919
We were building a container orchestrator like before Kubernetes, before Docker Swarm was really a thing.

s119
00:07:41.919 --> 00:07:43.280
We built our own orchestrator.

s120
00:07:43.280 --> 00:07:48.479
The founders had this vision that like, oh, Docker is going to make it much easier to ship on-prem software.

s121
00:07:48.479 --> 00:07:52.160
And when I say onrem, I don't mean literally like a a rack in a colo.

s122
00:07:52.160 --> 00:07:57.840
It's more like, hey, look, bring the app to where the data is rather than sending the data up to some cloud vendor.

s123
00:07:57.840 --> 00:08:02.000
And Docker makes it much more much easier to package up apps and and and and move them around.

s124
00:08:02.000 --> 00:08:15.280
And so they had this thesis that like basically you could build a platform that the experience that you get when you use GitHub enterprise which is like you install it and it has this admin panel but then you just get GitHub running in your data center and your code never has to leave your your data center.

s125
00:08:15.280 --> 00:08:19.360
Suddenly you could build a generic SAS where everybody could have that.

s126
00:08:19.360 --> 00:08:23.039
So I did two years as an engineer there and then our head of sales.

s127
00:08:23.039 --> 00:08:31.840
We parted ways with our head of sales and uh honestly I was having a lot of arguments about the software factory with our CTO and it's kind of like almost like a too many cooks in the kitchen kind of thing.

s128
00:08:31.840 --> 00:08:38.080
I'm sure many listeners listen listeners have had this experience of like well yeah I know I have these tickets to build but like CI sucks.

s129
00:08:38.080 --> 00:08:42.399
I got to fix CI because it's too slow or it's like there's too many different builds and it's always breaking.

s130
00:08:42.399 --> 00:08:49.360
like I'm going to fix that and then I'm going to do the end is just like Dex, I need you to stop fixing the build pipeline and like do the tickets I gave you.

s131
00:08:49.360 --> 00:08:51.519
I'm sure you've had this experience perhaps.

s132
00:08:51.519 --> 00:08:51.760
Yeah.

s133
00:08:51.760 --> 00:08:56.480
And then and was this what led you to either forward deploy engineering?

s134
00:08:56.480 --> 00:08:56.959
Yeah.

s135
00:08:56.959 --> 00:08:58.560
So I like I really loved our customers.

s136
00:08:58.560 --> 00:09:03.360
Our customer our customers are Hashi Corp, Data Stacks, Puppet, all these really cool engineering brands.

s137
00:09:03.360 --> 00:09:04.800
TravisCI, CircleCI.

s138
00:09:04.800 --> 00:09:06.959
I was like yeah I actually love working with our customers.

s139
00:09:06.959 --> 00:09:08.320
Our customers are awesome.

s140
00:09:08.320 --> 00:09:10.480
And uh it was a great way to like get in the trenches.

s141
00:09:10.480 --> 00:09:27.040
a lot of really good engineers who were solving the hardest problem at the company which is like how do we take this 3 to 5year-old SAS platform and package it all up so that someone who knows nothing about our architecture can run it reliably in their own AWS VPC in their own on-prem data center whatever it was

s142
00:09:27.040 --> 00:09:44.880
and so I spent I was our first kind of customerf facing engineer and it was in about three months I we closed I met with like every company customer that was like kind of in the pipeline but wasn't moving saleswise is and we closed like 12 deals in 3 months and the CEO was like, "Holy crap, Dex.

s143
00:09:44.880 --> 00:09:47.120
Like the the investors are taking my calls again.

s144
00:09:47.120 --> 00:09:54.800
Like I don't I know you want to get back to coding, but like I need you to go hire three people and like build this team out cuz I think you might have been like born for this."

s145
00:09:54.800 --> 00:09:55.360
Wow.

s146
00:09:55.360 --> 00:09:55.600
Yeah.

s147
00:09:55.600 --> 00:10:05.680
So I did that for about four years, built that or to like 25 people and then Zer happened and uh it got a lot smaller and we kind of realized like, hey, we have a product that's like pretty good uh and we've been solving

s148
00:10:05.680 --> 00:10:08.800
what lots of early startups do is like, okay, there's some usability issues.

s149
00:10:08.800 --> 00:10:14.320
is we'll throw we'll get a bunch of smart people, throw them in the trenches with our customers, great for sales, great for retention, all this stuff.

s150
00:10:14.320 --> 00:10:17.920
And it was like, oh, we actually like the margins on that aren't aren't good enough.

s151
00:10:17.920 --> 00:10:24.880
And so we basically were like, cool, we actually just need to make the product way more usable, do a more PLG-shaped thing, make it productled growth,

s152
00:10:24.880 --> 00:10:25.600
product led growth.

s153
00:10:25.600 --> 00:10:28.959
Make it a little more self-service so you don't need an expert to teach you how to use it.

s154
00:10:28.959 --> 00:10:29.519
And I was like, cool.

s155
00:10:29.519 --> 00:10:32.959
If that's the most important thing, then I want to go be a product manager because I have tons of opinions.

s156
00:10:32.959 --> 00:10:35.360
I've now spent four years in the trenches with our customers.

s157
00:10:35.360 --> 00:10:41.519
I have a laundry list of roadmap things that I think would make the product way easier to use and adopt and implement and deploy

s158
00:10:41.519 --> 00:10:44.480
and and now you went the full ar you went towards a dark side.

s159
00:10:44.480 --> 00:10:44.959
Exactly.

s160
00:10:44.959 --> 00:10:45.440
Yeah, I did.

s161
00:10:45.440 --> 00:10:47.519
I was like this is going to kill my street cred isn't it?

s162
00:10:47.519 --> 00:10:56.399
But uh I was really glad you know I think a lot of engineers are afraid that if they go do a customerf facing thing they lose all their credibility and like yes I wasn't coding for 10 hours a day.

s163
00:10:56.399 --> 00:10:59.519
I was coding for like three or four hours on a Saturday for fun.

s164
00:10:59.519 --> 00:11:02.959
Not uh but I mean we were helping people build YAML we were building CLIs.

s165
00:11:02.959 --> 00:11:09.040
We owned a lot of the tooling that customers use, but it was like the last mile delivery side of it, not the core platform.

s166
00:11:09.040 --> 00:11:17.040
And like on a more personal note, I had spent the last like most of my 20s feeling like okay, a little bit introverted, a little bit like socially awkward.

s167
00:11:17.040 --> 00:11:24.160
What I what a lot of engineers I'm sure experience and uh I had talked to my uncle's a music producer.

s168
00:11:24.160 --> 00:11:27.760
So he used to work with like Randy Newman and a bunch of like really famous musicians.

s169
00:11:27.760 --> 00:11:28.320
Oh wow.

s170
00:11:28.320 --> 00:11:28.720
Yeah.

s171
00:11:28.720 --> 00:11:34.959
This guy Mitchell F. And he he I was sitting with dinner with him at some point and when I was I think it was when I was still in undergrad, but he gave me this lecture.

s172
00:11:34.959 --> 00:11:40.079
He was basically like if you want to be really good at something, you have to make it the only thing you do.

s173
00:11:40.079 --> 00:11:46.720
The guy playing guitar nights and weekends trying to get his band off off the ground will probably never achieve greatness.

s174
00:11:46.720 --> 00:11:52.399
The people who become great are the people who basically make it like if I don't play guitar, I don't eat.

s175
00:11:52.399 --> 00:11:57.200
And you go and you sit on the street all day and you play for 14 hours a day or whatever it is.

s176
00:11:57.200 --> 00:11:59.360
That's the only way to become great.

s177
00:11:59.360 --> 00:12:12.399
So, I said, "Okay, instead of trying to like read self-help books about how to be less introverted and less socially awkward, like what if I just made it my freaking job to just talk to people and make friends and like help people and solve their problems and uh I think it worked out.

s178
00:12:12.399 --> 00:12:12.959
I recommend it.

s179
00:12:12.959 --> 00:12:16.959
I think everyone should spend a year or two at least doing something really like customerf facing."

s180
00:12:16.959 --> 00:12:26.079
Did you do this because you felt that it was holding you back be being introverted or or like what what what and I I know you got the motivation from the whole musician motivation.

s181
00:12:26.079 --> 00:12:34.480
I I get it on one part, but what was it that you said like is a customerf facing thing that I'm I'm going to be doing it because clearly you were pretty great at like writing code by that point.

s182
00:12:34.480 --> 00:12:36.320
You could argue you were doing it night and day.

s183
00:12:36.320 --> 00:12:42.160
So where where did you find that like I actually I think like customerf facing or like getting this introvert off of me?

s184
00:12:42.160 --> 00:12:45.360
Did you feel that I was holding you back or you just wanted to be good at it?

s185
00:12:45.360 --> 00:12:50.560
It was just kind of a thing that was like interfering with my like general life satisfaction.

s186
00:12:50.560 --> 00:12:52.560
And it was also like I'm not a very type A person.

s187
00:12:52.560 --> 00:12:53.680
I'm very disorganized.

s188
00:12:53.680 --> 00:13:05.120
is I don't know if people call it like okay I'm like ADHD now that's why I can run 30 quads in parallel or whatever it is but it was like I was really bad at email and calendars and spreadsheets I just like didn't care about these didn't understand them and so like another side effect of this was

s189
00:13:05.120 --> 00:13:18.880
like it just forced me to be organized and keep a lot of things going and so like I don't know there's like weird benefits you get from like stepping outside your comfort zone and learning like industrial disciplines that are separate from what you've been doing and so the opportunity presented itself and I was like oh I like

s190
00:13:18.880 --> 00:13:24.480
working I'll try this for a little bit started going really well I'm like cool let's keep let's see let's see how far this thread goes

s191
00:13:24.480 --> 00:13:27.200
and then afterwards you're now in your second startup.

s192
00:13:27.200 --> 00:13:38.959
You you became a founder and you also got involved in in AI pretty early as I as it was even before it was so obvious that it would change how it would change how we develop software, right?

s193
00:13:38.959 --> 00:13:48.720
Well, I would say I was I was later than I could have been because we started the company uh me and a buddy in Chicago started a company in the data engineering space in about 2020 November 20.

s194
00:13:48.720 --> 00:13:50.320
We decided in like August of 2015,

s195
00:13:50.320 --> 00:13:51.040
this is Metalytics.

s196
00:13:51.040 --> 00:13:51.920
Metalytics.

s197
00:13:51.920 --> 00:13:56.399
Um, technically still the same company as human layer, we just like pivoted the the the mission.

s198
00:13:56.399 --> 00:14:05.040
But, uh, yeah, the the the advice I got from every angel investor that, you know, people who just knew CTOs I'd worked for before and stuff, they were just like, look, hitting a lot of heads, wins.

s199
00:14:05.040 --> 00:14:13.839
I don't know if you know like the whole DBT data engineering fiverr that whole arc where it was like this huge party and tons of investor money going into all these different companies

s200
00:14:13.839 --> 00:14:23.519
and then within by like 2021 2022 there was kind of the zer thing and just this general realization that the TAM for those sorts of tools is not as big as everyone

s201
00:14:23.519 --> 00:14:24.000
market

s202
00:14:24.000 --> 00:14:30.560
yes the total addressable market for those sort of tools was was not as quite as big as uh as we all thought it was.

s203
00:14:30.560 --> 00:14:32.639
Um so it was it was a hard place to raise money.

s204
00:14:32.639 --> 00:14:34.399
It was a hard place to get customers.

s205
00:14:34.399 --> 00:14:34.560
Yeah.

s206
00:14:34.560 --> 00:14:38.720
And then I I met you at while you were at Human Layer NSF at an event.

s207
00:14:38.720 --> 00:14:41.839
We you actually talked and we chatted afterwards.

s208
00:14:41.839 --> 00:14:49.680
But by by that, this was about a year ago, you were already you you started to have some really strong opinions on using AI.

s209
00:14:49.680 --> 00:14:54.160
And one of them was this now famous 12 factor agents manifesto.

s210
00:14:54.160 --> 00:14:56.160
Is are we calling it a manifesto now?

s211
00:14:56.160 --> 00:14:57.600
I'm I'm calling it a manifesto.

s212
00:14:57.600 --> 00:14:58.399
It's a manifesto.

s213
00:14:58.399 --> 00:14:59.120
I'm calling it.

s214
00:14:59.120 --> 00:15:00.079
Let's talk about this.

s215
00:15:00.079 --> 00:15:04.240
This was 12 engineuring principles to build reliable production ready apps.

s216
00:15:04.240 --> 00:15:07.839
uh how did you come up with this and maybe we can also talk about some of them.

s217
00:15:07.839 --> 00:15:08.160
Yeah.

s218
00:15:08.160 --> 00:15:16.000
So um I'll I'll kind of like go to like around August the co-founder I was working with kind of burned out and left and it was very we were on good terms.

s219
00:15:16.000 --> 00:15:16.880
It was very mutual.

s220
00:15:16.880 --> 00:15:26.240
Um and I decided to start messing with AI stuff and I was building a AI agents and what was really in fog right then was like the lang chain the crew AI these like agent frameworks.

s221
00:15:26.240 --> 00:15:30.399
Um and it seemed like there was a ton of you go you go in the crew AI discord there's 10,000 people.

s222
00:15:30.399 --> 00:15:34.240
It's like, okay, this feels like the right shape and this there's clearly this eco.

s223
00:15:34.240 --> 00:15:37.600
You go in every single one of those projects, they have a Chroma DB plugin.

s224
00:15:37.600 --> 00:15:39.440
They have like a Composeio plugin.

s225
00:15:39.440 --> 00:15:43.920
There's like clearly like this is the this is the shared interface that everybody is building for.

s226
00:15:43.920 --> 00:15:45.600
I say, okay, what's missing from all of this?

s227
00:15:45.600 --> 00:15:52.079
The agents can call tools, but it's really hard to like control which tools they call.

s228
00:15:52.079 --> 00:15:56.959
And if it's a chatbot, obviously, you can show approved deny in the UI of your application.

s229
00:15:56.959 --> 00:16:01.199
But I kind of was obsessed with what I would call like outer loop agents or proactive agent.

s230
00:16:01.199 --> 00:16:03.440
Agents that would run in the background, get triggered by events.

s231
00:16:03.440 --> 00:16:10.079
I mean, OpenClaw is basically like the biggest manifestation of this of like you have a heartbeat, it wakes up, it sees if there's any work to do, it tries to do stuff.

s232
00:16:10.079 --> 00:16:16.000
And my thought was like, I'm not going to trust that agent to do anything meaningful.

s233
00:16:16.000 --> 00:16:27.680
if I can't get like a Slack message or an iMessage or something when it wants to do something and kind of guarantee deterministically that I can approve or deny that or deny it with feedback and say actually no do it like this.

s234
00:16:27.680 --> 00:16:33.440
So we played in that space for a while and talked to a lot of founders and founding engineers and builders.

s235
00:16:33.440 --> 00:16:36.560
We came into YC in the fall of 2024 with this idea.

s236
00:16:36.560 --> 00:16:42.639
We're building out this API platform and it was sort of like pedag duty but like it wasn't who's on call to fix the servers.

s237
00:16:42.639 --> 00:16:52.880
It was like who's on call to this like routing mechanism for like who needs to approve this agent and can they like escalate it or delegate it or defer it all this stuff and we built it for this ecosystem

s238
00:16:52.880 --> 00:17:04.160
crew AI link chain fi there's so many grip tape there was so many in that in that time and then I talked to tons of AI engineers who were actually building really interesting things and like actually making money

s239
00:17:04.160 --> 00:17:17.280
doing six figure contracts shipping AI to the enterprise and all of them had tried that stuff for like a month or two and then they had thrown it out and they were just writing all a API calls by hand and they were building more things that look more like pipelines and workflows

s240
00:17:17.280 --> 00:17:21.760
than these sort of like hands-off call tools in a loop kind of thing.

s241
00:17:21.760 --> 00:17:28.720
And so I talked to a hundred people and I spent a lot of time a lot a lot of time hanging out with one of my best friends uh Vib from uh Boundary.

s242
00:17:28.720 --> 00:17:36.720
So they build a programming they built like this like protobuffs for AI thing and they're I think they're about to launch their like full fat like programming language touring complete thing.

s243
00:17:36.720 --> 00:17:46.000
But he had this way of thinking about agents and building with models and building with inference where it was a lot more about understanding what structured output really is under the hood.

s244
00:17:46.000 --> 00:17:51.039
And every single step in your AI workflow is just tokens in tokens out.

s245
00:17:51.039 --> 00:17:57.039
And your job as an engineer is figure out, okay, what tokens do I need to put in to maximize the chance that the tokens out are going to be good.

s246
00:17:57.039 --> 00:18:10.240
and kind of distilled all these ideas into about 12 principles and wrote about it on GitHub, posted just like this like 12-page GitHub repo, threw it on HackerNews, got like five, it was on the front page for like two days and it I think it really resonated with a lot of people.

s247
00:18:10.240 --> 00:18:10.480
Yeah.

s248
00:18:10.480 --> 00:18:16.240
So, I I'll just quickly read the 12 principles and and then let's talk about like one or two that resonate.

s249
00:18:16.240 --> 00:18:22.240
Sub 12 are natural language of tool calls, own your prompts, own your context window.

s250
00:18:22.240 --> 00:18:24.240
Tools are just structured outputs.

s251
00:18:24.240 --> 00:18:26.720
Unify execution state and business state.

s252
00:18:26.720 --> 00:18:28.960
Launch pause resume with simple APIs.

s253
00:18:28.960 --> 00:18:30.799
Contact humans with tool calls.

s254
00:18:30.799 --> 00:18:32.320
Own your control flow.

s255
00:18:32.320 --> 00:18:34.240
Compact errors into context window.

s256
00:18:34.240 --> 00:18:35.679
Small focused agents.

s257
00:18:35.679 --> 00:18:36.640
Trigger from anywhere.

s258
00:18:36.640 --> 00:18:38.080
Meet users where they are.

s259
00:18:38.080 --> 00:18:40.240
Make your agent a stateless reducer.

s260
00:18:40.240 --> 00:18:41.120
The stateless.

s261
00:18:41.120 --> 00:18:46.400
Yeah, the stateless reducer one was a little actually someone hit me up on Twitter and uh corrected me.

s262
00:18:46.400 --> 00:18:52.320
It's actually it's actually a transducer because there's technically multiple steps in the workflow, but there we go.

s263
00:18:52.320 --> 00:19:00.720
But but but of this one, this this was a year ago, so like which is like forever in uh in in how the tooling is is evolving.

s264
00:19:00.720 --> 00:19:05.840
Which ones still stick with you where you're like, "All right, these were good that that still seem to hold off."

s265
00:19:05.840 --> 00:19:09.760
Yeah, I think I'm going spent most of March writing it, published this in April.

s266
00:19:09.760 --> 00:19:13.679
Uh, and then Swix hit me up from AI.engineer and he said, "Hey, can you come?

s267
00:19:13.679 --> 00:19:15.039
You want to come talk about this."

s268
00:19:15.039 --> 00:19:19.120
So, I gave this talk 12 factor agents in like June 6th, I think.

s269
00:19:19.120 --> 00:19:23.679
And, uh, small room maybe like it was packed, but it was like maybe a hundred people.

s270
00:19:23.679 --> 00:19:36.799
That was the year at AI engineer where like the lower physically like on on the on the second basement floor was all the super corporate stuff and you go up a level is a little bit more and then like on the top floor is all the like weird cutting edge like startup stuff that like you probably shouldn't

s271
00:19:36.799 --> 00:19:38.080
care about yet kind of thing.

s272
00:19:38.080 --> 00:19:42.160
So we were up there on the top of this like weird way of thinking about agents.

s273
00:19:42.160 --> 00:19:48.880
Uh and then about a week later or two weeks later uh Toby Licki from Shopify says I really like this idea of like context engineering.

s274
00:19:48.880 --> 00:19:51.840
And I'm like I I wrote about this two months ago.

s275
00:19:51.840 --> 00:19:52.480
This is great.

s276
00:19:52.480 --> 00:19:59.360
Toby gets it and then a week later Andre Carpathi is like well I really like I think what we should think about is not prompt engineering but context engineering.

s277
00:19:59.360 --> 00:20:00.880
And I was like, "Yes, that's my."

s278
00:20:00.880 --> 00:20:01.600
Anyways, I don't know.

s279
00:20:01.600 --> 00:20:08.400
If you ask Gemini, depends what day it is, they will tell you either me or Toby or Andre came up with context engineering.

s280
00:20:08.400 --> 00:20:09.440
You can't really own a word.

s281
00:20:09.440 --> 00:20:12.160
Like I don't no one remembers who invented the word prompt engineering.

s282
00:20:12.160 --> 00:20:15.039
But of all the factors, factor three of own your context window.

s283
00:20:15.039 --> 00:20:26.000
And basically the only way you can whether it's agentic or a single step at a pipeline, the only way you can impact the quality of your output from AI is by caring a lot about what the inputs and crafting them.

s284
00:20:26.000 --> 00:20:30.799
Let's talk about context engineering, which I am going to credit you that you coined it.

s285
00:20:30.799 --> 00:20:33.919
I I did some research and like I think you were earlier by a few days.

s286
00:20:33.919 --> 00:20:34.720
So there we go.

s287
00:20:34.720 --> 00:20:35.440
You you coined it.

s288
00:20:35.440 --> 00:20:37.760
We're adding we're adding to the we're adding to SEO juice.

s289
00:20:37.760 --> 00:20:39.280
We'll have it in a transcript.

s290
00:20:39.280 --> 00:20:40.960
Dex coined context engineering.

s291
00:20:40.960 --> 00:20:47.520
Well, and and like a asterisk on that is basically like I learned about context engineering from talking to these hundred engineers and founders.

s292
00:20:47.520 --> 00:20:52.000
I just kind of like what was the same about what they were all doing and I put a name on it.

s293
00:20:52.000 --> 00:20:53.440
So like I didn't invent doing it.

s294
00:20:53.440 --> 00:21:06.000
I was just like I think we I think there's this thing and like vocabulary and names are really important and having like clean ways to talk about the problem especially when like a lot of the content about AI right now is so much hype and jargon that is like meaningless.

s295
00:21:06.000 --> 00:21:12.400
I was like okay I think there's a word here that is useful to builders that explains how they should be thinking about building their software.

s296
00:21:12.400 --> 00:21:14.000
So what is context engineering?

s297
00:21:14.000 --> 00:21:17.919
It's kind of like deabstracting a lot of the abstractions that have been layered on top.

s298
00:21:17.919 --> 00:21:28.720
So you have rag, you have memory, you have agentic history, you have structured output, you have all these things that are like different ideas in the frame of agentic programming.

s299
00:21:28.720 --> 00:21:36.159
And at the end of the day, they're all like different ways to pass tokens into a model and ask it to produce usually some structured output.

s300
00:21:36.159 --> 00:21:46.640
And understanding that is a lot more powerful than trying to learn memory and trying to pick some agent framework off the shelf and some memory framework off the shelf.

s301
00:21:46.640 --> 00:21:48.159
I mean, those are these things are all really good.

s302
00:21:48.159 --> 00:21:51.520
If you want to get to like 80%, you want to get a really good demo.

s303
00:21:51.520 --> 00:21:56.000
But when you have to go from 80% to 95% or 99%.

s304
00:21:56.000 --> 00:22:00.320
You need to go down a level and think about what's everything we're putting into the context window.

s305
00:22:00.320 --> 00:22:03.360
What order is it going in depending on which model we're doing?

s306
00:22:03.360 --> 00:22:04.640
And all of this stuff matters.

s307
00:22:04.640 --> 00:22:07.360
You have all of these levers that you can pull.

s308
00:22:07.360 --> 00:22:14.720
And it just felt like the right abstraction for thinking about how do I get AI to do the thing I want as accurately as possible.

s309
00:22:14.720 --> 00:22:18.080
Why is context engineering started to become more more talked about?

s310
00:22:18.080 --> 00:22:19.280
It it was about a year ago.

s311
00:22:19.280 --> 00:22:25.919
Was it did it have to do with the the context the the context window that we could pass on to LLMs pretty much.

s312
00:22:25.919 --> 00:22:34.080
Did it start to expand or did did we just start to realize that we can do a lot more by passing on from you know the easiest one is of course system prompts

s313
00:22:34.080 --> 00:22:44.480
but of course whenever you build an LLM behind the scenes you will pass additional context as well not just to prompt the user you will add a bunch of stuff that's I guess a dirty secret of any any

s314
00:22:44.480 --> 00:22:48.880
LM but why do you think the focus is moving on to like all right context is important

s315
00:22:48.880 --> 00:23:00.799
I think it always was important I think what had to happen is a ton of smart people again like all these builders I talked to a ton of smart people had to like focus really hard on producing like I want to make software that I can sell.

s316
00:23:00.799 --> 00:23:06.080
I want to make something that is accurate enough that I'm proud of and I can sell to an enterprise and they're going to be happy with it.

s317
00:23:06.080 --> 00:23:15.120
And there's just like the the the the easiest way to get to really high quality AI applications is by thinking at that token level.

s318
00:23:15.120 --> 00:23:24.080
Thinking about a string of different LLM calls like rather than just tools in the loop and it's kind of open-ended and very flexible but not that reliable.

s319
00:23:24.080 --> 00:23:36.559
thinking of agents as as workflows, as pipelines, as some mix between maybe a couple tools in a loop versus just, hey, I have my tools and I have my model and I have my system prompt and these are the only levers I have.

s320
00:23:36.559 --> 00:23:38.240
And it's actually no, you have way more levers.

s321
00:23:38.240 --> 00:23:42.960
It's going to take more work and you're going to have to like understand the LLM with a deeper intuition.

s322
00:23:42.960 --> 00:23:53.360
But it was a thing that we always needed and it just took time for people to build with this technology to figure out that like this is the layer of abstraction that allows you to break through the quality ceiling.

s323
00:23:53.360 --> 00:23:58.159
And how are cost and context engineering connected?

s324
00:23:58.159 --> 00:23:58.720
Yeah.

s325
00:23:58.720 --> 00:23:59.520
Um I don't know.

s326
00:23:59.520 --> 00:24:06.960
I was I was talking about this with uh someone this morning um about like when you're working with LM, one of the things I I like to say is kind of like

s327
00:24:06.960 --> 00:24:09.200
make it run, make it right, make it fast.

s328
00:24:09.200 --> 00:24:16.480
see if the world's best LLM at the time I think we did a podcast episode that at the time it was like 03 see if 03 can solve your problem

s329
00:24:16.480 --> 00:24:24.000
and then give it to people and see if they want that and then if people want it and you use it a lot then go do a bunch of context engineering

s330
00:24:24.000 --> 00:24:33.279
because your engineering time is always the bottleneck like humans trying to figure out and solve problems and build evals and improve and try different dimensions or set up jeep or whatever it is

s331
00:24:33.279 --> 00:24:44.080
is always going to be more expensive than just using a smarter model until you have millions of requests a A and then it's like, okay, we're going to do a bunch of context engineering, break this up into three calls, and get it to work on GPT40.

s332
00:24:44.080 --> 00:24:48.400
And then we're going to take two of those and make those two work on GPT40 and using old model names.

s333
00:24:48.400 --> 00:25:03.039
But the point is like for a certain task in your workflow, can you get GPT OSS 12B, which is like 1/ 1,000th of the cost of Opus, can you get it to solve parts of the problem so that the tokens and the things you're using the smartest frontier models for

s334
00:25:03.039 --> 00:25:05.919
are just the things that you really need, that level of intelligence?

s335
00:25:05.919 --> 00:25:18.000
But you shouldn't go build all of that and overengineer it until you've proved that you need it that it's valuable that it's like okay this is now I mean we get to Eli Goldrat and like what is the the he had this book the goal right it was about how to model your factory and I'm sure we'll

s336
00:25:18.000 --> 00:25:29.039
get to that when we talk about software factories it was like what is the bottleneck in your system and one day it will be latency and cost but it's probably not that when you first start out and context engineering is how you move from the

s337
00:25:29.039 --> 00:25:36.320
you you add human effort to the equation to improve the efficiency the speed the price the cost efficiency of your system.

s338
00:25:36.320 --> 00:25:37.039
Interesting.

s339
00:25:37.039 --> 00:25:43.840
And then one thing that came up more recently and a lot later uh recently is harness engineering.

s340
00:25:43.840 --> 00:25:45.520
What is harness engineering?

s341
00:25:45.520 --> 00:25:53.679
So I made a post in like October I think about or maybe November of of like hey there's this new thing that I see is like I'm calling it harness engineering.

s342
00:25:53.679 --> 00:26:01.440
My definition that I had at the time is not what actually this guy Viv who's at lang chain now does a lot of really good writing on agents and how to think about harness.

s343
00:26:01.440 --> 00:26:06.320
He had written something called harness engineering like a couple weeks before me but I hadn't read it at that point.

s344
00:26:06.320 --> 00:26:12.240
And my take was basically like okay when you build an agent you use use context engineering.

s345
00:26:12.240 --> 00:26:19.600
When you use an agent because we gave this talk in August of 2025 about like how to apply context engineering to how you use coding agents.

s346
00:26:19.600 --> 00:26:30.400
And that kind of evolved into this idea of like how do you take a harness like cloud code like codeex how do you engineer against the integration points of that harness.

s347
00:26:30.400 --> 00:26:34.960
So commands, MCPs, skills, how you organize your codebase.

s348
00:26:34.960 --> 00:26:40.720
How do you kind of optimize the environment that the coding agent runs in to like get the best results?

s349
00:26:40.720 --> 00:26:43.760
The same way with context engineer, how do you optimize the inputs to every single prompt?

s350
00:26:43.760 --> 00:26:51.440
Well, harness engineering just is like how do I raise the floor so that every single turn of this thing, the results are as good as possible.

s351
00:26:51.440 --> 00:26:55.520
And the term got super blurry and some people think harness engineering means building a harness.

s352
00:26:55.520 --> 00:26:58.640
And some people think hard harness engineering means building around a harness.

s353
00:26:58.640 --> 00:27:09.600
I actually like what Martin Fowler came up with uh as usual he's very good at naming things and he kind of defined the you have the LLM and then you have the inner harness which is like the thing the the tool definitions

s354
00:27:09.600 --> 00:27:23.600
and the integration points that like say like a cloud code or a codeex or a amp actually exposes that's your inner harness and then you have the outer harness which is the stuff that you the human do to customize that for your specific needs your codebase your languages

s355
00:27:23.600 --> 00:27:26.880
etc that's the best definition I think we have for harness engineering

s356
00:27:26.880 --> 00:27:31.039
it's interesting how naming is still so so important, isn't it?

s357
00:27:31.039 --> 00:27:40.159
Well, it's like as soon as you name anything, people are most people are I'm actually surprised that context engineering still means the same thing to most people that it did a year ago and that it's even still relevant.

s358
00:27:40.159 --> 00:27:51.360
Like that's honestly the the craziest thing to me is like you wrote how many things that were written about AI 15 months ago still matter or still interesting um or are still like

s359
00:27:51.360 --> 00:27:53.600
have good advice baked into them.

s360
00:27:53.600 --> 00:28:04.880
Stuff changes a I think context engineering has been so long lived because it's it's grounded in the fundamentals of how transformer attention works and until we have post transformer

s361
00:28:04.880 --> 00:28:15.760
models or linear attention or whatever it is which who knows when that's going to happen context engineering will be interesting and important to anyone building on AI and can we talk about the physics of of context

s362
00:28:15.760 --> 00:28:26.559
uh you you you had a you had a tweet uh this this one the the context reality check this is a graph of uh as you get to 1 million context

s363
00:28:26.559 --> 00:28:29.200
just the the quality just drops it.

s364
00:28:29.200 --> 00:28:30.240
It goes down.

s365
00:28:30.240 --> 00:28:32.240
What do we need to know about like the context?

s366
00:28:32.240 --> 00:28:35.679
Again, we we now have models that do have a 1 million context window.

s367
00:28:35.679 --> 00:28:41.840
Maybe we'll have even longer ones, but when you start to just put in more stuff into the context, it starts to become less efficient.

s368
00:28:41.840 --> 00:28:48.559
Like what what do we know so far in terms of from a practical perspective of like someone who is using the context window to add on a bunch of stuff?

s369
00:28:48.559 --> 00:28:53.600
May that be MCP, may that be tools, may that be scales, may that be all of these things.

s370
00:28:53.600 --> 00:28:53.840
Yeah.

s371
00:28:53.840 --> 00:28:56.320
I mean, so the longer context windows are good.

s372
00:28:56.320 --> 00:28:57.679
You can talk to it for longer.

s373
00:28:57.679 --> 00:28:58.720
Like they're doing a good job.

s374
00:28:58.720 --> 00:29:08.720
But at the end of the day, like especially when you had like Opus, it was like Opus 4.5 and then Opus 4.51 mil or 4.6 and 4.61 mil.

s375
00:29:08.720 --> 00:29:12.480
You're not actually getting a like smarter model.

s376
00:29:12.480 --> 00:29:28.960
like the intelligence of the model is is what drives its ability to attend to all of the tokens in the context window to figure out on the next turn which parts of this 100k or 200k context window are the most relevant to making the decision of like what is the next tool we call and doing that over

s377
00:29:28.960 --> 00:29:30.240
and over again in a loop.

s378
00:29:30.240 --> 00:29:37.520
So I don't know there was some study that came out in 2025 which found that and again these are old models so like inflate your numbers but it was like frontier LLMs

s379
00:29:37.520 --> 00:29:44.559
can follow about 150 to 250 instructions before it starts to drop off.

s380
00:29:44.559 --> 00:29:48.399
Their ability to follow all the instructions just like drops off pretty quickly.

s381
00:29:48.399 --> 00:29:57.919
And I think Lori Vos I haven't actually looked at the data but they did a study with like the next generation models a year later and it looks like it's like much better the number of instructions you can get in.

s382
00:29:57.919 --> 00:30:02.480
In any case, you have like I split context engineering into like two categories.

s383
00:30:02.480 --> 00:30:14.240
You have like the the most people think about like the information budget of like okay I can do rag and I can pull out chunks of this document rather than putting the entire book into my context window.

s384
00:30:14.240 --> 00:30:16.080
I can just go grab the pages that matter.

s385
00:30:16.080 --> 00:30:29.360
But it's also your instruction budget is like if you give the model too many instructions and especially too many conflicting instructions and that's in your initial prompt and also like if you have a conversation you start going down a path and then you change your mind and you start going down a different you actually I don't want

s386
00:30:29.360 --> 00:30:30.960
to do any of that I want to do this.

s387
00:30:30.960 --> 00:30:36.720
It's like a it's a lot of computation the model has to do to notice that it has to ignore that whole thing.

s388
00:30:36.720 --> 00:30:50.799
And when both of those things are kind of far back enough in the context window that they're only half getting attended to, your likelihood that it's like actually going to like remember the exact instructions you gave it 100,000 tokens ago is like it goes down quite significantly.

s389
00:30:50.799 --> 00:31:05.200
This is all very interesting because as engineers there we are expected when you know when we're AI engineers which now a lot of software engineer meaning you just like use LMS to to build software like underneath there's an LLM layer somewhere you're an AI engineer congratulations

s390
00:31:05.200 --> 00:31:16.240
but it sounds like the expectation is to be you know to be a good to be a good software engineer preI you need to understand you know how to write good code and it helps when you understand a little bit of the underlying

s391
00:31:16.240 --> 00:31:26.880
we didn't need to do that that much over time but it it it never hurts but sounds Like right now we're in this phase that to be an engineer who can write an efficient

s392
00:31:26.880 --> 00:31:29.200
AI system that use LMS.

s393
00:31:29.200 --> 00:31:40.640
You need to understand the dynamics of the context you need to understand why stuffing your context one way or the other can be compute can introduce latency and all of these.

s394
00:31:40.640 --> 00:31:50.080
It sounds like it's kind of more of an intuition and of course there's some understanding but from talking to you you're like well it it it does this computation like I know you know cuz you tried it out right?

s395
00:31:50.080 --> 00:31:57.760
Yeah like I'm not I'm not a PhD in machine learning like I couldn't actually go like draw a mathematical proof of how this works but we know attention is quadratic

s396
00:31:57.760 --> 00:32:03.039
and the more stuff you put in the more it has to spread this attention out over everything.

s397
00:32:03.039 --> 00:32:10.960
This just feels like an absolute new area and like a little bit very different to like what we're used to like software engineering which is like pretty kind of like black and white, right?

s398
00:32:10.960 --> 00:32:12.720
That compiles or doesn't compile.

s399
00:32:12.720 --> 00:32:13.200
That's true.

s400
00:32:13.200 --> 00:32:14.720
I mean there's a different kind of intuition.

s401
00:32:14.720 --> 00:32:28.880
I was talking about this earlier as well is like there's a different kind of intuition that you that you develop over years as a software engineer and uh there's many categories of it but the one I'll I'll call attention to that is like a thing that you cannot teach you cannot do you cannot learn in a textbook.

s402
00:32:28.880 --> 00:32:34.559
The only way to learn it is like I know bad patterns in software because I have debugged them at three in the morning.

s403
00:32:34.559 --> 00:32:38.159
This is my buddy Jake from Netflix said this in his talk at AI engineer code.

s404
00:32:38.159 --> 00:32:45.919
It's just like there's no better way to learn what is good and what is bad and what works and what doesn't than suffering through the thing that doesn't work.

s405
00:32:45.919 --> 00:32:53.039
Well, speaking of suffering through the things that that doesn't work, uh a new paradigm uh that is spreading up is loops.

s406
00:32:53.039 --> 00:32:54.159
Loop engineering.

s407
00:32:54.159 --> 00:32:57.120
The idea that instead of writing prompts, just write loops.

s408
00:32:57.120 --> 00:32:57.760
Set up your loops.

s409
00:32:57.760 --> 00:33:05.679
And this all started with the Ralph Wiggum technique where it it will just well it I I guess that's an early version of loops that were just loops around

s410
00:33:05.679 --> 00:33:11.200
and now we're we're hearing with some of the big biggest labs talking about that they're actually just doing looping.

s411
00:33:11.200 --> 00:33:15.600
What is your take on have have you done some looping yourself?

s412
00:33:15.600 --> 00:33:17.279
Have you set up some loops?

s413
00:33:17.279 --> 00:33:22.399
And what do you think is good about it and what do you think is bad about it?

s414
00:33:22.399 --> 00:33:22.720
Yeah.

s415
00:33:22.720 --> 00:33:26.320
So I think of loops as I mean this could I could ramble on this for 10 minutes.

s416
00:33:26.320 --> 00:33:32.559
This is an entire talk, but I'll I'll try to I'll try to lay out some highle stuff and then we can dig in wherever you think is most interesting.

s417
00:33:32.559 --> 00:33:33.440
We had Ralph Wickham.

s418
00:33:33.440 --> 00:33:49.200
It was actually a year and four days ago was the first time I saw the Ralph Wickham demo and like Jeff Hunley was just like visiting SF and he just like came through and like dropped everybody's jaws with his like, "Yeah, I just ran Sonnet around the clock and spent six grand in six weeks and like I

s419
00:33:49.200 --> 00:33:51.200
built an entire Gen Z programming lang."

s420
00:33:51.200 --> 00:33:58.559
Look at it compiles and it has a stage two compiler where the compiler for the language is written in the language itself and all the insane.

s421
00:33:58.559 --> 00:34:13.040
And the core lesson from all of that I think was the idea of back pressure which is basically and I think a lot of people were doing this for a very have been doing this for a long time which is how do I let the model check its own work?

s422
00:34:13.040 --> 00:34:16.879
How do I automate the process of getting feedback into the model?

s423
00:34:16.879 --> 00:34:19.599
And there's lots and lots of different flavors of this.

s424
00:34:19.599 --> 00:34:21.520
You can have deterministic llinters.

s425
00:34:21.520 --> 00:34:22.960
You can have unit tests.

s426
00:34:22.960 --> 00:34:28.159
Like part of what made the programming language easy to build with Ralph is a programming language can be infinitely verified.

s427
00:34:28.159 --> 00:34:30.639
You write you write the code in the language, you compile it.

s428
00:34:30.639 --> 00:34:33.119
If the compiler fails, you go fix the compiler.

s429
00:34:33.119 --> 00:34:34.079
You run the program.

s430
00:34:34.079 --> 00:34:36.240
If the program fails, you go fix the compiler.

s431
00:34:36.240 --> 00:34:39.919
Like it's like it's very very verifiable.

s432
00:34:39.919 --> 00:34:50.000
And I think the lesson in loops engineering is like if you can make a problem very verifiable, you can kind of like treat it like a black box

s433
00:34:50.000 --> 00:34:55.839
and then have it loop because it will keep improving itself because of the verification loop is already there.

s434
00:34:55.839 --> 00:34:56.159
Exactly.

s435
00:34:56.159 --> 00:34:59.440
And so like you can do this with CI/CD is like I I do this every time I'm doing a release.

s436
00:34:59.440 --> 00:35:00.079
I'm like I'm tired.

s437
00:35:00.079 --> 00:35:01.200
The CI/CD is slow.

s438
00:35:01.200 --> 00:35:01.680
Cool.

s439
00:35:01.680 --> 00:35:07.040
Go research the codebase, make a change, make a pull request, run the test, see if it's faster, try again.

s440
00:35:07.040 --> 00:35:08.640
Run the p run the test.

s441
00:35:08.640 --> 00:35:22.960
push push to the branch check again see if it's faster and so it's like if it can verify its own work in a loop instead of design instead of saying let's try this approach or let's try that approach or suggest and being really back and forth you just say like my goal is to make CI faster

s442
00:35:22.960 --> 00:35:34.079
and you tell the model here's the steps here's the five here's the five steps you're going to write some code you're going to commit it you're going to push it you're going to launch a sub agent to watch the job until it's finished it's going to tell you what happened then you're going to decide what to do

s443
00:35:34.079 --> 00:35:38.960
next and so that's that's like the very simplest example I have of like designing loops

s444
00:35:38.960 --> 00:35:50.560
and you just set the goal which is cloud code and and I think codecs have both chip/go goal which is you just set the goal and it iterates until it reaches it or or as long as it makes progress towards it.

s445
00:35:50.560 --> 00:35:51.200
Exactly.

s446
00:35:51.200 --> 00:35:54.960
And so it's like if it's verifiable if you can measure this is auto research too.

s447
00:35:54.960 --> 00:36:03.760
Auto research is like hey go make this model twice as fast and like it's just a prompt that tells the model to like go to it over and over again and try things until it actually has good results.

s448
00:36:03.760 --> 00:36:05.119
So that's what I think of loops engineering.

s449
00:36:05.119 --> 00:36:05.440
I don't know.

s450
00:36:05.440 --> 00:36:18.160
We we do a very interesting kind of loops engineering where like the the challenge is like I think it's very easy to get very excited about building the thing that builds the thing or building the thing that builds the thing that builds the thing we talked about.

s451
00:36:18.160 --> 00:36:26.079
Uh and so people say, "Oh, we need to like redo everything as this big like aentic first factory, maybe even a dark factory."

s452
00:36:26.079 --> 00:36:30.160
And they're like redesigning their entire thing to be their infrastructure for the next 5 years.

s453
00:36:30.160 --> 00:36:37.440
And I'm sure one thing we know of in engineering uh and especially uh pragmatic engineering is uh how can you make this more incremental?

s454
00:36:37.440 --> 00:36:39.359
How can you make it more continuous?

s455
00:36:39.359 --> 00:36:46.560
Uh and a lot of people don't have the option to just hey I ran a Ralph loop for 3 days and it fixed every line error in our codebase.

s456
00:36:46.560 --> 00:36:48.320
Here's a 60,000line PR.

s457
00:36:48.320 --> 00:36:54.160
Who wants to review it and who wants to sign off on merging and deploying it and uh that there's not going to be any bugs?

s458
00:36:54.160 --> 00:36:54.800
Nobody.

s459
00:36:54.800 --> 00:37:01.839
So I think the the thing I'm most excited is actually like what we call like iterated loops or like slow loops where we basically have a cron job.

s460
00:37:01.839 --> 00:37:04.240
We have the loop the the the structure of the loop is really easy.

s461
00:37:04.240 --> 00:37:15.280
It's like run this llinter fix one thing commit and push and then we run that every night in our GitHub actions and we wake up every morning to one PR that makes the codebase a little bit better.

s462
00:37:15.280 --> 00:37:16.880
I I like the slow loops.

s463
00:37:16.880 --> 00:37:17.119
Yeah.

s464
00:37:17.119 --> 00:37:18.480
And it has two dimensions.

s465
00:37:18.480 --> 00:37:23.359
So you can add now we have a blueprint for it and actually Kyle just shipped a skill so that you can build these yourself.

s466
00:37:23.359 --> 00:37:26.240
you can add more like feedback mechanisms.

s467
00:37:26.240 --> 00:37:28.240
So, we have React Doctor for the front end.

s468
00:37:28.240 --> 00:37:32.720
We have another anti-attern that has no deterministic tooling, but Kyle's just like, "Here's what good looks like.

s469
00:37:32.720 --> 00:37:33.680
Here's what bad looks like.

s470
00:37:33.680 --> 00:37:34.880
Go fix one thing and bring it back."

s471
00:37:34.880 --> 00:37:36.240
It's like prop narrowing basically.

s472
00:37:36.240 --> 00:37:38.720
We have a bunch of optional props and most of them don't need to be optional.

s473
00:37:38.720 --> 00:37:44.320
It's like here's how to make the prop not optional so that you know that the code just is like cleaner and easier to reason about.

s474
00:37:44.320 --> 00:37:47.920
And so, you can add more conditions, more things of like fix one thing.

s475
00:37:47.920 --> 00:37:49.200
I want to wake up to a PR.

s476
00:37:49.200 --> 00:37:52.320
So, now we wake up to like four PRs because there's four separate things.

s477
00:37:52.320 --> 00:37:56.640
And then the other dimension you can do here is as you gain confidence, you can increase the scope.

s478
00:37:56.640 --> 00:37:58.880
Instead of fixing one thing, fix four things.

s479
00:37:58.880 --> 00:38:05.200
And so these are like other ways to think about loops where it's like something that's not a human triggers it to start.

s480
00:38:05.200 --> 00:38:22.800
Whether it's, you know, an alert from Sentry, whether it's a user feedback like support ticket, whether it's PM writes a ticket, whether it's a test is failing, any of or it's a cron, it runs on a schedule, but it's like the trigger should be something that you don't have to like press a button on and there's a

s481
00:38:22.800 --> 00:38:25.520
defined workflow and it makes everything a little bit better.

s482
00:38:25.520 --> 00:38:29.520
Dex just described letting agents fix things without a human pressing a button.

s483
00:38:29.520 --> 00:38:35.119
But what if a bug is too difficult not just for an agent but also for human to reproduce let alone fix?

s484
00:38:35.119 --> 00:38:37.599
This is where presenting sponsor anticysis comes in.

s485
00:38:37.599 --> 00:38:45.920
I was recently pairing with the antithesis team where we did a walkthrough of how they helped fix a nasty bug inc the open source key value store used by Kubernetes.

s486
00:38:45.920 --> 00:38:48.160
This is a bug that actually happened incd.

s487
00:38:48.160 --> 00:38:53.440
The team noticed that the linearization validation assertion failed during the regular anticis runs.

s488
00:38:53.440 --> 00:38:57.040
This is not good because the linearization guarantees strong consistency.

s489
00:38:57.040 --> 00:38:58.560
So this needs to be fixed.

s490
00:38:58.560 --> 00:39:03.359
So what the ETCD team did was run a casualty analysis inside anticysis.

s491
00:39:03.359 --> 00:39:07.359
This generates this graph which is a bug probability graph.

s492
00:39:07.359 --> 00:39:12.079
Here the x-axis is virtual time and the y-axis is probability.

s493
00:39:12.079 --> 00:39:19.040
Now we see that something happened just before virtual time 24 that caused a huge jump in the probability that the bug would occur.

s494
00:39:19.040 --> 00:39:22.560
Going deeper, we can look at the entire set of timelines.

s495
00:39:22.560 --> 00:39:29.040
Vertical lines going down represent events branching off from the same state and the purple dots are where the buck happens.

s496
00:39:29.040 --> 00:39:33.760
If we look closely enough, we see that all of the failures come from one parent branch.

s497
00:39:33.760 --> 00:39:34.480
Gotcha.

s498
00:39:34.480 --> 00:39:36.320
This is such a useful debugging tool.

s499
00:39:36.320 --> 00:39:42.079
In the end, the team was able to figure out that process pauses were causing the bug using all these anticis debugging tools.

s500
00:39:42.079 --> 00:39:46.240
This non-deterministic bug was diagnosed in a deterministic way.

s501
00:39:46.240 --> 00:39:47.359
How cool is that?

s502
00:39:47.359 --> 00:39:50.480
Oh, and this is an actual bug that then got fixed incd.

s503
00:39:50.480 --> 00:39:53.520
You can see the bug and the fix in ATCD's GitHub repo.

s504
00:39:53.520 --> 00:39:58.720
Honestly, the tools that Antys is built for debugging feel pretty darn futuristic, but they are also really powerful.

s505
00:39:58.720 --> 00:40:01.760
Head over to antithesis.com/pragmatic to learn more.

s506
00:40:01.760 --> 00:40:04.880
I'd also like to talk about our season sponsor, Sentry.

s507
00:40:04.880 --> 00:40:09.920
Sentry is a tool I use for application monitoring on all of my projects, including their pragmatic engine back end.

s508
00:40:09.920 --> 00:40:13.200
I've used it for 10 years now, starting with when I worked at Uber.

s509
00:40:13.200 --> 00:40:18.079
A neat Sentry feature I'm liking is their SE AI agent, which helps investigate production errors.

s510
00:40:18.079 --> 00:40:20.800
For example, here's an actual error I had in my application.

s511
00:40:20.800 --> 00:40:24.480
I can just ask Seir what might be the root cause and it brings context.

s512
00:40:24.480 --> 00:40:27.440
And it can also make a plan to fix it right from the web interface.

s513
00:40:27.440 --> 00:40:31.440
And a nice thing is how Seir also works great from Slack as well, not just from the web.

s514
00:40:31.440 --> 00:40:36.079
One place I find even more handy to use Sentry is from codeex or clock code using Sentry MCP.

s515
00:40:36.079 --> 00:40:40.320
Also, you can set up neat automations like when a resolve sentry issue resurfaces.

s516
00:40:40.320 --> 00:40:47.839
You can kick off a cursor agent or GitHub copilot agent to investigate the regression, read the relevant code, and open a PR with a suggested fix.

s517
00:40:47.839 --> 00:40:55.280
I'm not a fan for using AI tools just for the sake of it, but I really like the practical integrations where I can fix errors faster and with more context.

s518
00:40:55.280 --> 00:41:00.960
Check out Sentry at centry.io/pragmatic and start monitoring and fixing regressions today.

s519
00:41:00.960 --> 00:41:05.200
And with this, let's get back to Dex and to Agentic loops that trigger themselves.

s520
00:41:05.200 --> 00:41:12.880
Now, you said we can get more ambitious and we can add more things to it, but I'm I'm going to quote you with uh with one of your tweets

s521
00:41:12.880 --> 00:41:27.920
which says, "This may surprise you that this is coming from me, but I think we're in for a 1 to three year period where stuff might break at 3:00 a.m. and you're relying on loops to fix it and nobody understands what's under the hood, and you're looking at an ex existential threat to your company."

s522
00:41:27.920 --> 00:41:28.400
Yes.

s523
00:41:28.400 --> 00:41:29.680
Uh yeah, that one was great.

s524
00:41:29.680 --> 00:41:30.800
That one did a lot of numbers.

s525
00:41:30.800 --> 00:41:31.648
Uh

s526
00:41:31.648 --> 00:41:32.319
[laughter]

s527
00:41:32.319 --> 00:41:33.040
it resonated.

s528
00:41:33.040 --> 00:41:43.520
Here's the other side of it is like I think that the today with today's models, today's programming languages, today's infrastructure, you might get away with not reading the code.

s529
00:41:43.520 --> 00:41:48.079
Problem with loops is like at a certain point you're going to generate so much code that you can't read it anymore.

s530
00:41:48.079 --> 00:41:50.480
This is the strong VM dark factory.

s531
00:41:50.480 --> 00:41:52.880
This is like Ryan Leopo's like harness engineering.

s532
00:41:52.880 --> 00:41:54.880
Just spend as many tokens as possible.

s533
00:41:54.880 --> 00:41:55.520
We tried this.

s534
00:41:55.520 --> 00:42:01.520
We built a lights off software factory in July of 2025 and by November we had shut it down.

s535
00:42:01.520 --> 00:42:10.800
I think it takes about three to six months of you shipping all the time with nobody reading the code before you realize like, wow, this is getting way worse and it's easier to start over than it is to fix it.

s536
00:42:10.800 --> 00:42:15.680
Like the models have made the codebase so bad that it is actually going to be easier to just like rethink this from scratch.

s537
00:42:15.680 --> 00:42:18.640
And maybe that's okay because we have AI and it's easier to rebuild things from nothing.

s538
00:42:18.640 --> 00:42:21.760
And like usually when engineers say like, "Oh, we can't fix this.

s539
00:42:21.760 --> 00:42:22.800
We have to rebuild it."

s540
00:42:22.800 --> 00:42:24.800
The feedback is like, "No, just refactor in place.

s541
00:42:24.800 --> 00:42:26.880
Just constantly keep the codebase getting better."

s542
00:42:26.880 --> 00:42:27.839
You mentioned what I said.

s543
00:42:27.839 --> 00:42:31.760
You'll notice what I said was not use loops to ship the features that users want.

s544
00:42:31.760 --> 00:42:39.760
We use loops to actually improve the codebase quality and we read all the code because we care about how it's architected and we care not just about the system architecture

s545
00:42:39.760 --> 00:42:52.240
but what I would call the program design which I think is something people are going to where are the interfaces where are the seams how are we doing dependency injection all of these things that like make your codebase more maintainable over time and keep you from

s546
00:42:52.240 --> 00:42:57.040
falling into this trap of like okay well now if I change something over here I broke something over here.

s547
00:42:57.040 --> 00:43:09.200
This is the classic problem of software engineering that like software engineering was invented in the 1970s because we realized we needed techniques for avoiding that problem of like this giant ball of spaghetti.

s548
00:43:09.200 --> 00:43:24.560
And I don't think the models are smart enough and I don't think we actually have the training and the benchmarking and the eval techniques to get models to write code that is more maintainable over time versus they're all trained on SWE and SWEBench looking things, right?

s549
00:43:24.560 --> 00:43:28.480
All of the benchmarks are basically like here's a commit in Django.

s550
00:43:28.480 --> 00:43:31.599
Here's an issue that was filed around that time.

s551
00:43:31.599 --> 00:43:33.760
see if you can create the fix that the human created.

s552
00:43:33.760 --> 00:43:44.160
And it's Django and it's Apache and it's there's a hundred repos in Go and C++ and Typescript and Java and all these different languages, but they're all it's like the problem with training models on maintainability

s553
00:43:44.160 --> 00:43:56.880
is like the cost function of bad architecture and bad program design can't be evaluated by running the unit test because it hits you 3 to 6 months later when you're like, "Holy crap, like no one can make it's this software has become so hard to change."

s554
00:43:56.880 --> 00:44:04.319
Is this not similar to how senior software engineers why it took years for someone to become a senior?

s555
00:44:04.319 --> 00:44:12.319
Because typically and in some environments you became a you can become a senior faster typically fast moving where there's a bunch of issues and you have to keep fixing it.

s556
00:44:12.319 --> 00:44:16.160
Sometimes you know some people are working in the same place for 10 years and they're still not that level.

s557
00:44:16.160 --> 00:44:25.440
The point was it it just takes time for you to understand the m the small mistake that you make right now that snowballs into like something disastrous later and

s558
00:44:25.440 --> 00:44:35.040
you get hit by it and you realize like okay things like you know like testing matters architecture matters tech depth can actually be a killer you know we don't talk about it anymore but we used to talk about how techdub

s559
00:44:35.040 --> 00:44:45.520
kills or slows down companies so badly preai that their competitors can overtake them or they're just like stuck with a 2-year refactor not shipping any new features and the competition

s560
00:44:45.520 --> 00:44:48.240
you shifts a bunch bunch of other stuff and now they're ahead.

s561
00:44:48.240 --> 00:44:57.680
And I I will say like it is possible that GPT7 will fix this, but if you are turning the lights off in your software factory and you're saying like,

s562
00:44:57.680 --> 00:44:58.560
"Hey, you know what?

s563
00:44:58.560 --> 00:44:59.760
Like we're not going to read the code.

s564
00:44:59.760 --> 00:45:00.160
It's fine.

s565
00:45:00.160 --> 00:45:01.119
The models are smart enough.

s566
00:45:01.119 --> 00:45:05.680
If we give it the right feedback and just throw enough tokens at the problem, it will keep getting better."

s567
00:45:05.680 --> 00:45:07.680
This is what led to this tweet.

s568
00:45:07.680 --> 00:45:15.920
that might work, but if nobody read the code in three months and you replace all of your all of your like code review with loops of like, hey, if a user complains, we give it to an agent.

s569
00:45:15.920 --> 00:45:17.920
If something crashes, we give it to an agent.

s570
00:45:17.920 --> 00:45:20.240
If a if a PM writes a ticket, we give it to an agent.

s571
00:45:20.240 --> 00:45:24.400
If a CEO writes an obnoxious essay about what we should be building in Slack, we give it to an agent.

s572
00:45:24.400 --> 00:45:24.831
Yeah.

s573
00:45:24.831 --> 00:45:25.520
[laughter]

s574
00:45:25.520 --> 00:45:28.319
And then you stop reading the code because that's going to produce way too much.

s575
00:45:28.319 --> 00:45:29.280
Like, no one can read it.

s576
00:45:29.280 --> 00:45:31.599
And like the the the PR reviews become the bottleneck.

s577
00:45:31.599 --> 00:45:35.760
So, you replace that with aentic testing and agentic uh agentic code review.

s578
00:45:35.760 --> 00:45:40.720
Uh, but none of these things have intuition for software architecture because we haven't trained it in yet.

s579
00:45:40.720 --> 00:45:46.480
And so you're going to wake up one day and you're going to have an issue with this happened to us and like we got through it and at the time like it was still worth it.

s580
00:45:46.480 --> 00:45:58.319
It was like spent 3 weeks onboarding back into the codebase that we had stopped reading 3 months ago because no matter how much sophisticating expert prompting we could not get Opus, I think it was Opus 4.1 at the time.

s581
00:45:58.319 --> 00:46:01.359
We could not get Opus 4.1 to actually find the root cause.

s582
00:46:01.359 --> 00:46:11.040
We had to go spend several days digging through the code and figuring out like, oh, there's just actually a primary key that's being routed through this whole thing that needs to be changed to a different type of object and it needs it.

s583
00:46:11.040 --> 00:46:11.760
This actually happened to you.

s584
00:46:11.760 --> 00:46:12.480
This happened to us.

s585
00:46:12.480 --> 00:46:13.200
Yeah.

s586
00:46:13.200 --> 00:46:14.800
And when it happened, I was like, you know what?

s587
00:46:14.800 --> 00:46:15.280
That sucked.

s588
00:46:15.280 --> 00:46:16.000
That was terrible.

s589
00:46:16.000 --> 00:46:16.480
But we did it.

s590
00:46:16.480 --> 00:46:17.440
We solved it.

s591
00:46:17.440 --> 00:46:26.400
And uh it's still worth it's still worth not reading the code for most of the time at the cost of every once in a while I'm going to have to spend two weeks fixing an issue by hand.

s592
00:46:26.400 --> 00:46:34.160
And I don't believe that anymore because I think the amount of code we're able to write now is actually like 10xed or 100xed and I think the problem's just getting worse.

s593
00:46:34.160 --> 00:46:36.400
So let's talk about software factories.

s594
00:46:36.400 --> 00:46:36.800
Yeah.

s595
00:46:36.800 --> 00:46:44.480
In your mind, cuz I feel it's an overloaded word, but what do you think of a software factory before AI and now post AI?

s596
00:46:44.480 --> 00:46:47.520
Do you know the first definition of software factory the first time it was used?

s597
00:46:47.520 --> 00:46:50.640
No. It was a NATO conference in 1968.

s598
00:46:50.640 --> 00:46:52.240
Oh, Grady Buch would know about this.

s599
00:46:52.240 --> 00:46:52.800
Yeah, exactly.

s600
00:46:52.800 --> 00:46:53.200
Yeah, great.

s601
00:46:53.200 --> 00:46:54.400
You should ask Grady about it.

s602
00:46:54.400 --> 00:47:00.160
They talked about the idea of like, okay, you actually need to build a system of steps and like just like a factory floor.

s603
00:47:00.160 --> 00:47:05.359
You have like the coding part and the testing part and the validation part and the integration part.

s604
00:47:05.359 --> 00:47:06.560
We had no CI/CD.

s605
00:47:06.560 --> 00:47:22.160
we barely had version control like but you needed a factory and then it was adopted by like um Toshiba and a bunch of companies and then the the next moment was like DevOps and you have like this idea of like okay we're going to do CI/CD we're going to automate we're chef and anible puppet whatever all these

s606
00:47:22.160 --> 00:47:38.720
technologies is like instead of having dudes running around data centers like resizing discs and stuff or clicking around the AWS console yeah exactly it was like cool we build loops the server hits 90% disc space that sends an alert to Nagios Nagios triggers a chef front chefs makes the disc the disc bigger

s607
00:47:38.720 --> 00:47:40.000
feedback loops, right?

s608
00:47:40.000 --> 00:47:41.440
This has been around for a while.

s609
00:47:41.440 --> 00:47:50.640
And in 2018, I want to say this guy Nick Chalane who was uh he was like the CTO or chie ch ch ch ch ch ch ch ch ch ch ch ch ch ch ch ch ch ch ch ch ch ch ch ch ch ch chief software officer of the air force, he wrote this 100page

s610
00:47:50.640 --> 00:47:54.800
essay of hey the DoD needs a software factory,

s611
00:47:54.800 --> 00:47:55.920
the department of defense.

s612
00:47:55.920 --> 00:47:58.240
Yeah, the the department of defense and the air force.

s613
00:47:58.240 --> 00:48:00.400
And he called it the dev sec ops factory.

s614
00:48:00.400 --> 00:48:04.400
And he said we need all the things that all of the good enterprises are using.

s615
00:48:04.400 --> 00:48:05.680
We need Jenkins.

s616
00:48:05.680 --> 00:48:07.839
We need like code quality scanning.

s617
00:48:07.839 --> 00:48:08.880
We need security scanning.

s618
00:48:08.880 --> 00:48:09.920
We need CI/CD.

s619
00:48:09.920 --> 00:48:11.040
We need to be able to ship.

s620
00:48:11.040 --> 00:48:13.520
We're shipping once every three months or once a year.

s621
00:48:13.520 --> 00:48:16.400
We need to be able to ship every day like all these other companies.

s622
00:48:16.400 --> 00:48:30.800
And the way we do that is we actually embrace all these automations and technologies so that engineers are are 90% of the issues are caught by automations instead of people actually like manually checking it or manually reading the code or manually integrating modules together.

s623
00:48:30.800 --> 00:48:31.760
Wow.

s624
00:48:31.760 --> 00:48:34.640
Talk about forward thinking in in the government.

s625
00:48:34.640 --> 00:48:34.880
I know.

s626
00:48:34.880 --> 00:48:47.680
Oh no, as I was surprised like oh nice like this is I mean and that was part of it is like hey look we're falling behind in like you know I don't know exactly all the reason but I I imagine also about like attracting really good talent is like hey look if we have like the modern software

s627
00:48:47.680 --> 00:48:58.160
stack and we're building things fast and we care about efficiency and we care about people's using people's time well we care about them spending time on the hard parts of the job not manually looking for SQL injections like you could automate that.

s628
00:48:58.160 --> 00:49:00.800
So this was software factories pre AAI.

s629
00:49:00.800 --> 00:49:01.440
Pre AI.

s630
00:49:01.440 --> 00:49:04.240
Now I've heard the term a lot more because of AI.

s631
00:49:04.240 --> 00:49:04.559
Yeah.

s632
00:49:04.559 --> 00:49:05.280
Is it the same?

s633
00:49:05.280 --> 00:49:06.640
Is it different?

s634
00:49:06.640 --> 00:49:10.079
So this is really hard to say without a drawing, but I'll try to draw it out.

s635
00:49:10.079 --> 00:49:13.680
At the core of a software factory, you have like a source of work.

s636
00:49:13.680 --> 00:49:21.760
Most you you can imagine a linear a Jira a the st source of truth your object whether it's a spreadsheet or whatever is you have like what stages is the work in.

s637
00:49:21.760 --> 00:49:22.240
Yep.

s638
00:49:22.240 --> 00:49:26.160
And prei you would take, you know, you would maybe do some architecture review planning.

s639
00:49:26.160 --> 00:49:30.559
You would maybe do some sprint planning and then people would take tickets off the queue and they would go build them.

s640
00:49:30.559 --> 00:49:42.640
And then you would make a pull request and people would review it and you would run CI checks and then you would send it to prod and then it would make contact with your users and your users would complain about stuff and that would go to your support team and back into your work tracker and

s641
00:49:42.640 --> 00:49:48.000
it would crash and you would have issues and that would go into your monitoring stack and that would go into your tracker and that was your loop.

s642
00:49:48.000 --> 00:49:51.040
And then people would take stuff off the tracker based on priorities.

s643
00:49:51.040 --> 00:50:06.400
product managers, engineering managers, engineers prioritizing work and then we go and do that and the first change is is like this long wind lot lots of phases and this is also why when like a developer shifts a bug but by the time it comes back to you it might be two or 3 months or even longer and

s644
00:50:06.400 --> 00:50:15.520
by the time it get fixed it might be a year or two and you know this is why when you're using a piece of software it's like that annoying bug and you talk with customer support but it's just a very

s645
00:50:15.520 --> 00:50:20.400
like long latencies at each each part of the the factory if you will.

s646
00:50:20.400 --> 00:50:20.640
Yeah.

s647
00:50:20.640 --> 00:50:30.720
And the the step where someone pulls a work item off a queue and starts working on it is, you know, couple hours to a couple days before it actually gets integrated into everything else and touches user.

s648
00:50:30.720 --> 00:50:32.480
And that's in a in a in a great world, right?

s649
00:50:32.480 --> 00:50:35.760
Sometimes you go build it and then you merge it and then it actually gets released 3 months later.

s650
00:50:35.760 --> 00:50:46.640
But we're going to assume we're in a fairly modern like we're somewhere like the a Netflix or a meta where engineers are capable of shipping 100 times a day or a thousand times a day, but it still takes 2 three hours to do the work.

s651
00:50:46.640 --> 00:50:52.480
And now with an identic factory, what you do is you take out that person building the thing and you replace it with an agent building the thing.

s652
00:50:52.480 --> 00:50:54.720
And so you have orchestration to trigger things.

s653
00:50:54.720 --> 00:51:01.200
You have a sandbox, you have an LLM, you have an inner harness, you have an outer harness, which is like the dev environment you build for the agent.

s654
00:51:01.200 --> 00:51:05.680
And maybe you give it a browser, you give it a video recorder if you use like things like cursor background agents.

s655
00:51:05.680 --> 00:51:09.359
They've kind of built this outer harness around the inner harness that is the coding agent.

s656
00:51:09.359 --> 00:51:11.280
And then you make PRs with that.

s657
00:51:11.280 --> 00:51:25.280
problem there is that like okay now now it takes 10 minutes to do a build instead of two hours or two days and so now the bottleneck is code review so okay let's throw a bunch of AI agents at code review and let's do agentic testing so that like we can basically catch a lot of the easy

s658
00:51:25.280 --> 00:51:36.559
stuff and humans are only focused on the most like important critical core parts of the codebase and then the next level up of your agentic factory is you do the top it's like okay then it gets deployed it goes to prod

s659
00:51:36.559 --> 00:51:50.720
and a user complains you just hook your support queue right up to the agent someone complains about something agent tries to fix it and instead of looking at a ticket and then saying okay go send you just close that loop and instead every time something goes wrong you just get a PR and then every time something crashes

s660
00:51:50.720 --> 00:52:04.079
in Sentry or Data Dog or whatever it goes into the tracker it gets picked up by an agent and you get a PR this is the ramp inspect thing this is the the only difference is like then you have so much code to review and people say well let's try turning the lights off let's just take all

s661
00:52:04.079 --> 00:52:12.319
the human testing and review steps out and we'll say okay cool if users complain then it's broken and if users don't complain and it's working and we're not going to read the code.

s662
00:52:12.319 --> 00:52:14.960
We're going to use we're going to treat the whole system as a black box.

s663
00:52:14.960 --> 00:52:22.319
So, you said you tried this out uh when it was like Opus Formula and you you built the software factory was running beautifully until it just blew up on your faces.

s664
00:52:22.319 --> 00:52:23.920
How do you think of this model?

s665
00:52:23.920 --> 00:52:28.720
cuz I I can see an ideal world where it works, but clearly we're not in an ideal world.

s666
00:52:28.720 --> 00:52:38.000
Like where do you think we are like and could some of this actually work at some point or you know like like what what progress are you seeing right now and and

s667
00:52:38.000 --> 00:52:43.440
what is the the today the situation like how much of this do you believe we can automate or should we automate?

s668
00:52:43.440 --> 00:52:44.079
Yep.

s669
00:52:44.079 --> 00:52:47.599
So if you know me, you follow my stuff, you know I stand for three things.

s670
00:52:47.599 --> 00:52:55.599
Number one is like cutting through the hype and the jargon and going trying things and talking to people who are using things and figuring out which parts of this actually work and are valuable.

s671
00:52:55.599 --> 00:52:57.119
Number two, we talked about words.

s672
00:52:57.119 --> 00:53:03.119
I try to find and protect useful bits of language because I think it helps us all move forward.

s673
00:53:03.119 --> 00:53:10.720
And when you take a useful word like agents or you take a useful word like software factory and then you semantically diffuse it, this is another Martin Fowler word.

s674
00:53:10.720 --> 00:53:16.720
You make it mean everybody likes the word and it all becomes hype and everyone starts agents means nothing anymore.

s675
00:53:16.720 --> 00:53:22.400
agents could be a chatbot, it could be a Slackbot, it could be a coding agent, it could be tools in a loop, whatever it is.

s676
00:53:22.400 --> 00:53:29.839
So, I like to protect important useful words and like help help us all like elevate the conversation out of that hype and jargon.

s677
00:53:29.839 --> 00:53:35.760
And then I care a lot about going one level down beneath where I'm generally working.

s678
00:53:35.760 --> 00:53:45.280
I think there's always this is the same thing with context engineering is like I was rarely actually going and like building LLMs or understanding or training LLMs but knowing how they're trained how transformers works

s679
00:53:45.280 --> 00:53:54.319
informs how you build at one layer up and for the software factory my version of that is I spent the last couple weeks going really deep on uh reinforcement

s680
00:53:54.319 --> 00:54:14.319
learning with uh verifiable rewards RLVR which is like this very productionized like it's not like RH RHF is still like fairly academic and pure RLVR are is this like it's a machine in these labs of how we train these models and I'm studying like the benchmarks for coding agents and the techniques for training them and how we

s681
00:54:14.319 --> 00:54:29.280
like give it a small problem have it solve it delete the test changes it made revert them apply a test patch see if it passed and then even the frontier this year we have like we can get into this later but like frontier code and marathon these new benchmarks that are supposed to be like better at evaluating

s682
00:54:29.280 --> 00:54:36.319
models's ability to maintain a codebase over time and write maintainable code um and they are better But I don't think they're sufficient.

s683
00:54:36.319 --> 00:54:42.559
But it's basically this idea that like the only thing that made claude code good was reinforcement learning.

s684
00:54:42.559 --> 00:54:46.160
And the dimension along which it got good was like we made a model.

s685
00:54:46.160 --> 00:54:48.559
We trained the model and the harness together.

s686
00:54:48.559 --> 00:54:53.040
And so the model got really good at calling the specific tools in that harness.

s687
00:54:53.040 --> 00:54:57.920
Really good at reading files, writing files, searching for files, all this stuff through doing these problems.

s688
00:54:57.920 --> 00:55:02.960
And that was what made it feel so much better than all the other CLI coding agents that came before it.

s689
00:55:02.960 --> 00:55:04.720
And so people like, "Okay, that was so much better."

s690
00:55:04.720 --> 00:55:05.920
And they're just going to keep getting better.

s691
00:55:05.920 --> 00:55:08.960
But it's like it got really good in one dimension.

s692
00:55:08.960 --> 00:55:13.680
And the dimension that they're not getting better in because it's hard, expensive.

s693
00:55:13.680 --> 00:55:29.920
Maybe we need to like get a lot more creative with how we design these these verifiers and benchmarks is in how do I make code that in three months is going to like improve the productivity of humans and agents, mostly agents, but humans and agents in the codebase instead of making it worse over time.

s694
00:55:29.920 --> 00:55:31.839
And so you think that part is just missing?

s695
00:55:31.839 --> 00:55:34.800
We haven't seen too much improvement.

s696
00:55:34.800 --> 00:55:38.480
I haven't seen obviously no one knows what the labs are doing internally cuz it's all very secret.

s697
00:55:38.480 --> 00:55:45.040
But I think if we looking at where the bench the benchmarks tend to reflect where the labs are, right?

s698
00:55:45.040 --> 00:55:52.400
If there is no benchmark that can convey to me did this model write code that is going to make my codebase better or worse.

s699
00:55:52.400 --> 00:55:56.640
The best we have is I I think frontier code from the cognition team is really interesting.

s700
00:55:56.640 --> 00:56:01.680
They have like did the test pass and then they have like two layers of model review.

s701
00:56:01.680 --> 00:56:08.319
So they have a judge model that checks okay is the patch the model made similar to the patch that is like the golden answer set.

s702
00:56:08.319 --> 00:56:19.440
So even if the model didn't write the exact code that the benchmark was expecting did was it functionally equivalent and the next one is like a like code quality review from another judge model and

s703
00:56:19.440 --> 00:56:23.119
like that's better but it's not it's not sufficient.

s704
00:56:23.119 --> 00:56:34.799
And this is why I also think agentic code review is like yes it will catch things and it will raise your floor but I don't believe like the model writing the code is the same model reading the code and if you ask a model hey is this code good it's going to be like oh yeah it's great

s705
00:56:34.799 --> 00:56:47.359
comprehensive it's got unit tests you've tried this I'm sure and you say okay review this PR that my coworker wrote and tell me everything that's wrong with it I was like oh it has this problem and this problem and this is this is sickopantic and they want to tell you what you want to hear and so like

s706
00:56:47.359 --> 00:57:02.079
it's really hard for me to trust a model to evaluate the quality of of of code that's written And so I I I have some ideas on like, okay, can you build a benchmark where the model builds 20 features in a row and maintains the codebase the whole time and it doesn't know what features are coming.

s707
00:57:02.079 --> 00:57:08.000
You treat it like a real product team where you don't know what you're going to build next week until you get there and you find out what's most important

s708
00:57:08.000 --> 00:57:16.640
and then can we try to evaluate like can we build a problem like that that's hard enough that most frontier models fail by issue six or seven.

s709
00:57:16.640 --> 00:57:24.880
Is it fair to say that you know like we've had the software factory like before AI it was just like lots of loop it was like the the PM

s710
00:57:24.880 --> 00:57:39.119
giving the ticket to the dev the dev building it deploying to production user customers using it customer support getting tickets and then you creating PM triaging and it kind of goes around like in this loop is it fair to say that

s711
00:57:39.119 --> 00:57:54.160
the software factory of how a company a team builds and maintains software that is changing because now everyone's replacing some parts of it, you know, maybe the the least advanced teams will just be devs are starting to use cloud code or codecs to write faster.

s712
00:57:54.160 --> 00:57:55.839
They're not spending as much time on there.

s713
00:57:55.839 --> 00:57:58.720
Some others are also having the deployment the feedback.

s714
00:57:58.720 --> 00:58:02.960
Some some actually have the agents already oneshotting bucks.

s715
00:58:02.960 --> 00:58:10.559
So like is it fair to say that that the software factory is just is just changing everywhere maybe at different speeds but everyone I think every team who is

s716
00:58:10.559 --> 00:58:16.160
building production software they're like they're frantically experimenting trying and everyone's at a different pace.

s717
00:58:16.160 --> 00:58:24.000
You'll have the AI native starters where most of this will have agents in them and you'll have the the laggers who are or more more cautious ones.

s718
00:58:24.000 --> 00:58:26.640
They have agents in a few places but not in the others.

s719
00:58:26.640 --> 00:58:33.680
Well, and I think that's the key is like if you want to do loops engineering, you should build one loop at a time and you should keep them small and contained.

s720
00:58:33.680 --> 00:58:38.240
Basically, I think everything except stop reading the code is really good advice.

s721
00:58:38.240 --> 00:58:43.040
Take support tickets and turn them into tickets in your system and then maybe turn those into PRs.

s722
00:58:43.040 --> 00:58:43.520
Great.

s723
00:58:43.520 --> 00:58:51.119
The advice that I have and like what we kind of like are chasing at human layer is like how can I add another checkpoint in that factory?

s724
00:58:51.119 --> 00:59:02.640
So instead of having one human re view point where you're reviewing PRs and sometimes they're 100 lines and sometimes they're a thousand lines but it's quite a lot of effort for especially if it's bad especially if it needs rework.

s725
00:59:02.640 --> 00:59:13.920
It's quite a lot of effort for a human to be like okay this is wrong go change it in this way and then you loop back to the agent and then you come with another one and like doing a lot of loops on there once once the direction has been committed to it's really hard to steer off

s726
00:59:13.920 --> 00:59:16.960
like you're better off just kind of restarting from scratch.

s727
00:59:16.960 --> 00:59:19.520
How do you build like controls and mechanisms around that?

s728
00:59:19.520 --> 00:59:35.839
And then my take is like if you do a little bit of human agent planning and like discussion before you hand it to the impletor whether it's I mean planning and specs whatever you want to call it again this is spec driven development is another word that has become kind of very like muddled as far as what

s729
00:59:35.839 --> 00:59:53.680
it means but basically how can we spend an hour before we start building so that the PR when we read it only takes 20 minutes because the code is perfect instead of not touching it just literally saying every user reported issue becomes a PR through the loop and then we read that PR and it takes six hours

s730
00:59:53.680 --> 00:59:55.920
because there's back and forth and we have to make changes and things.

s731
00:59:55.920 --> 00:59:58.480
It's all I'm all about like let's find leverage.

s732
00:59:58.480 --> 01:00:01.920
And so you basically you have three options in the software factory world.

s733
01:00:01.920 --> 01:00:16.319
If you're going to go all in on aentic software factories, you can turn the lights off and just let everything flow and pray that you don't create too much slop and pray that the next generation of models comes fast enough before you create a giant pile of ash.

s734
01:00:16.319 --> 01:00:20.319
you can slow way down and read every PR and read every line of code.

s735
01:00:20.319 --> 01:00:30.880
Uh, and then you're only going to really get modest benefits from AI because that becomes I I think you should expect maybe 30 to 50% lift in productivity is kind of what I see when we go into teams

s736
01:00:30.880 --> 01:00:44.720
or you can find the right leverage points where humans can actually an hour spent over here in planning can save you four hours in in implementation in terms of fixing and going back and and getting the design right.

s737
01:00:44.720 --> 01:00:46.720
And that's what I call like seeking leverage.

s738
01:00:46.720 --> 01:00:55.920
If you can find the right leverage points for the agents to guide the work, then you can actually move like two to three times faster while maintaining a like 99%

s739
01:00:55.920 --> 01:01:01.119
like accuracy to like if the humans were carefully writing this code by hand, how would it come out?

s740
01:01:01.119 --> 01:01:03.440
Now jumping a little bit back to ideas.

s741
01:01:03.440 --> 01:01:05.040
I will come back to this.

s742
01:01:05.040 --> 01:01:09.280
This was earlier maybe it was last year but you had the research plan implement.

s743
01:01:09.280 --> 01:01:14.319
Can we talk about the original research plan implement framework and then also what you've learned about this approach?

s744
01:01:14.319 --> 01:01:16.319
what what you got wrong about it.

s745
01:01:16.319 --> 01:01:16.880
Yeah, sure.

s746
01:01:16.880 --> 01:01:17.119
Yeah.

s747
01:01:17.119 --> 01:01:21.599
So, um I mean the first time we talked about RPI was in August of 2025.

s748
01:01:21.599 --> 01:01:28.319
Um and it was basically like the research was this thing of like, hey, before you go build anything, go read lots and lots of code.

s749
01:01:28.319 --> 01:01:31.599
Use a bunch of sub aent sub aents in parallel, understand all the code.

s750
01:01:31.599 --> 01:01:35.760
It was this technique that like worked really well for hard problems in complex code bases.

s751
01:01:35.760 --> 01:01:40.319
You just ask Claude uh to do a thing that that's it would read three files and make a change.

s752
01:01:40.319 --> 01:01:41.440
It would have no context.

s753
01:01:41.440 --> 01:01:42.640
So, you start the research.

s754
01:01:42.640 --> 01:01:44.400
You don't even tell it what you're working on.

s755
01:01:44.400 --> 01:01:58.559
You just tell it, "Hey, can you tell me how this system works and this system and how they connect together and then you get a markdown dock out and this is the context engineering part is like that would take a 100,000 tokens of context, but you would get a 10k token dock out of it that summarized it.

s756
01:01:58.559 --> 01:02:06.400
Then you would start a new context window and you would do planning and the planning would be and actually realize like the plans that we were building last summer were actually terrible.

s757
01:02:06.400 --> 01:02:07.760
But it would basically be this long.

s758
01:02:07.760 --> 01:02:09.760
You would say, "Okay, now here's what we're building.

s759
01:02:09.760 --> 01:02:11.119
Here's the research doc.

s760
01:02:11.119 --> 01:02:12.640
build a plan to implement it.

s761
01:02:12.640 --> 01:02:22.160
And uh in retrospect, now that we see like everyone is obsessed with how do I get agents to work for longer, I think the reason why in like May, June, July, August of 2025

s762
01:02:22.160 --> 01:02:31.760
that a lot of people became really interested in planning was it was a very powerful lever to get agents to work for longer.

s763
01:02:31.760 --> 01:02:37.440
If you said, "Build me a B2B SAS for uh burrito delivery," you'd get like a homepage and that's it.

s764
01:02:37.440 --> 01:02:40.079
But if you said, "Build me a plan," it would build out this big plan.

s765
01:02:40.079 --> 01:02:42.720
And then in the next context window, you'd say, "Hey, here's the plan.

s766
01:02:42.720 --> 01:02:44.079
Here's all the changes we're going to make.

s767
01:02:44.079 --> 01:02:46.799
Go imple it would actually keep going until the plan was done."

s768
01:02:46.799 --> 01:02:52.559
So the plan was a really good way to anchor an agent and remind it that like, hey, you're not done until this is all finished.

s769
01:02:52.559 --> 01:02:53.680
So that was the original RPI.

s770
01:02:53.680 --> 01:02:56.640
And the plan doc, what was bad about it is it didn't give you leverage.

s771
01:02:56.640 --> 01:03:02.960
The plan was every single line of code that was going to change like in diff blocks and like all the new stuff to write.

s772
01:03:02.960 --> 01:03:05.119
And so like people would review these plans.

s773
01:03:05.119 --> 01:03:05.920
We recommended this.

s774
01:03:05.920 --> 01:03:07.200
We told people to read the plans.

s775
01:03:07.200 --> 01:03:08.480
We read all our plans.

s776
01:03:08.480 --> 01:03:11.839
And then eventually I found myself like I just kind of skimmed the plans.

s777
01:03:11.839 --> 01:03:14.799
And so you're not really using it as a way to resteer the agent.

s778
01:03:14.799 --> 01:03:15.920
It's just kind of there.

s779
01:03:15.920 --> 01:03:18.319
And then you go write the code and there's a crap.

s780
01:03:18.319 --> 01:03:26.160
Some people would review the plans and the code and it's like okay well the plan was took you 20 minutes to read and then the pull request takes you 20 minutes to read and they're different.

s781
01:03:26.160 --> 01:03:30.400
And so you actually doubled the amount of time you're spending reading code instead of like doing less of it.

s782
01:03:30.400 --> 01:03:31.680
You've anti- leverage.

s783
01:03:31.680 --> 01:03:48.400
And hang on was spec different development not related to this the one that Amazon Kira for example and and GitHub workflows again a year ago did which was it also it first generated a plan and it had the human review it and then it started to and you could edit it as well and then it went off

s784
01:03:48.400 --> 01:03:51.280
and implement this part and it it looked beautifully on the surface.

s785
01:03:51.280 --> 01:03:56.640
It it should have worked great but it's tossed into the garbage outside of some m some maintenance projects.

s786
01:03:56.640 --> 01:03:58.160
I I think it just didn't work.

s787
01:03:58.160 --> 01:04:02.240
like all all the feedback I got, people just stopped using it because it just didn't really work that well.

s788
01:04:02.240 --> 01:04:06.720
It just rhymes to the RPI framework a little bit, the original one, right?

s789
01:04:06.720 --> 01:04:17.119
Well, so our thing too, like the biggest difference between RPI and specri development and some people refer to RPI as specriven dev because for some people SD all it means is

s790
01:04:17.119 --> 01:04:20.160
I use a bunch of markdown files while I'm coding and forget what's in them.

s791
01:04:20.160 --> 01:04:23.599
I just specri those are my specs and I'm using them to drive development.

s792
01:04:23.599 --> 01:04:34.480
There was this OpenAI researcher who talked about specri dev and like hey stop reading the code just write the specs and treat like the coding part as compiling specs into code.

s793
01:04:34.480 --> 01:04:36.880
that part never really materialized.

s794
01:04:36.880 --> 01:04:38.960
Maybe with GPT7, you know.

s795
01:04:38.960 --> 01:04:52.079
Um, but the challen I'm on a GitHub issue in specit uh that has been open for a year and every couple weeks I get there's a new email on the thread of people complaining about this problem of like, okay, I edit my specs and then I edit the code and then the code drifts and the specs how

s796
01:04:52.079 --> 01:04:59.280
do I keep the specs up to date as the code is changing and it's basically like you now have two sources of truth and it's it stops being useful.

s797
01:04:59.280 --> 01:05:07.920
And so like that's why when RPI the idea of the docs is they were all for a while we kept them around but after two or three months we're like oh these are actually like tactical execution docs.

s798
01:05:07.920 --> 01:05:17.680
I do the research I do the plan I do the implementation I throw the docs out and the next time I need research I just do it from scratch because tokens are cheap and my time is expensive

s799
01:05:17.680 --> 01:05:24.000
and the amount of time I might waste if I reuse a research that is no longer in sync with the real state of the codebase.

s800
01:05:24.000 --> 01:05:25.760
So we just create it live every time.

s801
01:05:25.760 --> 01:05:28.319
This is why it's like context engineering still matters.

s802
01:05:28.319 --> 01:05:38.000
Creating artifacts that compress the state of the codebase and compress the intent of the builder into small things that can be reused in the future for the scope of a task

s803
01:05:38.000 --> 01:05:49.359
is like a very powerful like tactical approach, but it's not a thing like I I have very few opinions on like what sorts of docs that you should leave lying around your codebase that are like evergreen.

s804
01:05:49.359 --> 01:05:57.839
I've seen people try to maintain parody between documentation or specs and the code itself and I don't think anyone actually like found it very useful.

s805
01:05:57.839 --> 01:06:08.559
Like you can do it and it works but it's like the ratio of the effort it takes to keep them up to date and the and trivially you could do this with AI probably but I've never known anyone who was like yeah this is great and we're glad we have it.

s806
01:06:08.559 --> 01:06:17.839
Like you could do it and it might help but I I don't think anyone found it useful enough to like maintain a system to keep the specs and the code in sync versus just using the code as the source of truth always.

s807
01:06:17.839 --> 01:06:34.319
Now you mentioned something interesting which is with context engineering you need to sometimes compact and you've previously co talked about intentional compaction that when context is noisy deliberately compress the useful part into a clear like markdown artifact verify it and then start a fresh conversation.

s808
01:06:34.319 --> 01:06:43.440
Can we talk about this kind of compaction and why it's important and and it sounds like it's going to be a building block where it already is for context engineering, right?

s809
01:06:43.440 --> 01:06:43.760
Yeah.

s810
01:06:43.760 --> 01:06:46.799
No, frequent intentional compaction is the building block.

s811
01:06:46.799 --> 01:06:53.200
It is it is completely comes from context engineering is context engineering is like how do we get the most out of today's models?

s812
01:06:53.200 --> 01:06:57.520
How do we change what we're putting into the model into the context window into the agentic chat?

s813
01:06:57.520 --> 01:07:07.280
How do we control that in such a way that we get the best results possible which means doing as much work as possible in the smart zone the you know first 100,000 tokens of the context window.

s814
01:07:07.280 --> 01:07:14.559
And uh this intentional frequent intentional compaction is basically like okay the research step we're going to go read a bunch of code and turn it into a doc.

s815
01:07:14.559 --> 01:07:15.440
That's our compaction.

s816
01:07:15.440 --> 01:07:17.200
We take that forward in the next session.

s817
01:07:17.200 --> 01:07:25.119
We're going to read we're going to read the ticket and the intent and turn that into a design document that we call is like okay here's the highle spec of what we want to do.

s818
01:07:25.119 --> 01:07:34.400
Here's a high level like current state desired end state and then a bunch of design questions the model has kind of like a very thorough maybe even overengineered like plan mode.

s819
01:07:34.400 --> 01:07:38.640
And then you take the research and the design and you do a new session, new context one.

s820
01:07:38.640 --> 01:07:39.680
You're like, cool.

s821
01:07:39.680 --> 01:07:46.319
You you've compressed the intent and you've compressed the state of the codebase so that you can then do your planning of like, okay, we know what the end state looks like.

s822
01:07:46.319 --> 01:07:48.319
We know where we're going.

s823
01:07:48.319 --> 01:07:50.799
Now, let's break down how we're going to get there.

s824
01:07:50.799 --> 01:07:56.720
All of these different steps of the process exist because models have shortcomings in each of these phases.

s825
01:07:56.720 --> 01:07:58.079
So, the research is pretty hands-off.

s826
01:07:58.079 --> 01:07:59.119
I don't read the research docs.

s827
01:07:59.119 --> 01:08:02.400
It's just like go read a bunch of code and then like make a doc out of it.

s828
01:08:02.400 --> 01:08:03.920
Models are pretty damn good at that.

s829
01:08:03.920 --> 01:08:07.039
If you ask it to find a bug and have opinions about the codebase, that's different.

s830
01:08:07.039 --> 01:08:11.920
But if you just ask it what is the intent and how do this stuff fit together, uh that's usually pretty straightforward.

s831
01:08:11.920 --> 01:08:18.239
But designing the end state of the of the software, the architecture and the program design, models are not great at.

s832
01:08:18.239 --> 01:08:21.359
They make a lot of like they make decisions and sometimes they're right and sometimes they're wrong.

s833
01:08:21.359 --> 01:08:23.279
So we have want to have a human in the loop there.

s834
01:08:23.279 --> 01:08:29.759
And then the steps to get there, I we talked about this before, but models love making what I call like horizontal plans.

s835
01:08:29.759 --> 01:08:33.040
If you ask a model like build a plan of steps to go build this app, it's like cool.

s836
01:08:33.040 --> 01:08:37.199
We're going to do the database and then we're going to do the services layer, then we're going to do the API and then we're going to do the front end.

s837
01:08:37.199 --> 01:08:43.120
It's like, well, that actually kind of sucks because we're going to be on the other side of 2,000 lines of code and let's imagine this is an existing codebase, right?

s838
01:08:43.120 --> 01:08:45.359
We're going to make changes to all these different parts of the system.

s839
01:08:45.359 --> 01:08:46.799
I can't test it till the end.

s840
01:08:46.799 --> 01:08:50.400
And so what I would do is like, okay, how would I have built this if I were building by hand?

s841
01:08:50.400 --> 01:08:53.839
Well, okay, I would probably create a mock API endpoint with fake data.

s842
01:08:53.839 --> 01:08:57.199
And then I would go kind of get the front end kind of how I want it to look.

s843
01:08:57.199 --> 01:09:01.520
And then I would actually go like build a services layer and actually wire the data through.

s844
01:09:01.520 --> 01:09:04.400
And then I would make a database migration and make my new table.

s845
01:09:04.400 --> 01:09:06.960
And then I would actually add a lot of business logic.

s846
01:09:06.960 --> 01:09:09.040
And then I would add a bunch of error handling.

s847
01:09:09.040 --> 01:09:17.040
And it's completely orthogonal to how model like models would write the database layer and all the error handling without ever like anyone's ever touched or seen the code or whatever it is.

s848
01:09:17.040 --> 01:09:21.359
And so this is another place where we like we like to have humans involved because humans have really good taste and judgment.

s849
01:09:21.359 --> 01:09:32.400
Like I would rather read five separate little mini diffs of like things that I can manually verify and explore than read 2,000 lines of code and be like well it's not working.

s850
01:09:32.400 --> 01:09:33.440
I don't know where.

s851
01:09:33.440 --> 01:09:34.880
You don't know where cuz you wrote the code.

s852
01:09:34.880 --> 01:09:36.000
You were supposed to get it right.

s853
01:09:36.000 --> 01:09:37.520
We talk about compaction context engineer.

s854
01:09:37.520 --> 01:09:42.319
It's like how can you stay in the smart zone of the context window which is again the dumb zone.

s855
01:09:42.319 --> 01:09:46.719
I will say disclaimer it's really good training wheels if you don't have intuition about this.

s856
01:09:46.719 --> 01:09:49.359
So let's just define these things.

s857
01:09:49.359 --> 01:09:51.920
What is a smart zone and what is a dumb zone?

s858
01:09:51.920 --> 01:09:56.400
So, it's it's it's a little bit blurriier than like I would like I would like it to be.

s859
01:09:56.400 --> 01:10:02.080
I think in November we we talked about the first 40% of the context window, but then we had million smart zone.

s860
01:10:02.080 --> 01:10:02.239
Yeah.

s861
01:10:02.239 --> 01:10:03.600
Then we had million token context window.

s862
01:10:03.600 --> 01:10:09.199
So then I changed it to like the first 100,000 tokens if it's a really like 4.8 I usually will go up to like 200k.

s863
01:10:09.199 --> 01:10:15.520
But basically the the thing Jeff Huntley had and Ralph Wickham was like the less context window you use the better outcomes you'll get.

s864
01:10:15.520 --> 01:10:30.000
And basically the smart smart zone mean meaning if you have context in that first part it should work a lot better and then like the dumb zone is like once you have stuff there it's kind of forget about it like it'll be confused it's not going to do much like it'll degrade.

s865
01:10:30.000 --> 01:10:30.159
Yeah.

s866
01:10:30.159 --> 01:10:35.600
And there are times and this is an intuition thing like I will often go up to 3 400k tokens.

s867
01:10:35.600 --> 01:10:44.480
Four is rare but I will go up to 250 300k tokens for certain types of work where my intuition tells me that I can keep working without without degrading the performance.

s868
01:10:44.480 --> 01:10:56.640
But if you don't have good LLM intuition, like 100K for smaller models, 200K for these like really beefy like Codeex and Opus 4.8 models is usually a good like training wheel guideline

s869
01:10:56.640 --> 01:11:01.280
of like if you pass there, your quality of results may be degrading.

s870
01:11:01.280 --> 01:11:07.679
The biggest tell I see for this is often the uh model's trying to get the test to pass and your 200k token.

s871
01:11:07.679 --> 01:11:08.320
Well, let me try this.

s872
01:11:08.320 --> 01:11:08.960
Okay, let me try that.

s873
01:11:08.960 --> 01:11:22.560
and it's like trying a bunch of stuff and it's getting more and more extreme and it's like thing oh let me delete your end file and try again like this is where things get really really weird and so it's like if you start to see certain types of if I'm like oh we're at 300k tokens and I

s874
01:11:22.560 --> 01:11:35.120
need to like fix the unit test I'm like cool write everything we did to a file or even I'll just do like a a built-in compaction depending on the model and then I'm starting a new session at 30k or 50k tokens and I'm like cool we're going to do a hard thing which is you're going to get

s875
01:11:35.120 --> 01:11:46.719
this freaking test to pass and you're not going to be stupid about it by the One thing that you said like about the the the model being dumb is you said that if the model ever tells you you are absolutely right

s876
01:11:46.719 --> 01:11:57.199
you should start over and we've all had that when it tells me like oh you know you didn't you're absolutely right and I'm like we just get annoyed but why should we start over what's happening there in your

s877
01:11:57.199 --> 01:11:58.400
um observations

s878
01:11:58.400 --> 01:12:03.704
yeah that's great yeah and the new the new you're absolutely right I think is uh you're right to push back on that right yes

s879
01:12:03.704 --> 01:12:04.239
[laughter]

s880
01:12:04.239 --> 01:12:05.199
that's opus right

s881
01:12:05.199 --> 01:12:09.040
yeah opus is like you didn't run the test did you right could push back on that.

s882
01:12:09.040 --> 01:12:09.840
I totally did it.

s883
01:12:09.840 --> 01:12:12.719
But no, for me, you're absolutely right was always what the model would respond.

s884
01:12:12.719 --> 01:12:14.640
If you were like, "That's totally wrong.

s885
01:12:14.640 --> 01:12:15.040
You did it."

s886
01:12:15.040 --> 01:12:23.120
Like you if you if you said something where you were angry or frustrated or just wanted to point out that it's done something wrong, it would respond with, "You're absolutely right."

s887
01:12:23.120 --> 01:12:27.840
And most of us have had the experience of it says that and then it continues to do the wrong thing.

s888
01:12:27.840 --> 01:12:33.760
So, it's like once it starts doing dumb things because there's there's four things in your context window that matter.

s889
01:12:33.760 --> 01:12:36.239
There's like the size of it, how many tokens?

s890
01:12:36.239 --> 01:12:39.600
There's like the quality of the information is like is there any incorrect information?

s891
01:12:39.600 --> 01:12:42.960
Like if the model had some thinking trace where it decided the wrong thing was true.

s892
01:12:42.960 --> 01:12:44.239
Is there missing information?

s893
01:12:44.239 --> 01:12:46.880
Does this like have context missing that it should have?

s894
01:12:46.880 --> 01:12:48.400
And then there's the trajectory.

s895
01:12:48.400 --> 01:12:52.080
And the trajectory is very subtle, but you may have had sessions.

s896
01:12:52.080 --> 01:12:54.320
The trajectory meaning you're prompting

s897
01:12:54.320 --> 01:12:56.000
the actual history of everything.

s898
01:12:56.000 --> 01:13:00.320
I call it trajectory is like the actual history of like what the agent has done in the past.

s899
01:13:00.320 --> 01:13:01.920
And so if I say, "Hey, make this change."

s900
01:13:01.920 --> 01:13:05.440
and the agent makes the change and then it runs the test and then they're broken and then it fixes the test.

s901
01:13:05.440 --> 01:13:17.440
I have very high confidence the next change I asked it to make, it's going to follow that path again because it's like, okay, here's a conversation and the last time the user asked me to do a thing, I made the change, I ran the test, test broken, fixed the test, and then I told the user.

s902
01:13:17.440 --> 01:13:22.320
But if I say make a change and it makes a change, it doesn't run the tests, then I'm on a different trajectory.

s903
01:13:22.320 --> 01:13:26.239
And if I say, okay, make another change, it's like basically the they're auto reggressive.

s904
01:13:26.239 --> 01:13:29.280
So they're they're predicting the ne what's the next message in this conversation.

s905
01:13:29.280 --> 01:13:40.400
And so the example we we talked about in uh No Vibes Allowed was of course the like hey the model makes a mistake and then you yelled at it and then it made another mistake and then you yelled at it and then it's like cool what's the next message in this conversation.

s906
01:13:40.400 --> 01:13:46.159
Well look if I read the history I should probably make another mistake so the human can yell at me.

s907
01:13:46.159 --> 01:13:50.080
So I was like okay that's a great that's a great example of like uh time to start over.

s908
01:13:50.080 --> 01:13:53.760
Let's talk about some observations on how software engineuring is changing.

s909
01:13:53.760 --> 01:14:01.040
One thing you talked about recently on the evolution of the coding meta is going from token harder to token smarter.

s910
01:14:01.040 --> 01:14:05.520
Can we talk about what you mean by token harder and token smarter?

s911
01:14:05.520 --> 01:14:05.760
Yeah.

s912
01:14:05.760 --> 01:14:12.480
So token harder is I mean I'm in a I'm in a group chat called hyperengineering and it's all like people trying to max out their cloud subs.

s913
01:14:12.480 --> 01:14:12.880
Oh wow.

s914
01:14:12.880 --> 01:14:13.199
Okay.

s915
01:14:13.199 --> 01:14:14.051
It's just like

s916
01:14:14.051 --> 01:14:15.040
[laughter]

s917
01:14:15.040 --> 01:14:16.560
that sounds like a fun is it fun place?

s918
01:14:16.560 --> 01:14:19.040
It's a fun place but it's like all token harder.

s919
01:14:19.040 --> 01:14:21.360
It's like look at all the side projects I built.

s920
01:14:21.360 --> 01:14:25.040
It's look at everything that uh I I I've gotten my Claude token.

s921
01:14:25.040 --> 01:14:27.040
I've got six six cloud code accounts.

s922
01:14:27.040 --> 01:14:29.840
I've gotten all of them maxed out every 5 hour period.

s923
01:14:29.840 --> 01:14:35.040
I've timed it out so I always use all the tokens and it starts up immediately when the limit resets.

s924
01:14:35.040 --> 01:14:48.239
And so it's like I mean getting into Eli Goldrat and the goal is like optimizing for utilization and efficiency of one node in your factory rather than the end to end goal of like how do we ship value and things that people like that are stable and like will last a long time.

s925
01:14:48.239 --> 01:14:56.560
But that's my idea of token harder and it's the same thing with the dark factory thing is like hey if you if you if you remove humans from code review you can push more tokens through the system.

s926
01:14:56.560 --> 01:14:59.920
So we talk about software factories but what is the dark factory?

s927
01:14:59.920 --> 01:15:06.880
Ah so the dark factory is this comes from this idea of like there are factories where uh everything is automated by robotics.

s928
01:15:06.880 --> 01:15:13.360
So you can imagine like a car factory where it's all robots building the cars and they don't have lights because there's no humans.

s929
01:15:13.360 --> 01:15:15.360
Oh, so that's where it comes from.

s930
01:15:15.360 --> 01:15:16.000
The dark factory.

s931
01:15:16.000 --> 01:15:16.159
Yeah.

s932
01:15:16.159 --> 01:15:17.280
You walk in there's no lights.

s933
01:15:17.280 --> 01:15:18.560
There's not even light switches.

s934
01:15:18.560 --> 01:15:23.920
So, it will be the fully automated software factory where it it it will be like no human input basically.

s935
01:15:23.920 --> 01:15:24.719
No human input.

s936
01:15:24.719 --> 01:15:27.199
Raw materials go in, cars come out.

s937
01:15:27.199 --> 01:15:27.600
Yep.

s938
01:15:27.600 --> 01:15:38.400
And I think in in a micro like you can have many loops that are dark in your in your thing of like, hey, if uh if the code review agent comes back with a problem, you loop that back to the builder agent, it fixes it and comes back and that's dark.

s939
01:15:38.400 --> 01:15:40.080
You don't need a human loop for that.

s940
01:15:40.080 --> 01:15:45.120
But the full dark factory where you don't read any code, yeah, it's a good way to maximize your token utilization.

s941
01:15:45.120 --> 01:15:57.280
And it's like if if your belief is like my job is to extract as much intelligence out of the machine god as I can because that's how I get the most value and the most leverage on my time then token harder.

s942
01:15:57.280 --> 01:16:03.040
Um and my take is basically what we talked about before token smarter is like okay how do I move faster?

s943
01:16:03.040 --> 01:16:12.880
How do I get as much value out of as AI as I can without having to turn the lights off while still maintaining control and taste and judgment and understanding the system architecture

s944
01:16:12.880 --> 01:16:24.640
and having a lot of like applying my hard one opinions through 10 years of software engineering to the design of the program so that I can feel confident that the code's going to get better and more maintainable over time.

s945
01:16:24.640 --> 01:16:27.679
It's the same thing of like you look at like the S sur team inside Google.

s946
01:16:27.679 --> 01:16:42.640
They brought out this book SR site reliability engineering and the whole take was like hey we're going to go from one data center to five data centers and we need the same sixperson team to be able to manage five data centers and we need the same sixperson team to be able to manage 50 data centers next year

s947
01:16:42.640 --> 01:16:57.920
and it's basically how do we apply software to this problem so that instead of scaling linearly of like okay every data center needs five devops people so we need to scale the people with the things how do we continually automate the parts that we don't need so a little bit orthogonal and maybe even like contra contradictory to

s948
01:16:57.920 --> 01:17:08.719
what I just said, but this idea of like how do you find leverage and the way the way well I I think what you were saying there is like when Google did that never seek to remove those SRE

s949
01:17:08.719 --> 01:17:10.000
from the process at all.

s950
01:17:10.000 --> 01:17:14.800
They just said like look can we think ahead and scale yourselves and they actually grew the team.

s951
01:17:14.800 --> 01:17:16.000
It wasn't actually six people.

s952
01:17:16.000 --> 01:17:20.000
It was more like I think Google specifically said, "Okay, we have five data centers.

s953
01:17:20.000 --> 01:17:21.440
Next year we'll have 50.

s954
01:17:21.440 --> 01:17:22.400
There's six of you.

s955
01:17:22.400 --> 01:17:24.000
We do not want to have 60 people.

s956
01:17:24.000 --> 01:17:26.080
We don't want and and then management layer and all that.

s957
01:17:26.080 --> 01:17:33.840
It's like how can we do it with like 12 or like or like 10 and then when we'll have 500 and now actually their SRE has grown but but

s958
01:17:33.840 --> 01:17:34.480
of course yeah

s959
01:17:34.480 --> 01:17:44.080
but but they never you know I I think as engineers like we feel pretty threatened when someone says like all right we just want to have zero engineers like I mean that's not a fun place to work at but

s960
01:17:44.080 --> 01:17:45.120
what it sounds like

s961
01:17:45.120 --> 01:17:49.280
it's not a possible place to work at if they have zero engineers neither of us can work there right

s962
01:17:49.280 --> 01:17:58.000
but do understand the token smarter is like let's keep humans in the loop let's keep adding value and figure out what are the parts which are not as relevant,

s963
01:17:58.000 --> 01:17:59.679
boring, where we don't need it.

s964
01:17:59.679 --> 01:18:07.920
And so like one developer can probably do more than before, but you are built to like be part of this whole thing and the lights are on in a factory.

s965
01:18:07.920 --> 01:18:08.159
Yeah.

s966
01:18:08.159 --> 01:18:20.000
And it's like basically I think I think what I'm trying to get to is like the connection here is like S builts a thing where like headcount scales at like a square root function or a logarithmic function whereas their output scales like linearly

s967
01:18:20.000 --> 01:18:24.560
and you want the same that the way you do that is with good architecture and good program design.

s968
01:18:24.560 --> 01:18:35.840
And so in order to like avoid this problem where you have to throw more people or more tokens at at the problem, if you design good software in such a way that it gets more maintainable and more scalable over time

s969
01:18:35.840 --> 01:18:40.960
and like just today it doesn't feel like like basically you need humans in the loop to be able to do that.

s970
01:18:40.960 --> 01:18:44.480
Let's talk about uh AI slop.

s971
01:18:44.480 --> 01:18:48.800
At one point you wrote, "Yeah, AI can write your code, but it can also write your specs and PRDs."

s972
01:18:48.800 --> 01:18:51.679
But the same the same rule is always slop in, slop out.

s973
01:18:51.679 --> 01:18:54.640
If you outsource your thinking, you're gonna get garbage.

s974
01:18:54.640 --> 01:18:55.360
Yep.

s975
01:18:55.360 --> 01:19:11.199
Um, so yeah, that's basically the idea is like the way we think about like getting high quality outputs is like yeah, you could write the code by hand or you could sit with a model and work back and forth and go maybe a little bit faster and you have control and every time it makes a change, you

s976
01:19:11.199 --> 01:19:15.679
go read the change and if it's bad, you tell it, nope, we want it like this and you kind of incrementally slowly.

s977
01:19:15.679 --> 01:19:23.679
This is like kind of the stage two or stage three version of working with agents where like the agents writing all your code, but you're kind of very much in the loop.

s978
01:19:23.679 --> 01:19:26.320
And this will make you go faster, but it won't make you go that much faster.

s979
01:19:26.320 --> 01:19:32.320
It won't make you go anywhere near there's like there's like that level and then there's like the maximum speed you can go while still caring about the code.

s980
01:19:32.320 --> 01:19:35.120
And then there's like the maximum speed you can go if you turn the lights off.

s981
01:19:35.120 --> 01:19:42.640
And so we always think about it as like in terms of leverage is like, okay, let me take everything starts with like a sentence or a voice note ramble like I want to build this thing.

s982
01:19:42.640 --> 01:19:54.719
is going to work like this or whatever it is to let's say like on average like two sentences I got to fix this thing or there's a support ticket I got to fix this thing if you can turn that with AI into a one pager and then turn that one page and make sure that's correct and then

s983
01:19:54.719 --> 01:20:10.960
turn that one pager into a three-pager and make sure that's correct and then turn that three-pager into a 10-page like detailed outline then you can write a 100 pages worth of code and it's maybe not perfect you shouldn't like sweat over these documents and make sure they're perfect but you're increasing the chance

s984
01:20:10.960 --> 01:20:14.239
that like you're decreasing the uncertainty of the outputs.

s985
01:20:14.239 --> 01:20:21.360
It's like you can think of like you have like a line of like where it's going and then you have like the probabilities of where like it might go in that range

s986
01:20:21.360 --> 01:20:27.840
if you are kind of reviewing along the way as you get more and more detailed into how what you're building and how you want it to be built.

s987
01:20:27.840 --> 01:20:33.280
You kind of collapse the uncertainty and the set of end states that you could land in.

s988
01:20:33.280 --> 01:20:45.760
That's me doing the physics thing of like you got to superimpose all these probabilities and like I don't know I have this thing that like I think people who really like playing real-time strategy games uh are probably going to be really good with AI because you kind of have to like

s989
01:20:45.760 --> 01:20:46.159
I don't know.

s990
01:20:46.159 --> 01:20:53.199
Matt Matt PCO was just talking about fog of war and like things that are at the frontier of like there's stuff we don't know about this problem yet.

s991
01:20:53.199 --> 01:20:58.080
How can we find that out and how can I make the best decision now knowing what I have seen?

s992
01:20:58.080 --> 01:21:04.080
there's a I've seen a couple pieces of information and so there's a 30% chance it's this and there's a 40% chance it's this.

s993
01:21:04.080 --> 01:21:05.280
How could I get more information?

s994
01:21:05.280 --> 01:21:11.840
So in my head I can like recalculate those probabilities and decide what's the most likely path that's going to lead us to success.

s995
01:21:11.840 --> 01:21:20.239
Speaking of the most likely path that leads you to success, let's talk about your company that's you've you've just come out of stealth human layer.

s996
01:21:20.239 --> 01:21:24.719
What is human layer and what is the probability that you're setting up for success?

s997
01:21:24.719 --> 01:21:25.840
That's a good question.

s998
01:21:25.840 --> 01:21:42.480
100% 100% probability uh maybe 110 but uh no uh so human layer is it's an AI IDE it's a collaboration platform and it is building blocks for your software factory and the basic pitch is like engineers solving hard problems and complex code bases basically

s999
01:21:42.480 --> 01:21:56.080
there's two categories of builders there's like vibe coders building side projects and then there's people building production software where the stakes are high and if something breaks we're going to get fined millions of dollars or you know we're going to lose millions of dollars of money for the company and there's a whole spectrum in between there.

s1000
01:21:56.080 --> 01:22:04.159
But it's like if you're kind of in the left half of that spectrum, you're building software that matters and it has to last and be around for a while,

s1001
01:22:04.159 --> 01:22:14.239
then you're helping people like that solve problems two to three times faster without descending into slop is like how do you maintain that near human level of quality and move two to three times faster?

s1002
01:22:14.239 --> 01:22:18.719
And what were the ideas that you you built and that you came with?

s1003
01:22:18.719 --> 01:22:20.639
one idea that we're really excited about right now.

s1004
01:22:20.639 --> 01:22:35.840
I mean, it all comes from this RPI and this like using specs to like I mean I've kind of been hinting at it this whole time, right, of like okay cool like start really high level and zoom in layer by layer and resteer and like find find that leverage that helps you move faster and increase the chance

s1005
01:22:35.840 --> 01:22:40.080
that your agent's going to build exactly what you want or something that's really high quality.

s1006
01:22:40.080 --> 01:22:45.600
The other thing I think that's really interesting that where I just posted yesterday I said, "Hey chat, should we uh kill the poll request?"

s1007
01:22:45.600 --> 01:22:54.239
And that's uh something I can't talk too much about, but basically the idea is like the IDE of the future needs to be rethought from the ground up for agents.

s1008
01:22:54.239 --> 01:23:01.280
And it might not even be a like I don't know a lot of editors kind of started with the text field and bolted on an agents tab.

s1009
01:23:01.280 --> 01:23:04.400
And then eventually you've seen like cursor 3. I can't even find the text editor.

s1010
01:23:04.400 --> 01:23:05.040
I know it exists.

s1011
01:23:05.040 --> 01:23:09.679
People have told me you can get to a text view of files, but it's also very agent first.

s1012
01:23:09.679 --> 01:23:17.040
And so we started from the ground up of like what is an IDE designed for helping a developer interact with and manage the work of agents.

s1013
01:23:17.040 --> 01:23:25.360
And then we zoomed out and said how do we make this collaborative and build in a sync engine and durable streams and all of these like pieces of tech that enable

s1014
01:23:25.360 --> 01:23:32.960
me to get human input and feedback on what I'm doing with agents in real time rather than waiting for the pull request time.

s1015
01:23:32.960 --> 01:23:42.080
And great engineering teams have been doing this for decades of like, hey, we're gonna have a design review where we're going to talk about how we're going to build the thing as like a two-page Google doc or whatever 10page what, however,

s1016
01:23:42.080 --> 01:23:50.800
BRD er architecture requirements document and then you go to sprint planning and you break it down into little tickets and you decide who's going to do what.

s1017
01:23:50.800 --> 01:23:52.560
It's like AI can help with all of this.

s1018
01:23:52.560 --> 01:23:58.239
You should, if you're just using AI to write the code, you're missing out on a lot of the benefits that AI can bring to your SDLC.

s1019
01:23:58.239 --> 01:24:01.280
And a lot of people say like, "Well, we don't need any of those meetings anymore because we have the loop.

s1020
01:24:01.280 --> 01:24:02.239
We have the dark factory.

s1021
01:24:02.239 --> 01:24:03.360
Things just fly around the loop."

s1022
01:24:03.360 --> 01:24:11.440
But it's like, "Okay, but if you want to actually move faster and maintain quality, then like you should have these checkpoints before you go to actually write the code and you should use AI to help with that."

s1023
01:24:11.440 --> 01:24:20.880
So, we built this like cloud platform that's kind of has like a Google Doc style component where you can comment and the agent can surface like mockups and mermaid diagrams and HTML and all these things.

s1024
01:24:20.880 --> 01:24:24.000
So, basically, how do we make agents like Big Figma style?

s1025
01:24:24.000 --> 01:24:25.280
Everything's in the cloud.

s1026
01:24:25.280 --> 01:24:26.080
Everything's collaborative.

s1027
01:24:26.080 --> 01:24:27.679
I see all my co-workers sessions.

s1028
01:24:27.679 --> 01:24:28.560
they see all of mine.

s1029
01:24:28.560 --> 01:24:37.360
It's almost like the benefit that Slack had over email was that you didn't have to be in every conversation to know what was happening.

s1030
01:24:37.360 --> 01:24:39.520
You could maintain you could see all these channels light up.

s1031
01:24:39.520 --> 01:24:40.480
You could check on them.

s1032
01:24:40.480 --> 01:24:41.760
Okay, I don't care about any of that.

s1033
01:24:41.760 --> 01:24:44.560
But if you saw a conversation that you cared about, you could jump in on that.

s1034
01:24:44.560 --> 01:24:53.040
And it's like how do we do that for engineering work versus like we really had these like very strict even when we called it agile it's very waterfall like

s1035
01:24:53.040 --> 01:25:01.760
PRD ard tickets everyone goes and builds for a day and then you get the PR back and then one person reviews it.

s1036
01:25:01.760 --> 01:25:17.520
How do you create this more just like soup and like what is the data model for that world where you have like agentic traces, you have documents, you have tasks and projects that group these things, you have actual git diffs being streamed everywhere where it's like why would I review all the code at once

s1037
01:25:17.520 --> 01:25:23.760
when I can just always every everybody's work lives in a shared environment that anyone can go interact with.

s1038
01:25:23.760 --> 01:25:41.199
I mean what it reminds me is like what you know GitHub that did the software teams before GitHub and its competitors you might have a tracker somewhere but most teams were just kind of like in inside the company you didn't know what one one team was I I remember pre- GitHub like you know you had individual teams

s1039
01:25:41.199 --> 01:25:55.040
they some of them had like a board with stickers but no one else in the company knew what they were doing they were all working isolation and now when you have GitHub or even the internal version of of GitHub inside a company you can always see when when you go to a team you you see the pull

s1040
01:25:55.040 --> 01:26:08.480
request flying you you can join in you have history it's all it is all kind of connected and it it came together and now it's like you know for a very long time I was like you duh you're going to use GitHub or or people will copy it so do I sense that you're trying to build something

s1041
01:26:08.480 --> 01:26:25.120
like this this workflow for like when you have the the software factories which are like dark factories and loops at a bunch of places how can we have this this new way of working which which will feel natural but like coming up with it like is is hard work and it's it's counterintuitive.

s1042
01:26:25.120 --> 01:26:37.280
How can we do something that accomplishes what GitHub did but like 10x better like more specifically like more continuous and more real time and more collaborative than like these discrete units of work that is like the poll request.

s1043
01:26:37.280 --> 01:26:42.560
Well, I now I'm starting to understand why you're saying maybe we should kill the poll request because pull request was invented by GitHub, right?

s1044
01:26:42.560 --> 01:26:53.120
like it's it is not part of Git, but they did it as a way for you to do a code review merge before it goes in and be able to modify it or or like just reject it, etc.

s1045
01:26:53.120 --> 01:27:00.719
And it's probably a lot better than whatever we had before, which I guess was like emailing your git patch to Lionus and ask him to merge it into the kernel or whatever.

s1046
01:27:00.719 --> 01:27:01.360
They still do that.

s1047
01:27:01.360 --> 01:27:02.400
It work it works for them.

s1048
01:27:02.400 --> 01:27:02.960
That's the point.

s1049
01:27:02.960 --> 01:27:03.920
But it only works for them.

s1050
01:27:03.920 --> 01:27:04.080
Yeah.

s1051
01:27:04.080 --> 01:27:05.679
I don't know anybody else who does that.

s1052
01:27:05.679 --> 01:27:08.880
I mean, I'm sure even before Get Up for you, you guys had what, like CVS or

s1053
01:27:08.880 --> 01:27:10.400
CVS?

s1054
01:27:10.400 --> 01:27:12.719
So, if you had a lot of money for Microsoft,

s1055
01:27:12.719 --> 01:27:19.120
they made us use subversion at in undergrad because the guy who invented subversion uh was a you Chicago guy.

s1056
01:27:19.120 --> 01:27:21.920
The year after I graduated, they switched everybody to Git.

s1057
01:27:21.920 --> 01:27:26.080
And I was like, damn, I learned a useless thing just for somebody's ego.

s1058
01:27:26.080 --> 01:27:30.800
specifically for AI startups or startups building on top of AI or building AI products.

s1059
01:27:30.800 --> 01:27:43.120
How important do you think location and network is especially you are based in in the the valley we see research that AI startups are more frequently funded from here than normal startups as well.

s1060
01:27:43.120 --> 01:27:49.520
Do do you see this advantage and also do you see some disadvantages of being a specific may that be Silicon Valley or elsewhere?

s1061
01:27:49.520 --> 01:27:51.520
I don't have really strong opinions on this.

s1062
01:27:51.520 --> 01:27:55.360
Actually, like Paul Graham gave a talk in Sweden about why SF is cool.

s1063
01:27:55.360 --> 01:27:58.960
Rather than just regurgitate that, I will I will forward people onto that one.

s1064
01:27:58.960 --> 01:28:08.639
Um, we can put it in the show notes or whatever, but he talks about all of the dynamics of Silicon Valley and the pay it forward culture and the like people take you way more seriously just because you're based here.

s1065
01:28:08.639 --> 01:28:10.080
I lived in Chicago for a long time.

s1066
01:28:10.080 --> 01:28:13.600
I have a lot of really good friends from high school, from college, from growing up in LA.

s1067
01:28:13.600 --> 01:28:19.120
And never before have I felt like so locked in with like my people more.

s1068
01:28:19.120 --> 01:28:22.080
Never have I felt more seen, more connected.

s1069
01:28:22.080 --> 01:28:23.600
Like there's just so many people here.

s1070
01:28:23.600 --> 01:28:30.400
Again, talking about the founder thing, people who care deeply, who are incredibly competent, who like we have all the same types of problems.

s1071
01:28:30.400 --> 01:28:31.920
We love all the same types of things.

s1072
01:28:31.920 --> 01:28:36.400
Like I don't do land parties where we play video games, but all my buddies will come over and we'll sit in the office till 11.

s1073
01:28:36.400 --> 01:28:39.920
We'll just do co-working and like hack on cool fun projects and stuff.

s1074
01:28:39.920 --> 01:28:42.000
And like you can't do that anywhere else.

s1075
01:28:42.000 --> 01:28:47.679
There's not enough like uh critical mass for that to just happen organically everywhere you go.

s1076
01:28:47.679 --> 01:28:48.960
and and I absolutely love it.

s1077
01:28:48.960 --> 01:28:50.000
I wouldn't trade it for anything.

s1078
01:28:50.000 --> 01:28:53.440
Yeah, I think a critical mass nails it on on the head.

s1079
01:28:53.440 --> 01:28:57.280
When it comes to hiring, what types of folks are you hiring for specifically?

s1080
01:28:57.280 --> 01:29:06.639
Cuz I'm interested in how hiring changes and and what what a standout engineer means for you and how you are trying to, you know, confirm that those traits exist.

s1081
01:29:06.639 --> 01:29:12.320
In general, we we are looking um for people who are have really strong software fundamentals.

s1082
01:29:12.320 --> 01:29:19.360
So, understand distributed systems, understand like the core fundamentals of CS and operating systems and these kind of things.

s1083
01:29:19.360 --> 01:29:24.480
I mean, you don't have to be a PhD in freaking kernel design or whatever, but it's a lot easier.

s1084
01:29:24.480 --> 01:29:28.560
We can we can teach we can teach somebody, I think, to be a really good AI developer in a few months.

s1085
01:29:28.560 --> 01:29:34.320
You can build enough intuition where you are, you know, accelerated off the ground and you can go like keep growing there.

s1086
01:29:34.320 --> 01:29:38.560
It's really hard to teach someone a CS undergrad program in in 3 months.

s1087
01:29:38.560 --> 01:29:49.520
And what's a problem space that you're excited about in in software engineering or even product engineering or building products that you think in the next few years is going to be one of the interesting things that you're going to be attacking?

s1088
01:29:49.520 --> 01:29:59.360
My co-founder could talk more about this, but like there's a lot of interesting things happening in in real time in cloud and sandboxes in sync and kind of like

s1089
01:29:59.360 --> 01:30:03.120
using these new building blocks that have gotten really solid in the last couple years.

s1090
01:30:03.120 --> 01:30:05.040
We're big fans of the electric SQL team.

s1091
01:30:05.040 --> 01:30:06.560
where users have durable streams.

s1092
01:30:06.560 --> 01:30:14.159
It's like how can you build systems that kind of are a lot more spread out and distributed and almost like decentralized.

s1093
01:30:14.159 --> 01:30:16.880
This is really interesting for coding because you want to be able to run coding agents anywhere.

s1094
01:30:16.880 --> 01:30:24.960
You want to be able to run them for a short time, for a long time, on demand, on a schedule, all these things and have them all be part of this kind of like

s1095
01:30:24.960 --> 01:30:25.440
brain.

s1096
01:30:25.440 --> 01:30:30.239
So I don't know, parts of what we're doing are really boring like all our data is in Postgress and then parts of what we're doing is really interesting.

s1097
01:30:30.239 --> 01:30:32.159
Um, but there's a lot of distributed systems problems.

s1098
01:30:32.159 --> 01:30:33.840
There's a lot of infrastructure problems.

s1099
01:30:33.840 --> 01:30:47.120
Like we are building tools for AI, but there's a lot of problems in building collaboration platforms that are really really hard and there's a lot of new tech that makes it easier and more interesting, but it's still uh by far from an easy problem.

s1100
01:30:47.120 --> 01:31:02.639
It sounds like what you're saying is like the infr layers to some extent a new infrar being built and it'll take some time and but it'll be like just new new blocks and it will eventually become the primitives like for cloud we have primitives already but it took freaking decade to get those together or more.

s1101
01:31:02.639 --> 01:31:02.800
Yeah.

s1102
01:31:02.800 --> 01:31:06.560
You had AWS in what like 2008 2006.

s1103
01:31:06.560 --> 01:31:06.960
Yeah.

s1104
01:31:06.960 --> 01:31:09.520
Uh and then you got Kubernetes a decade later.

s1105
01:31:09.520 --> 01:31:09.840
Yep.

s1106
01:31:09.840 --> 01:31:14.080
And as closing, what's a book or or reading that you would recommend?

s1107
01:31:14.080 --> 01:31:15.679
Something that you personally enjoyed.

s1108
01:31:15.679 --> 01:31:20.159
Nowadays, we talk a lot about refactoring by Martin Fowler classic.

s1109
01:31:20.159 --> 01:31:31.199
I think it's because we spent a lot of time uh improving the design of existing code and trying to figure out how to get models to build code that is easy to maintain and like easy to read and easy to understand and easy to to build on.

s1110
01:31:31.199 --> 01:31:35.440
I feel like I probably have a better answer than that, but that's that's what's top of mind these days.

s1111
01:31:35.440 --> 01:31:37.920
Uh we're reading a lot of like classics of software engineering.

s1112
01:31:37.920 --> 01:31:43.920
Refactoring clean code, the pragmatic programmer, like all that stuff is I think is more relevant now than it has ever been.

s1113
01:31:43.920 --> 01:31:44.480
Love it.

s1114
01:31:44.480 --> 01:31:45.840
Well, Dex, thanks so much.

s1115
01:31:45.840 --> 01:31:46.639
This was fun.

s1116
01:31:46.639 --> 01:31:47.520
This was a blast, dude.

s1117
01:31:47.520 --> 01:31:48.800
Thanks for thanks for having me on.

s1118
01:31:48.800 --> 01:31:49.360
This was great.

s1119
01:31:49.360 --> 01:31:50.880
Uh I had a lot of fun.

s1120
01:31:50.880 --> 01:31:53.520
I don't know about you, but I really enjoyed this conversation.

s1121
01:31:53.520 --> 01:31:55.920
Dex is such a big believer in gender coding.

s1122
01:31:55.920 --> 01:32:03.023
Yet, he's the one warning us that if you stop reading the code, you have about 3 to 6 months before your codebase becomes easier to rewrite than to

s1123
01:32:03.023 --> 01:32:03.040
[music]

s1124
01:32:03.040 --> 01:32:03.760
fix.

s1125
01:32:03.760 --> 01:32:05.679
And this comes from first hasn't experience.

s1126
01:32:05.679 --> 01:32:10.639
His team built a light software factory, ran it, and then had to shut it down.

s1127
01:32:10.639 --> 01:32:12.193
I also like the idea of the slow

s1128
01:32:12.193 --> 01:32:12.480
[music]

s1129
01:32:12.480 --> 01:32:13.520
loop.

s1130
01:32:13.520 --> 01:32:16.719
Loop engineering feels like a somewhat meaningless term to me.

s1131
01:32:16.719 --> 01:32:19.760
What Dex's team does is actually pretty boring.

s1132
01:32:19.760 --> 01:32:25.520
A cron job runs every night, fixes one issue or one anti-attern, and opens one small pull request.

s1133
01:32:25.520 --> 01:32:28.569
The team wakes up to a codebase that's a little bit better every morning,

s1134
01:32:28.569 --> 01:32:28.639
[music]

s1135
01:32:28.639 --> 01:32:30.719
and dev still needs to review and prove it.

s1136
01:32:30.719 --> 01:32:34.239
This is a practice that honestly any engineering team could just adopt today.

s1137
01:32:34.239 --> 01:32:36.400
Finally, I really enjoy the history lesson.

s1138
01:32:36.400 --> 01:32:39.049
The term software factory comes from a NATO conference in

s1139
01:32:39.049 --> 01:32:39.120
[music]

s1140
01:32:39.120 --> 01:32:40.400
1968.

s1141
01:32:40.400 --> 01:32:51.360
The idea of software used to build software with analogies to a factory is more than 60 years old and every generation of our industry has tried to automate more of the loop of building software.

s1142
01:32:51.360 --> 01:32:55.199
AI agents are just yet one more attempt, although probably the most successful one.

s1143
01:32:55.199 --> 01:33:01.600
Do check out show notes below for the related the pragmatic engineer deep dives that go even deeper into AI engineering and other related topics.

s1144
01:33:01.600 --> 01:33:05.249
If you enjoy this podcast, please do subscribe on your favorite podcast platform and

s1145
01:33:05.249 --> 01:33:05.280
[music]

s1146
01:33:05.280 --> 01:33:06.320
on YouTube.

s1147
01:33:06.320 --> 01:33:08.880
A special thank you if you also leave a rating on the show.

s1148
01:33:08.880 --> 01:33:12.159
Thanks and see you in the next