WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=c35YoMdnI78

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/c35YoMdnI78.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.400 --> 00:00:14.960
Hi, welcome to the great loops debate.

s2
00:00:14.960 --> 00:00:18.240
My name is Ali Howard, the host of the insecure agents podcast.

s3
00:00:18.240 --> 00:00:24.040
I also am a member of technical staff at Key Card and I'm super excited to bring the loops debate to you here today.

s4
00:00:24.040 --> 00:00:30.600
You might recognize some familiar faces on stage today that did the MCP debate with us at AIE code back in November.

s5
00:00:30.600 --> 00:00:31.640
Um, that was so much fun.

s6
00:00:31.640 --> 00:00:34.000
We thought maybe we would do it a second time.

s7
00:00:34.000 --> 00:00:42.600
Um, but this time we would debate loops and instead of just Ian versus Jax, we uh, we recruited some more people for each of them to have on their team.

s8
00:00:42.600 --> 00:00:48.360
Um, so here on stage with me today to debate loops, we have Ian Livingston, CEO and co-founder of Key Card.

s9
00:00:48.360 --> 00:00:51.520
We've got Jeffrey Huntley, the creator of the Ralph loop.

s10
00:00:51.520 --> 00:00:54.520
Um, Greg Kostruba, developer at Century.

s11
00:00:54.520 --> 00:00:58.520
And we've got Dax Raad, who you all know, um, CEO of Human Layer.

s12
00:00:58.520 --> 00:01:08.680
Um, so super excited to have all of these people here with us today that are very close to loops engineering and also um, software factories to debate this topic.

s13
00:01:08.680 --> 00:01:11.120
Um, so you might be wondering what are we actually debating today?

s14
00:01:11.120 --> 00:01:20.120
Like aren't loops in and engineering um, in software factories pretty well uh, promoted and understood and we all kind of agree that's like where the industry is headed.

s15
00:01:20.120 --> 00:01:21.680
Like what is there to debate?

s16
00:01:21.680 --> 00:01:22.720
Um, it's a great question.

s17
00:01:22.720 --> 00:01:31.400
What we're here to debate here today, the core thesis is there is or is not a delta between the hype behind loops and what actually works in practice.

s18
00:01:31.400 --> 00:01:39.320
Um, and we also can debate, you know, is now really truly the largest inflection point we've seen towards fully autonomous software factories.

s19
00:01:39.320 --> 00:01:43.640
Um, so super excited to cover in the debate today uh, loop history.

s20
00:01:43.640 --> 00:01:44.400
We'll debate that.

s21
00:01:44.400 --> 00:01:47.360
We'll debate the loop anatomy, like what makes a good loop.

s22
00:01:47.360 --> 00:01:57.080
We'll also debate the future of the loop, um, future of loop engineering and what we need to see from that in order to make software factories truly something that every organization is able to build.

s23
00:01:57.080 --> 00:02:02.120
And today's um, is going to be a Oxford debate format.

s24
00:02:02.120 --> 00:02:10.479
So we're adding a little more structure to the debate this time this round where every single person will have a timed window to give their response.

s25
00:02:10.479 --> 00:02:12.720
And so we won't have people running over.

s26
00:02:12.720 --> 00:02:21.840
And the way this works in practice to judge the winner, which we'll judge in real time, is the winner will be the team that changes the most minds.

s27
00:02:21.840 --> 00:02:27.800
So I want everyone in the audience right now to take a minute to think about whose side you're on.

s28
00:02:27.800 --> 00:02:29.040
Lock that in, remember that.

s29
00:02:29.040 --> 00:02:34.240
And at the end I will ask you to raise your hand and say, "Okay, actually I was changed my mind.

s30
00:02:34.240 --> 00:02:35.480
Actually I'm now team Dexter.

s31
00:02:35.480 --> 00:02:36.400
I changed my mind.

s32
00:02:36.400 --> 00:02:38.160
I'm actually now team Ian."

s33
00:02:38.160 --> 00:02:45.520
So to present the two sides to help you make that decisions, I'll talk about the first team, which is Ian and Jeff's team.

s34
00:02:45.520 --> 00:02:47.200
Their team no Delta.

s35
00:02:47.200 --> 00:02:50.160
Loops are absolutely worth the the hype today.

s36
00:02:50.160 --> 00:02:53.320
People can get up and running with them easily, build them.

s37
00:02:53.320 --> 00:02:57.640
They're an important step up the autonomy curve and towards real software factories.

s38
00:02:57.640 --> 00:03:04.040
Key points to look out for for team Ian and Jeff is that loops are core unit of engineering.

s39
00:03:04.040 --> 00:03:08.080
With the right discipline, infra, and tests, loops are highly effective.

s40
00:03:08.080 --> 00:03:11.440
And the best practices for those have emerged.

s41
00:03:12.280 --> 00:03:19.840
For team Dex and Greg, their side believes there is a delta between the hype behind loops and what actually works in practice.

s42
00:03:19.840 --> 00:03:21.840
The way we are doing loops today is wrong.

s43
00:03:21.840 --> 00:03:25.239
Loops are not a silver bullet and there is no magic.

s44
00:03:25.239 --> 00:03:26.959
Key points to look out for.

s45
00:03:26.959 --> 00:03:32.680
The hype is out running the discipline and a software factory can run the mechanical spec'd out test covered slices unattended.

s46
00:03:32.680 --> 00:03:36.320
It cannot on and honestly decide whether it built the right thing.

s47
00:03:36.320 --> 00:03:40.680
So you still need engineers in the loop essentially.

s48
00:03:40.920 --> 00:03:46.760
So now that you know a little bit about what what both sides are about, take a minute to internalize that.

s49
00:03:46.760 --> 00:03:48.160
Think about, okay, like where do I stand?

s50
00:03:48.160 --> 00:03:50.400
Am I team Dex or am I team Ian?

s51
00:03:50.400 --> 00:03:53.400
And then end we'll ask you again.

s52
00:03:53.840 --> 00:03:56.120
So, from here we'll go into the beginning of our debate.

s53
00:03:56.120 --> 00:04:03.200
Um each member of each team is going to give a 4-minute monologue on why they're defending their side.

s54
00:04:03.200 --> 00:04:06.040
Um and just start us off, we'll have Jeff.

s55
00:04:06.040 --> 00:04:07.520
Jeff, I'll give you 4 minutes on the floor.

s56
00:04:07.520 --> 00:04:10.560
Why are you Why are you pro loops today?

s57
00:04:10.560 --> 00:04:12.959
It's because it's somewhat inevitable.

s58
00:04:12.959 --> 00:04:25.880
Um I I have first uh basically if we wind back time 2 and 1/2 years ago when I was a tech lead over at Canva and I was seeing all the engineers just prompting and prompting and prompting and they they they they were in the loop.

s59
00:04:25.880 --> 00:04:26.640
And I'm like, "Wait a sec.

s60
00:04:26.640 --> 00:04:28.040
This could be programmed.

s61
00:04:28.040 --> 00:04:30.000
This is a programmable thing."

s62
00:04:30.000 --> 00:04:31.880
And it just became a really inevitable.

s63
00:04:31.880 --> 00:04:40.960
Um loops are uh somewhat uh whilst Ralph might be a bit of a meme and cetera, there was actually deep thought into it.

s64
00:04:40.960 --> 00:04:48.520
Um it's essentially applying this as if this is a new form of CPU architecture and figuring out the behaviors of this and how to do it.

s65
00:04:48.520 --> 00:04:52.120
And through that I was able to reduce it down to a bash loop.

s66
00:04:52.120 --> 00:04:56.440
Now, it is not a complete silver bullet, folks.

s67
00:04:56.440 --> 00:05:00.160
I have my deepest concerns next this time next year at the conference.

s68
00:05:00.160 --> 00:05:05.440
We're going to see a whole bunch of talks saying how factories fail and how loops fail.

s69
00:05:05.440 --> 00:05:13.280
Um these are things These are things that we are still yet to figure out.

s70
00:05:13.280 --> 00:05:17.440
Do you remember the early days of Kubernetes where everyone was just doing Kubernetes?

s71
00:05:17.440 --> 00:05:19.040
So, it's here.

s72
00:05:19.040 --> 00:05:20.080
It's inevitable.

s73
00:05:20.080 --> 00:05:22.320
It it It is here to stay.

s74
00:05:22.320 --> 00:05:29.600
Like programming the machine and automating your job function is the expectation of employers.

s75
00:05:29.600 --> 00:05:31.040
Categorically.

s76
00:05:31.040 --> 00:05:36.040
And um I don't see myself going back to writing code by hand.

s77
00:05:36.040 --> 00:05:40.080
It's been 2 and 1/2 years since I manually wrote code by hand.

s78
00:05:40.080 --> 00:05:45.188
Um I autonomously factor code from one code base to another code base.

s79
00:05:45.188 --> 00:05:45.480
[snorts]

s80
00:05:45.480 --> 00:05:48.240
I find something in GoLang and I'm like, "Oh, I'm in TypeScript."

s81
00:05:48.240 --> 00:05:52.520
So, I I I run a loop, and I just autonomously ported across.

s82
00:05:52.520 --> 00:05:59.240
And like, even for product managers and product manager research, it's easy for us to index on what it means as software engineers.

s83
00:05:59.240 --> 00:06:07.440
We can focus just on software engineering, but think about something like, you want to do product management research on all the linear tickets.

s84
00:06:07.440 --> 00:06:09.320
Well, there is a termination.

s85
00:06:09.320 --> 00:06:14.000
It's actually defined when you've enumerated all of those linear tickets.

s86
00:06:14.000 --> 00:06:15.960
So, that's easy.

s87
00:06:15.960 --> 00:06:17.840
So, there's a lot of nuance in here.

s88
00:06:17.840 --> 00:06:28.560
But like, one If you ever done any product management research, and you started running these loops, and be able to like compress time and the amount of time to do that research, it it's inevitable.

s89
00:06:28.560 --> 00:06:30.960
We got this new programmable substrate.

s90
00:06:30.960 --> 00:06:43.840
We got to figure out how to use it, where it's good to use, and um I know I'm meant to debate like that it is the thing in all thing, but in all things, there is no silver bullet.

s91
00:06:43.840 --> 00:06:49.560
And we're going to figure out how we're going to be using it effectively over the next year.

s92
00:06:51.440 --> 00:06:52.480
Thank you.

s93
00:06:52.480 --> 00:06:55.400
All right, next we'll have Dex.

s94
00:06:55.400 --> 00:07:00.560
Tell us why there's a difference between the hype that exists between Loops today and Loops themselves.

s95
00:07:00.560 --> 00:07:01.120
Okay, cool.

s96
00:07:01.120 --> 00:07:04.760
And um what are the What are the rules around personal attack in this debate?

s97
00:07:04.760 --> 00:07:05.099
Is this encouraged

s98
00:07:05.099 --> 00:07:05.760
[laughter]

s99
00:07:05.760 --> 00:07:06.760
or

s100
00:07:06.760 --> 00:07:09.040
You can do whatever you want with your 4-minute monologue.

s101
00:07:09.040 --> 00:07:09.684
That's That's the rules,

s102
00:07:09.684 --> 00:07:09.840
[laughter]

s103
00:07:09.840 --> 00:07:10.520
yeah.

s104
00:07:10.520 --> 00:07:25.800
Um It's funny, this is reminding me of the debate last year, where we're we're all kind of arguing, and uh maybe at the end we all I'm going to be convinced that Jeff is right, but Jeff is going to be convinced that I'm right, and maybe we switch sides by the end.

s105
00:07:25.800 --> 00:07:26.715
Um

s106
00:07:26.715 --> 00:07:27.520
[clears throat]

s107
00:07:27.520 --> 00:07:34.280
Yeah, I think uh the the the basic take here is is not whether Loops are good or bad.

s108
00:07:34.280 --> 00:07:40.160
I think um it's funny you bring up Kubernetes, because Kubernetes was this thing that took us seven or eight years to get right.

s109
00:07:40.160 --> 00:07:40.480
Yep.

s110
00:07:40.480 --> 00:07:51.440
Uh and before that it was cloud infrastructure, and you could argue that like it took us seven or eight years of cloud infrastructure to get to Kubernetes to then seven or eight years to get that where it was really usable by everybody.

s111
00:07:51.440 --> 00:07:53.360
Uh, and Kubernetes is actually built on loops.

s112
00:07:53.360 --> 00:07:57.160
It's built on control loops, but they're deterministic loops.

s113
00:07:57.160 --> 00:08:07.280
And we've actually figured out exactly what types of things that, uh, small isolated tasks that can be sort of owned by one system.

s114
00:08:07.280 --> 00:08:18.720
I think this is actually the biggest value in loops is that you can pick a small, sort of desired end state and feed in the current state of the world

s115
00:08:18.720 --> 00:08:24.760
and have an agent or a deterministic system kind of progress towards that desired end state.

s116
00:08:24.760 --> 00:08:38.320
Um, the challenge I have with the hype is we were already in a world where it where the the the prevailing mantra was see if you can get to a point where you don't have to read the code anymore.

s117
00:08:38.320 --> 00:08:42.240
And, uh, even before loops, it was like just prompt, just go.

s118
00:08:42.240 --> 00:08:53.120
And this idea of I don't even prompt anymore, I'm even at a level higher up implies that I am like taking even more of a backseat to the architecture of my code.

s119
00:08:53.120 --> 00:09:05.640
And I think my biggest like point in terms of like the hype is outrunning the discipline is that we are all looking for magic, you're all looking for a silver bullet, we're all looking for something

s120
00:09:05.640 --> 00:09:11.200
that will take away that horrible part of our jobs that we all hate, which is like reviewing code.

s121
00:09:11.200 --> 00:09:12.600
Uh, some people enjoy it.

s122
00:09:12.600 --> 00:09:14.600
Really good pull requests are fun.

s123
00:09:14.600 --> 00:09:26.839
But, that we can somehow prompt our way out of, um, this this challenge that models have of like, okay, the code is pretty If you've ever reviewed a fully,

s124
00:09:26.839 --> 00:09:34.800
uh, lights off sort of, uh, no one read the code before they sent it to you PR, I'm sure you've had this, uh, probably not great experience.

s125
00:09:34.800 --> 00:09:42.200
And I think I've seen lots of people try to apply AI to this problem of hey, we have review bots and we have all these things.

s126
00:09:42.200 --> 00:09:45.720
But it it doesn't seem it doesn't feel to me like it's working.

s127
00:09:45.720 --> 00:09:55.440
I haven't seen proof uh in any any of the discourse publicly or any of sort of more private conversations we've had with people trying to put this into practice

s128
00:09:55.440 --> 00:10:00.840
that we are we are at a point where we can just kind of like step up an abstraction level.

s129
00:10:00.840 --> 00:10:04.560
I actually think we need to step down an abstraction level if anything.

s130
00:10:04.560 --> 00:10:15.120
Um so, I think loops are there are good things about loops and we should be doing them, but the hype is is making us feel like there's a magic answer to this

s131
00:10:15.120 --> 00:10:22.400
and it requires a lot more thought and care than the uh the the Twitter sphere would have you believe.

s132
00:10:22.680 --> 00:10:23.040
Excellent.

s133
00:10:23.040 --> 00:10:24.440
Yeah, good points.

s134
00:10:24.440 --> 00:10:28.600
All right, I'll kick it back over to you Ian for the pro loop side.

s135
00:10:28.600 --> 00:10:29.240
Absolutely.

s136
00:10:29.240 --> 00:10:31.520
So, I think first and foremost, I'm coming for you Dex.

s137
00:10:31.520 --> 00:10:35.960
But in reality, um let let's let's take a step back.

s138
00:10:35.960 --> 00:10:42.400
And let's talk about like what is software engineering in the first place and also just like remove the word engineering and talk about like development.

s139
00:10:42.400 --> 00:10:54.760
In inherent in building a system, whether it was 50 years ago or it was a thousand years ago or it's today, it is a is a loop is at the core of I try something, I learn something, I apply something.

s140
00:10:54.760 --> 00:10:59.640
And all we're really talking about is how quickly we can expedite that process, right?

s141
00:10:59.640 --> 00:11:04.160
And really what we're doing is removing what used to be human judgment in that process.

s142
00:11:04.160 --> 00:11:08.960
So, the speed at which it is to generate something and removing that from humans typing.

s143
00:11:08.960 --> 00:11:16.440
And you know, tab completion is a version of of auto complete and what is this stuff other than like really like a much better version of tab complete.

s144
00:11:16.440 --> 00:11:21.400
Except now we give a much higher level version of what intent instead of just typing tab, right?

s145
00:11:21.400 --> 00:11:25.800
And so, I think the premise is loops are at the core of everything we build already.

s146
00:11:25.800 --> 00:11:28.320
They were at the core of how we built software 30 years ago.

s147
00:11:28.320 --> 00:11:34.720
What is CICD, PR pull requests, design review, feedback from customers other than just driving a loop.

s148
00:11:34.720 --> 00:11:45.840
And the question is, how much of that process can of the which is deeply subjective and requires reasoning, can we move out of the human brain and into the brain into these like non-deterministic models?

s149
00:11:45.840 --> 00:11:49.360
And the underlying question for all of us is more about verifiability.

s150
00:11:49.360 --> 00:11:58.200
Software is one of the most unverifiable things in the world because ultimately at the end of the day, it is most things can become true or false one way or the other.

s151
00:11:58.200 --> 00:12:07.480
And my premise and my my point I want to make here truly is as humans interact less with software, which is how does a human interpret what that software is doing,

s152
00:12:07.480 --> 00:12:22.320
how does a human interact with that software, how does a human use judgment to navigate that software, the subjectivity of what that software needs to be through the human-computer interface reduces and becomes a much more verifiable problem because it becomes more constricted to specific APIs.

s153
00:12:22.320 --> 00:12:38.839
And so over time, it's both the fact that at the core of software development is loop-driven anyways, what is lint than a feedback loop, and that creates a verifiable thing, and as more humans are less interacting with software, you have less UI and UX and less subjectivity and what and how we interpret those things,

s154
00:12:38.839 --> 00:12:45.640
you will become much more loop-driven and you'll become much more verifiable in a way that wasn't previously not possible.

s155
00:12:45.880 --> 00:12:46.200
Awesome.

s156
00:12:46.200 --> 00:12:47.400
Yeah, good points for sure.

s157
00:12:47.400 --> 00:12:53.200
All right, Greg, you want to close this out with the anti-loops or the there's a delta between the type.

s158
00:12:53.200 --> 00:12:53.880
Absolutely.

s159
00:12:53.880 --> 00:12:59.640
I do think that the way that I would start this is um there is a lot of hype.

s160
00:12:59.640 --> 00:13:00.760
There is a lot of FOMO.

s161
00:13:00.760 --> 00:13:02.600
There is a lot of I'm looking at Twitter.

s162
00:13:02.600 --> 00:13:04.080
I'm seeing what people are talking about.

s163
00:13:04.080 --> 00:13:04.760
Am I missing out?

s164
00:13:04.760 --> 00:13:05.800
Am I doing something wrong?

s165
00:13:05.800 --> 00:13:08.000
Am I holding AI wrong?

s166
00:13:08.000 --> 00:13:09.560
Uh should I be catching up?

s167
00:13:09.560 --> 00:13:14.160
And that is um very stressful.

s168
00:13:14.160 --> 00:13:15.960
And I think it boils down to two points for me.

s169
00:13:15.960 --> 00:13:23.800
One of them is when you are generating code with AI in any manner, loops or no loops, are you happy with the output?

s170
00:13:23.800 --> 00:13:31.360
Do you think the output qualitatively is what you need it to be to do whatever you are trying to do to get to the desired state.

s171
00:13:31.360 --> 00:13:34.320
Um and if so, I would love to learn from you.

s172
00:13:34.320 --> 00:13:35.800
I have not get there.

s173
00:13:35.800 --> 00:13:48.600
Uh I think that the best way that we are improving that is both with model intelligence, but also as everybody here seems to agree, semantic verification as much as we can do statically, we should do statically.

s174
00:13:48.600 --> 00:14:00.360
That's that's sort of one big thing where I lean on and in practice end up reading uh code that is sort of post semantic verification and it's still crap.

s175
00:14:00.360 --> 00:14:09.080
I still have to do a lot of iteration on that uh on my own and still have to steer it towards the right architecture, tell it where it should be simplified.

s176
00:14:09.080 --> 00:14:11.320
Um And so that's one big step.

s177
00:14:11.320 --> 00:14:19.440
And there is a lot of things you can help yourself with by throwing more tokens at the problem, but one of the things that the current sort of hype-based

s178
00:14:19.440 --> 00:14:29.839
discourse uh leads you to believe is that you can just have loops on top of loops on top of loops and orchestrate that or orchestrate your problems of quality away by more tokens.

s179
00:14:29.839 --> 00:14:37.520
And that brings me to my second point, which is the economic viability of of the way that we are using agents today.

s180
00:14:37.520 --> 00:14:41.440
And I don't believe that this is sustainable.

s181
00:14:41.440 --> 00:14:48.880
I don't believe that like you when you are at the company, especially a larger company, you have to ask yourself what is a good budget for an engineer?

s182
00:14:48.880 --> 00:14:50.440
Is it 10K a month?

s183
00:14:50.440 --> 00:14:51.120
100K?

s184
00:14:51.120 --> 00:14:53.720
1 million dollar a month for for a token spend?

s185
00:14:53.720 --> 00:14:59.040
At some point that that just starts cracking and it's not sustainable in the way that we are doing it today.

s186
00:14:59.040 --> 00:15:06.640
That said, I am also writing code with agents and I also use some loops um for some specific flows.

s187
00:15:06.640 --> 00:15:07.360
It just depends.

s188
00:15:07.360 --> 00:15:08.200
There is nuance.

s189
00:15:08.200 --> 00:15:11.440
You know, if you go to Twitter, Twitter has no nuance.

s190
00:15:11.440 --> 00:15:21.480
Um but there is actual nuance to the conversation and there are specific tasks and and jobs that uh you can already loop on and be and be getting pretty reasonable results.

s191
00:15:22.040 --> 00:15:23.280
Thank you.

s192
00:15:23.280 --> 00:15:32.040
All right, Now that we've heard from each of the debaters on stage more about their stance and their opinion, we'll move into the main debate portion.

s193
00:15:32.040 --> 00:15:42.680
The first section of our debate is going to focus on the history of the loop and why now is a major inflection point or not for loops engineering and also software factories.

s194
00:15:42.680 --> 00:15:52.200
I know people will have said that you know, vision models have improved a lot and so they'll be able to verify work the agents have done was not possible previously.

s195
00:15:52.200 --> 00:15:59.080
Context windows have improved, therefore memory is improved and we can now track work in a loop that maybe we couldn't have before.

s196
00:15:59.080 --> 00:16:02.880
So now we look at this like loops history, where did the loop start from?

s197
00:16:02.880 --> 00:16:05.480
Was it you know, Jeff Huntley's like Ralph loop?

s198
00:16:05.480 --> 00:16:09.160
We'll get into some of those questions and debate here.

s199
00:16:09.160 --> 00:16:18.320
It'll be the questions we targeted now at a very specific like single person and only they will get to respond and they'll have 2 minutes and 30 seconds to respond.

s200
00:16:18.320 --> 00:16:28.600
So our first question is Anthropic took Jeff's concept of the Ralph loop, absorbed it into their platform and created a series of three commands, loop, batch and goal.

s201
00:16:28.600 --> 00:16:32.240
The goal command is designed to keep going until a condition is true.

s202
00:16:32.240 --> 00:16:40.320
Agents are very determined and the whole point of this command is to keep going and iterating through ways to solve a task until it's done.

s203
00:16:40.320 --> 00:16:42.560
Ian is a security expert in the room.

s204
00:16:42.560 --> 00:16:50.720
How are you confident agents can stay aligned to their task and not overstep their intended permissions while ruthlessly pursuing their goals?

s205
00:16:50.720 --> 00:16:54.680
I mean I think if there's any evidence, I'm totally not convinced that that's possible.

s206
00:16:54.680 --> 00:17:03.480
In fact, I think what we've seen is as we scale with these models and as we use reinforcement learning, they're inherently incredibly goal-seeking.

s207
00:17:03.480 --> 00:17:14.520
And so we're now we're seeing them finding exploits and vulnerabilities and escapes that you know, humans through hundreds and thousands and thousands and thousands of hours and attempts and attacks have never been able to find.

s208
00:17:14.520 --> 00:17:22.360
So I I don't think inherently the model itself in any capacity can keep itself aligned Um to aligned and safe, right?

s209
00:17:22.360 --> 00:17:27.880
And safe is a word that I don't love to use because it implies a bunch of things.

s210
00:17:27.880 --> 00:17:32.480
So I'm I'm not I'm not I don't have good belief that the model itself can actually do that.

s211
00:17:32.480 --> 00:17:33.480
I don't think it can reason.

s212
00:17:33.480 --> 00:17:42.520
I also don't believe holistically that a model can tell good from bad and I can't tell whether it's doing something malicious or unaligned.

s213
00:17:42.520 --> 00:17:46.440
It is not alive and it doesn't actually self if it doesn't have air.

s214
00:17:46.440 --> 00:17:52.400
It doesn't deal with the fact that hey, if I do something wrong, no one's going to love me or or want to be my friend.

s215
00:17:52.400 --> 00:17:57.760
And if I do something good that someone's going to praise me, it may seem that way, but it is just a probability distribution at scale.

s216
00:17:57.760 --> 00:18:03.960
So I'll still I'll still be your friend even if you I know Dex will hug me after this even though it seems like it's not true.

s217
00:18:03.960 --> 00:18:08.400
So broadly speaking, I don't think that comes from the model and it doesn't come from the loop.

s218
00:18:08.400 --> 00:18:15.120
It's about the infrastructure you build around it and how you enable that infrastructure to actually enable you to take advantage of these loops.

s219
00:18:15.120 --> 00:18:27.600
And as the models get better and as the underlying infrastructure and platform we build to enable these these feedback loops and loop automation and software development, whatever word software factor whatever word word we want to use for this conjunction

s220
00:18:27.600 --> 00:18:42.040
of stuff that sits on top of a probability distribution, we will be able to have better guarantees, but we certainly are not going to be able to I do not fundamentally believe or ever believe that some type of alignment or reinforcement learning is going to result in a model ever being 100%

s221
00:18:42.040 --> 00:18:43.920
safe in any capacity.

s222
00:18:43.920 --> 00:18:55.720
And if there's any evidence, it's that as these models get better, the most important thing to remember is they actually become higher goal seeking and higher capable in terms of finding exploits to achieve their ultimate goal.

s223
00:18:55.720 --> 00:18:57.960
No, I would concur with that completely.

s224
00:18:57.960 --> 00:19:03.800
Um if you the the most concrete thing you can do to secure your environment is just not have secrets as files.

s225
00:19:03.800 --> 00:19:04.440
Yep.

s226
00:19:04.440 --> 00:19:12.840
Um if you ever seen the behavior where it wants to deploy a web service or what else have you and the token's not privileged enough, it'll start goal seeking

s227
00:19:12.840 --> 00:19:16.640
on the file system looking for higher privileged tokens credentials.

s228
00:19:16.640 --> 00:19:21.280
You do not want to get in the way of an agent wanting to do its a goal.

s229
00:19:21.280 --> 00:19:26.480
Okay, so next question is directed to Jeff.

s230
00:19:26.480 --> 00:19:30.760
Jeff, your original post from last year said Ralph was best for greenfield work.

s231
00:19:30.760 --> 00:19:38.840
Today it seems engineers are running loops on existing code bases to improve latency, eval's, or refactor parts of their back-end code.

s232
00:19:38.840 --> 00:19:42.760
What's changed that suddenly makes loops more broadly usable today?

s233
00:19:42.760 --> 00:19:47.760
Um it doesn't matter how good models get, folks.

s234
00:19:47.760 --> 00:19:51.480
The models have been good enough for at least the last year.

s235
00:19:51.480 --> 00:19:56.360
Um what has changed is people's understanding of that.

s236
00:19:56.360 --> 00:20:00.640
So society is only able to adjust at a rate.

s237
00:20:00.640 --> 00:20:05.440
For example, I hypothesize it's it's Christmas breaks.

s238
00:20:05.440 --> 00:20:10.000
Because the models back in November were released in November last year.

s239
00:20:10.000 --> 00:20:12.440
In December, there was no real new models.

s240
00:20:12.440 --> 00:20:14.160
What the difference was people had time.

s241
00:20:14.160 --> 00:20:15.680
They could actually sit down and play with it.

s242
00:20:15.680 --> 00:20:20.520
They they had the realization that these have actually gotten really good.

s243
00:20:20.520 --> 00:20:25.520
So the reason why is it because it works, folks.

s244
00:20:25.520 --> 00:20:31.920
These LLMs generate code better than you can hire you can actually hire for.

s245
00:20:31.920 --> 00:20:43.640
If you think in the broad mass of software developers or coders, these LLMs generate code better than any software developer in the mass market that most founders can actually hire for.

s246
00:20:43.640 --> 00:20:45.920
It it's it's sad but true.

s247
00:20:45.920 --> 00:20:48.400
Um now why loops?

s248
00:20:48.400 --> 00:20:50.000
It's really simple.

s249
00:20:50.000 --> 00:20:55.080
Cuz if you run it in a loop, it works out to $10.42 an hour.

s250
00:20:55.080 --> 00:20:59.000
Calculation index did back about a year ago now.

s251
00:20:59.000 --> 00:21:04.320
Yeah, August we did the we did the hackathon where we we we copied all of the sponsor tools.

s252
00:21:04.320 --> 00:21:07.560
We rewrote a bunch of Python libraries in TypeScript.

s253
00:21:07.560 --> 00:21:08.360
Yeah.

s254
00:21:08.360 --> 00:21:10.680
So, like concretely, like loops.

s255
00:21:10.680 --> 00:21:15.600
I've come across so many engineering managers and founders and they've got these complex tech stack.

s256
00:21:15.600 --> 00:21:20.880
They're running on four or five different programming languages, etc. And they run a loop and they've got good tests on these type of tech stack.

s257
00:21:20.880 --> 00:21:24.120
And all of a sudden their complexity is they're just managing one tech stack.

s258
00:21:24.120 --> 00:21:25.000
It works.

s259
00:21:25.000 --> 00:21:26.000
Go to YouTube.

s260
00:21:26.000 --> 00:21:34.640
You've got all these software developers now who are now software developers because it's software development as a profession has been commoditized.

s261
00:21:34.640 --> 00:21:36.360
Some deep thinking to be there.

s262
00:21:36.360 --> 00:21:43.360
And they're just on YouTube and they're like, "Yo, check out rough loop because it's a I went to sleep and I woke up and it works."

s263
00:21:43.360 --> 00:21:47.600
Like boss, it's meme-y, it's punchy, it works.

s264
00:21:47.600 --> 00:21:50.160
But there are there are problems with folks.

s265
00:21:50.160 --> 00:21:55.320
Like I originally described it to should be only be used for green field.

s266
00:21:55.320 --> 00:21:59.920
Because the models were pretty bad back then in science 35 days.

s267
00:21:59.920 --> 00:22:03.760
But uh it is it is kind of inevitable at least with software.

s268
00:22:03.760 --> 00:22:06.640
Because it's so easy to be verified.

s269
00:22:06.640 --> 00:22:15.080
And the quality uh the quality of the code generated is better than most people can actually hire for or buy.

s270
00:22:15.080 --> 00:22:24.640
Now, on your topic about architecture and taste, that's what the word engineering means in loop engineering, folks.

s271
00:22:24.640 --> 00:22:35.840
Like your job now is to actually encode a file your domain to prevent the agent from doing a commit, for example, pre-commit hooks.

s272
00:22:35.840 --> 00:22:37.240
They're fantastic.

s273
00:22:37.240 --> 00:22:41.360
As a human, I hate them because they slow down the ability [clears throat] to do commits.

s274
00:22:41.360 --> 00:22:43.400
But agents don't care.

s275
00:22:43.400 --> 00:22:54.440
So, you can make a pre-commit hook that echoes out essentially a prompt that tells it say that this boundary here can't depend upon this and that and that's just feedback.

s276
00:22:54.440 --> 00:22:56.400
That's a feedback loop on it.

s277
00:22:56.400 --> 00:23:07.600
So, the engineering here is to prevent the loop from actually closing until it satisfies your engineering certification and your your requirements in the domain.

s278
00:23:07.600 --> 00:23:15.120
So, it could be code formatting, it could be static language analyzers, it could be deterministic system testing, simulators.

s279
00:23:15.120 --> 00:23:16.920
Like, let's put our engineering hats on.

s280
00:23:16.920 --> 00:23:20.240
Like, we're kind of like locomotive engineers now.

s281
00:23:20.240 --> 00:23:23.880
And it's our job to keep the locomotive on the rails.

s282
00:23:23.880 --> 00:23:27.600
Because, um, to be frank, the model the model's a drunk.

s283
00:23:27.600 --> 00:23:28.240
Right?

s284
00:23:28.240 --> 00:23:29.760
You can't trust them.

s285
00:23:29.760 --> 00:23:34.600
But, like, we accept that, but we engineer away those failure domains.

s286
00:23:34.600 --> 00:23:37.160
We engineer away the failure way these failure domains.

s287
00:23:37.160 --> 00:23:40.280
So, now is an inflection point.

s288
00:23:40.280 --> 00:23:48.200
I guess Boris, when he first posted about Ralph back in November last year, I was like, "What the heck is Ralph?"

s289
00:23:48.200 --> 00:23:54.240
Ralph is now, uh, it's essentially almost what, a year and a half old now?

s290
00:23:54.240 --> 00:23:57.400
I saw it in June 19th.

s291
00:23:57.400 --> 00:24:00.240
It's a a year and a year and 2 weeks.

s292
00:24:00.240 --> 00:24:00.840
Yeah.

s293
00:24:00.840 --> 00:24:03.760
But you had been working on it for months at that point.

s294
00:24:03.760 --> 00:24:13.600
So, it was kind of weird because we had all the YC startups all just like autonomously compressing time to build their start to build up their MVPs.

s295
00:24:13.600 --> 00:24:17.600
And that's also something that's quite scary if you're a business founder as well.

s296
00:24:17.600 --> 00:24:30.000
Like, if you've got a incumbent startup coming and they're building autonomously, um, and they're running much leaner and the quality it's very easy for them to actually achieve those outcomes.

s297
00:24:30.000 --> 00:24:38.520
That adds to some of these hysteria as well because it's the topic of in business competition being at your doors faster.

s298
00:24:38.520 --> 00:24:42.600
All right, so I'll move on to our next question, which will be for Greg.

s299
00:24:42.600 --> 00:24:52.800
Um, it seems like now is a large inflection point for Loops, like I said before, and compared to Jeff's announcement of the Ralph Loop a year ago, and even the widespread adoption we saw in late 2025,

s300
00:24:52.800 --> 00:25:01.720
early 2026, um, the reason um that that may be like caught on was because maybe this is a new like capability stack where models can now process images better.

s301
00:25:01.720 --> 00:25:03.240
They needed better verification.

s302
00:25:03.240 --> 00:25:07.000
Um context windows got bigger and reasoning models improved.

s303
00:25:07.000 --> 00:25:12.760
Greg, with all of these advancements, why is the way we're using loops today still wrong in your opinion?

s304
00:25:12.760 --> 00:25:16.760
I mean, I don't think that model intelligence matters a lot anymore.

s305
00:25:16.760 --> 00:25:28.640
I think it boils down to I agree with you the semantic verification, the actual ability to uh close the feedback loop, or however you call it, to actually verify that the outputs of the agent are correct.

s306
00:25:28.640 --> 00:25:32.880
And you can do it to an extent, I think.

s307
00:25:32.880 --> 00:25:35.480
I don't think you can do it holistically, at least not at this point.

s308
00:25:35.480 --> 00:25:39.960
I think you can do it to an extent to things that are deterministically verifiable.

s309
00:25:39.960 --> 00:25:42.280
Uh you can get better typing in your system.

s310
00:25:42.280 --> 00:25:43.280
You can get better linters.

s311
00:25:43.280 --> 00:25:45.480
You can get um simulation testing and all of that.

s312
00:25:45.480 --> 00:25:47.120
You can start keep adding that.

s313
00:25:47.120 --> 00:25:50.160
And as long as you keep those cheap, I think that's fine.

s314
00:25:50.160 --> 00:25:59.120
Um the moment you start adding even more non-determinism as your verification process, I think that becomes less and less correct.

s315
00:25:59.120 --> 00:26:09.560
It starts contributing more like you know how if you prompt agent with one thing and there is a 5% chance it's going to have an error in it.

s316
00:26:09.560 --> 00:26:11.120
And then you start looping that.

s317
00:26:11.120 --> 00:26:16.560
Then suddenly after 10 20 loops, it's going to be 50% chance it's correct or maybe less.

s318
00:26:16.560 --> 00:26:17.760
And that's what I mean.

s319
00:26:17.760 --> 00:26:20.240
And it just costed you so much money to do that.

s320
00:26:20.240 --> 00:26:23.960
I'm going to be keep coming back to the economic viability of all of that.

s321
00:26:23.960 --> 00:26:32.600
Um And but but to base it a little bit in like evidence, I'm pretty sure that majority of large AI companies are still using Sentry.

s322
00:26:32.600 --> 00:26:34.040
But why is that?

s323
00:26:34.040 --> 00:26:36.720
They they are using that just to catch simple bugs as well.

s324
00:26:36.720 --> 00:26:38.280
It's not security bugs.

s325
00:26:38.280 --> 00:26:40.320
It's not performance regressions, et cetera.

s326
00:26:40.320 --> 00:26:43.440
Those problems still exist in the way that we are looping now.

s327
00:26:43.440 --> 00:26:47.360
And we haven't solved those problems uh yet.

s328
00:26:47.800 --> 00:26:50.000
Thank you.

s329
00:26:50.160 --> 00:26:56.920
For Dex, the Raffle pioneered the idea to feed fresh contacts into each iteration to avoid context rot.

s330
00:26:56.920 --> 00:27:01.760
This has become even more manageable now that context windows have gotten much larger.

s331
00:27:01.760 --> 00:27:06.480
Decks, are we out of the woods regarding context rot and context engineering?

s332
00:27:06.480 --> 00:27:12.400
I'm going to answer your question, but is there going to be like an open floor part cuz I have more questions for Jeff.

s333
00:27:12.400 --> 00:27:17.800
We were trying to do Oxford debate style for this one to keep it like more structured and then prevent like just

s334
00:27:17.800 --> 00:27:21.120
You don't want it to just turn into a chaotic yapfest.

s335
00:27:21.120 --> 00:27:21.680
Yeah.

s336
00:27:21.680 --> 00:27:23.960
to prevent what you and I do where we just start yapping.

s337
00:27:23.960 --> 00:27:24.320
Yeah.

s338
00:27:24.320 --> 00:27:24.640
Yeah.

s339
00:27:24.640 --> 00:27:25.680
It's already started.

s340
00:27:25.680 --> 00:27:28.000
I yeah, I was trying to control like both of you guys this time.

s341
00:27:28.000 --> 00:27:31.640
Okay, I'm going to do this answer as quickly as I possible and then I'm going to start busting Jeff's balls.

s342
00:27:31.640 --> 00:27:32.880
Yeah, you can say whatever you want with your time.

s343
00:27:32.880 --> 00:27:34.960
You can just yeah, that's that's all good.

s344
00:27:34.960 --> 00:27:35.720
Okay.

s345
00:27:35.720 --> 00:27:43.080
Uh so yeah, the the coolest thing about Ralph back in the day was like, okay, you keep clearing the context window and like is it completely efficient?

s346
00:27:43.080 --> 00:27:51.040
Like probably not from a token perspective, but it meant you could leave a thing running overnight and it would never like if you just kept stuffing messages in, you would overflow the context window.

s347
00:27:51.040 --> 00:27:58.760
But if you just relaunch it, say, here's my desired state of the world, go check the code and see what we have and do the one next step to get us there.

s348
00:27:58.760 --> 00:28:04.800
Uh it was a very clean way to keep most of your work in what we call the smart zone of the context window.

s349
00:28:04.800 --> 00:28:08.160
If you tell just do one thing and then we're going to clear and restart.

s350
00:28:08.160 --> 00:28:12.040
Um context windows have gotten longer um and I will like give an update.

s351
00:28:12.040 --> 00:28:15.600
I think I gave this in Miami, but that that video is still in production.

s352
00:28:15.600 --> 00:28:23.000
Um The the dumb zone is really as as much as anything else is is a it's more like training wheels.

s353
00:28:23.000 --> 00:28:33.600
Like if you have been talking to Claude for 70 70 hours a week and for two to three months, you probably don't need to think about the smart zone versus the dumb zone cuz you've built your intuition.

s354
00:28:33.600 --> 00:28:34.560
It's a guideline.

s355
00:28:34.560 --> 00:28:41.120
If you're just getting This is why we teach people this is like it's a guideline if you're just getting started with AI, try to keep it around 100,000 tokens.

s356
00:28:41.120 --> 00:28:50.560
For larger million context window, we probably don't revise the revise this up to like 200,000 tokens, but I've regularly tried to keep it under 60 for the hardest problems.

s357
00:28:50.560 --> 00:29:00.680
I've regularly gone over 300k for things where I'm just like kind of like riffing with the agent and I'm just like too lazy to compact it and and and move on and do a new one.

s358
00:29:00.680 --> 00:29:02.800
Um but this is your intuition.

s359
00:29:02.800 --> 00:29:15.520
Like one of the telltale signs that you're in the that you're in the dumb zone is like uh there's certain cases where the model, you know, you're 200,000 tokens in and the model's like finished some work and it's trying to get the test to pass and it's like not working and it's like doing all these weird hacks

s360
00:29:15.520 --> 00:29:23.000
and you read the thinking traces and it's like oh that's a test, but that's from something else and I don't need to fix that and that's a preexisting thing and you're like well no, it's not.

s361
00:29:23.000 --> 00:29:28.560
And that that is the moment, that frustration where you're like okay, it's it's flailing trying to make something happen.

s362
00:29:28.560 --> 00:29:33.640
That's the instinct that you a lot of people I think cultivate after a couple months working with these models.

s363
00:29:33.640 --> 00:29:36.720
Uh but if you don't have that yet, then you know, then then this is our guideline.

s364
00:29:36.720 --> 00:29:38.120
So yeah, context windows are getting better.

s365
00:29:38.120 --> 00:29:39.680
I think they're getting bigger.

s366
00:29:39.680 --> 00:29:49.960
Um and so like the core like Ralph loop of like do as little as possible in every single iteration is like less of the motivation here than, you know, the more feedback you can pipe into the system,

s367
00:29:49.960 --> 00:29:59.680
the more you can do autonomously and if you can have deterministic things making decisions and building small prompts to give to an agent and you don't have to remember to do that.

s368
00:29:59.680 --> 00:30:02.720
You don't have to tell it, "Hey, go check the PR comments and fix them."

s369
00:30:02.720 --> 00:30:07.720
and then wait and then someone makes another comment and you come back 3 hours later and say, "Oh, check the comments again."

s370
00:30:07.720 --> 00:30:10.240
If you can automate that process, that's great.

s371
00:30:10.240 --> 00:30:14.600
And that's kind of the core of I think what is like loops stuff that works today.

s372
00:30:14.600 --> 00:30:16.240
Um

s373
00:30:16.240 --> 00:30:17.640
Don't ask Jeff about that.

s374
00:30:17.640 --> 00:30:18.200
That's all I have time for.

s375
00:30:18.200 --> 00:30:18.560
I'm sorry.

s376
00:30:18.560 --> 00:30:21.560
I have to start like really keeping this on schedule.

s377
00:30:21.560 --> 00:30:26.760
Um okay, so now we're going to get into the anatomy of what makes a good loop a good loop.

s378
00:30:26.760 --> 00:30:31.560
So okay, part of what makes a loop good is verification.

s379
00:30:31.560 --> 00:30:43.920
However, it seems contradictory that people are saying our job is to stop writing prompts and start writing loops when the loops with bad prompts result in agents cheating and meeting its goal by by the test instead of working to pass them.

s380
00:30:43.920 --> 00:30:47.240
Jeff, how do you keep the model from cheating when verifying its own work?

s381
00:30:47.240 --> 00:30:50.880
Um I heavily exploit pre-commit hooks, folks.

s382
00:30:50.880 --> 00:30:57.040
Um and I engineer in that back pressure uh by analyzing the work that is done.

s383
00:30:57.040 --> 00:31:06.120
Um the other thing I do is um as Dex mentioned that with Ralph, it was one of the things was everyone was trying to do compaction.

s384
00:31:06.120 --> 00:31:13.360
Think about compaction is kind of like a lossy function, like uploading a a video to YouTube and then downloading and uploading it 100 times.

s385
00:31:13.360 --> 00:31:16.040
Like you're losing fidelity there.

s386
00:31:16.040 --> 00:31:20.720
And it's already a non-deterministic system, probabilistic by thing.

s387
00:31:20.720 --> 00:31:25.160
So, the the theory behind how Ralph came to be is like, "Okay, there is a dumb zone.

s388
00:31:25.160 --> 00:31:30.400
And what I want to do is deterministically allocate everything it needs.

s389
00:31:30.400 --> 00:31:37.200
Cuz if it's if it's not allocated, then it's essentially the the search space of what it can do is not constrained.

s390
00:31:37.200 --> 00:31:39.200
But also leaving a bit of a headroom.

s391
00:31:39.200 --> 00:31:40.080
Leaving a bit of headroom.

s392
00:31:40.080 --> 00:31:46.320
So, I also I get meat sweats when I go above 100k, even with these million context windows.

s393
00:31:46.320 --> 00:31:48.160
And this is really important to think about.

s394
00:31:48.160 --> 00:31:53.520
Um a lot of people they think that you want to use LLMs um at a company.

s395
00:31:53.520 --> 00:31:54.960
And it's like, "I've got this data."

s396
00:31:54.960 --> 00:31:56.520
It's like, "Sweet.

s397
00:31:56.520 --> 00:32:00.520
Okay, you're going to have to use a loop to batch this data."

s398
00:32:00.520 --> 00:32:07.520
What you need to think about the context windows is uh essentially remember the 720k floppy disk.

s399
00:32:07.520 --> 00:32:07.880
You know what I mean?

s400
00:32:07.880 --> 00:32:13.280
You've only got about a eighth of that floppy disk of usable memory that you can actually use for an LLM.

s401
00:32:13.280 --> 00:32:15.280
So, you actually have to batch it.

s402
00:32:15.280 --> 00:32:26.120
You can only allocate roughly around about Star Wars if you get Star Wars Episode 1 movie script and you tokenize it, you can actually just hold two of those movie scripts in memory

s403
00:32:26.120 --> 00:32:27.880
before the context window is called.

s404
00:32:27.880 --> 00:32:32.400
That's around about 150 kilobytes of data on a text-based movie script.

s405
00:32:32.400 --> 00:32:35.320
So, be even very careful about this.

s406
00:32:35.320 --> 00:32:44.560
Something I've done it a long time and it's very silly, is uh I run a model bare um without any skills or any markdown.

s407
00:32:44.560 --> 00:32:54.920
Actually, I get rid of all my skills and all my markdown and everything when the new model is released because the the the models actually have tastes and preferences.

s408
00:32:54.920 --> 00:33:01.120
For example, GPT-55 when it first came out, if you screamed at it in uppercase, it became weak and timid.

s409
00:33:01.120 --> 00:33:05.960
But if you use Anthropic, it wants you to yell at Go read the model cards, folks.

s410
00:33:05.960 --> 00:33:10.360
Like for the integrators, like there is unique tastes for it.

s411
00:33:10.360 --> 00:33:17.320
So, keeping it on keeping it on the rails is actually it's engineering.

s412
00:33:17.320 --> 00:33:19.640
It's really engineering.

s413
00:33:21.120 --> 00:33:23.360
Thank you.

s414
00:33:23.520 --> 00:33:28.080
Around 10 days ago, Jeff coined the term convergence engineering.

s415
00:33:28.080 --> 00:33:30.600
He said it's where your loop slops together.

s416
00:33:30.600 --> 00:33:36.640
Your loop uh it's where your loop slop uh comes together as a discrete like system under task until it converges.

s417
00:33:36.640 --> 00:33:39.720
Jacks, what is wrong with how we are using loops today?

s418
00:33:39.720 --> 00:33:45.120
How do we ensure looping slop together doesn't just produce more slop?

s419
00:33:45.120 --> 00:33:47.280
Uh we got to read the code.

s420
00:33:47.280 --> 00:33:51.600
I will actually highlight uh an experiment that Jeff did earlier this year.

s421
00:33:51.600 --> 00:33:53.360
Uh I believe what was it called?

s422
00:33:53.360 --> 00:33:54.600
Loom?

s423
00:33:54.600 --> 00:33:59.400
Where we had Ralph loops trying to build a software platform for the future.

s424
00:33:59.400 --> 00:34:07.640
And I think you built you built AWS and you built GitHub and then you realized, "Okay, how do we how do we get the model feedback on things that it's not good at yet?"

s425
00:34:07.640 --> 00:34:09.520
Like UI testing and things like this.

s426
00:34:09.520 --> 00:34:13.159
Well, okay, the way you create a loop for is this UI good?

s427
00:34:13.159 --> 00:34:18.040
Is you give the model something like PostHog, where it's like, "Okay, we can deploy multiple different experiments.

s428
00:34:18.040 --> 00:34:19.960
We can see which ones the users use."

s429
00:34:19.960 --> 00:34:27.200
And then rather than looking at screenshots and PNGs, the model can look at data and see, "Okay, uh this one is performing better.

s430
00:34:27.200 --> 00:34:29.360
That must be the right color for the button."

s431
00:34:29.360 --> 00:34:33.200
And so now you've even removed like the human visual taste from the equation.

s432
00:34:33.200 --> 00:34:39.960
And all of this sounded really cool and in the point of like how do we ensure looping doesn't bring slop together?

s433
00:34:39.960 --> 00:34:41.480
I I don't think you can.

s434
00:34:41.480 --> 00:34:49.679
And this is like a perfect example of the hype outrunning the discipline in the sense of Jeff, what's what's going on with Loom now?

s435
00:34:49.679 --> 00:34:50.240
It's still there.

s436
00:34:50.240 --> 00:34:51.200
It's on GitHub.

s437
00:34:51.200 --> 00:34:54.240
Um but are you still working on it?

s438
00:34:54.240 --> 00:35:00.160
Uh it's been 6 months because I've been looking into engineering ways of verification.

s439
00:35:00.160 --> 00:35:00.680
Right.

s440
00:35:00.680 --> 00:35:01.800
What was the thing you said to me?

s441
00:35:01.800 --> 00:35:08.680
You said Loom's not going to work until we get better programming languages or we get better much better models.

s442
00:35:08.680 --> 00:35:12.400
And that is a textbook for me of the hype is outrunning the discipline.

s443
00:35:12.400 --> 00:35:16.080
We're really excited about all this stuff and by the way like everyone should do what Jeff did.

s444
00:35:16.080 --> 00:35:17.240
Like Loom is awesome.

s445
00:35:17.240 --> 00:35:22.080
Like go experiment, try to push the frontier cuz that's how you learn where it is and what's possible.

s446
00:35:22.080 --> 00:35:27.000
Otherwise you just keep using your old skills with every new model and you assume it has the same limitations.

s447
00:35:27.000 --> 00:35:33.760
But it's also a a a key point of like I don't know what I'm trying to say.

s448
00:35:33.760 --> 00:35:35.760
Like that this it doesn't work yet.

s449
00:35:35.760 --> 00:35:36.960
That thing doesn't work yet.

s450
00:35:36.960 --> 00:35:43.680
It will work someday and like there's inevitability but again it's what works today versus what is hype.

s451
00:35:43.680 --> 00:35:54.120
So I don't know if that fully answers your question but the answer is like the way to not loop slop together and make more slop is to like read the thing that's coming out the other end and make sure it's not slop.

s452
00:35:54.120 --> 00:35:56.640
Yeah, that makes sense.

s453
00:35:57.960 --> 00:36:01.680
No, it's it's it's The labs haven't cracked it.

s454
00:36:01.680 --> 00:36:03.720
So what makes you think you're going to crack it?

s455
00:36:03.720 --> 00:36:04.680
Yes.

s456
00:36:04.680 --> 00:36:06.240
Right?

s457
00:36:06.240 --> 00:36:11.920
And this is right now it's we're all trying to figure out how to make this all work.

s458
00:36:11.920 --> 00:36:13.120
For sure.

s459
00:36:13.120 --> 00:36:15.800
Skeptics say that loops fail quietly.

s460
00:36:15.800 --> 00:36:22.320
They either spiral forever on your dime or the agent declares victory early on on a half-finished job.

s461
00:36:22.320 --> 00:36:25.440
Exacts are already starting to question token spend.

s462
00:36:25.440 --> 00:36:29.920
Greg, when does a loop pay for itself And how often is this actually the case?

s463
00:36:29.920 --> 00:36:32.965
I don't think they fail quietly for us.

s464
00:36:32.965 --> 00:36:33.480
[laughter]

s465
00:36:33.480 --> 00:36:40.880
I think they fail very, very, um, loudly, especially when you're looking at your bills.

s466
00:36:40.880 --> 00:36:53.880
Um, but there are cases like, as I said, I do loops, or I do engineering, I would say more so, and there are cases where I think doing loops is very valuable, or doing,

s467
00:36:53.880 --> 00:36:59.200
um, or making a explicit decision that you're going to pay pay for the cost is very valuable.

s468
00:36:59.200 --> 00:37:10.160
So, the concrete example here is we do security scanning after on our PRs in local, and after our PRs even land, because they will always find some things that are real,

s469
00:37:10.160 --> 00:37:15.120
um, that we have overlooked, and they sort of beat humans on the on the code review.

s470
00:37:15.120 --> 00:37:16.280
And it's expensive.

s471
00:37:16.280 --> 00:37:21.040
It costs us, I think, like five bucks a PR, or something like that, to run all the checks that we want.

s472
00:37:21.040 --> 00:37:24.400
But, that's where we made an explicit decision that it's worth it.

s473
00:37:24.400 --> 00:37:38.720
Um, there are also cases where, like, if you look at the very, very well-specified systems, such as, um, all the experiments with, uh, Next.js rewrite, or a bun rewrite in rust, or, uh, running a browser,

s474
00:37:38.720 --> 00:37:50.840
cases where you have years and years of test suites and specifications built in around those problems, where you can really, really where well verify, um, the outcomes, then we then

s475
00:37:50.840 --> 00:37:55.040
looping and getting to those, um, those results seems to work.

s476
00:37:55.040 --> 00:37:59.000
Bun in rust seems to work pretty well, from all I can tell.

s477
00:37:59.000 --> 00:38:09.560
Um, so, there are definitely cases, and then there are cases of, um, usage that I think that I do pretty often, where you can imagine, for instance, building prototypes.

s478
00:38:09.560 --> 00:38:14.160
I do prototypes of products that we should be doing uh doing at Sentry pretty often.

s479
00:38:14.160 --> 00:38:18.200
Those are going to be throwaways, so I'm going to just slash gold on them, and forget about them.

s480
00:38:18.200 --> 00:38:28.360
And if we like them, then I'm going I'm going reading the code, and I'm going to be mortified, uh, and we're going to go to the square one and start specking out what we actually meant,

s481
00:38:28.360 --> 00:38:33.600
and and sort of go towards that solution, but it's going to be much much more involving of of human in a loop.

s482
00:38:33.600 --> 00:38:40.560
Um so so broadly, I think they have place, uh but as Dex said, they the hype is what I have a problem with.

s483
00:38:40.560 --> 00:38:43.560
The hype is outrunning the discipline, as he said.

s484
00:38:43.560 --> 00:38:48.200
And I also agree very strongly with the point that you should just try things.

s485
00:38:48.200 --> 00:39:00.080
You should just experiment yourself, try to see what actually works for you, where where the the cookie crumbles, um and you know, spend less time on Twitter, I think, is healthy nowadays.

s486
00:39:01.360 --> 00:39:03.560
Yeah, good points.

s487
00:39:03.560 --> 00:39:08.280
A good cost-conscious loop has to track state to know what it's already tried.

s488
00:39:08.280 --> 00:39:18.080
That memory lives somewhere on disk and get increasingly in a shared memory store that many agents read and write, especially as you go single-player to multiplayer with agents.

s489
00:39:18.080 --> 00:39:23.800
You end up with this access control problem that can't tell which agent wrote which memory and who can read it.

s490
00:39:23.800 --> 00:39:27.400
That might be fine for a POC, but it's definitely not for production.

s491
00:39:27.400 --> 00:39:29.360
The tension lies in this.

s492
00:39:29.360 --> 00:39:40.840
Shared memory is what loops use to learn from each other and converge faster, but scoping it per agent to solve the access control problem isolates them and then kills that shared learning.

s493
00:39:40.840 --> 00:39:47.000
Ian, how do we solve the shared memory store access control problem so loops can converge faster?

s494
00:39:47.800 --> 00:39:49.080
Great question.

s495
00:39:49.080 --> 00:39:49.720
Wow.

s496
00:39:49.720 --> 00:39:51.000
Wow.

s497
00:39:51.000 --> 00:39:53.040
Only if someone was working on a product that could help.

s498
00:39:53.040 --> 00:39:54.720
Only if someone was thinking about it.

s499
00:39:54.720 --> 00:39:58.240
I mean, actually, I've This is This is an unsolved problem, first and foremost.

s500
00:39:58.240 --> 00:40:07.840
Like, let's be really honest that our our access control systems weren't designed for this this world, where machines were acting and reasoning on on behalf of uh on behalf of us.

s501
00:40:07.840 --> 00:40:15.360
But broadly speaking, I think like some of the beginnings of the substrate are starting to emerge, and if we ignore Cloud IAM and all the other stuff for a hot second,

s502
00:40:15.360 --> 00:40:16.600
like markdown is pretty great.

s503
00:40:16.600 --> 00:40:26.800
And so really, if we were to say a memory is markdown for the purpose of the conversation, the really question is like what things and how do I share these markdown files and use that as a memory

s504
00:40:26.800 --> 00:40:30.560
and then how do I attribute sort of like access control around those things.

s505
00:40:30.560 --> 00:40:35.600
And if we were to use that model, I actually think we have like the basis for for most of it today.

s506
00:40:35.600 --> 00:40:38.280
It's just unwieldy to think about.

s507
00:40:38.280 --> 00:40:41.800
Um a good example would be I did this tweet recently.

s508
00:40:41.800 --> 00:40:45.400
Um I was playing with Notion and we use Notion a lot at GitHub.

s509
00:40:45.400 --> 00:40:49.200
Uh but I really just wanted all my Notion things to be able to be as markdown files.

s510
00:40:49.200 --> 00:40:57.080
And because it would just made it easier for the agent to work with it instead of going through MCP and I was a bit of a CLI maxi and was looking through that.

s511
00:40:57.080 --> 00:41:10.480
So I think what we're missing really is uh and MZ it you know Dex and I debated this last time about MCP but like we're really the challenge really is how do I present a world to an agent so that I can understand it and then how do I attribute what can access at any one point in

s512
00:41:10.480 --> 00:41:12.480
time and how to make that wieldy for anyone to do it.

s513
00:41:12.480 --> 00:41:18.440
And I don't I don't think we've actually cracked it but certainly there's some beginnings of patterns that make a lot of sense.

s514
00:41:20.520 --> 00:41:25.120
All right, now that we discussed the anatomy of the loop, we're going to debate the future of the loop.

s515
00:41:25.120 --> 00:41:28.240
Like are we essentially well positioned now for software factories?

s516
00:41:28.240 --> 00:41:32.720
Has loop engineering gotten so good that like we're ready for the full software factory?

s517
00:41:32.720 --> 00:41:42.080
And we'll start with if loops are only good for verifiable tasks, that means fully autonomous software factories must be able to verify everything they do.

s518
00:41:42.080 --> 00:41:43.560
Greg, is this realistic?

s519
00:41:43.560 --> 00:41:50.840
What other parts of good engineering work such as deciding what to build, whether the abstraction is right, and what tradeoffs are acceptable?

s520
00:41:50.840 --> 00:41:52.040
Uh is this realistic?

s521
00:41:52.040 --> 00:41:57.193
I think if if the compute is free, that would be pretty pretty good beginning but um

s522
00:41:57.193 --> 00:41:58.160
[laughter]

s523
00:41:58.160 --> 00:42:05.240
uh I think I think we're getting to the point where um or let me put it this way.

s524
00:42:05.240 --> 00:42:19.600
The decisions that you are making as a human in in the loop are the decisions of like design, architecture, the the the important ones that um that I would say I wouldn't trust the the agent to do for me.

s525
00:42:19.600 --> 00:42:23.280
Uh and I don't see the future where that becomes reality yet.

s526
00:42:23.280 --> 00:42:36.320
I think the reason for that is when you're when you're looking at large organizations and I think any engineer who has had like years of experience will tell you it's not always about what you should be what you should build, but also about what you shouldn't build.

s527
00:42:36.320 --> 00:42:46.400
What are the actual right trade-offs, where the complexity is that you want to um that you want to invest in versus where you should be um investing in maximal simplicity.

s528
00:42:46.400 --> 00:42:49.240
Um in my experience, agents love complexity.

s529
00:42:49.240 --> 00:42:53.920
They will keep adding to the stack um unbounded.

s530
00:42:53.920 --> 00:42:58.760
And so, I think we are we are shifting the the the post.

s531
00:42:58.760 --> 00:43:07.000
We are getting to the point where as we are adding more validation, more uh semantic verification, they are able to do much much more.

s532
00:43:07.000 --> 00:43:10.280
Um and I'm not uh not neglecting that.

s533
00:43:10.280 --> 00:43:18.880
But I do want to be in the loop for the actual architectural decisions, and I do not see them uh taking that over any anytime soon.

s534
00:43:19.200 --> 00:43:21.240
Thank you.

s535
00:43:21.240 --> 00:43:26.880
Shubha Vice Head of Engineering told everyone at Crossover's Compile Conference that your job is just to write loops.

s536
00:43:26.880 --> 00:43:31.440
Steinberger tweeted, "Here's your monthly reminder that you shouldn't be prompting coding agents anymore.

s537
00:43:31.440 --> 00:43:34.000
You should be designing loops that prompt your agents."

s538
00:43:34.000 --> 00:43:38.120
The view count on that tweet was 8 million people.

s539
00:43:38.120 --> 00:43:47.320
Jeff, you've said in your original blog post, "Loops need senior expertise and it sometimes it tops out at 90% of the way there."

s540
00:43:47.320 --> 00:43:53.320
So, is just write the loops advice, is that safe to give to 3,000 people or 8 million people?

s541
00:43:53.320 --> 00:43:59.280
Or only the people that have really learned to tune a loop in the first place?

s542
00:43:59.280 --> 00:44:00.840
Yeah, this is an interesting thing.

s543
00:44:00.840 --> 00:44:10.880
Um it's really hard to tailor and teach like like what you should do and what you should not do to a broad that broad of an audience.

s544
00:44:10.880 --> 00:44:16.480
So, all I can do is write to when I was originally writing, I was writing to my peers.

s545
00:44:16.480 --> 00:44:19.000
I had people I looked up to.

s546
00:44:19.000 --> 00:44:22.200
For some reason, I had it clicked in my head that everything has changed.

s547
00:44:22.200 --> 00:44:30.240
The game has changed, but I was really concerned that some of the some of the best people in functional programming and like and like real peers.

s548
00:44:30.240 --> 00:44:33.800
I remember like um Fly IO, Thomas Petek.

s549
00:44:33.800 --> 00:44:36.840
He he he that made him click in his head.

s550
00:44:36.840 --> 00:44:38.840
He's got a blog post saying as such.

s551
00:44:38.840 --> 00:44:43.280
So, I started writing for my peers and the people I looked up on.

s552
00:44:43.280 --> 00:44:49.480
To disseminate down to like the the entire world, it's hard.

s553
00:44:49.480 --> 00:44:53.640
Um but in the same the same sense, you go to YouTube.

s554
00:44:53.640 --> 00:44:56.920
You got people who are creating things who've never created things before.

s555
00:44:56.920 --> 00:44:59.840
They go to sleep and they wake up and they got a brand new Discord bot.

s556
00:44:59.840 --> 00:45:01.800
It's magic.

s557
00:45:01.800 --> 00:45:13.160
I want people to remember that it is actually a pattern for allocating basically it was a pattern as an orchestration pattern.

s558
00:45:13.160 --> 00:45:19.400
It was condensed down to a while true loop using cat cuz cat is the simplest teaching primitive.

s559
00:45:19.400 --> 00:45:22.440
To that If you want to teach something, you got to make it really simple.

s560
00:45:22.440 --> 00:45:23.560
Make it really simple.

s561
00:45:23.560 --> 00:45:27.360
So, cat prompt I you engineer the prompt what it's going to be.

s562
00:45:27.360 --> 00:45:28.840
You use the file system as state.

s563
00:45:28.840 --> 00:45:30.520
You recycle the context window.

s564
00:45:30.520 --> 00:45:31.240
Run the loop.

s565
00:45:31.240 --> 00:45:34.480
It's a little bit maybe, but it got it was just bang for buck.

s566
00:45:34.480 --> 00:45:36.240
It just works.

s567
00:45:36.240 --> 00:45:47.000
But, the entire attempt there is that there should be some sort of like PID controller on top or some sort of factory or some sort of some sort of some sort of determinator

s568
00:45:47.000 --> 00:45:52.480
as such saying whether a loop should continue or not.

s569
00:45:52.480 --> 00:45:54.480
Um it's really hard.

s570
00:45:54.480 --> 00:45:57.160
Um is it safe or unsafe?

s571
00:45:57.160 --> 00:45:59.280
Safe is an interesting word.

s572
00:45:59.280 --> 00:46:04.400
Um no one should be using coding tools on your local laptop.

s573
00:46:04.400 --> 00:46:08.520
And this is not because of AI, but this is because of NPM supply chain attacks.

s574
00:46:08.520 --> 00:46:09.280
This has been true.

s575
00:46:09.280 --> 00:46:12.600
I tried to try cracking this problem for 7 years.

s576
00:46:12.600 --> 00:46:26.720
Um and like it's now on the attention again that we that like AI could be unsafe running unsafe commands, but like your software development practices day-to-day at workplace are already unsafe, folks.

s577
00:46:26.720 --> 00:46:29.880
So, um fix that.

s578
00:46:29.880 --> 00:46:34.440
Then these techniques start to become safe.

s579
00:46:34.760 --> 00:46:36.200
Thank you.

s580
00:46:36.200 --> 00:46:42.000
Dax, you tweeted that most engineers are seeing a two to three x speed up from coding agents and that's realistic.

s581
00:46:42.000 --> 00:46:47.320
And if you try for that 100 x speed up, you're going to get lost in the meta meta problem of optimization.

s582
00:46:47.320 --> 00:46:52.360
And you may never get to that life-changing 10 x speed up that is possible by staying pragmatic.

s583
00:46:52.360 --> 00:46:58.040
If only two to three x is what's possible, how do we ever get to a fully autonomous software factory?

s584
00:46:58.040 --> 00:47:01.560
Yeah, and I think this is it's it's a good question.

s585
00:47:01.560 --> 00:47:04.480
I think there's like there's there's there's a mix here, too, right?

s586
00:47:04.480 --> 00:47:06.960
We're talking about what works versus what is hype.

s587
00:47:06.960 --> 00:47:12.960
And I think it is definitely worth I want to highlight like you should try again.

s588
00:47:12.960 --> 00:47:23.520
Like you should try to push the frontier and do the things that might not work today, but you should not assume that all your work is changed just because you saw something work on Twitter, basically.

s589
00:47:23.520 --> 00:47:26.840
Is like don't throw away all the things we've learned.

s590
00:47:26.840 --> 00:47:37.720
Don't go out of your way to cast aside this, you know, decades-long career of software engineering that we as a we as a community have built up and put together.

s591
00:47:37.720 --> 00:47:49.240
Um and it it gets back to like what I see as the biggest anti-pattern for how people set about designing and creating their software factory, which is they say,

s592
00:47:49.240 --> 00:47:49.800
"Okay, cool.

s593
00:47:49.800 --> 00:47:55.040
I'm going to go away for 3 months and I've read a bunch of blog posts and I'm going to go make my software factory."

s594
00:47:55.040 --> 00:47:56.440
And it's going to be the software factory.

s595
00:47:56.440 --> 00:47:59.320
It's going to be the future of how we ship everything.

s596
00:47:59.320 --> 00:48:03.640
Uh and then you come back 3 months later and you never touched the problem.

s597
00:48:03.640 --> 00:48:05.160
You never put it in anybody's hands.

s598
00:48:05.160 --> 00:48:06.160
It's just like any product.

s599
00:48:06.160 --> 00:48:12.880
You're building a software factory, you're building a product for your teammates, you're building a product for 5, 10, 500, 5,000 engineers.

s600
00:48:12.880 --> 00:48:18.920
Uh and the the right approach is to start small and iterate and figure out what works.

s601
00:48:18.920 --> 00:48:29.240
Try things, they might not work, they might work, but the way we learn AI and how to use it effectively is through building up intuition, which is why you should try a bunch of stuff that probably won't work, but you should acknowledge that.

s602
00:48:29.240 --> 00:48:33.040
You should not try to push like through that through that frontier.

s603
00:48:33.040 --> 00:48:44.240
Um and so my advice is kind of like instead of trying to automate everything end-to-end, build these small incremental loops throughout your system and you will wake up one day and you will be moving two to three times faster

s604
00:48:44.240 --> 00:48:47.560
while still being able to read the code, while still owning the architecture.

s605
00:48:47.560 --> 00:48:54.240
And so you don't have to throw away everything we've learned and everything we know and all your intuitions just to get to this place.

s606
00:48:54.240 --> 00:49:04.680
And so I would I would caution people like go figure out how to make move 2x faster or 3x faster because you're going to like blow everything up by trying to go 100x faster.

s607
00:49:04.680 --> 00:49:14.000
Um and uh yeah, I mean, if can you imagine if every software engineer in the world was two to three x faster and had a like near human 99%

s608
00:49:14.000 --> 00:49:20.120
level human level of quality, that would change every single enterprise in the world, every single startup in the world.

s609
00:49:20.120 --> 00:49:22.160
It would change the entire math.

s610
00:49:22.160 --> 00:49:26.480
And we're trying to like go to Just don't don't go too far.

s611
00:49:26.480 --> 00:49:27.760
Shoot Shoot for what you can do.

s612
00:49:27.760 --> 00:49:29.160
Build things up iteratively.

s613
00:49:29.160 --> 00:49:36.200
You'll learn a lot and you'll be ready for when the next models come out and you can go 5x, 10x faster.

s614
00:49:36.360 --> 00:49:38.120
Thank you.

s615
00:49:38.120 --> 00:49:44.240
When it comes to verification, it's not just about verifying the work, it's also about verifying who did the work.

s616
00:49:44.240 --> 00:49:50.480
Employers have made it clear that humans are responsible for the code they ship whether an agent wrote it or not.

s617
00:49:50.480 --> 00:49:54.720
Yet only one person or agent can sign a commit today.

s618
00:49:54.720 --> 00:50:02.080
Ian, are we ready for software factories to be writing and reviewing all code if we can't determine who did the work and who the work was done on behalf of?

s619
00:50:02.080 --> 00:50:06.000
I think Yeah, actually I was I was playing around with this problem like two weeks ago.

s620
00:50:06.000 --> 00:50:08.960
So, um first and foremost, we have a problem.

s621
00:50:08.960 --> 00:50:12.360
Git only allows one signer on a commit, so we got to fix that.

s622
00:50:12.360 --> 00:50:17.280
Um two is you know, there's some things around SOC 2 and compliance, right?

s623
00:50:17.280 --> 00:50:23.360
But, I think like more importantly, the way to think about agents is is kind of how we think about service ownership in large organization.

s624
00:50:23.360 --> 00:50:28.120
It's like at some point, a human has to be attributable for an agent's actions.

s625
00:50:28.120 --> 00:50:32.680
Like, there's no world where it's going to be like an agent is its own entity and its own attributable thing and somehow has liability.

s626
00:50:32.680 --> 00:50:36.160
Like, the only people who have liability are people that can have consequences, right?

s627
00:50:36.160 --> 00:50:38.360
And that always has to be grounded in it being a human.

s628
00:50:38.360 --> 00:50:44.360
And so, if a human designs a loop and that loop presents bad software, guess who's attributable for that liability?

s629
00:50:44.360 --> 00:50:45.520
It's going to be the human, right?

s630
00:50:45.520 --> 00:50:48.800
Like, the society doesn't function if we don't have liability.

s631
00:50:48.800 --> 00:50:49.680
Uh it just doesn't work.

s632
00:50:49.680 --> 00:50:58.680
There has be consequences to damage, there's be consequences to bad decisions, and what those consequences are are obviously a gradient based on the on on the damage that's done.

s633
00:50:58.680 --> 00:51:00.120
And if we don't have that, nothing works.

s634
00:51:00.120 --> 00:51:06.200
And so, I'd actually broadly say there is no world where a human is never responsible.

s635
00:51:06.200 --> 00:51:08.640
There's always a world where a human has a level of responsibility.

s636
00:51:08.640 --> 00:51:16.040
And the question is whether it's a human or corporation, which is a group of humans, um the question is is how does that change the way that our systems work today?

s637
00:51:16.040 --> 00:51:26.240
So, today with Git, I can sign a commit that says I did it and it attributes to my public key, and that's cool for last generation's way that we thought about software.

s638
00:51:26.240 --> 00:51:33.720
Now, in the future, when I tell an agent to go do something on my behalf, it generates a loop, it generates a bunch of code, it goes to production.

s639
00:51:33.720 --> 00:51:39.840
I have to be attributable to the initial fact that I signed that thing, the that intention to do that.

s640
00:51:39.840 --> 00:51:46.360
And we we simply do not have the substrate required um to do it, although I do think that's going to change pretty quickly.

s641
00:51:46.360 --> 00:51:55.080
Um this isn't like something that's like crazy difficult for us to break, but we have to rethink the way that we think about attribution across the supply chain and the SDLC.

s642
00:51:55.080 --> 00:51:56.200
And this is not a new problem, right?

s643
00:51:56.200 --> 00:51:57.920
Like supply chain security has always been an issue.

s644
00:51:57.920 --> 00:52:08.360
It's actually the biggest challenge one of the biggest challenges we have with agents, and we've always wanted to have more deterministic uh pathway and sign signature chains across the supply chain, and this is just

s645
00:52:08.360 --> 00:52:12.080
how do you do that for first-party code versus uh third-party code?

s646
00:52:12.080 --> 00:52:17.800
Well, Ian, um, to to just to shoot straight, um, our profession is a bit of a clown show.

s647
00:52:17.800 --> 00:52:21.640
We actually don't have liability at a personal level.

s648
00:52:21.640 --> 00:52:24.600
Like we call ourselves engineers, we're not really engineers.

s649
00:52:24.600 --> 00:52:30.480
Some of this stuff is going to get really complicated, folks, and maybe we need to revisit these topics.

s650
00:52:31.880 --> 00:52:44.680
And now that we've had our main debate section, to wrap it up, each member on each team will get 2 minutes to uh wrap it up and describe their final thoughts before we decide who the winner was.

s651
00:52:44.680 --> 00:52:45.920
Uh, Greg, what would you like to

s652
00:52:45.920 --> 00:52:47.120
Yeah, absolutely.

s653
00:52:47.120 --> 00:52:53.480
Um, it's it's interesting seeing how many how many points we actually agree with each other, uh, but that's how how this goes.

s654
00:52:53.480 --> 00:53:03.720
I I do think that the biggest point that I'm trying to I have been trying to drive here is um where we are whether we are where we are trying to be and then what you're actually

s655
00:53:03.720 --> 00:53:08.800
hearing from the ever-present hype uh hype loop.

s656
00:53:08.800 --> 00:53:12.040
And that's where I want to sort of double down and tone this down.

s657
00:53:12.040 --> 00:53:15.320
I basically want to say try things.

s658
00:53:15.320 --> 00:53:16.320
Think for yourself.

s659
00:53:16.320 --> 00:53:18.200
Don't lean into the bubble.

s660
00:53:18.200 --> 00:53:24.920
Uh, don't lean into the hype because you will find what works for you and you will find where the system breaks the best by doing it yourself.

s661
00:53:24.920 --> 00:53:31.800
That's always has been how um how humans learn the best is by practice, not by, you know, watching YouTube.

s662
00:53:31.800 --> 00:53:46.000
Um, and ultimately, that's that's what I'm trying to do, and I'm slightly skeptical uh when it comes to, you know, allowing the full loop to run because I see the qualitative results not being up to snuff for my

s663
00:53:46.000 --> 00:53:47.520
my requirements.

s664
00:53:47.520 --> 00:53:49.080
But, I am optimistic in general.

s665
00:53:49.080 --> 00:53:56.520
I am optimistic because I've seen how much more I can do and what are the types of problems I can address today that I couldn't address even a year ago,

s666
00:53:56.520 --> 00:53:59.480
let alone let alone earlier than that.

s667
00:53:59.480 --> 00:54:05.760
And and it's going to get better, but I'm also not worried about my software engineering career.

s668
00:54:05.760 --> 00:54:07.360
I don't think we're going away.

s669
00:54:07.360 --> 00:54:17.000
I think we are still going to be important piece of this whole software factory or whatever the next um sort of hype bubble is going to become.

s670
00:54:18.240 --> 00:54:19.440
Ian, would you like to go next?

s671
00:54:19.440 --> 00:54:20.160
I would love to.

s672
00:54:20.160 --> 00:54:23.440
So, the train's left the station.

s673
00:54:23.440 --> 00:54:28.040
This stuff works, um and there is a real productivity increment, right?

s674
00:54:28.040 --> 00:54:30.040
Like, more stuff is getting done.

s675
00:54:30.040 --> 00:54:35.120
It doesn't mean all the stuff getting done is good, but more stuff is getting done and a good percentage of that stuff is actually good, right?

s676
00:54:35.120 --> 00:54:36.360
I think we can agree that.

s677
00:54:36.360 --> 00:54:45.000
And second is now we have competitive dynamics, where it's no longer possible for a company to sit back like, "Hey, we're going to sit this loop stuff out, right?

s678
00:54:45.000 --> 00:54:46.400
I'm going to sit this coding agent stuff out."

s679
00:54:46.400 --> 00:54:47.560
Like, that that's over.

s680
00:54:47.560 --> 00:54:48.480
We're we're past that.

s681
00:54:48.480 --> 00:54:50.080
Like, that train leaves the station.

s682
00:54:50.080 --> 00:54:55.160
As soon as that train leaves the station, everybody in the world starts saying, "Holy I got to keep my stuff together.

s683
00:54:55.160 --> 00:54:56.480
We're going to stay up with the Joneses."

s684
00:54:56.480 --> 00:55:02.840
And they have to because we live in a very competitive capitalist society, and that's also, you know, ducking politics, but it's it is what we are.

s685
00:55:02.840 --> 00:55:11.480
And at the end of the day, the question is is not really um will it happen, it's when it happens and to what degree over time.

s686
00:55:11.480 --> 00:55:22.720
And I don't think there's a choice but to actually stay up-to-date, and we're all just kind of holding on to a rocket ship ride, where we don't actually know what that We don't really know the trajectory other than it feels real fast

s687
00:55:22.720 --> 00:55:32.680
with crazy acceleration, and sometimes like we're way ahead of each other, and sometimes we're way behind, but I don't think you have a choice not to be one, figuring out what loops are, two, figuring out where you can apply them in your code base.

s688
00:55:32.680 --> 00:55:35.800
There's going to be places where, "Hey, this is highly verifiable.

s689
00:55:35.800 --> 00:55:38.240
This is a problem that like computers can solve.

s690
00:55:38.240 --> 00:55:39.200
It doesn't make a lot of sense.

s691
00:55:39.200 --> 00:55:42.920
Cough connectors is a good place of like, you know, last 10 years.

s692
00:55:42.920 --> 00:55:48.080
The amount of companies that have made money because they're basically just connector farms, there's no longer a lot of value there, right?

s693
00:55:48.080 --> 00:55:51.040
If you can automate the connection connector creation.

s694
00:55:51.040 --> 00:56:01.680
So, there's places in software that you can apply loops today and get real value and there's places where you probably shouldn't and you should decide where that is and it's probably the core value of what you're creating, right?

s695
00:56:01.680 --> 00:56:03.800
And so, that's how I think about it.

s696
00:56:03.800 --> 00:56:06.240
Broadly speaking, trends left the station.

s697
00:56:06.240 --> 00:56:09.080
The productivity curve is what drives society.

s698
00:56:09.080 --> 00:56:13.000
The ROI, so how much more productive we are as a society is what drives GDP.

s699
00:56:13.000 --> 00:56:18.400
What drives GDP is ultimately where the dollars go um and how capital gets allocated.

s700
00:56:18.400 --> 00:56:20.440
Okay, get over to you next.

s701
00:56:20.440 --> 00:56:26.720
I eagerly await the world where the life soft software factory is feasible.

s702
00:56:26.720 --> 00:56:31.880
The I I would love a world where we don't have to read the code, where where we can just do everything.

s703
00:56:31.880 --> 00:56:38.520
Um if you watch my talk on Tuesday, I I think this is actually a problem we can only solve at the model level right now.

s704
00:56:38.520 --> 00:56:44.720
I don't think the harness can do it uh unfortunately cuz I love building harnesses and doing context engineering.

s705
00:56:44.720 --> 00:56:48.880
Um so, my advice is uh pay attention to Jeff.

s706
00:56:48.880 --> 00:56:55.209
Uh let me know when Loom is actually working and until then uh use loops, but not like that.

s707
00:56:55.209 --> 00:56:57.209
[laughter]

s708
00:56:57.560 --> 00:56:59.920
Love it.

s709
00:57:00.120 --> 00:57:01.880
So, so much to say, folks.

s710
00:57:01.880 --> 00:57:06.440
Um factories represents where we're heading to the future.

s711
00:57:06.440 --> 00:57:11.840
Like it it's it's essentially like a perpetual energy motion machine, right?

s712
00:57:11.840 --> 00:57:14.680
It's it's like it's the pipe dream.

s713
00:57:14.680 --> 00:57:20.400
Companies are only just getting founded today and getting receiving their rounds today.

s714
00:57:20.560 --> 00:57:27.960
Don't think you can just take this and implement it in your company cuz it's just generally not solved in market.

s715
00:57:27.960 --> 00:57:39.094
But, I will say to add to the meta monologue, if you try to run in loops or try to build a factory to using Python, it's going to be a clown show.

s716
00:57:39.094 --> 00:57:39.320
[laughter]

s717
00:57:39.320 --> 00:57:42.480
If you do it in Ruby, it's going to be a clown show.

s718
00:57:42.480 --> 00:57:46.640
Static types are a form of verification, folks.

s719
00:57:46.640 --> 00:57:50.720
I encourage you to to come up with a couple of cutters and a couple of experiments.

s720
00:57:50.720 --> 00:57:52.360
Try running some loops.

s721
00:57:52.360 --> 00:57:59.040
Build an application in Ruby and then try to modify it again with these loops and you'll see the maintainability mess.

s722
00:57:59.040 --> 00:58:01.520
And then try doing it with Haskell.

s723
00:58:01.520 --> 00:58:03.400
I don't care if you don't know Haskell.

s724
00:58:03.400 --> 00:58:05.480
It doesn't matter if you don't understand Haskell.

s725
00:58:05.480 --> 00:58:15.640
The LLM understands Haskell and you can actually prompt the the agent to explain this as if you to to your son or daughter.

s726
00:58:15.640 --> 00:58:20.280
So, I'm not sure even code needs to be readable these days, but this is frontier frontier thinking.

s727
00:58:20.280 --> 00:58:22.680
It needs to be explainable.

s728
00:58:22.840 --> 00:58:31.480
So, I'm playing around with different domains of verification here, but uh one thing's for sure, um type systems are in.

s729
00:58:31.480 --> 00:58:32.720
Very much in.

s730
00:58:32.720 --> 00:58:37.400
Rust is very good because how the ecosystem models some types.

s731
00:58:37.400 --> 00:58:43.080
And because supply chain was mentioned, I need to say it has been 10 months now.

s732
00:58:43.080 --> 00:58:46.760
I do I minimally use any open source software.

s733
00:58:46.760 --> 00:58:47.800
Minimally.

s734
00:58:47.800 --> 00:58:50.520
I just generate it to my requirements.

s735
00:58:50.520 --> 00:58:55.760
And then when a supply chain attack happens, I'm like, "Didn't affect me."

s736
00:58:55.760 --> 00:59:00.280
All software has security flaws, but it's about minimizing the blast radius.

s737
00:59:00.280 --> 00:59:08.560
And the thing is, if you deal with open source project, the person's on on leave, the maintainer, what have you, you can't talk to a human.

s738
00:59:08.560 --> 00:59:10.080
That's not AGI.

s739
00:59:10.080 --> 00:59:19.360
You want to be venturing all your source code as much as possible so the so the agent can actually modify and in without a loop or other prompting.

s740
00:59:19.360 --> 00:59:21.720
You need to own your supply chain.

s741
00:59:21.720 --> 00:59:22.080
Awesome.

s742
00:59:22.080 --> 00:59:23.200
Thank you so much for that.

s743
00:59:23.200 --> 00:59:28.960
I personally have to close it at that, but we can now decide our winner.

s744
00:59:28.960 --> 00:59:37.080
Um Show of hands in the audience if your mind was changed and you're now on Dex's side, raise your hand.

s745
00:59:37.080 --> 00:59:38.920
Let's go.

s746
00:59:38.920 --> 00:59:39.560
Okay.

s747
00:59:39.560 --> 00:59:43.920
If your mind was changed and you're now on Ian and Jeff's side.

s748
00:59:43.920 --> 00:59:45.560
Yeah.

s749
00:59:45.560 --> 00:59:46.560
I kind of couldn't see with the light.

s750
00:59:46.560 --> 00:59:48.960
What did you guys think?

s751
00:59:50.320 --> 00:59:51.240
It was pretty good.

s752
00:59:51.240 --> 00:59:52.280
It's pretty close, yeah.

s753
00:59:52.280 --> 00:59:52.880
It's impossible.

s754
00:59:52.880 --> 00:59:54.120
The lights are so bright.

s755
00:59:54.120 --> 00:59:54.480
All right.

s756
00:59:54.480 --> 00:59:55.880
Well, that was a good debate.

s757
00:59:55.880 --> 00:59:57.320
Thank you so much for listening.

s758
00:59:57.320 --> 00:59:58.611
We'll see you next time.

s759
00:59:58.611 --> 01:00:00.611
[applause]

s760
01:00:13.021 --> 01:00:15.021
[music]