WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=q4WC4TowkBo

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/q4WC4TowkBo.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.720 --> 00:00:14.120
Hey, welcome.

s3
00:00:14.120 --> 00:00:16.600
Welcome to Beyond Done.

s4
00:00:16.600 --> 00:00:18.480
Uh we're going to get started here with this workshop.

s5
00:00:18.480 --> 00:00:26.800
We got a lot to pack in and uh we want to make sure that we have time for questions and um uh getting getting like everyone situated.

s6
00:00:26.800 --> 00:00:29.475
So, we're going to go ahead and get started.

s7
00:00:29.475 --> 00:00:30.080
[clears throat and cough]

s8
00:00:30.080 --> 00:00:39.480
So, what we mean by Beyond Done, just to like jump straight into it, um you probably saw a tweet about this incident that happened a while back uh where

s9
00:00:39.480 --> 00:00:43.280
an agent just started deleting production databases.

s10
00:00:43.280 --> 00:00:46.400
And in 9 seconds, everything was gone.

s11
00:00:46.400 --> 00:00:56.320
Um that is what we want to help you prevent here today uh while still trusting your agents and not having to look at every line, even look at every PR.

s12
00:00:56.320 --> 00:00:59.440
Like this is where we think things are going and how do we get there?

s13
00:00:59.440 --> 00:01:12.640
How do we make this a reality where instead of having that broken piece, we can have you calm, have uh everything work, and let you to some degree trust your agents, but build that trust over time.

s14
00:01:12.640 --> 00:01:15.640
I'd say also that it's not just about preventing errors, right?

s15
00:01:15.640 --> 00:01:21.520
But this is also what we're going to look at today foundational to achieving true scale with it with agents as well.

s16
00:01:21.520 --> 00:01:22.720
So, I'm Nick Nisi.

s17
00:01:22.720 --> 00:01:24.800
Uh I'm an engineering manager at WorkOS.

s18
00:01:24.800 --> 00:01:28.320
Uh I live in Omaha, Nebraska and uh I've been there for about 2 years.

s19
00:01:28.320 --> 00:01:30.520
And I do a lot with agents.

s20
00:01:30.520 --> 00:01:33.240
Uh I haven't written code since July 30th, 2025.

s21
00:01:33.240 --> 00:01:38.600
I know because that's the day I got back from Laracon and I saw PHP and I was like, "Never again."

s22
00:01:38.600 --> 00:01:40.920
And uh here I am.

s23
00:01:40.920 --> 00:01:42.880
Uh and with me is my best friend, Zach.

s24
00:01:42.880 --> 00:01:43.800
Hey, I'm Zach Prosser.

s25
00:01:43.800 --> 00:01:46.800
I'm an engineer on the Applied AI team at WorkOS.

s26
00:01:46.800 --> 00:01:50.480
Super excited to be here with you all today in New York.

s27
00:01:51.351 --> 00:01:51.720
[clears throat]

s28
00:01:51.720 --> 00:01:53.240
Now, uh we had this up before.

s29
00:01:53.240 --> 00:01:55.120
This repo is your workbench.

s30
00:01:55.120 --> 00:01:57.640
Um all of the slides are in this repo.

s31
00:01:57.640 --> 00:01:59.240
You can get it from here.

s32
00:01:59.240 --> 00:02:03.440
You can also view it at um workos.cloud/umbeyond-done.

s33
00:02:06.200 --> 00:02:11.600
Uh, I know that's a mouthful, but the next PR or the next QR code, I believe, will take you there.

s34
00:02:11.600 --> 00:02:14.520
Um, sorry, talks.workos.cloud/beyond-done.

s35
00:02:15.320 --> 00:02:19.440
Uh, and I just wanted to call it out because on that, I'll switch to it real quick.

s36
00:02:19.440 --> 00:02:21.800
Um, all of the slides are out there.

s37
00:02:21.800 --> 00:02:27.400
There's also a page for the workshops, like the the exercises that we're going to go through.

s38
00:02:27.400 --> 00:02:29.520
And so, for each piece of it, you can go through it.

s39
00:02:29.520 --> 00:02:30.640
You can go through it on your own time.

s40
00:02:30.640 --> 00:02:31.600
You can go through it later.

s41
00:02:31.600 --> 00:02:34.720
This is a a step-by-step guide to go through that.

s42
00:02:34.720 --> 00:02:41.760
There's also a glossary uh, covering all of the terms, big and small, uh, some that you're familiar with.

s43
00:02:41.760 --> 00:02:49.160
Uh, hopefully most of that you're familiar with, but if there's any that you're not, uh, you can talk in here or sorry, you can look at the the glossary

s44
00:02:49.160 --> 00:02:50.200
for those terms.

s45
00:02:50.200 --> 00:02:57.640
There's also an ask anything button, and you can talk to a reg that's trained on this uh, slide deck, and uh, hopefully get your answers there.

s46
00:02:57.640 --> 00:03:00.160
But, you can also just talk to a human, us.

s47
00:03:00.160 --> 00:03:00.840
Indeed.

s48
00:03:00.840 --> 00:03:02.160
Yeah, we'll leave leave this up as well.

s49
00:03:02.160 --> 00:03:08.160
We find that a lot of folks want to ask questions and for whatever reason can't, so be sure to check that out and use that.

s50
00:03:08.160 --> 00:03:08.640
Cool.

s51
00:03:08.640 --> 00:03:09.800
Give it just another second.

s52
00:03:09.800 --> 00:03:12.840
I see some cameras pointed at QR codes.

s53
00:03:12.840 --> 00:03:16.200
Security would be so mad at us.

s54
00:03:17.040 --> 00:03:17.880
All right.

s55
00:03:17.880 --> 00:03:28.840
Um, so when we're thinking about uh, being done with work, uh, agents being done, uh, there's lots of things that go into us trusting that.

s56
00:03:28.840 --> 00:03:32.320
And easily the first one is just is the code right?

s57
00:03:32.320 --> 00:03:33.920
Is the code what I would have written?

s58
00:03:33.920 --> 00:03:36.480
Did it generate a good PR?

s59
00:03:36.480 --> 00:03:40.280
And then I go through that like a normal normal code reviewer.

s60
00:03:40.280 --> 00:03:43.120
Um, but that takes a long time.

s61
00:03:43.120 --> 00:03:56.800
And these agents can write code a lot faster than us, so we don't want to rely on that too much because it's uh, it just there a a signal-to-noise ratio that starts coming into play and we just have too much code to review.

s62
00:03:56.800 --> 00:03:58.560
We're not writing any code anymore.

s63
00:03:58.560 --> 00:04:02.240
And we're less familiar with that code, so we become further abstracted away from it.

s64
00:04:02.240 --> 00:04:04.160
So, we need to find other ways to do that.

s65
00:04:04.160 --> 00:04:05.960
And there's plenty of ways that you can do that as well.

s66
00:04:05.960 --> 00:04:15.800
There's you know, using tools like browser tools like Playwright for example, to say like, "Hey, I have this website and I submitted a form.

s67
00:04:15.800 --> 00:04:16.880
The submit worked.

s68
00:04:16.880 --> 00:04:19.880
I checked that and it showed me a success message.

s69
00:04:19.880 --> 00:04:21.560
That's a great test to write.

s70
00:04:21.560 --> 00:04:26.560
Not even really anything agentic about it, but it is something that agents can use to verify their work.

s71
00:04:26.560 --> 00:04:28.960
And they can self-report that.

s72
00:04:28.960 --> 00:04:32.600
And you can trust that often times, but not all the time.

s73
00:04:32.600 --> 00:04:39.440
And so, that's really what we want to drive is like, how do we make it so that we can actually trust these reports?

s74
00:04:39.960 --> 00:04:40.320
Absolutely.

s75
00:04:40.320 --> 00:04:52.680
So, everything that we've got here for you today in this workshop as Nick mentioned is backed by a repo and it is interactive, so be you know, feel free to take an opportunity to actually run this to experiment with it to take something back that you might you know,

s76
00:04:52.680 --> 00:04:55.640
use at work or at least experience in a different way.

s77
00:04:55.640 --> 00:04:56.600
Yep.

s78
00:04:56.600 --> 00:05:08.880
In this so like if you just go into the repo like I'm doing and you do PNPM snack if I can type snack up uh This is why I have shortcuts for everything.

s79
00:05:08.880 --> 00:05:10.600
P is my PNPM.

s80
00:05:10.600 --> 00:05:20.120
Um it will pop up the browser and we can look at it and we can see this Midnight Snack Club app that we made.

s81
00:05:20.120 --> 00:05:23.600
Um and you can play with it right now and I can test this myself.

s82
00:05:23.600 --> 00:05:25.520
Let me just put in my email address.

s83
00:05:25.520 --> 00:05:31.200
I'll put in um zach@spamhim .com.

s84
00:05:31.200 --> 00:05:31.800
Thanks, buddy.

s85
00:05:31.800 --> 00:05:32.840
Yep.

s86
00:05:32.840 --> 00:05:35.000
Uh and look at that, success.

s87
00:05:35.000 --> 00:05:35.960
It saved the email.

s88
00:05:35.960 --> 00:05:37.040
The email is here.

s89
00:05:37.040 --> 00:05:37.880
I can see it.

s90
00:05:37.880 --> 00:05:43.000
If I had a test that was looking to confirm that the email went on the page it worked great.

s91
00:05:43.000 --> 00:05:44.960
So, that's what these examples look like.

s92
00:05:44.960 --> 00:05:47.880
You can kind of see it um, as you're going.

s93
00:05:47.880 --> 00:05:53.760
Uh, but if you reload that page, just like this, nothing's there.

s94
00:05:53.760 --> 00:05:57.600
So, even though, oops, what did I do?

s95
00:06:01.080 --> 00:06:02.040
Oh.

s96
00:06:02.040 --> 00:06:02.960
Switch back.

s97
00:06:02.960 --> 00:06:06.120
Uh, if you reload that, uh, it shows that nobody's here.

s98
00:06:06.120 --> 00:06:07.760
I can't drive Chrome, apparently.

s99
00:06:07.760 --> 00:06:13.240
Um, and that's because it did update, but it didn't persist that anywhere.

s100
00:06:13.240 --> 00:06:14.880
So, we didn't have any kind of verification.

s101
00:06:14.880 --> 00:06:18.840
We're just checking UI, seeing, oh, did that actually do anything?

s102
00:06:18.840 --> 00:06:25.640
And yes, it did do something to the UI, but we didn't check the other half, and so it was never persisted, and we lost that email.

s103
00:06:25.640 --> 00:06:27.680
So, sorry, Zach, you're not signed up.

s104
00:06:27.680 --> 00:06:28.760
It's okay with me.

s105
00:06:28.760 --> 00:06:30.960
Um, yeah, I think this is a really critical point.

s106
00:06:30.960 --> 00:06:38.080
So, uh, just show of hands, who in this room is is building with a ton with agents, just to start with the first question.

s107
00:06:38.080 --> 00:06:40.200
Okay, awesome, like more than half of the room.

s108
00:06:40.200 --> 00:06:46.800
Uh, how many folks are running multiple terminal tabs or multiple chat sessions or multiple agents, like everybody that put their hands up the first time, right?

s109
00:06:46.800 --> 00:06:51.000
And so, this is what we're all experiencing, at least what I'm experiencing, what Nick's experiencing.

s110
00:06:51.000 --> 00:07:06.120
If this is the 18th, you know, lane of work that you're driving today, and you have like, you know, diffracted attention to actually spend on it, these are the subtle bugs that will creep in because the agent's like, yeah, I did what you said, the code looks good, it builds, it runs, the test pass, and it's still

s111
00:07:06.120 --> 00:07:06.640
wrong.

s112
00:07:06.640 --> 00:07:16.240
And so, what we're going to ask and explore in this workshop together is, how do we actually fix this problem so that the agents that we're now all managing as like herds of

s113
00:07:16.240 --> 00:07:20.680
insane token machines uh, can actually figure out what they need to do themselves and when they've made a mistake.

s114
00:07:20.680 --> 00:07:21.320
Yep.

s115
00:07:21.320 --> 00:07:23.480
And you'll notice that we are aren't really looking at the code for this.

s116
00:07:23.480 --> 00:07:30.645
We're not really going to look at the code too much for this because, uh, like I said, I saw PHP and I'm scared, so we don't look at code anymore.

s117
00:07:30.645 --> 00:07:31.280
[clears throat]

s118
00:07:31.280 --> 00:07:39.160
Um, but all of this builds up to this great quote by, uh, a famous potato on, uh, X, call, uh, her her name is Lauren 10.

s119
00:07:39.160 --> 00:07:47.120
Uh, and she has this quote, if any agent, sorry, if an agent can verify its own work, nothing can't verify if an agent can't verify its own work, nothing else matters.

s120
00:07:47.120 --> 00:07:56.280
Yeah, another way to say it is is that and I think this is true in in my experience and others as well is that we're kind of converging on the point that verification now, especially as model

s121
00:07:56.280 --> 00:08:00.600
capabilities advance rapidly is is one of the key foundational things to focus on, right?

s122
00:08:00.600 --> 00:08:03.760
It's like reliability and verification and evals.

s123
00:08:03.760 --> 00:08:11.560
Especially as we're starting to see more and more harnesses come online that have the concept of RSI or you know, recursive self-improvement baked into them.

s124
00:08:11.560 --> 00:08:18.800
It feels like we're shifting to a world where we may all be maintaining multiple self-evolving code bases eventually, right?

s125
00:08:18.800 --> 00:08:30.800
And in that world, how do we set the eval foundation so that the agent actually knows what good looks like and can do those hill climbs on its own and then how are we going to give the agent the feedback it needs to understand that it's failed something.

s126
00:08:30.800 --> 00:08:37.440
Yeah, until we can solve that, we're not going to trust these agents to really accelerate our work to its fullest extent.

s127
00:08:37.440 --> 00:08:49.640
And so we need to solve that in various degrees and it's not just going from zero to one, but it's a gradient and getting us closer to one to where we can fully trust these agents is is the goal.

s128
00:08:49.720 --> 00:08:55.040
And I'll also just share that you know, again, anecdotally, this is what we're all experiencing at WorkOS.

s129
00:08:55.040 --> 00:08:59.320
We've got an applied AI team that's building a ton of different stuff for other teams and for ourselves.

s130
00:08:59.320 --> 00:09:08.600
And it is just a night and day experience when you are iterating on a code base that has, you know, the foundational eval in place, a baseline that can be run and trusted.

s131
00:09:08.600 --> 00:09:17.280
You know, we'll get into some of the tips and tricks, but the ability to actually see that it did or did not do something correctly in the same way that we would as human developers, right?

s132
00:09:17.280 --> 00:09:24.440
So we're, you know, of we've the the metaphor of us all becoming agent harness managers is is, you know, well trod.

s133
00:09:24.440 --> 00:09:30.720
This is how we ensure that each one of them is actually doing correct work and we're not burning money and time essentially.

s134
00:09:30.720 --> 00:09:34.240
Yeah, this is the equivalent of saying like make no mistakes, right?

s135
00:09:34.240 --> 00:09:39.440
Yeah, if you want to have some verification in there, you can't just like guess at what it needs to be.

s136
00:09:39.440 --> 00:09:45.880
You need to give it clear, defined goals to get to and give it a way to observe and check those goals.

s137
00:09:45.880 --> 00:09:55.400
If not, the agent's going to do something and it's going to spend some tokens and it might get to the end that you expect it to, but there's no guarantee of that.

s138
00:09:55.400 --> 00:10:04.160
Every for every detail that you don't fill in in terms of an outcome and what you actually want to get to, it's just going to infer that or make something up and

s139
00:10:04.160 --> 00:10:15.080
that assumption that it makes may not be the assumptions that you make or that you were expecting because you're a rational logical human and you were just not aligned on that yet.

s140
00:10:15.080 --> 00:10:26.680
And so um you need to have these systems in place to allow for it to observe, understand what you want, observe that, repair that and then recheck that.

s141
00:10:26.680 --> 00:10:37.560
Yeah, and I think just again, anecdotally like we're all finding that you know, the tired way to do it is to have a giant massive PR and have one agent session that's running for three or four days and say fix this, fix this, fix this, make it better.

s142
00:10:37.560 --> 00:10:51.760
And what is working, you know, essentially perfectly for folks is decomposing tasks really cleanly, giving the each task an agent in the prompt exactly what it needs to do to know that it is actually done and to verify the state of done.

s143
00:10:51.760 --> 00:10:53.160
And those will just rip.

s144
00:10:53.160 --> 00:11:00.520
I mean, we have people on our team that are 900 PR's a month and and no incidents and like improving KPIs for the software.

s145
00:11:00.520 --> 00:11:07.720
So just anecdotally it is um something that we all kind of have to take a step back and think about.

s146
00:11:07.720 --> 00:11:10.200
I'm not approaching this code base anymore in the traditional way.

s147
00:11:10.200 --> 00:11:14.000
I'm almost gardening multiple code bases and letting them self evolve.

s148
00:11:14.000 --> 00:11:16.000
Yeah.

s149
00:11:16.800 --> 00:11:19.800
And this is something that we build and we hold our own tools too as well.

s150
00:11:19.800 --> 00:11:23.880
This is how Zach and I stay productive at Work OS and how we get our own work done.

s151
00:11:23.880 --> 00:11:29.000
We've each built kind of our own examples of this and our own ways of approaching this.

s152
00:11:29.000 --> 00:11:31.080
And we really think that that's like a good thing right now.

s153
00:11:31.080 --> 00:11:37.120
It's too early to really classify on anything, so experimentation is key, finding what sticks and then running with that.

s154
00:11:37.120 --> 00:11:45.200
And uh one thing that we're both running with is the verification piece or like driving everything from that verification loop.

s155
00:11:45.200 --> 00:11:51.640
Uh because uh as as it says in this quote here for for mine, uh status comes from records, not model pros.

s156
00:11:51.640 --> 00:11:54.840
Um so it's it's, you know, show me the receipts.

s157
00:11:54.840 --> 00:11:58.640
Show me the actual tangible proof that it actually did it uh what I expect.

s158
00:11:58.640 --> 00:12:01.320
Don't just tell me, "Yeah, it passes, it looks good."

s159
00:12:01.320 --> 00:12:03.240
Um because there's nothing to back that up.

s160
00:12:03.240 --> 00:12:13.640
I need actual screenshots, videos, test results, smoke tests, um any all of the above to prove to me without me having to look at the code that it's actually doing what I want.

s161
00:12:13.640 --> 00:12:15.480
Because at the end of the day, I don't care about the code as much.

s162
00:12:15.480 --> 00:12:19.520
I care about the result and what it's doing and what value it's delivering.

s163
00:12:19.520 --> 00:12:23.440
And it can have bad code, honestly, and deliver that result pretty well.

s164
00:12:23.440 --> 00:12:28.200
Uh or it could have really good code that doesn't uh give me that result at all.

s165
00:12:28.200 --> 00:12:34.720
And I need to find the right balance in there, which is giving me the result that I want, not the code.

s166
00:12:34.720 --> 00:12:35.400
Yeah, indeed.

s167
00:12:35.400 --> 00:12:36.560
Uh so I've been doing this as well.

s168
00:12:36.560 --> 00:12:38.920
I I built a tool that I call context or CTX.

s169
00:12:38.920 --> 00:12:50.080
It started out as a cross-agent harness memory layer cuz I'm rapidly experimenting, as all of us are, with like ChatGPT, Cloud Code, uh Pi, etc. And I wanted them to all be able to write and read from memories, and that started working.

s170
00:12:50.080 --> 00:13:01.080
And then the more that I was running more more multiple agents and, you know, average day is like 18 or 30 terminal tabs, the more that I realized uh it might be an interesting experiment to try to centralize authority for verification

s171
00:13:01.080 --> 00:13:01.920
in a web service.

s172
00:13:01.920 --> 00:13:04.920
So the CTX is a simple web app that runs on Cloudflare.

s173
00:13:04.920 --> 00:13:06.120
It was vibe coded initially.

s174
00:13:06.120 --> 00:13:07.120
It works.

s175
00:13:07.120 --> 00:13:08.480
Uh it started as memory.

s176
00:13:08.480 --> 00:13:12.360
It was able to hand out secrets to agents without them ever seeing it or getting the credential.

s177
00:13:12.360 --> 00:13:13.160
And that worked, too.

s178
00:13:13.160 --> 00:13:16.280
And then now as an additional experiment, I'm creating verifiers.

s179
00:13:16.280 --> 00:13:19.920
And the way that that works is the agents are assigned work, as we're all doing with prompts.

s180
00:13:19.920 --> 00:13:21.480
They go off and do that work.

s181
00:13:21.480 --> 00:13:26.640
They come back through the MCP connection to CTX and say, "This is what I've completed.

s182
00:13:26.640 --> 00:13:27.760
Uh am I done?

s183
00:13:27.760 --> 00:13:28.760
Please verify it.

s184
00:13:28.760 --> 00:13:36.000
So, let's say in the example of um uh one of the workflows needs to verify that a PR was created correctly.

s185
00:13:36.000 --> 00:13:39.400
And so, that is going to run an evaluator on a known verifier.

s186
00:13:39.400 --> 00:13:45.880
That could just be a simple CLI to start with as it is in my repo, and then that becomes a hardened piece that you can reuse again and again.

s187
00:13:45.880 --> 00:13:51.600
It's going to the GitHub API and verifying that that PR truly does exist and is healthy and that all the merge checks pass.

s188
00:13:51.600 --> 00:13:55.520
And only then will it mark that node satisfied in the workflow.

s189
00:13:55.520 --> 00:14:01.120
Otherwise, it will redirect to it will redirect the agent and say, "These are the things that are still failing."

s190
00:14:01.120 --> 00:14:01.360
Right?

s191
00:14:01.360 --> 00:14:09.160
And so, again, an experiment, but this is kind of how we've been trying to play around with this and see which patterns are are going to stick and and what works and what not.

s192
00:14:09.160 --> 00:14:15.600
Um but it's also just kind of a fun time to do that five coding because uh I require uh weird pixel art across all of my apps.

s193
00:14:15.600 --> 00:14:18.720
So, uh you can see in this case there was a simple workflow.

s194
00:14:18.720 --> 00:14:23.440
And one of the verification gate is that requires that all pull requests were merged.

s195
00:14:23.440 --> 00:14:25.560
And so, that work is done by the centralized authority.

s196
00:14:25.560 --> 00:14:37.280
It goes out and uses its verifier, checks the API or whatever the artifact is, and then finally delivers the status over here on the right sidebar if you go to even down, and references the actual repository checked and the PR, right?

s197
00:14:37.280 --> 00:14:47.400
So, simple example, but you could imagine having standardized verifiers in a library or in skills or in tooling that you share amongst your your org, and this could be one of the ways that you start to build toward accelerating,

s198
00:14:47.400 --> 00:14:49.800
you know, PRs and and less and less human oversight.

s199
00:14:49.800 --> 00:14:51.800
Mhm.

s200
00:14:53.280 --> 00:14:55.839
Uh and then a similar tool uh that I built is called Ryker.

s201
00:14:55.839 --> 00:15:06.680
This is more like a engineering-focused Grokbot or Muse uh that's trying to be extremely flexible in how you hand work to it, but then once you hand work to it, its job is to babysit it.

s202
00:15:06.680 --> 00:15:12.280
And so, I don't want to have to babysit all of the PRs that I'm I'm bringing to fruition.

s203
00:15:12.280 --> 00:15:22.880
At some point there's a like this level of diminishing returns for me and my focus on on the work that I'm doing because I end up just like babysitting the PR, you know, fixing the tests, fixing the the reviews

s204
00:15:22.880 --> 00:15:25.920
uh that are coming in, the the review comments and things like that.

s205
00:15:25.920 --> 00:15:36.120
So, I want to be able to get the idea down of like what we should be doing and how we can verify that that work is done uh and then I can hand it off to Riker.

s206
00:15:36.120 --> 00:15:47.160
It can go do the work of, you know, making sure that the the reviews are done, uh that when it fixes code uh from PR reviews that it's still matching our overall goal of working software

s207
00:15:47.160 --> 00:15:55.600
uh in some way, and then uh can just verify that and and alert me back when it needs me instead of me having to baby sit it.

s208
00:15:55.600 --> 00:16:08.800
Um and so, with that it's just um you know, it builds a check, it figures out what tools it needs to run, it will run them, uh verify that they all pass, it'll keep that evidence and show it to me in different ways,

s209
00:16:08.800 --> 00:16:10.560
uh and then decide when to stop.

s210
00:16:10.560 --> 00:16:17.720
So, if it can verify yes, it was working, we've now fixed this bug, for example, um we put them together and it works.

s211
00:16:17.720 --> 00:16:34.440
Uh so, it's really just keeping me out of having to be in the minutia of shipping the the like final 10% uh and just focusing on where I'm valuable as a human, which is why we're doing the work and how we're and uh to an extent how we're doing it.

s212
00:16:36.520 --> 00:16:46.200
Okay, so maybe just as a mental experiment or a thought experiment, if you will, let's just kind of show of hands like or feel free to shout out like how would we start to think as a human developer

s213
00:16:46.200 --> 00:16:49.520
of verifying a web app like maybe in the before times before an LLM.

s214
00:16:49.520 --> 00:16:52.080
Yeah, what are things that you would do to verify?

s215
00:16:52.080 --> 00:16:54.440
And we'll we'll throw some easy ones up here to get it started.

s216
00:16:54.440 --> 00:16:59.320
So, like refresh the page, is the application still visible, right?

s217
00:16:59.960 --> 00:17:02.080
What's that?

s218
00:17:02.120 --> 00:17:02.880
Absolutely, yep.

s219
00:17:02.880 --> 00:17:04.959
Open a new tab, see if it's healthy, right?

s220
00:17:04.959 --> 00:17:06.920
What are the response codes I'm getting back from the API?

s221
00:17:06.920 --> 00:17:09.199
Are they expected based on what I sent it?

s222
00:17:09.199 --> 00:17:10.000
Absolutely.

s223
00:17:10.000 --> 00:17:17.000
Local host or local storage, checking persistence in some way, console logs, um things like that.

s224
00:17:17.000 --> 00:17:18.920
Network, uh, like you said.

s225
00:17:18.920 --> 00:17:21.959
And, um, like reading what's actually on the page.

s226
00:17:21.959 --> 00:17:23.360
Do the contents of the page actually

s227
00:17:23.360 --> 00:17:24.600
Interacting with it, right?

s228
00:17:24.600 --> 00:17:31.560
And the reason we're we're saying this is like this is really to set up the the transition from we were the human developers doing this all the time.

s229
00:17:31.560 --> 00:17:35.280
This was part of our core responsibility when we're hand chiseling code, if you will.

s230
00:17:35.280 --> 00:17:44.520
And then, what is the delta between doing this and then in the the early stages of LLM time saying, "Here's three paragraphs of text.

s231
00:17:44.520 --> 00:17:46.640
Go execute these instructions."

s232
00:17:46.640 --> 00:17:57.760
The delta between those two things is the agent's ability to verify that it actually did what you asked it to do in a real application and that its changes did not break anything.

s233
00:17:57.760 --> 00:17:58.000
Right?

s234
00:17:58.000 --> 00:18:03.240
Otherwise, you're just getting the confabulations that LLMs are designed to give you cuz they're they're trying to process the text.

s235
00:18:03.240 --> 00:18:05.760
They're trying to make you happy.

s236
00:18:08.120 --> 00:18:11.160
Uh, your agent can't do any of this just in a terminal alone.

s237
00:18:11.160 --> 00:18:13.000
Uh, it needs tools to do that.

s238
00:18:13.000 --> 00:18:15.040
Uh, fortunately, tools are very easy to add to

s239
00:18:15.040 --> 00:18:16.400
agents now.

s240
00:18:16.400 --> 00:18:18.880
Um, and we just have to tell them how to use it.

s241
00:18:18.880 --> 00:18:22.760
We'll tell them specifically what we're trying to to prove.

s242
00:18:22.760 --> 00:18:30.240
Um, when you do that, you don't want it to be just a paragraph of like, "Hey, do this very always verify the change in a browser."

s243
00:18:30.240 --> 00:18:32.480
Whatever that means, that's pretty vague.

s244
00:18:32.480 --> 00:18:33.840
Uh, "Remember to reload and check."

s245
00:18:33.840 --> 00:18:35.640
That's a little bit more specific.

s246
00:18:35.640 --> 00:18:37.200
Uh, "Check that the data persists."

s247
00:18:37.200 --> 00:18:38.920
Look at the console for errors.

s248
00:18:38.920 --> 00:18:43.520
Uh, getting a little more specific, but like that's a lot, um, to put in there and you can certainly do that.

s249
00:18:43.520 --> 00:18:44.680
That's what a lot of skills do.

s250
00:18:44.680 --> 00:18:47.680
That's what what a lot of just like saved prompts do.

s251
00:18:47.680 --> 00:18:52.880
Um, but if you forget anything or things change, that can leave it up to a lot of interpretation.

s252
00:18:52.880 --> 00:19:07.160
Um, one thing that might be better is to have a script that can prove it out in different ways, um, so that you can just run that over and over and it's doing more of a deterministic check across a number of things and then it can feed that output

s253
00:19:07.160 --> 00:19:12.040
back into the agent and it knows exactly what was wrong uh, and how to fix it quickly.

s254
00:19:12.040 --> 00:19:21.720
And it just closes that loop so that it's a much faster uh and the agent can just spin that, run it again, check, verify, run again, and just do that over and over until there's no more errors.

s255
00:19:21.720 --> 00:19:23.480
It's also uniform if you think about it, right?

s256
00:19:23.480 --> 00:19:30.120
Um we know that LLMs are non-deterministic, so even giving 30 agents the exact same prompt and you have a potential for variance in your outcomes.

s257
00:19:30.120 --> 00:19:35.200
And pinning it down to a single command that is backed by actual bash commands and bash calls, right?

s258
00:19:35.200 --> 00:19:38.160
Like that's that's significantly more deterministic.

s259
00:19:38.160 --> 00:19:48.280
And anecdotally, the folks uh like Lauren Tan and others that are achieving thousands of PRs ostensibly a month and uh building ambitious products that were not possible to build in that time before

s260
00:19:48.280 --> 00:19:53.960
have mentioned that this is the first thing that they they concentrated on that actually unlocked the velocity, right?

s261
00:19:53.960 --> 00:20:11.320
So, if your specific web app, for example, does not already have a bespoke CLI that allows any agent to fully flex the functionality that you care about verifying, then you're going to have a much rougher time and an unpaved path if you're using prompts, you know, alone uh compared to actually,

s262
00:20:11.320 --> 00:20:14.800
again, giving each agent the ability to say, "Yes, I actually built that feature.

s263
00:20:14.800 --> 00:20:16.040
It is actually performant.

s264
00:20:16.040 --> 00:20:20.240
It didn't cause any errors, and the application looks correct."

s265
00:20:21.640 --> 00:20:24.400
So, for this workshop, uh we made a little tool.

s266
00:20:24.400 --> 00:20:26.840
Uh we've been running it a couple of times, but it's called snack.

s267
00:20:26.840 --> 00:20:29.600
Uh and you can just run it with PNPM snack in that repo.

s268
00:20:29.600 --> 00:20:42.640
Uh it's not meant to be like a reusable tool, but just kind of like a template for how you could set up this uh specific for the applications that you're working on, and then use that as a way to tie into the LLMs that are actually working on that application with you.

s269
00:20:42.640 --> 00:20:44.120
And so, snack can do a couple of things.

s270
00:20:44.120 --> 00:20:46.840
It can uh manage the health of the application.

s271
00:20:46.840 --> 00:20:48.480
It can bring it up, it can take it down.

s272
00:20:48.480 --> 00:20:52.200
It can run a doctor command to see is it actually, you know, is the database running?

s273
00:20:52.200 --> 00:20:53.840
Does it have network connection?

s274
00:20:53.840 --> 00:20:54.720
Et cetera.

s275
00:20:54.720 --> 00:20:57.920
Um that quick drive or that quick check that it can do.

s276
00:20:57.920 --> 00:21:01.600
Uh then it has uh drive, so you can tell it to join.

s277
00:21:01.600 --> 00:21:04.960
Remember, this is our our uh midnight snack waitlist.

s278
00:21:04.960 --> 00:21:06.240
Uh so, it can join the waitlist.

s279
00:21:06.240 --> 00:21:10.400
It can reload the page, it can reset all of the data so it can't find it.

s280
00:21:10.400 --> 00:21:21.080
You can have it look up things, so take a snapshot, uh, look at the network storage, look at console logs, um, look at local storage, uh, screenshot it, run evals,

s281
00:21:21.080 --> 00:21:22.640
all of all of the above.

s282
00:21:22.640 --> 00:21:24.680
Uh, and then finally it can prove it in two ways.

s283
00:21:24.680 --> 00:21:30.000
It can actually run a prove command where it's checking for deterministic output of that.

s284
00:21:30.000 --> 00:21:39.080
And a judge, which is using a system one model to, uh, in like, get an idea of whether we're going in the right direction without it necessarily matching one to one.

s285
00:21:39.080 --> 00:21:49.000
Just to, like, fully develop this idea, when you do something like this in a repo that you haven't already had something like this before, the velocity gains that you unlock are not solely for you, right?

s286
00:21:49.000 --> 00:21:57.840
It's for everyone else on your team that then comes along and is also able to issue significantly more PRs significantly faster and significantly safer, right?

s287
00:21:57.840 --> 00:22:01.600
More safely, so that, um, you know, everybody on the team actually benefits.

s288
00:22:01.600 --> 00:22:06.200
And again, that's like where we we keep harping on this being like a foundational concept now.

s289
00:22:06.200 --> 00:22:07.160
Yeah.

s290
00:22:07.160 --> 00:22:17.200
Uh, and if you think about it, like, that's a really big piece of it it might even be more important than your software, uh, than the actual application is the way that you verify it, the way that you test it,

s291
00:22:17.200 --> 00:22:25.160
uh, the way that you prove that it works because that is what you could use potentially, like, a lot of people are doing and converting to Rust.

s292
00:22:25.160 --> 00:22:38.320
Um, you could take your app, uh, like the Bun team did, convert it to Rust, uh, Go, TypeScript, convert it to Go, uh, not necessarily with the same level of LLM help, but, um, they did use LLMs to help with that.

s293
00:22:38.320 --> 00:22:43.480
Um, and then I think Theo just last week converted it to Rust again, so.

s294
00:22:43.480 --> 00:22:43.880
Yep.

s295
00:22:43.880 --> 00:22:47.960
Uh, when you want that, if you have all of those verifiers in place, then your source code doesn't matter as much.

s296
00:22:47.960 --> 00:22:50.800
You can write it again and prove that it works.

s297
00:22:50.800 --> 00:22:51.480
Keep going.

s298
00:22:51.480 --> 00:22:53.480
Absolutely.

s299
00:22:54.240 --> 00:22:57.240
And so, um, these tools give your agents their eyes.

s300
00:22:57.240 --> 00:23:01.800
Or you if you think about it, like, the tools that your agent use, that gives it the body.

s301
00:23:01.800 --> 00:23:10.840
The LLM is the brain and then the way that it reaches out and interacts physically, uh virtually on your computer is through these tools.

s302
00:23:10.840 --> 00:23:20.560
And so, things like uh join is it like actually having it run and do things and then it can check and see, you know, did I did I that actually succeed in some way?

s303
00:23:20.560 --> 00:23:29.200
Uh check storage, run reload, uh snapshot, etc. And so, let's just run a couple of these uh and see.

s304
00:23:29.200 --> 00:23:30.960
P snack up.

s305
00:23:30.960 --> 00:23:35.320
That brought up our um browser.

s306
00:23:35.320 --> 00:23:39.320
And then I'll open up another browser or another terminal.

s307
00:23:39.680 --> 00:23:41.520
I have too many terminals.

s308
00:23:41.520 --> 00:23:46.520
And run um P snack snapshot.

s309
00:23:47.600 --> 00:23:51.720
And that give us a snapshot of the page in as JSON that we can see.

s310
00:23:51.720 --> 00:24:01.440
If I put um Zack at zack.zack uh and add that to the page.

s311
00:24:01.440 --> 00:24:03.280
Hopefully, we'll see some Zacks in there.

s312
00:24:03.280 --> 00:24:04.480
There we go.

s313
00:24:04.480 --> 00:24:06.600
So, that's a way that we can verify.

s314
00:24:06.600 --> 00:24:11.280
Uh we can also do screenshot to make it a little easier.

s315
00:24:11.280 --> 00:24:14.640
That should pop up a screenshot.

s316
00:24:17.400 --> 00:24:18.400
And yeah.

s317
00:24:18.400 --> 00:24:25.880
So, that gave us the ability to, for example, not just like pass this to the LLM and have it do interpretation on it, but like proving back to us.

s318
00:24:25.880 --> 00:24:27.040
That's the end goal, right?

s319
00:24:27.040 --> 00:24:28.440
Is proving back to the human.

s320
00:24:28.440 --> 00:24:32.480
Being able to take screenshots, record videos, put those on the pull request.

s321
00:24:32.480 --> 00:24:35.520
You can do that with the the GH CLI now.

s322
00:24:35.520 --> 00:24:37.440
Um Chrome debug debugging tools.

s323
00:24:37.440 --> 00:24:38.520
Chrome debugging tools, yeah.

s324
00:24:38.520 --> 00:24:39.760
Where is the actual bottleneck?

s325
00:24:39.760 --> 00:24:42.080
Don't guess and, you know, spin your wheels for 3 hours.

s326
00:24:42.080 --> 00:24:45.200
Like actually figure out which part was causing the render to be slow.

s327
00:24:45.200 --> 00:24:46.080
Yeah.

s328
00:24:46.080 --> 00:25:01.800
So, the real key here is that there's no um it's it's not just code that you can look at to interpret whether this this application is running correctly, but you can build tools that will help you prove to you that it's working correctly without you having to read the code at all.

s329
00:25:01.800 --> 00:25:03.000
Uh because this is all we care about.

s330
00:25:03.000 --> 00:25:04.560
Is this what a user sees?

s331
00:25:04.560 --> 00:25:14.880
Is it correctly rendering in this browser at this width with dark mode, with light mode, etc. And you can run infinite tests on that and run them over and over and over in a loop,

s332
00:25:14.880 --> 00:25:23.480
uh which as Zach said unlocks your whole team to move faster because it you've given the agent the ability to run those checks itself as well.

s333
00:25:27.440 --> 00:25:37.080
So, [clears throat] if we run this uh if we ran it just like this where we ran the doctor uh and then we add uh ada@example.test and then we check the storage, it's there.

s334
00:25:37.080 --> 00:25:40.840
We run uh if we check the storage, it's not there.

s335
00:25:40.840 --> 00:25:42.880
Um we would see local storage is empty.

s336
00:25:42.880 --> 00:25:47.440
And if we rerun reload and then snapshot, we would verify that it is in fact empty.

s337
00:25:47.440 --> 00:25:55.600
So, this is a way that the tool itself could run in a loop, could check all of these things, and yes, confirm with before it even gets back to us.

s338
00:25:55.600 --> 00:25:57.760
Oh, I see that we're not actually persisting this.

s339
00:25:57.760 --> 00:25:58.800
I forgot to do that part.

s340
00:25:58.800 --> 00:26:01.640
And the agent can just move on from there.

s341
00:26:01.640 --> 00:26:06.040
Um real quick, we wanted to stop and take a a few questions if there are any.

s342
00:26:06.040 --> 00:26:06.720
Yes, sir.

s343
00:26:06.720 --> 00:26:08.160
Yeah.

s344
00:26:08.160 --> 00:26:21.160
Uh so, the question is uh are we uh using models to write JSON uh to to then like help us to make our code deterministic, to bring that back.

s345
00:26:21.160 --> 00:26:23.920
Um the answer for me is yes.

s346
00:26:23.920 --> 00:26:33.280
Uh I'm I'm using that and I'm running it and then I use uh like I'll I'll use unit tests around that to verify in different ways, catch edge cases,

s347
00:26:33.280 --> 00:26:42.160
uh and then verify that the code that that is like a real interpretation or real representation of what I'm expecting to get back from like a system one model,

s348
00:26:42.160 --> 00:26:46.040
uh and use that to prove that that I'm on the right track.

s349
00:26:46.040 --> 00:26:54.160
But, then have keep those as well to kind of like have a another check uh to make sure that it doesn't drift over time or things change.

s350
00:26:54.160 --> 00:26:56.040
The the the general process that we're talking about?

s351
00:26:56.040 --> 00:27:01.640
Yeah, so when what I'm talking about, uh, at least on our team, we we build a ton of applications that are internal tooling.

s352
00:27:01.640 --> 00:27:04.280
We have some customer-facing applications as well.

s353
00:27:04.280 --> 00:27:07.920
They tend to be uh, TypeScript and they tend to be deployed to Cloudflare.

s354
00:27:07.920 --> 00:27:09.720
So, they're you know, they're full-stack applications.

s355
00:27:09.720 --> 00:27:10.400
It works perfectly.

s356
00:27:10.400 --> 00:27:14.200
You can even have evals that are focused on the back end or database operations.

s357
00:27:14.200 --> 00:27:16.680
All right, you could get as granular as you want with it.

s358
00:27:16.680 --> 00:27:27.240
Um, but it's absolutely yeah, it works full-stack and and I expect that the way that we're all going to continue interfacing with software in the future is just going to be kind of like the gardeners making sure that they're evolving correctly

s359
00:27:27.240 --> 00:27:27.920
on our own.

s360
00:27:27.920 --> 00:27:28.440
Yeah.

s361
00:27:28.440 --> 00:27:39.040
That's why uh, we're not like giving out a tool because we couldn't give you a a tool at this time that works generically for every single application that everyone in this room is working on.

s362
00:27:39.040 --> 00:27:45.480
You kind of have to build up the tools that fit the app that you're working on and then give that to the models and just make sure that they know how to run them well.

s363
00:27:45.480 --> 00:27:46.960
Um, as a

s364
00:27:46.960 --> 00:27:59.120
I was so I was I was also going to say like your evals, what you actually need or care about as a team or company or business or going to wildly differ from, you know, if you're in like the interior design space versus some other place where like speed is the only thing that matters.

s365
00:27:59.120 --> 00:28:03.480
Um, I I think you're going to have like wildly different evals and there's a value to building those up with your team.

s366
00:28:03.480 --> 00:28:04.440
Yeah.

s367
00:28:04.440 --> 00:28:12.680
And there's there's lots of different ways to get the data to the like to the model in different ways for so that it can interpret it and it can keep up.

s368
00:28:12.680 --> 00:28:18.600
And that's like a a huge piece of like keeping the model in the loop as to what's going on.

s369
00:28:18.600 --> 00:28:20.880
Uh, the AMP team has a really great post on it.

s370
00:28:20.880 --> 00:28:24.720
I think it's called I can't remember the the post, but I'll get it out there.

s371
00:28:24.720 --> 00:28:27.080
I think it's called loop-driven develop I don't know.

s372
00:28:27.080 --> 00:28:27.840
I can't remember.

s373
00:28:27.840 --> 00:28:41.160
But uh, what they in in the example, they show like a canvas with a circle and a ball bouncing inside of this circle uh, and then like there's a little slit where the ball if it hits just right, it can fall out of the circle and then the game the game is over.

s374
00:28:41.160 --> 00:28:44.960
And uh, the whole idea of that is making things feedback loopable.

s375
00:28:44.960 --> 00:28:46.200
That's what it is.

s376
00:28:46.200 --> 00:29:00.440
Um, where you know, the model can't actually see the running canvas and where things are going, but if you, as a very contrived example, if you just print out the coordinates of that ball at every point, and you see a bug where, you know, once it gets to this specific point, it jumps to the other side of

s377
00:29:00.440 --> 00:29:01.360
the screen.

s378
00:29:01.360 --> 00:29:06.280
Well, if it's moving more than one pixel at a time, like per loop, something is wrong, potentially.

s379
00:29:06.280 --> 00:29:07.680
And so, there's a bug in there.

s380
00:29:07.680 --> 00:29:15.040
If you can surface that to the model, then it can help you debug it, where it doesn't actually have eyes to see the canvas and see the ball not

s381
00:29:15.040 --> 00:29:16.880
behaving well, like with physics.

s382
00:29:16.880 --> 00:29:17.280
So,

s383
00:29:17.280 --> 00:29:26.880
And also, once you do that, you can you're also then free to spin up 30 different variants of software that are, you know, speaking to that eval, and then pick the one that you want.

s384
00:29:26.880 --> 00:29:27.280
Yeah.

s385
00:29:27.280 --> 00:29:29.280
Um, so, that's the other thing that it unlocks, right?

s386
00:29:29.280 --> 00:29:45.920
It's really like foundational for just going, and especially if we expect agents to models to improve, you know, tokens to get cheaper, potentially, um, then, you know, you can imagine that we're all going to be like coming in on Monday and saying like I would like to see 35 versions of this new feature or application here.

s387
00:29:45.920 --> 00:29:46.160
Yep.

s388
00:29:46.160 --> 00:29:47.520
And pick one.

s389
00:29:47.520 --> 00:29:50.040
And so, how do we actually run or create that?

s390
00:29:50.040 --> 00:29:56.080
Well, in this case, like we said, we can't really prove it, uh, for everything, like universally.

s391
00:29:56.080 --> 00:29:59.280
It really has to be something that's unique to your application.

s392
00:29:59.280 --> 00:30:09.600
And so, for our midnight snack page, this is what the proof, uh, the proof command might look like, where it's basically using, uh, a tool like Playwright, um, to

s393
00:30:09.600 --> 00:30:13.920
check that a, uh, an email was actually saved when it gets on the waitlist.

s394
00:30:13.920 --> 00:30:23.920
So, uh, we can check that it doesn't have any on the waitlist, any emails, then we can fill in an email on the next line, and then we can get the button and click it,

s395
00:30:23.920 --> 00:30:37.720
uh, the join waitlist button, and then we can expect that the status, uh, element on the page has a waitlisted in there, uh, and then check the inner text of that to see if our email address is in there.

s396
00:30:37.720 --> 00:30:49.440
Um, and so, that's one simple way to do it, very easy if you've ever written, uh, an end-to-end test with like playwright or react testing library, any of those,

s397
00:30:49.440 --> 00:30:51.120
you've definitely used tools like this.

s398
00:30:51.120 --> 00:30:54.840
You can just give them back to the model to your agent and they can run them.

s399
00:30:54.840 --> 00:31:07.600
Give it to them as a tool and they can prove it out themselves and you've made this reproducible bit of evidence that then you can use in this one next time you use it next time that page changes it can run it again and verify that nothing broke.

s400
00:31:07.600 --> 00:31:14.280
It acts as a regression and keeps everyone on the team moving forward.

s401
00:31:15.941 --> 00:31:16.160
[clears throat]

s402
00:31:16.160 --> 00:31:21.720
One caution word of caution though, like a successful screenshot isn't proof.

s403
00:31:21.720 --> 00:31:33.360
Make sure that it's actually showing, you know, the state that you expect it to be in, that it actually like ran the commands to get to that state to actually be in there because otherwise it might just be like another lie,

s404
00:31:33.360 --> 00:31:40.600
but you're seeing a screenshot on the the PR and thinking, oh yeah, this is this is correct, but it might have be causing different side effects.

s405
00:31:40.600 --> 00:31:44.800
It might have taken a different path that's not actually possible within the UI.

s406
00:31:44.800 --> 00:31:50.680
So you want to make sure that it actually like runs start to finish to check that.

s407
00:31:52.760 --> 00:32:03.520
Yeah, and so once you build these loops that are, you know, durable, reliable, repeatable, uniform, the value of what you get or the benefit the main benefit is that a a failure starts to drive future work, right?

s408
00:32:03.520 --> 00:32:12.760
So if in one loop the agent fails, you're busy talking to somebody at work or you're starting yet another task, that agent can determine on its own, this is not quite correct.

s409
00:32:12.760 --> 00:32:14.000
Let me go take another stab at it.

s410
00:32:14.000 --> 00:32:15.440
Let me now run the command verify again.

s411
00:32:15.440 --> 00:32:16.880
Okay, now we're good.

s412
00:32:16.880 --> 00:32:26.840
If you think about what that means at scale that a single PR or an agent a single lane of work, if you will, and an agent, you know, self heals, self corrects, figures it out on its own,

s413
00:32:26.840 --> 00:32:33.160
you're not coming back to that particular PR at the end of the day and and then being like, oh, I still have to get this all the way through.

s414
00:32:33.160 --> 00:32:35.200
And if you just multiply that out, right?

s415
00:32:35.200 --> 00:32:40.880
That that's that's really where again, we just say this is foundational for speed and velocity.

s416
00:32:40.880 --> 00:32:41.680
Yep.

s417
00:32:41.680 --> 00:32:45.720
And how many of you write good error messages?

s418
00:32:45.720 --> 00:32:47.240
Show of hands.

s419
00:32:47.240 --> 00:32:48.120
Yeah, nobody knows.

s420
00:32:48.120 --> 00:32:48.952
We already know.

s421
00:32:48.952 --> 00:32:49.600
[laughter]

s422
00:32:49.600 --> 00:32:51.600
Sir, what's your message?

s423
00:32:51.600 --> 00:32:54.000
I got small hands.

s424
00:32:54.040 --> 00:32:54.520
There you go.

s425
00:32:54.520 --> 00:32:54.680
Yeah.

s426
00:32:54.680 --> 00:32:56.240
Solid.

s427
00:32:56.240 --> 00:32:57.480
Um well, you should.

s428
00:32:57.480 --> 00:33:05.920
Uh I think they're more important than ever because uh when your your uh app throws an error, guess what can interpret and run with that error really well?

s429
00:33:05.920 --> 00:33:06.600
An agent.

s430
00:33:06.600 --> 00:33:14.760
So, um if you make it so that your failure is explicit about why it failed and give as much detail as possible about how it got into that state,

s431
00:33:14.760 --> 00:33:17.720
then uh your agent can just like self-heal from that.

s432
00:33:17.720 --> 00:33:23.440
So, it's um very easy, very worthwhile to to do that.

s433
00:33:25.240 --> 00:33:28.640
Uh yeah, and so this is a concept that you know, Nick and I talk about, right?

s434
00:33:28.640 --> 00:33:35.200
And it sounds like we're being uh in some ways we're trying to wrap determinism around LLMs, uh but we actually don't want to do that.

s435
00:33:35.200 --> 00:33:38.800
And it's also something to the spirit of what the amp team is talking about.

s436
00:33:38.800 --> 00:33:41.640
Uh the agents are in in intelligent.

s437
00:33:41.640 --> 00:33:42.640
They're they're capable.

s438
00:33:42.640 --> 00:33:44.800
We want them to be creative, actually.

s439
00:33:44.800 --> 00:33:49.800
Because oftentimes if they have the right constraints, and then oftentimes constraints can be can equal freedom, right?

s440
00:33:49.800 --> 00:33:51.000
In a way.

s441
00:33:51.000 --> 00:33:55.400
Uh then they will come up with creative solutions and actually solve what you said.

s442
00:33:55.400 --> 00:33:57.000
That's why you give them the outcome.

s443
00:33:57.000 --> 00:34:01.560
That's why, you know, Nick's project he's experimenting with is outcome-driven development.

s444
00:34:01.560 --> 00:34:08.120
Um and so, it's not about over being overly prescriptive in how the agent achieves things.

s445
00:34:08.120 --> 00:34:10.919
Um that can actually lead to lower performance in some cases.

s446
00:34:10.919 --> 00:34:13.919
It's more about giving it the map and saying you must get here.

s447
00:34:13.919 --> 00:34:17.440
You'll know you're here when this command returns this.

s448
00:34:17.440 --> 00:34:18.200
Yeah.

s449
00:34:18.200 --> 00:34:20.520
You don't necessarily have to prescribe how it gets there, though.

s450
00:34:20.520 --> 00:34:25.480
And this is true more and more true every single day with like the frontier models.

s451
00:34:25.480 --> 00:34:35.000
Fable, Astra, Saul 6.1, uh Opus 5.5. They're all really good at being like you being creative is like the way I I describe that.

s452
00:34:35.000 --> 00:34:43.520
But, if you give them This is where I want to get to and don't necessarily tell them the path that they need to take, they might wander and make sure that they catch more edge cases.

s453
00:34:43.520 --> 00:34:47.360
They might figure something out or ask you a question that unlocks something.

s454
00:34:47.360 --> 00:34:52.200
But, if you're just like you have to get here and then here and then here, they may not sway from that.

s455
00:34:52.200 --> 00:34:56.320
And you want them to sway a little bit, but get to the right right end goal.

s456
00:34:56.320 --> 00:35:03.000
And like I said, this is something that we we've only come around to really appreciating in the latest batch of Frontier models.

s457
00:35:03.000 --> 00:35:04.280
Uh where it's really becoming possible.

s458
00:35:04.280 --> 00:35:06.320
So, going forward, it's going to just get better.

s459
00:35:06.320 --> 00:35:08.320
Yep.

s460
00:35:09.124 --> 00:35:09.800
[clears throat and cough]

s461
00:35:09.800 --> 00:35:17.440
So, when it's going, um you want to get into like, you know, the typical TDD like red, green, refactor loop with your agent.

s462
00:35:17.440 --> 00:35:25.960
Um we've talked You've heard about loops, you know, at every one of these AIE events, and um it's no different here, but you can use that within the verification piece itself.

s463
00:35:25.960 --> 00:35:30.800
Verify this, reproduce it, show me uh fix it, and then show me the proof of that.

s464
00:35:30.800 --> 00:35:32.120
And how are you going to show me the proof?

s465
00:35:32.120 --> 00:35:41.080
That's just as important as the fix that's going to go in, because that's what's going to leave leave you alone and allow you to trust the model to actually do the work.

s466
00:35:41.080 --> 00:35:47.760
And And then when it says, "Hey, I'm done," you know that it's done because it's proved it to you in some way.

s467
00:35:47.800 --> 00:35:49.800
Absolutely.

s468
00:35:50.480 --> 00:35:53.240
So, in this example, uh you can run Snack Doctor.

s469
00:35:53.240 --> 00:35:56.320
Let's just go ahead and run that real quick.

s470
00:35:57.200 --> 00:36:00.600
P Snack Doctor.

s471
00:36:01.320 --> 00:36:08.040
And so, this uh this ran, and it's showing us um you know, a bunch of checks that it did.

s472
00:36:08.040 --> 00:36:17.640
So, it's checking that we have Node, that Chromium's installed, uh that our session exists, uh that it's actually running, and that it successfully built.

s473
00:36:17.640 --> 00:36:22.200
So, that's like what we're running as a quick doctor check, just a bunch of different things.

s474
00:36:22.200 --> 00:36:27.560
So, then we run the next command, Snack Join Ada at example.test.

s475
00:36:29.560 --> 00:36:31.655
Sorry, we don't type anymore.

s476
00:36:31.655 --> 00:36:33.655
[laughter]

s477
00:36:35.760 --> 00:36:36.920
Okay.

s478
00:36:36.920 --> 00:36:39.000
Gave us the result back.

s479
00:36:39.000 --> 00:36:41.440
Snack storage.

s480
00:36:44.560 --> 00:36:45.520
Storage is empty.

s481
00:36:45.520 --> 00:36:48.200
It didn't actually put Ada in there.

s482
00:36:48.200 --> 00:36:56.320
So, this is the point at which in the real world your agent you've just gotten the value out of this entire system cuz at this moment your agent realizes I am not done

s483
00:36:56.320 --> 00:36:58.120
and I'm not going to drop this work on the floor.

s484
00:36:58.120 --> 00:36:58.880
Yep.

s485
00:36:58.880 --> 00:37:04.080
And if you reload it, do the same thing one more time.

s486
00:37:04.600 --> 00:37:05.040
Boom.

s487
00:37:05.040 --> 00:37:08.080
We have now given our app eyes.

s488
00:37:08.080 --> 00:37:09.000
It can see this.

s489
00:37:09.000 --> 00:37:09.840
It can run these.

s490
00:37:09.840 --> 00:37:23.360
It can make it can make sense and change the world of the application, the Midnight Snack application, and see the results of that and then be able to interpret what happened from there.

s491
00:37:24.710 --> 00:37:25.200
[clears throat]

s492
00:37:25.200 --> 00:37:25.920
So, you can try that.

s493
00:37:25.920 --> 00:37:34.880
You can If we like apply the persistence fix to it and add honest copy to it, then we can um run the proof.

s494
00:37:34.880 --> 00:37:37.240
Let's run that.

s495
00:37:41.440 --> 00:37:44.360
And that kicked off a browser and it failed.

s496
00:37:44.360 --> 00:37:47.400
I didn't actually run the test, but if we had run the fix.

s497
00:37:47.400 --> 00:37:51.320
If we had run the fix, it would show that.

s498
00:37:52.221 --> 00:37:53.240
[clears throat and cough]

s499
00:37:53.240 --> 00:38:00.320
Okay, one of the things that we're we're thinking about a lot lately and talking about and starting to roll out into more applications is this like concept of receipts.

s500
00:38:00.320 --> 00:38:09.440
Um you know, you can you can take like a ticket like a a help ticket system where it's there's a certain performance the way that a human would answer a ticket and consider it done well

s501
00:38:09.440 --> 00:38:12.120
as an excellent, you know, customer response or service.

s502
00:38:12.120 --> 00:38:18.320
And then what does the um like preserving that in some way, for example, right?

s503
00:38:18.320 --> 00:38:19.320
Uh why do we want to do that?

s504
00:38:19.320 --> 00:38:29.920
There's a ton of reasons and we'll get into some of them, but again, if you think about if it if we all or if it seems that in some way many many folks are heading towards, you know, running RSI

s505
00:38:29.920 --> 00:38:38.520
harnesses and self-evolving code bases, everything that we do today, everything that your agent does today, every run all day long, that is very valuable data.

s506
00:38:38.520 --> 00:38:44.520
It's very valuable data right now because you can feed it into LLMs and and get fixes that you need.

s507
00:38:44.520 --> 00:38:53.560
It's incredibly valuable data in the future as tooling improves and it gets easier and easier to set up RSI loops because then you've got a data set of exactly how your system performed

s508
00:38:53.560 --> 00:39:04.320
that you can create, you know, very comprehensive evals for and have agents hill climb on, which means loop and see their performance and then continue going until they get better and better and better, right?

s509
00:39:04.320 --> 00:39:05.080
Yep.

s510
00:39:05.080 --> 00:39:10.320
And at that point they're able to pass on their own using that evidence from the previous runs.

s511
00:39:12.132 --> 00:39:12.160
[clears throat]

s512
00:39:12.160 --> 00:39:18.320
So now, if we fix that code, it now persists, but it can also lie.

s513
00:39:18.320 --> 00:39:27.280
So after we reload the page, it could say wait listed, but what the screenshot might actually say is you're in, your seat is confirmed.

s514
00:39:27.280 --> 00:39:33.160
Those don't exactly match up, and so that might be like a failure right there because they don't exactly match.

s515
00:39:33.160 --> 00:39:37.080
So this is kind of getting away from the static testing piece of it.

s516
00:39:37.080 --> 00:39:48.800
Like how do we prove that it worked without having to be like very tightly coupled to the actual text on the screen for a very contrived example.

s517
00:39:48.800 --> 00:40:08.280
And so if we just run like, you know, P snack, judge, should run that and it failed because even though it worked and it says that we're in, Jev did not the the model did not see that.

s518
00:40:08.280 --> 00:40:11.840
And it's because the the confirmation text breaks the contract.

s519
00:40:11.840 --> 00:40:14.360
It's not exactly one-to-one.

s520
00:40:14.360 --> 00:40:20.520
So how do we fix that without adding a lot of tight coupling to everything that we're doing.

s521
00:40:20.520 --> 00:40:24.560
The answer is those system one models, which we'll get into here.

s522
00:40:24.560 --> 00:40:25.200
Yeah, absolutely.

s523
00:40:25.200 --> 00:40:32.120
So, just to introduce it for anyone that hasn't heard of it before, the you know, concept of system one, system two comes from psychology actually.

s524
00:40:32.120 --> 00:40:36.120
It's like a useful model to think about how the human brain works and how he thinks.

s525
00:40:36.120 --> 00:40:38.360
So, system one is super fast decisions.

s526
00:40:38.360 --> 00:40:39.760
It's It's, you know, autonomous.

s527
00:40:39.760 --> 00:40:45.480
Like, someone swerved at me while I'm driving and I just corrected without even thinking about it kind of thing, right?

s528
00:40:45.480 --> 00:40:48.720
Some I was about to lose my balance and I I corrected it.

s529
00:40:48.720 --> 00:40:52.240
And then system two is like I need to figure out what I'm going to do with my life.

s530
00:40:52.240 --> 00:40:55.623
So, I'm going to go book an Airbnb and like think about it for 3 days.

s531
00:40:55.623 --> 00:40:56.000
[laughter]

s532
00:40:56.000 --> 00:41:03.600
And so, you know, applied to the world of LLMs now we've got Jev and then immediately after we've got Cleft from Cloudflare, you know, 2 days later.

s533
00:41:03.600 --> 00:41:04.520
That's even faster.

s534
00:41:04.520 --> 00:41:14.200
And so, there's It's It's a really exciting time to think about all the ways that we can combine these tools together to make the new primitives that we're going to continue to see in evolving AI architectures.

s535
00:41:14.200 --> 00:41:14.760
Yeah.

s536
00:41:14.760 --> 00:41:19.600
To get Put it another way, that light right over there, if I said, "What color is that projecting?"

s537
00:41:19.600 --> 00:41:20.480
Quickly you would say yellow.

s538
00:41:20.480 --> 00:41:21.800
That's like system one thinking, right?

s539
00:41:21.800 --> 00:41:23.400
You just You know the color.

s540
00:41:23.400 --> 00:41:25.280
If I said, "Why is it projecting yellow?"

s541
00:41:25.280 --> 00:41:26.880
Well, because we're thinking.

s542
00:41:26.880 --> 00:41:32.040
Now we're thinking, you know, we could go down like a physics of why the light emits yellow.

s543
00:41:32.040 --> 00:41:32.958
Why are we here?

s544
00:41:32.958 --> 00:41:34.520
[laughter]

s545
00:41:34.520 --> 00:41:39.080
That's how we got to really what we want to talk to you about today.

s546
00:41:40.880 --> 00:41:46.280
So, in this we've got another game that we can play within within this.

s547
00:41:46.280 --> 00:41:48.640
It's a quick game with the snack.

s548
00:41:48.640 --> 00:41:51.640
We have saboteur and a detective.

s549
00:41:51.640 --> 00:41:55.800
And so, in this the the saboteur will pick a snack.

s550
00:41:55.800 --> 00:42:00.720
It'll put a secret bug in there, randomly putting a bug in there.

s551
00:42:00.720 --> 00:42:01.400
We don't know which.

s552
00:42:01.400 --> 00:42:02.440
It'll just be random.

s553
00:42:02.440 --> 00:42:07.160
And then you, the detective, find it with with snack alone.

s554
00:42:07.160 --> 00:42:09.040
And then prove what that is.

s555
00:42:09.040 --> 00:42:11.880
Name the bug and give give proof that you found it.

s556
00:42:11.880 --> 00:42:21.560
So, this is something that you can do if you've pulled down the website or pulled down the repo, um you can just run pnpm snack up --bug random and

s557
00:42:21.560 --> 00:42:25.120
uh give it uh let it run and see if you can find it.

s558
00:42:25.120 --> 00:42:25.920
So, let's try that.

s559
00:42:25.920 --> 00:42:29.560
We'll just give it some time to go.

s560
00:42:30.120 --> 00:42:31.880
Uh what was it?

s561
00:42:31.880 --> 00:42:32.600
--bug

s562
00:42:32.600 --> 00:42:34.640
bug random

s563
00:42:42.000 --> 00:42:44.560
Oh, it's already running.

s564
00:42:44.560 --> 00:42:46.160
Let me kill that.

s565
00:42:46.160 --> 00:42:48.640
Kill the right one.

s566
00:42:58.360 --> 00:43:00.040
Okay.

s567
00:43:00.040 --> 00:43:06.200
So, [clears throat] in this uh it ran uh it's got the app URL, it's running, it's not headless, so we can see it.

s568
00:43:06.200 --> 00:43:10.000
The bug uh says hidden until you stop up.

s569
00:43:10.000 --> 00:43:11.200
Yeah, right.

s570
00:43:11.200 --> 00:43:15.760
Um leave this running in another terminal, run pnpm snack doctor.

s571
00:43:15.760 --> 00:43:18.200
So, let's run that.

s572
00:43:21.320 --> 00:43:27.160
All right, so that ran the whole thing and we can see node's there, chrome's there, it's got a session.

s573
00:43:27.160 --> 00:43:37.040
The proof here, uh the last receipt says fail, but the source is now uh that proof is for code that no longer exists, so it wasn't able to prove that.

s574
00:43:37.040 --> 00:43:38.480
That was a previous run.

s575
00:43:38.480 --> 00:43:39.800
Let's run it again.

s576
00:43:39.800 --> 00:43:41.880
Same thing.

s577
00:43:41.880 --> 00:43:44.480
Uh can anyone tell me what the bug is?

s578
00:43:44.480 --> 00:43:46.600
Any ideas?

s579
00:43:53.560 --> 00:43:55.800
Wrong email?

s580
00:43:58.880 --> 00:44:00.400
Yep, exactly.

s581
00:44:00.400 --> 00:44:02.640
Port's wrong.

s582
00:44:06.040 --> 00:44:12.000
Uh I think it was running at 62174.

s583
00:44:12.120 --> 00:44:14.520
No, that was right.

s584
00:44:14.800 --> 00:44:16.480
Let me try it one more time.

s585
00:44:16.480 --> 00:44:20.240
We got less than a minute.

s586
00:44:39.840 --> 00:44:41.800
Uh, for that one, yeah, you do.

s587
00:44:41.800 --> 00:44:49.720
The tools make it optional though, so you can run through all the exercises if you don't have it and it'll just go, but um, it's also good opportunity to experiment with one if you do.

s588
00:44:49.720 --> 00:44:50.040
Yeah.

s589
00:44:50.040 --> 00:44:52.320
They're they're pretty cheap calls.

s590
00:44:52.320 --> 00:44:57.920
Uh, they give you $5 for free when you sign up and I think that'll last you like 3 years.

s591
00:44:57.920 --> 00:44:59.920
Yeah.

s592
00:45:02.040 --> 00:45:04.160
All right.

s593
00:45:06.960 --> 00:45:09.360
So, [clears throat] um, yeah, go ahead.

s594
00:45:09.360 --> 00:45:10.120
Next one.

s595
00:45:10.120 --> 00:45:11.280
Yeah.

s596
00:45:11.280 --> 00:45:14.280
Um, so, yeah, we ran it a couple times.

s597
00:45:14.280 --> 00:45:16.360
I think there was some problem with my output.

s598
00:45:16.360 --> 00:45:18.240
I'll debug that.

s599
00:45:18.240 --> 00:45:24.320
Uh, it's funny, I need some verification tests around my verification my contrived verification examples.

s600
00:45:24.320 --> 00:45:38.000
Um, but the, you know, the the outcomes that could come up are uh, that it named the bug uh, with output, uh, said it works but it didn't, uh, said it works but it did,

s601
00:45:38.000 --> 00:45:44.000
uh, blamed a bug that wasn't there, um, and then had no, it was the right bug or no evidence.

s602
00:45:44.000 --> 00:45:44.280
Right.

s603
00:45:44.280 --> 00:45:51.560
And this is just getting back to that point of uh, a contrived example, of course, but your team might score differently or have different criteria that you're looking at.

s604
00:45:51.560 --> 00:45:52.640
This is super flexible.

s605
00:45:52.640 --> 00:46:03.920
The most important thing to do is to start and to create one, create an eval baseline and to use it consistently and to evolve it together as a team, and then get to the point where your changes and your PRs automatically flex that

s606
00:46:03.920 --> 00:46:10.760
eval so that you can get to the point where you know if changes are safe to land or if they're improving things or if they're making things worse or if they're not changing things at all.

s607
00:46:10.760 --> 00:46:12.760
Yep.

s608
00:46:13.800 --> 00:46:17.560
Uh and every time it runs and it misses, um that's a new line that can go in the map.

s609
00:46:17.560 --> 00:46:26.880
So, it can give us ideas of like the steps that we're getting there, why we're getting there, and kind of like you know, every time it fails, it's getting us one step closer cuz we can fix that.

s610
00:46:26.880 --> 00:46:32.600
So, the first time uh is memory only, um like it's not storing anywhere but in memory.

s611
00:46:32.600 --> 00:46:34.680
Uh we're putting in a wrong email address.

s612
00:46:34.680 --> 00:46:36.480
Uh we've lost our confirmation.

s613
00:46:36.480 --> 00:46:42.160
Every time we can prove that we've caught that and then we don't run into that again.

s614
00:46:43.120 --> 00:46:53.920
Um but then the over promised one, uh the screenshot or the judge running uh can tell us that it um it actually did pass when it didn't.

s615
00:46:55.160 --> 00:46:56.440
Same thing with console error.

s616
00:46:56.440 --> 00:47:04.600
Uh there might not be anything in the console, um but it's in the receipt or it might not be it might be getting caught um before it gets to the console.

s617
00:47:04.600 --> 00:47:07.720
Um yeah, anything like that.

s618
00:47:07.720 --> 00:47:08.560
Or it could just be none.

s619
00:47:08.560 --> 00:47:09.560
There's nothing to find.

s620
00:47:09.560 --> 00:47:10.600
Everything works.

s621
00:47:10.600 --> 00:47:14.680
Um and that's a real answer as well.

s622
00:47:18.200 --> 00:47:26.000
Um yeah, if you run the doctor, kind of like what we were doing, it was showing this where it was showing the previous uh examples.

s623
00:47:26.000 --> 00:47:28.800
Um it's showing proof that no longer exists.

s624
00:47:28.800 --> 00:47:32.960
Uh that's because your code might have changed quite a bit between yesterday and today.

s625
00:47:32.960 --> 00:47:39.000
Um it might be different circumstances, like a a famous date bug, um for example.

s626
00:47:39.000 --> 00:47:40.840
It might be a number of different things.

s627
00:47:40.840 --> 00:47:50.000
So, uh we want to build the tools in such a way that they're flexible enough to uh not get caught by um issues like this.

s628
00:47:50.000 --> 00:48:02.880
Uh this is all kind of like going back to you know, basic unit testing examples, uh but bringing that kind of like deterministic check to the models in a way that they can self-heal and self-improve

s629
00:48:02.880 --> 00:48:03.640
from there.

s630
00:48:03.640 --> 00:48:05.640
Absolutely.

s631
00:48:06.618 --> 00:48:08.160
[clears throat and cough]

s632
00:48:08.160 --> 00:48:12.200
Okay, so this again to tie it back to what we used to do as human developers, right?

s633
00:48:12.200 --> 00:48:16.280
What we used to be able to see, how we would flex the application, how we would check everything.

s634
00:48:16.280 --> 00:48:19.280
Uh, that's really what we've been talking about and moving it into code.

s635
00:48:19.280 --> 00:48:31.640
So, by moving it into code and making it available in a deterministic CLI, then we can get to the same path and the same experience that we would flex as, you know, developers when in Chrome trying to fix something late at night.

s636
00:48:31.640 --> 00:48:41.320
And that's really what unlocks the the speed and the evolution and the ability to scale up the amount of code that you're delivering and to do that in a more safe way.

s637
00:48:41.800 --> 00:48:52.640
Uh, again, you know, if you uh, the recommendation is if you have a web app uh, and you're not already you don't already have a custom verifier CLI, that's one of the the highest leverage things you could start with for your

s638
00:48:52.640 --> 00:48:57.560
uh, code base, the foundational eval as well that shows what good looks like in your application.

s639
00:48:57.560 --> 00:49:06.760
And then also giving your agents the ability to see all aspects of performance and uh, networking and and traffic and and rendering, et cetera.

s640
00:49:06.760 --> 00:49:11.240
And so, giving it Chrome dev tools if you're in um, if you're in the web app land.

s641
00:49:11.240 --> 00:49:20.760
If you're shipping native clients if shipping iOS, you know, or OSX apps, then there's an alternative uh, projects and more coming out every day that you're then going to attach to there.

s642
00:49:20.760 --> 00:49:30.320
But the the the common, you know, foundation here across all of them is that you need to give your agent the ability to see absolutely everything that you would normally see as a developer.

s643
00:49:30.320 --> 00:49:37.560
Yeah, the more that you can give it, the more arms or eyes that you can give it, the more it's going to be able to take over from you.

s644
00:49:37.560 --> 00:49:41.480
Uh, and then you just have to give it the intuition about when to use those, how to use those.

s645
00:49:41.480 --> 00:49:46.760
Think about the way that you decide, okay, if I this bug came in, how would I reproduce this?

s646
00:49:46.760 --> 00:49:56.440
Think about that and if you can break it down and this is dependent on your application and the type of of code that you're writing, um, you can you might be able to do that in a way where it can like think, "Oh,

s647
00:49:56.440 --> 00:49:58.600
you know, for us it's a lot of sign-on things.

s648
00:49:58.600 --> 00:50:00.960
Like, is it properly setting a cookie?

s649
00:50:00.960 --> 00:50:03.440
Is it setting an expiration in that like all of that?"

s650
00:50:03.440 --> 00:50:04.520
the in the job, right?

s651
00:50:04.520 --> 00:50:10.200
You can go again like the this is applicable to every, you know, field and industry that you've got software for.

s652
00:50:10.200 --> 00:50:19.200
Um but it's it's really just all about like, "Hey, that world where it made sense for us all to hand-chisel code all the time is at least evolving or at least changing rapidly.

s653
00:50:19.200 --> 00:50:25.720
What are the foundational like what's the what are the same system fundamentals that we bring to this now so that we can accelerate software development?"

s654
00:50:25.720 --> 00:50:27.720
Yep.

s655
00:50:29.760 --> 00:50:30.880
Uh so, yeah.

s656
00:50:30.880 --> 00:50:32.280
Doing expensive call stacks.

s657
00:50:32.280 --> 00:50:44.240
Uh but the the important thing is that uh getting that output, making sure that it's giving you detailed output about what happened, uh the state of it, uh all of that so that it can be audited by the application

s658
00:50:44.240 --> 00:50:53.680
uh by the model, and then you can run it again, it can fix it, run it again, and verify that those changes actually took hold, and that there's no longer an issue.

s659
00:50:53.720 --> 00:50:57.880
Uh and yeah, and again uh can't overemphasize the value of keeping receipts.

s660
00:50:57.880 --> 00:51:01.160
So, if you have runs, uh you know, it can be the entire user journey.

s661
00:51:01.160 --> 00:51:02.280
Let's say it's an e-commerce app.

s662
00:51:02.280 --> 00:51:05.760
It's like when they log in, they hit the page, what happened, what was their flow?

s663
00:51:05.760 --> 00:51:06.720
Did they see an error at all?

s664
00:51:06.720 --> 00:51:11.560
Did they see 7 seconds of time to see your catalog, you know, come into view?

s665
00:51:11.560 --> 00:51:13.320
Uh were they able to put something in their cart?

s666
00:51:13.320 --> 00:51:14.760
Was there cart abandonment?

s667
00:51:14.760 --> 00:51:25.520
Um and so, tr- save that all now because every single one of those runs and receipts can be fed to LLMs, or you can even set up a system that's running all night long and is looking for

s668
00:51:25.520 --> 00:51:32.280
uh the latest receipts being produced, and is aggregating them, and saying, "Well, today we saw eight failures in the caching module.

s669
00:51:32.280 --> 00:51:35.840
So, I'm obviously going to, you know, issue a PR tonight to fix that properly."

s670
00:51:35.840 --> 00:51:43.120
And then, "Oh, I have the tools I need to prove in this PR that the issue I was trying to solve has been fixed."

s671
00:51:43.120 --> 00:51:49.920
Uh and then the recommendations that we're seeing from from folks, too, that are achieving like super high scale PR uh shipping without errors.

s672
00:51:49.920 --> 00:51:55.960
They're they're also able to, you know, add uh the stuff that is for us as humans in our oversight still.

s673
00:51:55.960 --> 00:52:01.360
So, not just screenshots of the feature, but like a recording of I walked through the app this way.

s674
00:52:01.360 --> 00:52:02.080
This was my flow.

s675
00:52:02.080 --> 00:52:03.320
This is what I experienced.

s676
00:52:03.320 --> 00:52:05.120
These were the states and I got to the goal.

s677
00:52:05.120 --> 00:52:14.200
So, that if I'm waking up and trying to rip through, you know, 15 important PRs in the the first thing in the morning, I can first start watching that and make sure okay, it actually got to the end and worked.

s678
00:52:14.200 --> 00:52:19.280
Then I can, you know, start looking at code uh if it's something that I'm actually going to like human review.

s679
00:52:19.280 --> 00:52:28.560
And uh again, all of that I think is going to be immensely valuable as we continue to see uh evolve self-evolving tooling, really.

s680
00:52:29.360 --> 00:52:37.720
And you know, think about what that means though because when we get to the point where there's self-evolving tooling is just like the default and people are using it, uh

s681
00:52:37.720 --> 00:52:45.960
there will be a specific advantage for folks that have preserved uh clean data sets of things that they can say this is what we are trying to get to in this application

s682
00:52:45.960 --> 00:52:49.000
and want you to get all the way there and rapidly improve it, right?

s683
00:52:49.000 --> 00:52:56.000
If you're not sitting on a mountain of that and your competitors are and they have similar apps, they're going to be able to go significantly faster in their development cycle.

s684
00:52:56.000 --> 00:52:56.560
Yep.

s685
00:52:56.560 --> 00:53:05.240
And this is just building up that trust so that we can continue to improve upon our app as we go, uh but we have these checks for everything that came before it.

s686
00:53:05.240 --> 00:53:15.120
So, we've got unit tests, we've got evals, we've got smoke tests, we've got whatever our application needs and we've got a way for the model to be able to interpret and run those and improve them

s687
00:53:15.120 --> 00:53:23.280
uh in such a way that if we add new features in, we can run all of the previous ones and make sure that we didn't actually break anything, uh no regressions,

s688
00:53:23.280 --> 00:53:32.280
and we can just mount and climb that evidence ladder, building evidence as we go, uh and ensure that our app is working uh exactly as expected.

s689
00:53:32.280 --> 00:53:37.360
And that really becomes the the spec that shows what our app does.

s690
00:53:38.520 --> 00:53:38.920
Okay, cool.

s691
00:53:38.920 --> 00:53:40.840
So, we've got uh just over 5 minutes.

s692
00:53:40.840 --> 00:53:42.800
I'm going to pause and see if there's additional questions.

s693
00:53:42.800 --> 00:53:43.840
Yes, sir.

s694
00:53:43.840 --> 00:53:54.040
So, does the scale I I would personally I'm curious with how you think about it.

s695
00:53:54.040 --> 00:54:02.400
I would I would personally almost lean away from skills cuz in my experience it's it's you know up [clears throat] to the model whether or not it's going to actually follow that path and use it

s696
00:54:02.400 --> 00:54:08.680
versus like having the CI actually enforce that these these commands get run as well.

s697
00:54:17.240 --> 00:54:19.720
Yeah, that's what

s698
00:54:19.720 --> 00:54:23.920
Like the answer or the question is how do you run these these checks normally, right?

s699
00:54:23.920 --> 00:54:25.920
Effectively.

s700
00:54:26.522 --> 00:54:28.522
[clears throat]

s701
00:54:37.200 --> 00:54:40.280
Yeah, it can you can run those you can run these regularly.

s702
00:54:40.280 --> 00:54:42.600
You can have them be like part of CI for example.

s703
00:54:42.600 --> 00:54:49.440
You can have them be various hooks that could be like a get hook although that's probably the last one I would reach for.

s704
00:54:49.440 --> 00:54:51.000
We could have it be like a cloud code hook.

s705
00:54:51.000 --> 00:54:53.720
It depends on what your team is using and how they're they're doing it.

s706
00:54:53.720 --> 00:55:01.800
I might build like a full pi extension to keep things running in the background and just kind of like feed in information as it's going.

s707
00:55:01.800 --> 00:55:11.120
There's a number of different ways that you can do that but the easiest way by far to get everyone across the board and I can say this now just put an agents.md

s708
00:55:11.120 --> 00:55:21.840
not a cloud.md in your in your app and then have it say like you know on change verify things by running this or like every time you change significant things or before you open a PR do this.

s709
00:55:21.840 --> 00:55:30.080
Like whatever wording that you want to put in there and then that's an easy way to get everyone across the board and they really kind of have to go out of their way to

s710
00:55:30.080 --> 00:55:31.880
ignore that or avoid it.

s711
00:55:31.880 --> 00:55:34.440
Hooks are also relevant too if you're doing the local development, right?

s712
00:55:34.440 --> 00:55:38.240
For the whole team cuz you can share hooks across the board.

s713
00:55:46.440 --> 00:55:47.560
Yeah, absolutely.

s714
00:55:47.560 --> 00:55:47.640
Yeah.

s715
00:55:47.640 --> 00:55:55.760
Well, CI is a is a I think a feature of it, but it's not like the the actual tool is like the rapid development that you get locally.

s716
00:55:55.760 --> 00:56:02.000
So, I give my agent a task to do and I can just quickly run uh like let the agent go.

s717
00:56:02.000 --> 00:56:03.760
You know, I basically tell it this is where I want to end up.

s718
00:56:03.760 --> 00:56:11.560
I want the waitlist to be I want to be able to add an email to the waitlist and then it gets persisted uh and saved and I can verify that.

s719
00:56:11.560 --> 00:56:12.880
Uh so, this is how you verify it.

s720
00:56:12.880 --> 00:56:14.280
It goes into this table, blah blah blah.

s721
00:56:14.280 --> 00:56:15.760
Like whatever it might be.

s722
00:56:15.760 --> 00:56:18.280
Um Yeah.

s723
00:56:18.280 --> 00:56:19.680
No, good question.

s724
00:56:19.680 --> 00:56:21.680
Sure.

s725
00:56:38.120 --> 00:56:51.800
Yeah, what's the benefit of running it in CI versus like a local Uh the benefits are definitely like a faster feedback loop, which allows us to run faster and allows me to

s726
00:56:51.800 --> 00:56:55.320
uh parallelize myself because I'm no longer working on just one thing at a time.

s727
00:56:55.320 --> 00:56:58.440
I'm working on at least right now I have three agents running.

s728
00:56:58.440 --> 00:57:01.120
Uh that's a very low day for me, but I'm at a conference.

s729
00:57:01.120 --> 00:57:02.414
Um

s730
00:57:02.414 --> 00:57:02.840
[laughter]

s731
00:57:02.840 --> 00:57:04.400
Normally, it's like 10 to 15.

s732
00:57:04.400 --> 00:57:06.760
He's really taking it easy today.

s733
00:57:06.760 --> 00:57:13.560
Um and and the the whole idea is like I'm going to set it off and I want I don't want it to be waiting on me and saying, "Yep, I did it."

s734
00:57:13.560 --> 00:57:16.160
and then I go running and it's like, "No, that's not actually correct."

s735
00:57:16.160 --> 00:57:22.040
and then I have to like set it off again and I go get distracted by another agent again and I just continue that loop.

s736
00:57:22.040 --> 00:57:24.120
I'm just like constantly moving between projects.

s737
00:57:24.120 --> 00:57:33.560
So, I want to give it the tools to where it can prove to itself that it did it and then prove to me that like show me the evidence in a way that I can trust

s738
00:57:33.560 --> 00:57:40.680
uh so that when I come back to it it might be in 5 minutes it might be in an hour or a couple of hours I want to know that it has done

s739
00:57:40.680 --> 00:57:51.040
the best possible job that it can to get done to a state of done so that I'm not just like re-prompting it and starting that whole loop again because I don't want to be that's that's not the fun part of my job.

s740
00:57:51.040 --> 00:58:01.120
Yeah, I would I would say another way to to put it is that and we haven't everyday achieved this perfectly but I want to have all of the different lanes that I started that day for stuff that I felt was important to ship that day.

s741
00:58:01.120 --> 00:58:05.600
I want to come back to them at the end and see them basic like correctly done.

s742
00:58:05.600 --> 00:58:11.720
What I don't want to do is have a whole long day and have them running in the background be doing other stuff and going to meetings and then come back

s743
00:58:11.720 --> 00:58:19.600
and then actually have to turn by turn nudge any of them because that is like a significant slow down and it it prevents me from going faster.

s744
00:58:19.600 --> 00:58:30.400
And another way to put this is I'm like I rely on CI and the tests that are running in CI and the code reviews like we how many people do automated code reviews with like Reptile, Code Rabbit, Codo, any of those?

s745
00:58:30.400 --> 00:58:31.160
Yeah.

s746
00:58:31.160 --> 00:58:31.920
I rely on those.

s747
00:58:31.920 --> 00:58:32.720
We use Reptile.

s748
00:58:32.720 --> 00:58:42.160
Reptile gives me a X out of five score and if it gives me a five out of five and all of my CI is passing I feel extremely confident that I'm not going to break production

s749
00:58:42.160 --> 00:58:43.800
when this gets shipped.

s750
00:58:43.800 --> 00:58:46.280
We've gotten to that level of trust with Reptile.

s751
00:58:46.280 --> 00:58:52.160
The more important thing for this is that I get to that state faster and that um

s752
00:58:52.160 --> 00:58:53.480
with less turns

s753
00:58:53.480 --> 00:58:54.800
With less turns, yes.

s754
00:58:54.800 --> 00:59:03.640
And that you know, I I might add a whole like I could add 10,000 lines of absolutely nothing and it does nothing and great it passed.

s755
00:59:03.640 --> 00:59:08.960
Reptile gave it a five out of five for some terrible reason and then it goes into production and it doesn't do anything it doesn't break anything.

s756
00:59:08.960 --> 00:59:11.480
Now it's just an extra 10,000 lines of nothing.

s757
00:59:11.480 --> 00:59:22.360
Um and so this is also to help me make sure that like I'm I'm telling the agent like exactly what I want and it gets there without going through a bunch of um

s758
00:59:22.360 --> 00:59:32.160
additional code to or additional like like it stays on track uh but it also knows how to fix itself to keep itself on track while I'm off doing something else.

s759
00:59:32.160 --> 00:59:37.200
Okay, we might have time for one more quick question, but I think we're Yes, yes, sir.

s760
00:59:41.720 --> 00:59:42.720
Yes.

s761
00:59:42.720 --> 00:59:44.720
Yes.

s762
00:59:53.040 --> 00:59:55.040
Yeah.

s763
00:59:55.600 --> 01:00:05.000
You could also You could also do depending on your stack like there's it's also you know there's various providers that you could easily also go to preview first and make and then verify it there in a different way.

s764
01:00:05.000 --> 01:00:07.560
So we'll then go to prod and then also immediately monitor.

s765
01:00:07.560 --> 01:00:18.640
It's another pattern we're starting to see emerge is that when folks have a deployment that was mostly agentically implemented, there's also at least one agent specifically concerned with that production rollout

s766
01:00:18.640 --> 01:00:22.840
looking for watching all the metrics, watching DataDog, watching feature flags, right?

s767
01:00:22.840 --> 01:00:26.840
Being able to say this is this is actually bad and we're we're reverting it.

s768
01:00:26.840 --> 01:00:38.240
Um so I I think that like I I truly understand and get the like queasiness around it like we all have or dealing with that, but I I I do see a path for that um because it just seems like it seems inevitable.

s769
01:00:38.240 --> 01:00:38.640
Yeah.

s770
01:00:38.640 --> 01:00:39.200
Yep.

s771
01:00:39.200 --> 01:00:42.200
That's all the time we have, but if you have more questions we'd be happy to chat.

s772
01:00:42.200 --> 01:00:47.000
We'll be in the expo hall booth six right to the right when you come in.

s773
01:00:47.000 --> 01:00:47.560
Come find us.

s774
01:00:47.560 --> 01:00:52.920
We'd be happy to chat to chat, show you what we're working on and talk to you about anything you'd like.

s775
01:00:52.920 --> 01:00:53.160
Yeah.

s776
01:00:53.160 --> 01:00:55.360
Thank you so much for coming and your attention and everything.

s777
01:00:55.360 --> 01:00:56.907
Thank you so much.

s778
01:00:56.907 --> 01:00:58.907
[applause]