Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS

AI Engineer · 61 min · 778 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hey, welcome.
00:14Welcome to Beyond Done.
00:16Uh we're going to get started here with this workshop.
00:18We got a lot to pack in and uh we want to make sure that we have time for questions and um uh getting getting like everyone situated.
00:26So, we're going to go ahead and get started.
00:29[clears throat and cough]
00:30So, what we mean by Beyond Done, just to like jump straight into it, um you probably saw a tweet about this incident that happened a while back uh where
00:39an agent just started deleting production databases.
00:43And in 9 seconds, everything was gone.
00:46Um that is what we want to help you prevent here today uh while still trusting your agents and not having to look at every line, even look at every PR.
00:56Like this is where we think things are going and how do we get there?
00:59How do we make this a reality where instead of having that broken piece, we can have you calm, have uh everything work, and let you to some degree trust your agents, but build that trust over time.
01:12I'd say also that it's not just about preventing errors, right?
01:15But this is also what we're going to look at today foundational to achieving true scale with it with agents as well.
01:21So, I'm Nick Nisi.
01:22Uh I'm an engineering manager at WorkOS.
01:24Uh I live in Omaha, Nebraska and uh I've been there for about 2 years.
01:28And I do a lot with agents.
01:30Uh I haven't written code since July 30th, 2025.
01:33I know because that's the day I got back from Laracon and I saw PHP and I was like, "Never again."
01:38And uh here I am.
01:40Uh and with me is my best friend, Zach.
01:42Hey, I'm Zach Prosser.
01:43I'm an engineer on the Applied AI team at WorkOS.
01:46Super excited to be here with you all today in New York.
01:51[clears throat]
01:51Now, uh we had this up before.
01:53This repo is your workbench.
01:55Um all of the slides are in this repo.
01:57You can get it from here.
01:59You can also view it at um workos.cloud/umbeyond-done.
02:06Uh, I know that's a mouthful, but the next PR or the next QR code, I believe, will take you there.
02:11Um, sorry, talks.workos.cloud/beyond-done.
02:15Uh, and I just wanted to call it out because on that, I'll switch to it real quick.
02:19Um, all of the slides are out there.
02:21There's also a page for the workshops, like the the exercises that we're going to go through.
02:27And so, for each piece of it, you can go through it.
02:29You can go through it on your own time.
02:30You can go through it later.
02:31This is a a step-by-step guide to go through that.
02:34There's also a glossary uh, covering all of the terms, big and small, uh, some that you're familiar with.
02:41Uh, hopefully most of that you're familiar with, but if there's any that you're not, uh, you can talk in here or sorry, you can look at the the glossary
02:49for those terms.
02:50There's also an ask anything button, and you can talk to a reg that's trained on this uh, slide deck, and uh, hopefully get your answers there.
02:57But, you can also just talk to a human, us.
03:00Indeed.
03:00Yeah, we'll leave leave this up as well.
03:02We find that a lot of folks want to ask questions and for whatever reason can't, so be sure to check that out and use that.
03:08Cool.
03:08Give it just another second.
03:09I see some cameras pointed at QR codes.
03:12Security would be so mad at us.
03:17All right.
03:17Um, so when we're thinking about uh, being done with work, uh, agents being done, uh, there's lots of things that go into us trusting that.
03:28And easily the first one is just is the code right?
03:32Is the code what I would have written?
03:33Did it generate a good PR?
03:36And then I go through that like a normal normal code reviewer.
03:40Um, but that takes a long time.
03:43And these agents can write code a lot faster than us, so we don't want to rely on that too much because it's uh, it just there a a signal-to-noise ratio that starts coming into play and we just have too much code to review.
03:56We're not writing any code anymore.
03:58And we're less familiar with that code, so we become further abstracted away from it.
04:02So, we need to find other ways to do that.
04:04And there's plenty of ways that you can do that as well.
04:05There's you know, using tools like browser tools like Playwright for example, to say like, "Hey, I have this website and I submitted a form.
04:15The submit worked.
04:16I checked that and it showed me a success message.
04:19That's a great test to write.
04:21Not even really anything agentic about it, but it is something that agents can use to verify their work.
04:26And they can self-report that.
04:28And you can trust that often times, but not all the time.
04:32And so, that's really what we want to drive is like, how do we make it so that we can actually trust these reports?
04:39Absolutely.
04:40So, everything that we've got here for you today in this workshop as Nick mentioned is backed by a repo and it is interactive, so be you know, feel free to take an opportunity to actually run this to experiment with it to take something back that you might you know,
04:52use at work or at least experience in a different way.
04:55Yep.
04:56In this so like if you just go into the repo like I'm doing and you do PNPM snack if I can type snack up uh This is why I have shortcuts for everything.
05:08P is my PNPM.
05:10Um it will pop up the browser and we can look at it and we can see this Midnight Snack Club app that we made.
05:20Um and you can play with it right now and I can test this myself.
05:23Let me just put in my email address.
05:25I'll put in um zach@spamhim .com.
05:31Thanks, buddy.
05:31Yep.
05:32Uh and look at that, success.
05:35It saved the email.
05:35The email is here.
05:37I can see it.
05:37If I had a test that was looking to confirm that the email went on the page it worked great.
05:43So, that's what these examples look like.
05:44You can kind of see it um, as you're going.
05:47Uh, but if you reload that page, just like this, nothing's there.
05:53So, even though, oops, what did I do?
06:01Oh.
06:02Switch back.
06:02Uh, if you reload that, uh, it shows that nobody's here.
06:06I can't drive Chrome, apparently.
06:07Um, and that's because it did update, but it didn't persist that anywhere.
06:13So, we didn't have any kind of verification.
06:14We're just checking UI, seeing, oh, did that actually do anything?
06:18And yes, it did do something to the UI, but we didn't check the other half, and so it was never persisted, and we lost that email.
06:25So, sorry, Zach, you're not signed up.
06:27It's okay with me.
06:28Um, yeah, I think this is a really critical point.
06:30So, uh, just show of hands, who in this room is is building with a ton with agents, just to start with the first question.
06:38Okay, awesome, like more than half of the room.
06:40Uh, how many folks are running multiple terminal tabs or multiple chat sessions or multiple agents, like everybody that put their hands up the first time, right?
06:46And so, this is what we're all experiencing, at least what I'm experiencing, what Nick's experiencing.
06:51If this is the 18th, you know, lane of work that you're driving today, and you have like, you know, diffracted attention to actually spend on it, these are the subtle bugs that will creep in because the agent's like, yeah, I did what you said, the code looks good, it builds, it runs, the test pass, and it's still
07:06wrong.
07:06And so, what we're going to ask and explore in this workshop together is, how do we actually fix this problem so that the agents that we're now all managing as like herds of
07:16insane token machines uh, can actually figure out what they need to do themselves and when they've made a mistake.
07:20Yep.
07:21And you'll notice that we are aren't really looking at the code for this.
07:23We're not really going to look at the code too much for this because, uh, like I said, I saw PHP and I'm scared, so we don't look at code anymore.
07:30[clears throat]
07:31Um, but all of this builds up to this great quote by, uh, a famous potato on, uh, X, call, uh, her her name is Lauren 10.
07:39Uh, and she has this quote, if any agent, sorry, if an agent can verify its own work, nothing can't verify if an agent can't verify its own work, nothing else matters.
07:47Yeah, another way to say it is is that and I think this is true in in my experience and others as well is that we're kind of converging on the point that verification now, especially as model
07:56capabilities advance rapidly is is one of the key foundational things to focus on, right?
08:00It's like reliability and verification and evals.
08:03Especially as we're starting to see more and more harnesses come online that have the concept of RSI or you know, recursive self-improvement baked into them.
08:11It feels like we're shifting to a world where we may all be maintaining multiple self-evolving code bases eventually, right?
08:18And in that world, how do we set the eval foundation so that the agent actually knows what good looks like and can do those hill climbs on its own and then how are we going to give the agent the feedback it needs to understand that it's failed something.
08:30Yeah, until we can solve that, we're not going to trust these agents to really accelerate our work to its fullest extent.
08:37And so we need to solve that in various degrees and it's not just going from zero to one, but it's a gradient and getting us closer to one to where we can fully trust these agents is is the goal.
08:49And I'll also just share that you know, again, anecdotally, this is what we're all experiencing at WorkOS.
08:55We've got an applied AI team that's building a ton of different stuff for other teams and for ourselves.
08:59And it is just a night and day experience when you are iterating on a code base that has, you know, the foundational eval in place, a baseline that can be run and trusted.
09:08You know, we'll get into some of the tips and tricks, but the ability to actually see that it did or did not do something correctly in the same way that we would as human developers, right?
09:17So we're, you know, of we've the the metaphor of us all becoming agent harness managers is is, you know, well trod.
09:24This is how we ensure that each one of them is actually doing correct work and we're not burning money and time essentially.
09:30Yeah, this is the equivalent of saying like make no mistakes, right?
09:34Yeah, if you want to have some verification in there, you can't just like guess at what it needs to be.
09:39You need to give it clear, defined goals to get to and give it a way to observe and check those goals.
09:45If not, the agent's going to do something and it's going to spend some tokens and it might get to the end that you expect it to, but there's no guarantee of that.
09:55Every for every detail that you don't fill in in terms of an outcome and what you actually want to get to, it's just going to infer that or make something up and
10:04that assumption that it makes may not be the assumptions that you make or that you were expecting because you're a rational logical human and you were just not aligned on that yet.
10:15And so um you need to have these systems in place to allow for it to observe, understand what you want, observe that, repair that and then recheck that.
10:26Yeah, and I think just again, anecdotally like we're all finding that you know, the tired way to do it is to have a giant massive PR and have one agent session that's running for three or four days and say fix this, fix this, fix this, make it better.
10:37And what is working, you know, essentially perfectly for folks is decomposing tasks really cleanly, giving the each task an agent in the prompt exactly what it needs to do to know that it is actually done and to verify the state of done.
10:51And those will just rip.
10:53I mean, we have people on our team that are 900 PR's a month and and no incidents and like improving KPIs for the software.
11:00So just anecdotally it is um something that we all kind of have to take a step back and think about.
11:07I'm not approaching this code base anymore in the traditional way.
11:10I'm almost gardening multiple code bases and letting them self evolve.
11:14Yeah.
11:16And this is something that we build and we hold our own tools too as well.
11:19This is how Zach and I stay productive at Work OS and how we get our own work done.
11:23We've each built kind of our own examples of this and our own ways of approaching this.
11:29And we really think that that's like a good thing right now.
11:31It's too early to really classify on anything, so experimentation is key, finding what sticks and then running with that.
11:37And uh one thing that we're both running with is the verification piece or like driving everything from that verification loop.
11:45Uh because uh as as it says in this quote here for for mine, uh status comes from records, not model pros.
11:51Um so it's it's, you know, show me the receipts.
11:54Show me the actual tangible proof that it actually did it uh what I expect.
11:58Don't just tell me, "Yeah, it passes, it looks good."
12:01Um because there's nothing to back that up.
12:03I need actual screenshots, videos, test results, smoke tests, um any all of the above to prove to me without me having to look at the code that it's actually doing what I want.
12:13Because at the end of the day, I don't care about the code as much.
12:15I care about the result and what it's doing and what value it's delivering.
12:19And it can have bad code, honestly, and deliver that result pretty well.
12:23Uh or it could have really good code that doesn't uh give me that result at all.
12:28And I need to find the right balance in there, which is giving me the result that I want, not the code.
12:34Yeah, indeed.
12:35Uh so I've been doing this as well.
12:36I I built a tool that I call context or CTX.
12:38It started out as a cross-agent harness memory layer cuz I'm rapidly experimenting, as all of us are, with like ChatGPT, Cloud Code, uh Pi, etc. And I wanted them to all be able to write and read from memories, and that started working.
12:50And then the more that I was running more more multiple agents and, you know, average day is like 18 or 30 terminal tabs, the more that I realized uh it might be an interesting experiment to try to centralize authority for verification
13:01in a web service.
13:01So the CTX is a simple web app that runs on Cloudflare.
13:04It was vibe coded initially.
13:06It works.
13:07Uh it started as memory.
13:08It was able to hand out secrets to agents without them ever seeing it or getting the credential.
13:12And that worked, too.
13:13And then now as an additional experiment, I'm creating verifiers.
13:16And the way that that works is the agents are assigned work, as we're all doing with prompts.
13:19They go off and do that work.
13:21They come back through the MCP connection to CTX and say, "This is what I've completed.
13:26Uh am I done?
13:27Please verify it.
13:28So, let's say in the example of um uh one of the workflows needs to verify that a PR was created correctly.
13:36And so, that is going to run an evaluator on a known verifier.
13:39That could just be a simple CLI to start with as it is in my repo, and then that becomes a hardened piece that you can reuse again and again.
13:45It's going to the GitHub API and verifying that that PR truly does exist and is healthy and that all the merge checks pass.
13:51And only then will it mark that node satisfied in the workflow.
13:55Otherwise, it will redirect to it will redirect the agent and say, "These are the things that are still failing."
14:01Right?
14:01And so, again, an experiment, but this is kind of how we've been trying to play around with this and see which patterns are are going to stick and and what works and what not.
14:09Um but it's also just kind of a fun time to do that five coding because uh I require uh weird pixel art across all of my apps.
14:15So, uh you can see in this case there was a simple workflow.
14:18And one of the verification gate is that requires that all pull requests were merged.
14:23And so, that work is done by the centralized authority.
14:25It goes out and uses its verifier, checks the API or whatever the artifact is, and then finally delivers the status over here on the right sidebar if you go to even down, and references the actual repository checked and the PR, right?
14:37So, simple example, but you could imagine having standardized verifiers in a library or in skills or in tooling that you share amongst your your org, and this could be one of the ways that you start to build toward accelerating,
14:47you know, PRs and and less and less human oversight.
14:49Mhm.
14:53Uh and then a similar tool uh that I built is called Ryker.
14:55This is more like a engineering-focused Grokbot or Muse uh that's trying to be extremely flexible in how you hand work to it, but then once you hand work to it, its job is to babysit it.
15:06And so, I don't want to have to babysit all of the PRs that I'm I'm bringing to fruition.
15:12At some point there's a like this level of diminishing returns for me and my focus on on the work that I'm doing because I end up just like babysitting the PR, you know, fixing the tests, fixing the the reviews
15:22uh that are coming in, the the review comments and things like that.
15:25So, I want to be able to get the idea down of like what we should be doing and how we can verify that that work is done uh and then I can hand it off to Riker.
15:36It can go do the work of, you know, making sure that the the reviews are done, uh that when it fixes code uh from PR reviews that it's still matching our overall goal of working software
15:47uh in some way, and then uh can just verify that and and alert me back when it needs me instead of me having to baby sit it.
15:55Um and so, with that it's just um you know, it builds a check, it figures out what tools it needs to run, it will run them, uh verify that they all pass, it'll keep that evidence and show it to me in different ways,
16:08uh and then decide when to stop.
16:10So, if it can verify yes, it was working, we've now fixed this bug, for example, um we put them together and it works.
16:17Uh so, it's really just keeping me out of having to be in the minutia of shipping the the like final 10% uh and just focusing on where I'm valuable as a human, which is why we're doing the work and how we're and uh to an extent how we're doing it.
16:36Okay, so maybe just as a mental experiment or a thought experiment, if you will, let's just kind of show of hands like or feel free to shout out like how would we start to think as a human developer
16:46of verifying a web app like maybe in the before times before an LLM.
16:49Yeah, what are things that you would do to verify?
16:52And we'll we'll throw some easy ones up here to get it started.
16:54So, like refresh the page, is the application still visible, right?
16:59What's that?
17:02Absolutely, yep.
17:02Open a new tab, see if it's healthy, right?
17:04What are the response codes I'm getting back from the API?
17:06Are they expected based on what I sent it?
17:09Absolutely.
17:10Local host or local storage, checking persistence in some way, console logs, um things like that.
17:17Network, uh, like you said.
17:18And, um, like reading what's actually on the page.
17:21Do the contents of the page actually
17:23Interacting with it, right?
17:24And the reason we're we're saying this is like this is really to set up the the transition from we were the human developers doing this all the time.
17:31This was part of our core responsibility when we're hand chiseling code, if you will.
17:35And then, what is the delta between doing this and then in the the early stages of LLM time saying, "Here's three paragraphs of text.
17:44Go execute these instructions."
17:46The delta between those two things is the agent's ability to verify that it actually did what you asked it to do in a real application and that its changes did not break anything.
17:57Right?
17:58Otherwise, you're just getting the confabulations that LLMs are designed to give you cuz they're they're trying to process the text.
18:03They're trying to make you happy.
18:08Uh, your agent can't do any of this just in a terminal alone.
18:11Uh, it needs tools to do that.
18:13Uh, fortunately, tools are very easy to add to
18:15agents now.
18:16Um, and we just have to tell them how to use it.
18:18We'll tell them specifically what we're trying to to prove.
18:22Um, when you do that, you don't want it to be just a paragraph of like, "Hey, do this very always verify the change in a browser."
18:30Whatever that means, that's pretty vague.
18:32Uh, "Remember to reload and check."
18:33That's a little bit more specific.
18:35Uh, "Check that the data persists."
18:37Look at the console for errors.
18:38Uh, getting a little more specific, but like that's a lot, um, to put in there and you can certainly do that.
18:43That's what a lot of skills do.
18:44That's what what a lot of just like saved prompts do.
18:47Um, but if you forget anything or things change, that can leave it up to a lot of interpretation.
18:52Um, one thing that might be better is to have a script that can prove it out in different ways, um, so that you can just run that over and over and it's doing more of a deterministic check across a number of things and then it can feed that output
19:07back into the agent and it knows exactly what was wrong uh, and how to fix it quickly.
19:12And it just closes that loop so that it's a much faster uh and the agent can just spin that, run it again, check, verify, run again, and just do that over and over until there's no more errors.
19:21It's also uniform if you think about it, right?
19:23Um we know that LLMs are non-deterministic, so even giving 30 agents the exact same prompt and you have a potential for variance in your outcomes.
19:30And pinning it down to a single command that is backed by actual bash commands and bash calls, right?
19:35Like that's that's significantly more deterministic.
19:38And anecdotally, the folks uh like Lauren Tan and others that are achieving thousands of PRs ostensibly a month and uh building ambitious products that were not possible to build in that time before
19:48have mentioned that this is the first thing that they they concentrated on that actually unlocked the velocity, right?
19:53So, if your specific web app, for example, does not already have a bespoke CLI that allows any agent to fully flex the functionality that you care about verifying, then you're going to have a much rougher time and an unpaved path if you're using prompts, you know, alone uh compared to actually,
20:11again, giving each agent the ability to say, "Yes, I actually built that feature.
20:14It is actually performant.
20:16It didn't cause any errors, and the application looks correct."
20:21So, for this workshop, uh we made a little tool.
20:24Uh we've been running it a couple of times, but it's called snack.
20:26Uh and you can just run it with PNPM snack in that repo.
20:29Uh it's not meant to be like a reusable tool, but just kind of like a template for how you could set up this uh specific for the applications that you're working on, and then use that as a way to tie into the LLMs that are actually working on that application with you.
20:42And so, snack can do a couple of things.
20:44It can uh manage the health of the application.
20:46It can bring it up, it can take it down.
20:48It can run a doctor command to see is it actually, you know, is the database running?
20:52Does it have network connection?
20:53Et cetera.
20:54Um that quick drive or that quick check that it can do.
20:57Uh then it has uh drive, so you can tell it to join.
21:01Remember, this is our our uh midnight snack waitlist.
21:04Uh so, it can join the waitlist.
21:06It can reload the page, it can reset all of the data so it can't find it.
21:10You can have it look up things, so take a snapshot, uh, look at the network storage, look at console logs, um, look at local storage, uh, screenshot it, run evals,
21:21all of all of the above.
21:22Uh, and then finally it can prove it in two ways.
21:24It can actually run a prove command where it's checking for deterministic output of that.
21:30And a judge, which is using a system one model to, uh, in like, get an idea of whether we're going in the right direction without it necessarily matching one to one.
21:39Just to, like, fully develop this idea, when you do something like this in a repo that you haven't already had something like this before, the velocity gains that you unlock are not solely for you, right?
21:49It's for everyone else on your team that then comes along and is also able to issue significantly more PRs significantly faster and significantly safer, right?
21:57More safely, so that, um, you know, everybody on the team actually benefits.
22:01And again, that's like where we we keep harping on this being like a foundational concept now.
22:06Yeah.
22:07Uh, and if you think about it, like, that's a really big piece of it it might even be more important than your software, uh, than the actual application is the way that you verify it, the way that you test it,
22:17uh, the way that you prove that it works because that is what you could use potentially, like, a lot of people are doing and converting to Rust.
22:25Um, you could take your app, uh, like the Bun team did, convert it to Rust, uh, Go, TypeScript, convert it to Go, uh, not necessarily with the same level of LLM help, but, um, they did use LLMs to help with that.
22:38Um, and then I think Theo just last week converted it to Rust again, so.
22:43Yep.
22:43Uh, when you want that, if you have all of those verifiers in place, then your source code doesn't matter as much.
22:47You can write it again and prove that it works.
22:50Keep going.
22:51Absolutely.
22:54And so, um, these tools give your agents their eyes.
22:57Or you if you think about it, like, the tools that your agent use, that gives it the body.
23:01The LLM is the brain and then the way that it reaches out and interacts physically, uh virtually on your computer is through these tools.
23:10And so, things like uh join is it like actually having it run and do things and then it can check and see, you know, did I did I that actually succeed in some way?
23:20Uh check storage, run reload, uh snapshot, etc. And so, let's just run a couple of these uh and see.
23:29P snack up.
23:30That brought up our um browser.
23:35And then I'll open up another browser or another terminal.
23:39I have too many terminals.
23:41And run um P snack snapshot.
23:47And that give us a snapshot of the page in as JSON that we can see.
23:51If I put um Zack at zack.zack uh and add that to the page.
24:01Hopefully, we'll see some Zacks in there.
24:03There we go.
24:04So, that's a way that we can verify.
24:06Uh we can also do screenshot to make it a little easier.
24:11That should pop up a screenshot.
24:17And yeah.
24:18So, that gave us the ability to, for example, not just like pass this to the LLM and have it do interpretation on it, but like proving back to us.
24:25That's the end goal, right?
24:27Is proving back to the human.
24:28Being able to take screenshots, record videos, put those on the pull request.
24:32You can do that with the the GH CLI now.
24:35Um Chrome debug debugging tools.
24:37Chrome debugging tools, yeah.
24:38Where is the actual bottleneck?
24:39Don't guess and, you know, spin your wheels for 3 hours.
24:42Like actually figure out which part was causing the render to be slow.
24:45Yeah.
24:46So, the real key here is that there's no um it's it's not just code that you can look at to interpret whether this this application is running correctly, but you can build tools that will help you prove to you that it's working correctly without you having to read the code at all.
25:01Uh because this is all we care about.
25:03Is this what a user sees?
25:04Is it correctly rendering in this browser at this width with dark mode, with light mode, etc. And you can run infinite tests on that and run them over and over and over in a loop,
25:14uh which as Zach said unlocks your whole team to move faster because it you've given the agent the ability to run those checks itself as well.
25:27So, [clears throat] if we run this uh if we ran it just like this where we ran the doctor uh and then we add uh ada@example.test and then we check the storage, it's there.
25:37We run uh if we check the storage, it's not there.
25:40Um we would see local storage is empty.
25:42And if we rerun reload and then snapshot, we would verify that it is in fact empty.
25:47So, this is a way that the tool itself could run in a loop, could check all of these things, and yes, confirm with before it even gets back to us.
25:55Oh, I see that we're not actually persisting this.
25:57I forgot to do that part.
25:58And the agent can just move on from there.
26:01Um real quick, we wanted to stop and take a a few questions if there are any.
26:06Yes, sir.
26:06Yeah.
26:08Uh so, the question is uh are we uh using models to write JSON uh to to then like help us to make our code deterministic, to bring that back.
26:21Um the answer for me is yes.
26:23Uh I'm I'm using that and I'm running it and then I use uh like I'll I'll use unit tests around that to verify in different ways, catch edge cases,
26:33uh and then verify that the code that that is like a real interpretation or real representation of what I'm expecting to get back from like a system one model,
26:42uh and use that to prove that that I'm on the right track.
26:46But, then have keep those as well to kind of like have a another check uh to make sure that it doesn't drift over time or things change.
26:54The the the general process that we're talking about?
26:56Yeah, so when what I'm talking about, uh, at least on our team, we we build a ton of applications that are internal tooling.
27:01We have some customer-facing applications as well.
27:04They tend to be uh, TypeScript and they tend to be deployed to Cloudflare.
27:07So, they're you know, they're full-stack applications.
27:09It works perfectly.
27:10You can even have evals that are focused on the back end or database operations.
27:14All right, you could get as granular as you want with it.
27:16Um, but it's absolutely yeah, it works full-stack and and I expect that the way that we're all going to continue interfacing with software in the future is just going to be kind of like the gardeners making sure that they're evolving correctly
27:27on our own.
27:27Yeah.
27:28That's why uh, we're not like giving out a tool because we couldn't give you a a tool at this time that works generically for every single application that everyone in this room is working on.
27:39You kind of have to build up the tools that fit the app that you're working on and then give that to the models and just make sure that they know how to run them well.
27:45Um, as a
27:46I was so I was I was also going to say like your evals, what you actually need or care about as a team or company or business or going to wildly differ from, you know, if you're in like the interior design space versus some other place where like speed is the only thing that matters.
27:59Um, I I think you're going to have like wildly different evals and there's a value to building those up with your team.
28:03Yeah.
28:04And there's there's lots of different ways to get the data to the like to the model in different ways for so that it can interpret it and it can keep up.
28:12And that's like a a huge piece of like keeping the model in the loop as to what's going on.
28:18Uh, the AMP team has a really great post on it.
28:20I think it's called I can't remember the the post, but I'll get it out there.
28:24I think it's called loop-driven develop I don't know.
28:27I can't remember.
28:27But uh, what they in in the example, they show like a canvas with a circle and a ball bouncing inside of this circle uh, and then like there's a little slit where the ball if it hits just right, it can fall out of the circle and then the game the game is over.
28:41And uh, the whole idea of that is making things feedback loopable.
28:44That's what it is.
28:46Um, where you know, the model can't actually see the running canvas and where things are going, but if you, as a very contrived example, if you just print out the coordinates of that ball at every point, and you see a bug where, you know, once it gets to this specific point, it jumps to the other side of
29:00the screen.
29:01Well, if it's moving more than one pixel at a time, like per loop, something is wrong, potentially.
29:06And so, there's a bug in there.
29:07If you can surface that to the model, then it can help you debug it, where it doesn't actually have eyes to see the canvas and see the ball not
29:15behaving well, like with physics.
29:16So,
29:17And also, once you do that, you can you're also then free to spin up 30 different variants of software that are, you know, speaking to that eval, and then pick the one that you want.
29:26Yeah.
29:27Um, so, that's the other thing that it unlocks, right?
29:29It's really like foundational for just going, and especially if we expect agents to models to improve, you know, tokens to get cheaper, potentially, um, then, you know, you can imagine that we're all going to be like coming in on Monday and saying like I would like to see 35 versions of this new feature or application here.
29:45Yep.
29:46And pick one.
29:47And so, how do we actually run or create that?
29:50Well, in this case, like we said, we can't really prove it, uh, for everything, like universally.
29:56It really has to be something that's unique to your application.
29:59And so, for our midnight snack page, this is what the proof, uh, the proof command might look like, where it's basically using, uh, a tool like Playwright, um, to
30:09check that a, uh, an email was actually saved when it gets on the waitlist.
30:13So, uh, we can check that it doesn't have any on the waitlist, any emails, then we can fill in an email on the next line, and then we can get the button and click it,
30:23uh, the join waitlist button, and then we can expect that the status, uh, element on the page has a waitlisted in there, uh, and then check the inner text of that to see if our email address is in there.
30:37Um, and so, that's one simple way to do it, very easy if you've ever written, uh, an end-to-end test with like playwright or react testing library, any of those,
30:49you've definitely used tools like this.
30:51You can just give them back to the model to your agent and they can run them.
30:54Give it to them as a tool and they can prove it out themselves and you've made this reproducible bit of evidence that then you can use in this one next time you use it next time that page changes it can run it again and verify that nothing broke.
31:07It acts as a regression and keeps everyone on the team moving forward.
31:15[clears throat]
31:16One caution word of caution though, like a successful screenshot isn't proof.
31:21Make sure that it's actually showing, you know, the state that you expect it to be in, that it actually like ran the commands to get to that state to actually be in there because otherwise it might just be like another lie,
31:33but you're seeing a screenshot on the the PR and thinking, oh yeah, this is this is correct, but it might have be causing different side effects.
31:40It might have taken a different path that's not actually possible within the UI.
31:44So you want to make sure that it actually like runs start to finish to check that.
31:52Yeah, and so once you build these loops that are, you know, durable, reliable, repeatable, uniform, the value of what you get or the benefit the main benefit is that a a failure starts to drive future work, right?
32:03So if in one loop the agent fails, you're busy talking to somebody at work or you're starting yet another task, that agent can determine on its own, this is not quite correct.
32:12Let me go take another stab at it.
32:14Let me now run the command verify again.
32:15Okay, now we're good.
32:16If you think about what that means at scale that a single PR or an agent a single lane of work, if you will, and an agent, you know, self heals, self corrects, figures it out on its own,
32:26you're not coming back to that particular PR at the end of the day and and then being like, oh, I still have to get this all the way through.
32:33And if you just multiply that out, right?
32:35That that's that's really where again, we just say this is foundational for speed and velocity.
32:40Yep.
32:41And how many of you write good error messages?
32:45Show of hands.
32:47Yeah, nobody knows.
32:48We already know.
32:48[laughter]
32:49Sir, what's your message?
32:51I got small hands.
32:54There you go.
32:54Yeah.
32:54Solid.
32:56Um well, you should.
32:57Uh I think they're more important than ever because uh when your your uh app throws an error, guess what can interpret and run with that error really well?
33:05An agent.
33:06So, um if you make it so that your failure is explicit about why it failed and give as much detail as possible about how it got into that state,
33:14then uh your agent can just like self-heal from that.
33:17So, it's um very easy, very worthwhile to to do that.
33:25Uh yeah, and so this is a concept that you know, Nick and I talk about, right?
33:28And it sounds like we're being uh in some ways we're trying to wrap determinism around LLMs, uh but we actually don't want to do that.
33:35And it's also something to the spirit of what the amp team is talking about.
33:38Uh the agents are in in intelligent.
33:41They're they're capable.
33:42We want them to be creative, actually.
33:44Because oftentimes if they have the right constraints, and then oftentimes constraints can be can equal freedom, right?
33:49In a way.
33:51Uh then they will come up with creative solutions and actually solve what you said.
33:55That's why you give them the outcome.
33:57That's why, you know, Nick's project he's experimenting with is outcome-driven development.
34:01Um and so, it's not about over being overly prescriptive in how the agent achieves things.
34:08Um that can actually lead to lower performance in some cases.
34:10It's more about giving it the map and saying you must get here.
34:13You'll know you're here when this command returns this.
34:17Yeah.
34:18You don't necessarily have to prescribe how it gets there, though.
34:20And this is true more and more true every single day with like the frontier models.
34:25Fable, Astra, Saul 6.1, uh Opus 5.5. They're all really good at being like you being creative is like the way I I describe that.
34:35But, if you give them This is where I want to get to and don't necessarily tell them the path that they need to take, they might wander and make sure that they catch more edge cases.
34:43They might figure something out or ask you a question that unlocks something.
34:47But, if you're just like you have to get here and then here and then here, they may not sway from that.
34:52And you want them to sway a little bit, but get to the right right end goal.
34:56And like I said, this is something that we we've only come around to really appreciating in the latest batch of Frontier models.
35:03Uh where it's really becoming possible.
35:04So, going forward, it's going to just get better.
35:06Yep.
35:09[clears throat and cough]
35:09So, when it's going, um you want to get into like, you know, the typical TDD like red, green, refactor loop with your agent.
35:17Um we've talked You've heard about loops, you know, at every one of these AIE events, and um it's no different here, but you can use that within the verification piece itself.
35:25Verify this, reproduce it, show me uh fix it, and then show me the proof of that.
35:30And how are you going to show me the proof?
35:32That's just as important as the fix that's going to go in, because that's what's going to leave leave you alone and allow you to trust the model to actually do the work.
35:41And And then when it says, "Hey, I'm done," you know that it's done because it's proved it to you in some way.
35:47Absolutely.
35:50So, in this example, uh you can run Snack Doctor.
35:53Let's just go ahead and run that real quick.
35:57P Snack Doctor.
36:01And so, this uh this ran, and it's showing us um you know, a bunch of checks that it did.
36:08So, it's checking that we have Node, that Chromium's installed, uh that our session exists, uh that it's actually running, and that it successfully built.
36:17So, that's like what we're running as a quick doctor check, just a bunch of different things.
36:22So, then we run the next command, Snack Join Ada at example.test.
36:29Sorry, we don't type anymore.
36:31[laughter]
36:35Okay.
36:36Gave us the result back.
36:39Snack storage.
36:44Storage is empty.
36:45It didn't actually put Ada in there.
36:48So, this is the point at which in the real world your agent you've just gotten the value out of this entire system cuz at this moment your agent realizes I am not done
36:56and I'm not going to drop this work on the floor.
36:58Yep.
36:58And if you reload it, do the same thing one more time.
37:04Boom.
37:05We have now given our app eyes.
37:08It can see this.
37:09It can run these.
37:09It can make it can make sense and change the world of the application, the Midnight Snack application, and see the results of that and then be able to interpret what happened from there.
37:24[clears throat]
37:25So, you can try that.
37:25You can If we like apply the persistence fix to it and add honest copy to it, then we can um run the proof.
37:34Let's run that.
37:41And that kicked off a browser and it failed.
37:44I didn't actually run the test, but if we had run the fix.
37:47If we had run the fix, it would show that.
37:52[clears throat and cough]
37:53Okay, one of the things that we're we're thinking about a lot lately and talking about and starting to roll out into more applications is this like concept of receipts.
38:00Um you know, you can you can take like a ticket like a a help ticket system where it's there's a certain performance the way that a human would answer a ticket and consider it done well
38:09as an excellent, you know, customer response or service.
38:12And then what does the um like preserving that in some way, for example, right?
38:18Uh why do we want to do that?
38:19There's a ton of reasons and we'll get into some of them, but again, if you think about if it if we all or if it seems that in some way many many folks are heading towards, you know, running RSI
38:29harnesses and self-evolving code bases, everything that we do today, everything that your agent does today, every run all day long, that is very valuable data.
38:38It's very valuable data right now because you can feed it into LLMs and and get fixes that you need.
38:44It's incredibly valuable data in the future as tooling improves and it gets easier and easier to set up RSI loops because then you've got a data set of exactly how your system performed
38:53that you can create, you know, very comprehensive evals for and have agents hill climb on, which means loop and see their performance and then continue going until they get better and better and better, right?
39:04Yep.
39:05And at that point they're able to pass on their own using that evidence from the previous runs.
39:12[clears throat]
39:12So now, if we fix that code, it now persists, but it can also lie.
39:18So after we reload the page, it could say wait listed, but what the screenshot might actually say is you're in, your seat is confirmed.
39:27Those don't exactly match up, and so that might be like a failure right there because they don't exactly match.
39:33So this is kind of getting away from the static testing piece of it.
39:37Like how do we prove that it worked without having to be like very tightly coupled to the actual text on the screen for a very contrived example.
39:48And so if we just run like, you know, P snack, judge, should run that and it failed because even though it worked and it says that we're in, Jev did not the the model did not see that.
40:08And it's because the the confirmation text breaks the contract.
40:11It's not exactly one-to-one.
40:14So how do we fix that without adding a lot of tight coupling to everything that we're doing.
40:20The answer is those system one models, which we'll get into here.
40:24Yeah, absolutely.
40:25So, just to introduce it for anyone that hasn't heard of it before, the you know, concept of system one, system two comes from psychology actually.
40:32It's like a useful model to think about how the human brain works and how he thinks.
40:36So, system one is super fast decisions.
40:38It's It's, you know, autonomous.
40:39Like, someone swerved at me while I'm driving and I just corrected without even thinking about it kind of thing, right?
40:45Some I was about to lose my balance and I I corrected it.
40:48And then system two is like I need to figure out what I'm going to do with my life.
40:52So, I'm going to go book an Airbnb and like think about it for 3 days.
40:55[laughter]
40:56And so, you know, applied to the world of LLMs now we've got Jev and then immediately after we've got Cleft from Cloudflare, you know, 2 days later.
41:03That's even faster.
41:04And so, there's It's It's a really exciting time to think about all the ways that we can combine these tools together to make the new primitives that we're going to continue to see in evolving AI architectures.
41:14Yeah.
41:14To get Put it another way, that light right over there, if I said, "What color is that projecting?"
41:19Quickly you would say yellow.
41:20That's like system one thinking, right?
41:21You just You know the color.
41:23If I said, "Why is it projecting yellow?"
41:25Well, because we're thinking.
41:26Now we're thinking, you know, we could go down like a physics of why the light emits yellow.
41:32Why are we here?
41:32[laughter]
41:34That's how we got to really what we want to talk to you about today.
41:40So, in this we've got another game that we can play within within this.
41:46It's a quick game with the snack.
41:48We have saboteur and a detective.
41:51And so, in this the the saboteur will pick a snack.
41:55It'll put a secret bug in there, randomly putting a bug in there.
42:00We don't know which.
42:01It'll just be random.
42:02And then you, the detective, find it with with snack alone.
42:07And then prove what that is.
42:09Name the bug and give give proof that you found it.
42:11So, this is something that you can do if you've pulled down the website or pulled down the repo, um you can just run pnpm snack up --bug random and
42:21uh give it uh let it run and see if you can find it.
42:25So, let's try that.
42:25We'll just give it some time to go.
42:30Uh what was it?
42:31--bug
42:32bug random
42:42Oh, it's already running.
42:44Let me kill that.
42:46Kill the right one.
42:58Okay.
43:00So, [clears throat] in this uh it ran uh it's got the app URL, it's running, it's not headless, so we can see it.
43:06The bug uh says hidden until you stop up.
43:10Yeah, right.
43:11Um leave this running in another terminal, run pnpm snack doctor.
43:15So, let's run that.
43:21All right, so that ran the whole thing and we can see node's there, chrome's there, it's got a session.
43:27The proof here, uh the last receipt says fail, but the source is now uh that proof is for code that no longer exists, so it wasn't able to prove that.
43:37That was a previous run.
43:38Let's run it again.
43:39Same thing.
43:41Uh can anyone tell me what the bug is?
43:44Any ideas?
43:53Wrong email?
43:58Yep, exactly.
44:00Port's wrong.
44:06Uh I think it was running at 62174.
44:12No, that was right.
44:14Let me try it one more time.
44:16We got less than a minute.
44:39Uh, for that one, yeah, you do.
44:41The tools make it optional though, so you can run through all the exercises if you don't have it and it'll just go, but um, it's also good opportunity to experiment with one if you do.
44:49Yeah.
44:50They're they're pretty cheap calls.
44:52Uh, they give you $5 for free when you sign up and I think that'll last you like 3 years.
44:57Yeah.
45:02All right.
45:06So, [clears throat] um, yeah, go ahead.
45:09Next one.
45:10Yeah.
45:11Um, so, yeah, we ran it a couple times.
45:14I think there was some problem with my output.
45:16I'll debug that.
45:18Uh, it's funny, I need some verification tests around my verification my contrived verification examples.
45:24Um, but the, you know, the the outcomes that could come up are uh, that it named the bug uh, with output, uh, said it works but it didn't, uh, said it works but it did,
45:38uh, blamed a bug that wasn't there, um, and then had no, it was the right bug or no evidence.
45:44Right.
45:44And this is just getting back to that point of uh, a contrived example, of course, but your team might score differently or have different criteria that you're looking at.
45:51This is super flexible.
45:52The most important thing to do is to start and to create one, create an eval baseline and to use it consistently and to evolve it together as a team, and then get to the point where your changes and your PRs automatically flex that
46:03eval so that you can get to the point where you know if changes are safe to land or if they're improving things or if they're making things worse or if they're not changing things at all.
46:10Yep.
46:13Uh and every time it runs and it misses, um that's a new line that can go in the map.
46:17So, it can give us ideas of like the steps that we're getting there, why we're getting there, and kind of like you know, every time it fails, it's getting us one step closer cuz we can fix that.
46:26So, the first time uh is memory only, um like it's not storing anywhere but in memory.
46:32Uh we're putting in a wrong email address.
46:34Uh we've lost our confirmation.
46:36Every time we can prove that we've caught that and then we don't run into that again.
46:43Um but then the over promised one, uh the screenshot or the judge running uh can tell us that it um it actually did pass when it didn't.
46:55Same thing with console error.
46:56Uh there might not be anything in the console, um but it's in the receipt or it might not be it might be getting caught um before it gets to the console.
47:04Um yeah, anything like that.
47:07Or it could just be none.
47:08There's nothing to find.
47:09Everything works.
47:10Um and that's a real answer as well.
47:18Um yeah, if you run the doctor, kind of like what we were doing, it was showing this where it was showing the previous uh examples.
47:26Um it's showing proof that no longer exists.
47:28Uh that's because your code might have changed quite a bit between yesterday and today.
47:32Um it might be different circumstances, like a a famous date bug, um for example.
47:39It might be a number of different things.
47:40So, uh we want to build the tools in such a way that they're flexible enough to uh not get caught by um issues like this.
47:50Uh this is all kind of like going back to you know, basic unit testing examples, uh but bringing that kind of like deterministic check to the models in a way that they can self-heal and self-improve
48:02from there.
48:03Absolutely.
48:06[clears throat and cough]
48:08Okay, so this again to tie it back to what we used to do as human developers, right?
48:12What we used to be able to see, how we would flex the application, how we would check everything.
48:16Uh, that's really what we've been talking about and moving it into code.
48:19So, by moving it into code and making it available in a deterministic CLI, then we can get to the same path and the same experience that we would flex as, you know, developers when in Chrome trying to fix something late at night.
48:31And that's really what unlocks the the speed and the evolution and the ability to scale up the amount of code that you're delivering and to do that in a more safe way.
48:41Uh, again, you know, if you uh, the recommendation is if you have a web app uh, and you're not already you don't already have a custom verifier CLI, that's one of the the highest leverage things you could start with for your
48:52uh, code base, the foundational eval as well that shows what good looks like in your application.
48:57And then also giving your agents the ability to see all aspects of performance and uh, networking and and traffic and and rendering, et cetera.
49:06And so, giving it Chrome dev tools if you're in um, if you're in the web app land.
49:11If you're shipping native clients if shipping iOS, you know, or OSX apps, then there's an alternative uh, projects and more coming out every day that you're then going to attach to there.
49:20But the the the common, you know, foundation here across all of them is that you need to give your agent the ability to see absolutely everything that you would normally see as a developer.
49:30Yeah, the more that you can give it, the more arms or eyes that you can give it, the more it's going to be able to take over from you.
49:37Uh, and then you just have to give it the intuition about when to use those, how to use those.
49:41Think about the way that you decide, okay, if I this bug came in, how would I reproduce this?
49:46Think about that and if you can break it down and this is dependent on your application and the type of of code that you're writing, um, you can you might be able to do that in a way where it can like think, "Oh,
49:56you know, for us it's a lot of sign-on things.
49:58Like, is it properly setting a cookie?
50:00Is it setting an expiration in that like all of that?"
50:03the in the job, right?
50:04You can go again like the this is applicable to every, you know, field and industry that you've got software for.
50:10Um but it's it's really just all about like, "Hey, that world where it made sense for us all to hand-chisel code all the time is at least evolving or at least changing rapidly.
50:19What are the foundational like what's the what are the same system fundamentals that we bring to this now so that we can accelerate software development?"
50:25Yep.
50:29Uh so, yeah.
50:30Doing expensive call stacks.
50:32Uh but the the important thing is that uh getting that output, making sure that it's giving you detailed output about what happened, uh the state of it, uh all of that so that it can be audited by the application
50:44uh by the model, and then you can run it again, it can fix it, run it again, and verify that those changes actually took hold, and that there's no longer an issue.
50:53Uh and yeah, and again uh can't overemphasize the value of keeping receipts.
50:57So, if you have runs, uh you know, it can be the entire user journey.
51:01Let's say it's an e-commerce app.
51:02It's like when they log in, they hit the page, what happened, what was their flow?
51:05Did they see an error at all?
51:06Did they see 7 seconds of time to see your catalog, you know, come into view?
51:11Uh were they able to put something in their cart?
51:13Was there cart abandonment?
51:14Um and so, tr- save that all now because every single one of those runs and receipts can be fed to LLMs, or you can even set up a system that's running all night long and is looking for
51:25uh the latest receipts being produced, and is aggregating them, and saying, "Well, today we saw eight failures in the caching module.
51:32So, I'm obviously going to, you know, issue a PR tonight to fix that properly."
51:35And then, "Oh, I have the tools I need to prove in this PR that the issue I was trying to solve has been fixed."
51:43Uh and then the recommendations that we're seeing from from folks, too, that are achieving like super high scale PR uh shipping without errors.
51:49They're they're also able to, you know, add uh the stuff that is for us as humans in our oversight still.
51:55So, not just screenshots of the feature, but like a recording of I walked through the app this way.
52:01This was my flow.
52:02This is what I experienced.
52:03These were the states and I got to the goal.
52:05So, that if I'm waking up and trying to rip through, you know, 15 important PRs in the the first thing in the morning, I can first start watching that and make sure okay, it actually got to the end and worked.
52:14Then I can, you know, start looking at code uh if it's something that I'm actually going to like human review.
52:19And uh again, all of that I think is going to be immensely valuable as we continue to see uh evolve self-evolving tooling, really.
52:29And you know, think about what that means though because when we get to the point where there's self-evolving tooling is just like the default and people are using it, uh
52:37there will be a specific advantage for folks that have preserved uh clean data sets of things that they can say this is what we are trying to get to in this application
52:45and want you to get all the way there and rapidly improve it, right?
52:49If you're not sitting on a mountain of that and your competitors are and they have similar apps, they're going to be able to go significantly faster in their development cycle.
52:56Yep.
52:56And this is just building up that trust so that we can continue to improve upon our app as we go, uh but we have these checks for everything that came before it.
53:05So, we've got unit tests, we've got evals, we've got smoke tests, we've got whatever our application needs and we've got a way for the model to be able to interpret and run those and improve them
53:15uh in such a way that if we add new features in, we can run all of the previous ones and make sure that we didn't actually break anything, uh no regressions,
53:23and we can just mount and climb that evidence ladder, building evidence as we go, uh and ensure that our app is working uh exactly as expected.
53:32And that really becomes the the spec that shows what our app does.
53:38Okay, cool.
53:38So, we've got uh just over 5 minutes.
53:40I'm going to pause and see if there's additional questions.
53:42Yes, sir.
53:43So, does the scale I I would personally I'm curious with how you think about it.
53:54I would I would personally almost lean away from skills cuz in my experience it's it's you know up [clears throat] to the model whether or not it's going to actually follow that path and use it
54:02versus like having the CI actually enforce that these these commands get run as well.
54:17Yeah, that's what
54:19Like the answer or the question is how do you run these these checks normally, right?
54:23Effectively.
54:26[clears throat]
54:37Yeah, it can you can run those you can run these regularly.
54:40You can have them be like part of CI for example.
54:42You can have them be various hooks that could be like a get hook although that's probably the last one I would reach for.
54:49We could have it be like a cloud code hook.
54:51It depends on what your team is using and how they're they're doing it.
54:53I might build like a full pi extension to keep things running in the background and just kind of like feed in information as it's going.
55:01There's a number of different ways that you can do that but the easiest way by far to get everyone across the board and I can say this now just put an agents.md
55:11not a cloud.md in your in your app and then have it say like you know on change verify things by running this or like every time you change significant things or before you open a PR do this.
55:21Like whatever wording that you want to put in there and then that's an easy way to get everyone across the board and they really kind of have to go out of their way to
55:30ignore that or avoid it.
55:31Hooks are also relevant too if you're doing the local development, right?
55:34For the whole team cuz you can share hooks across the board.
55:46Yeah, absolutely.
55:47Yeah.
55:47Well, CI is a is a I think a feature of it, but it's not like the the actual tool is like the rapid development that you get locally.
55:55So, I give my agent a task to do and I can just quickly run uh like let the agent go.
56:02You know, I basically tell it this is where I want to end up.
56:03I want the waitlist to be I want to be able to add an email to the waitlist and then it gets persisted uh and saved and I can verify that.
56:11Uh so, this is how you verify it.
56:12It goes into this table, blah blah blah.
56:14Like whatever it might be.
56:15Um Yeah.
56:18No, good question.
56:19Sure.
56:38Yeah, what's the benefit of running it in CI versus like a local Uh the benefits are definitely like a faster feedback loop, which allows us to run faster and allows me to
56:51uh parallelize myself because I'm no longer working on just one thing at a time.
56:55I'm working on at least right now I have three agents running.
56:58Uh that's a very low day for me, but I'm at a conference.
57:01Um
57:02[laughter]
57:02Normally, it's like 10 to 15.
57:04He's really taking it easy today.
57:06Um and and the the whole idea is like I'm going to set it off and I want I don't want it to be waiting on me and saying, "Yep, I did it."
57:13and then I go running and it's like, "No, that's not actually correct."
57:16and then I have to like set it off again and I go get distracted by another agent again and I just continue that loop.
57:22I'm just like constantly moving between projects.
57:24So, I want to give it the tools to where it can prove to itself that it did it and then prove to me that like show me the evidence in a way that I can trust
57:33uh so that when I come back to it it might be in 5 minutes it might be in an hour or a couple of hours I want to know that it has done
57:40the best possible job that it can to get done to a state of done so that I'm not just like re-prompting it and starting that whole loop again because I don't want to be that's that's not the fun part of my job.
57:51Yeah, I would I would say another way to to put it is that and we haven't everyday achieved this perfectly but I want to have all of the different lanes that I started that day for stuff that I felt was important to ship that day.
58:01I want to come back to them at the end and see them basic like correctly done.
58:05What I don't want to do is have a whole long day and have them running in the background be doing other stuff and going to meetings and then come back
58:11and then actually have to turn by turn nudge any of them because that is like a significant slow down and it it prevents me from going faster.
58:19And another way to put this is I'm like I rely on CI and the tests that are running in CI and the code reviews like we how many people do automated code reviews with like Reptile, Code Rabbit, Codo, any of those?
58:30Yeah.
58:31I rely on those.
58:31We use Reptile.
58:32Reptile gives me a X out of five score and if it gives me a five out of five and all of my CI is passing I feel extremely confident that I'm not going to break production
58:42when this gets shipped.
58:43We've gotten to that level of trust with Reptile.
58:46The more important thing for this is that I get to that state faster and that um
58:52with less turns
58:53With less turns, yes.
58:54And that you know, I I might add a whole like I could add 10,000 lines of absolutely nothing and it does nothing and great it passed.
59:03Reptile gave it a five out of five for some terrible reason and then it goes into production and it doesn't do anything it doesn't break anything.
59:08Now it's just an extra 10,000 lines of nothing.
59:11Um and so this is also to help me make sure that like I'm I'm telling the agent like exactly what I want and it gets there without going through a bunch of um
59:22additional code to or additional like like it stays on track uh but it also knows how to fix itself to keep itself on track while I'm off doing something else.
59:32Okay, we might have time for one more quick question, but I think we're Yes, yes, sir.
59:41Yes.
59:42Yes.
59:53Yeah.
59:55You could also You could also do depending on your stack like there's it's also you know there's various providers that you could easily also go to preview first and make and then verify it there in a different way.
60:05So we'll then go to prod and then also immediately monitor.
60:07It's another pattern we're starting to see emerge is that when folks have a deployment that was mostly agentically implemented, there's also at least one agent specifically concerned with that production rollout
60:18looking for watching all the metrics, watching DataDog, watching feature flags, right?
60:22Being able to say this is this is actually bad and we're we're reverting it.
60:26Um so I I think that like I I truly understand and get the like queasiness around it like we all have or dealing with that, but I I I do see a path for that um because it just seems like it seems inevitable.
60:38Yeah.
60:38Yep.
60:39That's all the time we have, but if you have more questions we'd be happy to chat.
60:42We'll be in the expo hall booth six right to the right when you come in.
60:47Come find us.
60:47We'd be happy to chat to chat, show you what we're working on and talk to you about anything you'd like.
60:52Yeah.
60:53Thank you so much for coming and your attention and everything.
60:55Thank you so much.
60:56[applause]