Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS

AI Engineer · 61 min · 881 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:11Hello everyone.
  2. 00:12Thank you very much for coming.
  3. 00:14We are very happy to be here with you today.
  4. 00:16We have a great program for you.
  5. 00:18My name is Zach Prosser and this is my friend Nick Nisi.
  6. 00:21We are both AI engineers at Work OS.
  7. 00:24Today we're going to share with you some of our best workflows and things we've learned from working with Claude and other agents every day.
  8. 00:31This will include voice coding, agent skills, hooks, and scheduled tasks.
  9. 00:36We hope there will also be time for questions and answers at the end.
  10. 00:39Yes.
  11. 00:39The most important thing we want you to take away from this is that most engineers simply look after one session at a time.
  12. 00:46It's like 2025.
  13. 00:46You're supposed to be operating a fleet of agents, and we're going to talk about the different ways we actually do that .
  14. 00:53This is not a marketing ploy or aspiration.
  15. 00:56These are the things we do every day to survive in this moment.
  16. 01:01Yes.
  17. 01:01Well, as we said, well , I'm Zach.
  18. 01:04I'm on the team for .
  19. 01:05artificial intelligence in Work OS.
  20. 01:07I would say my role is 60% implementing internal customer-facing tools and applications using AI throughout the day, and then about 40% AI education that elevates the rest of the organization.
  21. 01:17And I 'm Nick Nisi.
  22. 01:18I'm the engineering manager for the developer experience team, and we're a small but mighty team of two that manages over 25 repositories in eight languages, and we're only able to do this thanks to AI.
  23. 01:31This made it the only way I wouldn't go crazy.
  24. 01:36Great.
  25. 01:36So what we have for you today is a very interactive workshop.
  26. 01:39So if you want to sit back, relax, and just watch our presentations and talks, that's fine.
  27. 01:43If you want to participate in the practice, we have a repository you can use and, well, you can go through everything step by step.
  28. 01:50This has skills that will help you set up voice coding if you have never tried it before .
  29. 01:54We want you to think of it this way, it's a great opportunity and kind of a safe space if you want to try out new workflows before voice coding tasks, scheduled tasks, hooks, etc. It's a great place to actually get involved.
  30. 02:06And then we'll be available to help you or answer your questions.
  31. 02:10Yes, for sure.
  32. 02:11We will discuss these four things, with varying degrees of detail.
  33. 02:16And there will also be time for questions and we will answer them.
  34. 02:22But there are three things we will see, three main things we will see.
  35. 02:27Just slides.
  36. 02:28There's also this QR code you can scan.
  37. 02:31It leads to an active dictionary and contains a lot of terms that we will be talking about.
  38. 02:35Things like MCP and skills and whatnot are in Rag loops.
  39. 02:40Yes.
  40. 02:40They are all defined there, but there is also a little chat about ragas.
  41. 02:44So if you have questions about that , you can ask them there as well to help keep things moving.
  42. 02:48Yes.
  43. 02:48But you can also ask us.
  44. 02:51And then everyone...We'll also ask for your feedback and ways you can collaborate and use the skills we're learning today and show what you've learned in the next hour.
  45. 03:01And it's on this live board that we'll see.
  46. 03:04And if you go to the repository, you'll see it and you'll be able to contribute to it .
  47. 03:10Nick created this awesome MCP server which is part of a repository whose URL we'll have to share soon .
  48. 03:17But this will be an interview with you.
  49. 03:18This will be...This is optional.
  50. 03:20You don't have to do this.
  51. 03:21This will scan your computer if you agree.
  52. 03:23It will look at some of the conversations you've already had with agents and figure out what you've worked on, how you've worked, and give you some simple rubrics on how familiar you are with
  53. 03:34AI.
  54. 03:34Mostly for fun, because we'll do it again at the end and see if there's any movement.
  55. 03:38Yes.
  56. 03:39He is built like a coach.
  57. 03:40He's there so you can ask questions and he can try to help guide you.
  58. 03:46And he tries to answer in the way we would try to answer these questions.
  59. 03:49But you can also ask us, because AI isn't always perfect .
  60. 03:54And this is what data visualization looks like.
  61. 03:56So again, only sign up if you are interested and feel comfortable.
  62. 04:00When you're being interviewed through this MCP server or coaching tool, you send a simple post to the backend that we 've set up in Cloudflare, and it just checks the current room position, and then at the end of the session, we check that.
  63. 04:14The only data it sends are the questions you answer.
  64. 04:17It doesn't scan your system or anything like that.
  65. 04:20Yes.
  66. 04:21This is the beginning of the interactive part.
  67. 04:25And, uh, I'll mention again that if you want to follow along, a great way is to follow us, you know, here and talk, uh, listen to us here and go through everything step by step.
  68. 04:34We'll also be running on screen, but you can also clone this repository, which is designed so that you can run Claude in it and… That's where you'll get the MCP server, skills and tools, and where you can work on it ; and then it's a perfect , um, kind of artifact that you can take with you and
  69. 04:50continue practicing, um, after our session.
  70. 04:52Yes.
  71. 04:52And I think the way to leave this screen here is if you want to clone this repository, um, but we can ask one or two questions while we're at it, um, we'll leave this here if anyone has a question.
  72. 05:04Any questions?
  73. 05:05All right.
  74. 05:06We don't bite.
  75. 05:07Cool.
  76. 05:15All right.
  77. 05:15So if you're going the repository route, um, the first thing you want to do is run Claude, however you normally do it, you know, you usually just type Claude, and it'll ask you if you want to trust this repository.
  78. 05:25Trust the repository and then, uh, one of the first skills that's already loaded will be this setup, uh, skill that will help you install all the things for the workshop, including, uh, your first open source and free, uh, voice type, uh, coding model.
  79. 05:38So you can say, "set me up for workshop," or type that in and Claude will walk you through the steps to do it, and then once you're done with that, you can say, "run my workshop
  80. 05:49check."
  81. 05:50This will give you some idea of ​​ what stage you are at and what is happening.
  82. 05:55The repository is also setup with many skills and plugins already configured for Claude.
  83. 06:01So there's MCP, there's some skills that we created and some that we didn't create, like the ones from the Codex.
  84. 06:09This is just a simple way to get started and configure Claude's repository and environments in a way that is best for this workshop.
  85. 06:19And while we wait for a few people to set up the repository, let's raise our hands out of curiosity and see how many people are already using the voice encoding tool every day.
  86. 06:28Okay , great.
  87. 06:30Pretty good division.
  88. 06:31What voice encoding tool is this ?
  89. 06:32I mean whisper flow?
  90. 06:34Yes.
  91. 06:34Okay.
  92. 06:35Excuse me, which one?
  93. 06:37Whisper flow, but it got slow.
  94. 06:39Yes, I had the same problem.
  95. 06:40What has become faster and what is faster?
  96. 06:42Then I moved on to voice encoding.
  97. 06:44Great.
  98. 06:44Okay.
  99. 06:45The one we will install today is an interesting new tool.
  100. 06:48It's called Handy and it will install on your Mac.
  101. 06:51And the cool thing is that you can choose which open source or open weight model to use for transcription and you can strike a balance between language support and speed.
  102. 06:59But the cool thing is that there is no subscription because it runs on your computer.
  103. 07:02Nothing is sent to the cloud, it's completely private, right?
  104. 07:05So there are many advantages, and we will talk about the differences in choosing a voice encoding tool later.
  105. 07:10It is free.
  106. 07:11This was one of the main factors in his selection .
  107. 07:13But if you already have a tool installed that you use and prefer, go for it.
  108. 07:17Yes, go for it.
  109. 07:18Okay, voice coding .
  110. 07:20We are big supporters of voice coding.
  111. 07:22I would say that, besides learning about the LLM ecosystem, tools, and harnesses, this is probably one of the biggest leverage moves we've both made in recent years.
  112. 07:31I consider myself a pretty good machinist.
  113. 07:33I learned to type pretty fast playing EverQuest as a teenager and I could get around 90 words per minute if I was fully energized with caffeine, and I get around 190 with
  114. 07:43tools like whisper flow, right?
  115. 07:45So, it's much faster, and we'll talk about how it's not only much faster for a single task, but when you multiply that across multiple cards and agents that are running and checking and supervising the work and just guiding them, you gain a huge advantage very quickly.
  116. 07:58And not me.
  117. 07:59Let me be clear, and I failed.
  118. 08:01I can't write, but I make up for it with my Vim skills.
  119. 08:04To put it modestly.
  120. 08:05So, if you have n't tried it yet, you can definitely, you know, we encourage you to get out of your comfort zone, give it a try.
  121. 08:12I'll also share that it's not just about coding, you know, use it to respond to, you know, colleagues and clients and and and continue to make sure that we have that quality and that everything is formatted correctly, but you know, sometimes you can just discuss something architecturally and that's going to be the biggest gain that you're going
  122. 08:29to get from spending an hour on ideation.
  123. 08:31Nick has a very popular skill that's in this repository called ideation, and coming to clarity faster and focusing on results are some of the things that voice coding can help with.
  124. 08:40Yes.
  125. 08:41I want to give a special thank you to Swyx and the entire AI engineering team for this great event in San Francisco.
  126. 08:48This is just an example of use.
  127. 08:51Oh.
  128. 08:52Run it.
  129. 08:52Steph, I just want to get this going, guys.
  130. 08:56Please.
  131. 08:57Here you go.
  132. 08:57I think I was holding the wrong key.
  133. 09:01Maybe it's...
  134. 09:03Yeah, so it can be incredibly fast and we both found that there's really value in being able to express your thoughts quickly.
  135. 09:10They don't have to be perfect and can be messy and will be cleaned up later.
  136. 09:15But this just allows you to get a working artifact even faster.
  137. 09:19Yes.
  138. 09:19So, once you've got all that set up, let's try to fix the bug with voice.
  139. 09:26So in your encoding agent you can simply ask it to, for example, extract the external client into a separate module.
  140. 09:34And that's just an easy way to express it.
  141. 09:37In a sense, you eliminate the friction of having to type everything in and move at the speed of thought, something that, if you were a vimmer, you would be proud of.
  142. 09:45Actually.
  143. 09:46And then, you know, again, if we're both running a terminal throughout the workday and have eight or more sessions, all focused on different projects, you know, it has all the context and tools built specifically for that purpose.
  144. 09:59And you can drive past any one of them and while one is working in the background, you direct the next one to follow the path that will get you there faster.
  145. 10:08So it adds up really quickly .
  146. 10:10Mhm.
  147. 10:13Next slide.
  148. 10:14Yes.
  149. 10:17So you're talking about the result, not the keystrokes.
  150. 10:19So you want to think about what you want to achieve and not have it in your head, it kind of changes the way I interact with Claude and with my computer in general.
  151. 10:29Because what I'm really dealing with is more of a speed of thought, and it's a bit different than the words I would actually type if I were physically typing them, which allows me to get to the result faster.
  152. 10:40But I also find myself stopping and pausing in strange ways sometimes because I'm thinking about the next buffered set of words I'm about to say.
  153. 10:47Yes.
  154. 10:48Definitely.
  155. 10:49Well, again, it's just much faster if you mostly write.
  156. 10:53Most of these models that you'll come across, whether it's WhisperFlow or even the open source ones, are refined in such a way that they can recognize the exact filename that you're using.
  157. 11:03They know, you know, highly technical terminology, tools and utilities of the operating system, so you actually get the result much faster.
  158. 11:11Mhm.
  159. 11:14Okay.
  160. 11:15Now, a few words about why different tools are where they are , where some of them stand out?
  161. 11:20WhisperFlow is perfect and loved, and people all over the world are starting to embrace it and use it, even those who aren't, you know, traditional programmers.
  162. 11:28I would say one of the things where it benefits people the most is just the setup and ease of use, because if you can use a Mac and you can download the app on your Mac, you can use
  163. 11:38WhisperFlow.
  164. 11:38The trade-off is obviously the subscription, and people might not use it, especially if you're doing confidential work, whether it's coding or other types of, you know, document review, or you work in a regulated industry, there are downsides to sending everything you say, file
  165. 11:55names, etc., content and personal information from your computer to the cloud.
  166. 12:00And that's where, you know, using a tool like Handy can be much more convenient for people.
  167. 12:06Mhm.
  168. 12:07And that's exactly the tool we shared today in this workshop.
  169. 12:10We'll use Handy, and that's for local dictation on the device, because models are getting smaller and smaller with quantization, and it's now possible to run a very efficient speech-to-text model on consumer hardware.
  170. 12:25I want to bake a cake.
  171. 12:27And it consists of three steps.
  172. 12:29The first step is to prepare the dough.
  173. 12:31I don't actually know how to bake a cake.
  174. 12:33But the second step is to prepare the eggs, and the third is to put it in the microwave.
  175. 12:37There is sugar there.
  176. 12:38Okay.
  177. 12:39Well, you know what I mean.
  178. 12:41It understood.
  179. 12:43The reason to use a tool like this instead of the built-in dictation on the Mac is because it actually listens to what you're saying and knows I'm speaking in list form, so it creates a list.
  180. 12:55And that's a big part of it.
  181. 12:57It is also very good at understanding file names and functions, which it correctly types and automatically corrects them, as well as correcting errors, for example if it inserts the wrong word.
  182. 13:08I tried to write Swyx earlier, but I made a mistake and I was sure it would n't write it, so I could correct it , and the next time I say it, it will automatically correct it to Swyx.
  183. 13:18You can configure it so that all of these features define the context you're in.
  184. 13:25So he knows that if you're writing an email, you might want to write more formally.
  185. 13:28So I will use more formal text.
  186. 13:30But if you're in Slack, it'll be more casual, with punctuation and stuff like that.
  187. 13:35This can be adjusted depending on where the text is placed , which is really useful.
  188. 13:45Try it now.
  189. 13:46Did we tell you the command you need to run to configure this?
  190. 13:50Yes, it should be there already?
  191. 13:52Should be next, I think.
  192. 13:53Okay.
  193. 13:53Yes.
  194. 13:54Yes.
  195. 13:54So, you try this and talk to the repository.
  196. 13:56Here it is.
  197. 13:57Yes.
  198. 13:57So, you can just ask Clyde if you have this repository configured, just say " Configure Handy for me" and it should automatically configure it for you.
  199. 14:05Wi-Fi conference, interested?
  200. 14:06Yes.
  201. 14:06Would you like to open Handy in source control?
  202. 14:09What will it look like there?
  203. 14:10Yes.
  204. 14:11Yes.
  205. 14:11So, it looks like this.
  206. 14:12He has a cool little Mickey Mouse glove or hamburger helper glove.
  207. 14:18And these are just the settings.
  208. 14:21I don't know if I can make it any bigger , but um yeah.
  209. 14:28So, the main thing you'll be using is a way to start your transcription shortcut.
  210. 14:34In my case, I set it to my super key, which is control shift alt um and command together.
  211. 14:42Well, usually it's just a function in itself, often it 's one thing.
  212. 14:47And you can configure it however you want.
  213. 14:50Space option , I think.
  214. 14:51Yes, this is the default setting.
  215. 14:53Well, it's just a button that you hold down when you want to talk to him.
  216. 14:58And then you can configure various things like push-to-talk , what microphone it should use, and so on.
  217. 15:05If you want to get into the details, you can look at the models and then, you know, really dig into it , you can choose the perfect model for you.
  218. 15:12Choose the recommended settings the first time if you haven't used them before , but remember that there is an option to change them here and then the app will be downloaded from the cloud to your computer so you can run it there .
  219. 15:21So there's a lot more customization and control when you're using, you know, something like an open source product.
  220. 15:28Yes.
  221. 15:29But yeah, we'll give you a minute or two to practice this, and if you haven't tried Handy yet, try running the transcription key and saying something quickly on your computer.
  222. 15:39And don't worry about sounding funny, because everyone around you will sound similar and no one will laugh at you.
  223. 15:45So.
  224. 15:47Yes.
  225. 15:47And for comparison, Zach and I both work remotely.
  226. 15:50We both work from the comfort of our homes.
  227. 15:54So we don't have coworkers who might distract us or irritate us with constant conversations.
  228. 16:00But there are ways to do it.
  229. 16:03I know companies that make these little pen-sized microphones for everyone and you hold it very close and talk into it or mumble into it quite quickly.
  230. 16:12Zach and I also played around with DJI mini microphones, like the little wireless lavalier microphones.
  231. 16:19I've been to loud coffee shops and just pinned them to my shirt and whispered to them like that .
  232. 16:24And he can hear me completely without any problem and no one else.
  233. 16:27I look like a weird guy mumbling to himself in the corner.
  234. 16:31Yes.
  235. 16:31That's good.
  236. 16:32Uh, which is totally fine with me.
  237. 16:33Yes.
  238. 16:34And still the actual transcription accuracy is 100%.
  239. 16:38I highly recommend them if you haven't tried them yet .
  240. 16:40Um, but yeah, if you run this command, the skills in the repository will be found and Claude will follow all the correct steps to actually download Handy and run it.
  241. 16:48Yes.
  242. 16:49So.
  243. 16:49Does anyone need more time on this part or is everything okay?
  244. 16:53Question?
  245. 16:54Any questions so far?
  246. 16:56NO. Okay.
  247. 17:02All right.
  248. 17:03One thing we want to make clear is that we only have an hour.
  249. 17:07So we move forward, trying to fit as much into it as possible.
  250. 17:10But you have this repository and you can try it out too if we're going too fast.
  251. 17:15But also stop us if you have questions.
  252. 17:17Yes.
  253. 17:18For example, you could try to find the failing test in the repository I brought and fix the cause.
  254. 17:24And it will just run.
  255. 17:25I mean, that's exactly what you would type, but it's just a nice, easy way to get started.
  256. 17:30Zach too, I haven't tried it yet , but Zach is someone who will be using this remotely or away from the computer while you're looking around a bit, right?
  257. 17:38Oh, indeed.
  258. 17:39Yes.
  259. 17:39You can also install WhisperFlow on your phone, and that's really perfect because if you pair the remote control feature with the cloud, you can say, any session I run on my laptop or desktop will
  260. 17:51follow me on my phone.
  261. 17:52I should have access to it.
  262. 17:53You can, you know, work at your desk for 2-3 hours, do your morning check-ins and calls with colleagues, do some focused work, and then get up and go for a long walk.
  263. 18:02Well, I like to go to the forest for a while and maybe go for a walk of an hour and a half.
  264. 18:06I have a phone, so, oh, I just discovered, because I'm going, the shower rule, how to solve this problem.
  265. 18:11Now I take out my phone, search for that specific session, and record a 30-second voice memo.
  266. 18:17And then Cloud continues working in the background, making me look like a great employee who is literally not doing any work at the moment , and then I continue my hike and go home.
  267. 18:25This is recorded, right?
  268. 18:26I do n't think so.
  269. 18:28Well, but yes, I highly recommend it.
  270. 18:33Yes.
  271. 18:34Yes.
  272. 18:35So, if you have 10… Yes.
  273. 18:38What if Cloud tries to rename them ?
  274. 18:42Mhm.
  275. 18:43But sometimes it can be really hard, so I keep asking each of them, "Hey, what were we working on again?"
  276. 18:49Yes.
  277. 18:49“ What were we working on again?”
  278. 18:51Yes.
  279. 18:51The context is unknown.
  280. 18:53How do you deal with this?
  281. 18:55That's a great question, and this digital blacksmith has a tool for you called sessions.
  282. 19:01So the question...
  283. 19:02The question is, just to repeat it for the record, how do you manage when you have 10 different cloud sessions open and understand what's going on.
  284. 19:10And that is a difficult question.
  285. 19:12I have two solutions to this.
  286. 19:14I run everything through tmux and I built something called fleet.
  287. 19:19I now have 12 agents running simultaneously, and I list them by the project they're in and the session name they're in, and it's like a tmux window.
  288. 19:28It's on the left.
  289. 19:29And then in the middle is Claude's summary that he gave.
  290. 19:33It generates this automatically for each session, so it shows me an example of what we're doing.
  291. 19:38I can quickly look through it and think, "Oh, I need to do this , I've been messing around with dot files before ."
  292. 19:44I'll switch to this soon.
  293. 19:45"And then restore it.
  294. 19:47And that's something you can access .
  295. 19:49They're in JSON files.
  296. 19:50You can write your own tool.
  297. 19:52You can use a tool like that.
  298. 19:54Well, but the other thing is, if it 's not enabled by default, well, you can enable some setting, and I'll have to look carefully what it is, or you can ask Claude to just enable it.
  299. 20:03But it's this summary feature where, when you come back and focus on the window again, Claude will try to summarize and give you this italicized summary of where you are and maybe what you're waiting for.
  300. 20:15And that's a great way to do it.
  301. 20:18And then, in that same session, er, fleet, you can see these sessions here.
  302. 20:22It just tells me, er, that the sessions are finished.
  303. 20:25Like it's finished some task it was doing, and I can click here to go to it and it disappears.
  304. 20:30So it's like a notification system.
  305. 20:32I just didn't want everything screaming at me with different notifications.
  306. 20:35I know people use Zelda themes."
  307. 20:37Or video game sounds to keep track of it all.
  308. 20:40When you have 12 agents, that's way too many.
  309. 20:42So I just have a visual indicator.
  310. 20:45Did you want to show people where to find Fleet?
  311. 20:48Oh yeah.
  312. 20:49Just Nick Nisi/fleet on GitHub.
  313. 20:51And then, if you have fewer than 10 cards, or want a more advanced solution, you can just rename your sessions/ rename.
  314. 20:58You can also rename your cards in the terminal.
  315. 21:01So if you do both, fine, you can get to the point where you manage it yourself.
  316. 21:05Yes.
  317. 21:05Have you used the session tools available in Brew?
  318. 21:08Yes.
  319. 21:09Sure.
  320. 21:09So just install Brew sessions if you're interested.
  321. 21:12We'll have links to all of this stuff later.
  322. 21:15Good question.
  323. 21:16Thank you.
  324. 21:16Yes.
  325. 21:17Great.
  326. 21:17Now we'll move on to the next skill in the repository.
  327. 21:23If you type this or tell Claude, "Run my workshop ticket."
  328. 21:26"This is the part that's opt-in only.
  329. 21:29You don't feel obligated to do it, only if you want to participate, but the system will ask you a few simple questions.
  330. 21:34I think there are four of them, and you can answer them all in one line if you want, and just hit enter, and it'll be published in our system, and a moment later, a data visualization will appear showing where the room started and ended.
  331. 21:46We'll show you that in a moment.
  332. 21:47Yeah.
  333. 21:48Let's move on to the next section.
  334. 21:50And here we'll talk about loops.
  335. 21:51We're doing a little voice cooking right now, but we're still taking care of everything a little bit.
  336. 21:55The next step is not to eliminate us completely, but to eliminate the unnecessary things so that we can...
  337. 22:02We can focus on the things we're good at and focus on, and where we can focus and actively participate, not just babysit the agent while they're working.
  338. 22:11This allows us to let the agent go for a while while you do something else.
  339. 22:16And that something else could be a completely different agent in a different tab, doing something different, like I do with 12 agents at once, or It could be a walk in the woods, anything.
  340. 22:29To summarize, an agent is...
  341. 22:30there are many definitions, but one of them is a loop model.
  342. 22:34It's a model with tools in the loop, where you ask it to do something, and it thinks about it, then takes action, and then you want it to observe something and decide what to do next.
  343. 22:46And often it's back to that loop and repeats it over and over again.
  344. 22:50And there are several ways to do this.
  345. 22:53You can build your own...
  346. 22:54of course, your own.
  347. 22:55You can build the exact workflow that works exactly for you , your team, or your company.
  348. 23:00Claude is also available, as are others like Codex and others.
  349. 23:05They provide built-in helpers that make this very easy.
  350. 23:08We'll discuss two of them in detail here, namely the goal and the loop.
  351. 23:12There are ways to let Claude or your model run unattended for a while .
  352. 23:16Your agent can run unattended for a while and make decisions on its own.
  353. 23:21Yes.
  354. 23:21So I'll just say that if you often find yourself doing a lot of good work with Claude, but you notice that you constantly have to go back and say, "Okay, no , do it again."
  355. 23:29Okay, that wasn't entirely right.
  356. 23:31Okay, almost there.
  357. 23:32"You should think and strive for goals and loops.
  358. 23:35Yes.
  359. 23:36I will also say that from what we read and absorb, and from talking to people who, you know, practice this all the time , it seems that loops and similar solutions will be one of the keys to higher levels of productivity
  360. 23:49that will also be less demanding to monitor and oversee.
  361. 23:54So I highly recommend considering them.
  362. 23:56But it quickly becomes clear that a goal has a clear definition that I want you to meet, and I want you to work until that clear definition is met.
  363. 24:04It could be refactoring this simple file until these tests pass.
  364. 24:08It could be, for example, I want you to read this page repeatedly, you know , for a long time, until there's an update, and then let me know, or something like that, you know, change the price of a flight, etc. It has a clearly defined state that you're trying to reach, and at that point it ends.
  365. 24:22Mmm.
  366. 24:23A loop is: I want you to repeat a task, you know, over and over again, until I tell you to cancel it, or until...
  367. 24:28Set by The timer expired for me.
  368. 24:31You know the different ways to use them , but this is an important distinction.
  369. 24:37And it's really just repeating the same steps until they're completed.
  370. 24:41For example, for a goal, you might say, "The goal is that I want you to do this until all my tests are green."
  371. 24:47"I have a few failed tests.
  372. 24:49The goal is for all of them to pass.
  373. 24:50The goal is to achieve 98% test coverage.
  374. 24:54Whatever the goal, it's something tangible and measurable through the LLM model.
  375. 24:59If it succeeds, the program can work toward that goal, and when it reaches it, it will stop.
  376. 25:05But until then, it will try every possible way to achieve it.
  377. 25:10It will check whether it has achieved the goal, then fix everything possible and start the loop again, or improve its approach and start again.
  378. 25:18So, there's no need to restart the prompts; the program will just keep restarting until it reaches the goal, the ultimate goal.
  379. 25:29And really, you just have to internalize this—as we've all experienced, especially if you work with Claude and other agent tools—with any regularity, the program will throw away code well before the task is actually completed.
  380. 25:42The level of done .
  381. 25:43The "done" level, where you'll happily sign off and say, "This work is done," and send it to your colleagues for review.
  382. 25:50Correct?
  383. 25:50And that's the gap we're trying to fill with these loop, target, and ultimately schedule primitives.
  384. 25:56Yes.
  385. 25:57Goals consist of these four fields.
  386. 26:02None of them are optional.
  387. 26:04It needs to, for example, reproduce a bug with a failing test, implement the smallest possible fix, do a lint check, type check, make sure the tests run in green mode, and then it can open a request for proposal (PR) to submit it.
  388. 26:19And in fact, the idea of red-green refactoring is coming back.
  389. 26:22It's a really nice loop where you can write a failing test, write the minimum amount of code to make it pass, and then verify it.
  390. 26:30Many additional steps can also be performed automatically there .
  391. 26:34Things like checking or comparing against another model that reviews and reviews it.
  392. 26:43We then download this data and update it from there .
  393. 26:50So, as we've talked about, we break out early, we set up loops, and we set up repeatable tasks that we're constantly doing, code bases that we're interested in, to prevent
  394. 27:01them from lying and backing out before they actually complete the task.
  395. 27:07Yes, I was working on something important, like the entire agent loop, that wasn't part of the goal.
  396. 27:12It was my own loop and I wanted it to give me confirmation that it was actually doing the work, testing, and so on.
  397. 27:19So I just made sure you were running the test, and I said something like, "Oh, when you run the tests, touch this file called case tested.
  398. 27:27The file where the agent was called case.
  399. 27:30And if that file existed, that means it ran the tests."
  400. 27:34Well, Claude figured it out pretty early on and said, " All I have to do is touch this file."
  401. 27:38I don't actually need to run tests.
  402. 27:40"So it was like a lie, and he's a good programmer, right?
  403. 27:44He strives to be lazy, just like us.
  404. 27:46And that's exactly what he did.
  405. 27:47So you have to make the lazy path valid.
  406. 27:51So I had to add checksum verification to this file to make sure it works correctly.
  407. 27:58Yes.
  408. 27:59And then again to this visualization to make it work.
  409. 28:02The goal is a finite place we're aiming for, and as soon as you 're done, I want you to quit.
  410. 28:07The loop works within the loop.
  411. 28:09I think in some ways, the loop is a little more creative and more adaptable to broader goals.
  412. 28:15For example, with just one loop command, you can set up an auto- complete loop of the programming implementation for a given codebase, where you say, ' Read this on Slack.
  413. 28:25You have access to the MCP in Slack.'"
  414. 28:27Watch my colleague complain about bugs and ask for feature suggestions for this project.
  415. 28:31When you see them, add them to your Linear project as subtasks.
  416. 28:34Then take the Linear subtask off the queue, implement it, run all the tests it passes, and then implement it yourself, right?
  417. 28:41And then close them.
  418. 28:42"You could split it into two loops.
  419. 28:44That would be a little cleaner .
  420. 28:46The point is that it's incredibly, um, just reusable and adaptable to a lot of different situations.
  421. 28:52You could also think about regular reports that you need to generate, and we'll talk about when a schedule would be more appropriate for that.
  422. 28:58But there are tons of different things you can do with Loop, if you have n't tried them yet.
  423. 29:01Yes, definitely.
  424. 29:03And that's where we're heading.
  425. 29:04One thing that came up last week that we experimented with, but didn't have enough time to put it on slides, for example, is Claude Tag.
  426. 29:12It's just a simplification of being able to launch Loop and have some kind of trigger that starts it in Slack.
  427. 29:18It could be a message from a coworker or an update to a PR that starts it, and everything runs autonomously in a cloud environment.
  428. 29:27So it needs to be able to run in this independent loop without you to be able to continue.
  429. 29:32I'm working and being efficient.
  430. 29:34I have Claude Tag set up in my personal Slack right now, and about seven or eight Vercel bots."
  431. 29:40Eve, their new framework, Agentech, and they're doing a ton of different, you know, independent tasks for my company, and Claude Tag is the main interface for that.
  432. 29:49So, yeah.
  433. 29:49We see that it all just converges towards the same primitives, which you have to be very familiar with.
  434. 29:56Here are some examples of goals and loops in practice.
  435. 29:58For example, the goal would be for this test suite to be green.
  436. 30:02There are failing tests, so let's make everything green.
  437. 30:05That's something that can be measured.
  438. 30:06When there are no more failing tests, then the case is over.
  439. 30:09Another problem is that every file in the source directory has fewer than 200 lines.
  440. 30:14This is something that can be very easily measured by running various commands that Claude can execute to do this.
  441. 30:20A loop, for example, would mean I want to run tests every minute and make sure they all pass.
  442. 30:25There are much more efficient ways, so it's a bit of a stretch, but it's one way.
  443. 30:30Another way is to check the deployment and ping me when everything is OK.
  444. 30:34You could have a long process.
  445. 30:35Continuous Integration (CI) or a long build pipeline that deploys your code and you want to be notified.
  446. 30:40Again, there are simpler ways, but Claude could do this too and just set up a loop and check every 5 minutes and then run.
  447. 30:47It doesn't necessarily have to notify you.
  448. 30:49It could run an additional script or something after that happens.
  449. 30:53Loop and tell me when my favorite video game is finally available for release so I can sign up in advance, right?
  450. 30:58And send me an SMS.
  451. 30:59Tons of things you can do with that.
  452. 31:01Oh, right.
  453. 31:05Okay, and then we'll introduce the concept of a Git task tree.
  454. 31:08So, if you have any developer guidelines, you've probably used them before.
  455. 31:12Think of them as a way to give each independent agent an isolated and secure copy of the Git repository to work on simultaneously.
  456. 31:20There are some issues with that .
  457. 31:22We won't go into that .
  458. 31:24There are some experiments that are a bit different, but we've had success using Git task trees.
  459. 31:30So, you can run agents.
  460. 31:32parallel, and that means you can break down an entire afternoon, which is 5-6 hours of programming, implementation, and bug fixing, into separate blocks, and then assign those separate blocks to run in parallel, and get the time down to, you know, 45 minutes or an hour.
  461. 31:48Yes.
  462. 31:48How many of you use task trees these days ?
  463. 31:51Great.
  464. 31:52Half the room.
  465. 31:54Many use Claude's task tree {dash}{dash} to do this?
  466. 31:58{dash}{dash}.
  467. 31:59Task tree No. A couple.
  468. 32:02Good.
  469. 32:02Yes, it's one of the easiest ways to get started.
  470. 32:04You can just go into your repository, you know, whatever it is, and say " Claude Dash Dash" "Work, Tree," and the system will create a working tree and then launch an instance of Claude in it , so you can immediately start parallelizing your work and continue from there.
  471. 32:19Depending on your repository setup, this is either easy or hard , depending on how difficult it is for you to run concurrent versions of the repository locally.
  472. 32:30We have a Mono repository.
  473. 32:32I'm sure a lot of people work in Mono repositories and running multiple instances in different worktrees can be very difficult because it's gigabytes of node modules, for example.
  474. 32:42Virtual machines, Docker, and build systems, yes.
  475. 32:46Database, all that, the right ports, like mapped correctly.
  476. 32:49This is where cloud environments come in and can do all of this too.
  477. 32:54We've experimented with Cloud Tag, Devin, and others, but this is another way to parallelize this work, allowing agents to work independently in the same codebase without conflicts with each other, and then open independent,
  478. 33:06separate PRs that never transfer code changes from one to the other.
  479. 33:12And of course it runs in a loop until the end.
  480. 33:14We think of them as atoms.
  481. 33:18Each block is simply the same loop on a different, higher level, and it can be treated like literally anything.
  482. 33:24We definitely consider ourselves loop engineers now, because that's what we think about when we think about the atomic pieces of work we're actually trying to complete.
  483. 33:34And we could have a loop that fixes this error, while another loop fixes this, and another loop fixes this around, and think about how we handle all of this like that fleet
  484. 33:45tool I showed you, how we think about everyone in the loop.
  485. 33:48I literally loop between them , going from one to the other to keep everything running smoothly.
  486. 33:53Yes, definitely.
  487. 33:53You started a loop that lasted 2 hours.
  488. 33:56It's been a long time lately.
  489. 33:59Yes, I regularly do a lot of planning ahead on loops, and then I can just let Claude run and it will go on for two or three hours without any interruptions.
  490. 34:09It has all these checks built in, like keeping the size small, keeping it readable, reusing the code we have, running tests, getting feedback from Greptile, Code Rabbit, or whatever coding tool we're using.
  491. 34:22We use them all, honestly.
  492. 34:24And then we roll it back out and when everyone is happy , they hand it over to me and I can review it and approve it for someone else, a human.
  493. 34:34We still have human reviews in the loop for, um, a lot of things.
  494. 34:38And yes, er, but before the dust settles, it works.
  495. 34:41I've been shown that it works.
  496. 34:43The code looks clean.
  497. 34:45It looks like code I would have written myself and is ready for human review, rather than just redundant code I'm throwing over the fence.
  498. 34:52for someone else to worry about it.
  499. 34:57So now we'll let you try it out.
  500. 34:59We'll use loops and goals, and we can hand over the task and go away.
  501. 35:05So in this repository you can run {slash} target and then give it the command bun, bun playground targets check.ts and the target is that when you run this check it will show five out of five.
  502. 35:18So everything passes, everything works.
  503. 35:20And if it doesn't happen right away, Cloud will just turn around and keep running until it's done.
  504. 35:26I'll give you a minute or two to run this and see what the result looks like.
  505. 35:30Yes, question.
  506. 35:33Your program does not support my file size.
  507. 35:36I have a 3.5GB JSON file that I'm trying to read from.
  508. 35:42Is this true?
  509. 35:44Oh, cool.
  510. 35:45Which program is this?
  511. 35:48The one where I try to evaluate my Oh, okay.
  512. 35:51Okay.
  513. 35:513.5GB.
  514. 35:57What are you talking about?
  515. 35:59What are you talking about?
  516. 36:01I have my program running all day, every day.
  517. 36:03Cool.
  518. 36:05Great.
  519. 36:05For a year and a half.
  520. 36:06Cool.
  521. 36:08Yes.
  522. 36:09Okay.
  523. 36:10All right.
  524. 36:11just run compact often ?
  525. 36:13Is this how you cope?
  526. 36:16Yes.
  527. 36:22Uh yeah, when you're working with Cloud, um one thing I'll show you here in my, in this, um you can see at the bottom Yeah, okay.
  528. 36:35Oh.
  529. 36:36Yes.
  530. 36:38I see.
  531. 36:40Okay.
  532. 36:41I appreciate the bug report.
  533. 36:42Thank you.
  534. 36:43Yes, we will fix it.
  535. 36:44So yeah, as far as the recording goes, it was uh, it doesn't work on a 3.5GB JSONL file .
  536. 36:49We didn't anticipate this.
  537. 36:52Sorry.
  538. 36:52Thank you.
  539. 36:55Yes.
  540. 36:55No, it's fine.
  541. 36:56No, it's great.
  542. 36:57It's good.
  543. 36:59We don't know.
  544. 37:00I haven't seen this yet.
  545. 37:02Cool.
  546. 37:03Oh, question.
  547. 37:04Yes, sorry.
  548. 37:05After you.
  549. 37:06Okay.
  550. 37:06Um yeah, so what's what's So, like I think there are usage limits.
  551. 37:11Mhm.
  552. 37:12What is the impact of constantly running things in a loop?
  553. 37:15That's a great question.
  554. 37:17The question is, what effect does running things in a loop have on usage limits ?
  555. 37:21And that's something you definitely need to be aware of.
  556. 37:24Yes .
  557. 37:24I will say that this gentleman built it and it is actually in this repository, if you cloned it, it is in the cloud settings.
  558. 37:29It's called the idea generation plugin, the Nick Nies idea generation plugin.
  559. 37:33One of the best ways to deal with this is to simply plan ahead and get deep and lasting clarity on what exactly you're trying to build, what the end result looks like, what failure looks like, what you're not including in what's included, and then set up a
  560. 37:47loop that will definitely generate more token consumption.
  561. 37:50Yes.
  562. 37:51And the way that this plugin, which is why I often don't use built-in ones , I use my own, is that it tries to break everything down into these manageable
  563. 38:02specifications where...
  564. 38:02It can run them in independent context windows.
  565. 38:06In this case, he will try to think mainly about the size of the context.
  566. 38:10We are fortunate that there are currently no restrictions on use, but I see this becoming the case in the near future.
  567. 38:22It aims to perform tasks in small, atomic chunks that it can execute in independent context windows, without a lot of the associated hassle, making it easier to manage.
  568. 38:34Yes, running tasks in loops, and then there's things like in newer versions of Claude where the system creates a sort of little state machine that runs the whole loop in different ways, and there are gates
  569. 38:47in between.
  570. 38:49This may result in high token consumption.
  571. 38:51I've seen it use as many as four or five million tokens for a single task distributed across several workflows.
  572. 38:57It runs 35 agents and runs them all at the same time.
  573. 39:01Such things.
  574. 39:01Well, we'll also talk about...
  575. 39:02A little bit about verification gates in a moment.
  576. 39:04I think this is a different, you know, solution to this problem, which is really token growth or token waste.
  577. 39:09In the worst case, when you just loop, it's because of unrestricted exploration, with no one to tell you you're off course and no model to be able to determine
  578. 39:18that you're wasting, wasting a deadline, you know.
  579. 39:21So, you know, planning upfront and even burning extra tokens on getting it right, using maximal thinking, and then setting up the loop so that there are hard verification gates so they can't just get lost, that's one of two ways, in my opinion .
  580. 39:35Yes.
  581. 39:36So go ahead.
  582. 39:37We built a user interface or something, and then we came in and found out that Claude had screwed something up and it wasn't what we described.
  583. 39:45Hmm.
  584. 39:45I feel like I'm babysitting it.
  585. 39:47Have you developed the best workflow yet that doesn't completely eliminate human involvement in the loop?
  586. 39:51Yes.
  587. 39:52But we want the human to be in the loop at the end.
  588. 39:54Mhm.
  589. 39:56Definitely.
  590. 39:57Yes.
  591. 39:57What is the loop or workflow you have so I don't babysit the UI?
  592. 40:02I wanted the app to launch and test it for me first.
  593. 40:06Yes.
  594. 40:06Yes.
  595. 40:07Yes, great question.
  596. 40:08The question is: how not to babysit?
  597. 40:10What are some ways to make her act more independently without having to babysit her along the way because she makes a lot of mistakes?
  598. 40:19And the solution I came up with was to let her make those mistakes, but record them and track them so she doesn't make them again.
  599. 40:28For each loop that runs, I run a little retrospective agent that will analyze the loop's performance at the end and see, okay, what did we do here?
  600. 40:38Why did we waste so much time here?
  601. 40:40I had to tweak it a lot , or here I was running a lot of tools without changing anything and wondering what it was like with such messy loops.
  602. 40:50NO. NO. It's just...
  603. 40:52It's over after I stop using the computer.
  604. 40:55This is just at the end, after starting a loop during which I had to watch the kids a lot.
  605. 41:00We are working on it.
  606. 41:01I'm working on this with Claude to build a memory file that gets loaded back on the next session to give him some information.
  607. 41:08And he also deals with this to some extent.
  608. 41:11It's like a separate storage system that I play around with in Markdown files myself, but it's something that 's also curated.
  609. 41:18So he thinks about things in his short-term and long-term memory.
  610. 41:22So short-term things will be filtered over time.
  611. 41:25Long-term things are like fundamental assumptions that we keep coming back to and that I don't want you to ever forget, like in our design system.
  612. 41:32This is how you test this particular component, for example .
  613. 41:36These things are loaded and separated using progressive disclosure, so you know, if I don't talk about testing, it won't load the memory about testing.
  614. 41:46specifically.
  615. 41:47Or , if I'm not talking about the React part, it won't trigger any React-specific memories.
  616. 41:51It will only call them when we actually touch the React files.
  617. 41:54I'll also say, you know, specifically for the front end, the way I would approach this problem is Claude's design first, like ideation, the higher the expenses at the beginning.
  618. 42:02If you haven't tried it yet, this is a specific example of Claude tailored to design, and it can evolve for you like a design system, which is then saved across sessions and you can reuse it.
  619. 42:13So we start with the upfront planning and then, and we'll mention this a little later, the adversary review.
  620. 42:18You know, injecting hooks at the end of the implementation loop and specifically loading skills that are designed for UX and front-end testing, er, best practices, and you want that element to be as negative,
  621. 42:31meticulous, and detailed as possible, and for those first 25 builds to fail and say, "This isn't good enough; keep going and fix these things, right?"
  622. 42:41It's like, after you've collected those elements, you can design a loop where, yes, you go and make yourself a cup of coffee, you come back maybe 40 minutes later and you've burned a lot of tokens, but you're happy with it, and it's pretty close to a connection.
  623. 42:52We'll ask one more question and then move on.
  624. 43:08That's a great question.
  625. 43:09The question was: have you experimented with any passive voice or a system that listens to the conversations of people in the room and other operators, and then worked on it?
  626. 43:17Not yet.
  627. 43:18One thing I'll share that was incredibly—and this is a bit of a surprising answer—but it was incredibly, um, um, influential, so I think it will actually bear fruit, and it's worth it.
  628. 43:29To explore: We created a system that allows everyone in the organization to blog in a very unified way, with a unified voice, and that fixes, you know, um, the slop, and that, uh, allows everyone to create a draft blog that looks exactly like, you know, an expert blogger for Work OS would do it, and the most effective
  629. 43:47connector we added was Granola.
  630. 43:50Mmm.
  631. 43:50Because as soon as you filter through all the connections from people, people say, "Oh, this is just a little idea we just had while talking to a client.
  632. 43:57If we had a blog post about it, that would clear it up."
  633. 43:59"And that's all there is to it .
  634. 44:02They just feed it into one MCP call, and the blog bot runs, and what comes out is excellent.
  635. 44:06So I think what you mentioned, passive listening, ambient listening , and also using the computer, will soon get pretty weird when you combine the two and start saying, here's your year, and these 16 specific actions actually drove your business.
  636. 44:19So now I'm going to do it at high speed , you know, etc. Yeah.
  637. 44:23And the only other thing I would mention is experimenting with the Devon and Claude tags and having them appear in ambient in Claude or in Slack messages.
  638. 44:32That's more annoying than helpful right now.
  639. 44:34Yeah.
  640. 44:35But yeah, that's...
  641. 44:37Oh, we need to tinker a bit here.
  642. 44:39So there's a test purpose in this as well.
  643. 44:41The idea is to take this repository with you if you want to continue practicing loops and targets , and you can use the playground.
  644. 44:47Scripts there, and then you can adapt them to your programs.
  645. 44:49Yeah.
  646. 44:50Never tell him, to try harder.
  647. 44:51Uh, he doesn't know what you mean.
  648. 44:53Uh, always give him some kind of...
  649. 44:55You can get mad at him.
  650. 44:56It's good for the soul.
  651. 44:57It's cathartic .
  652. 44:58You should do it.
  653. 44:59We have a whole channel called " bot hate" where we just yell at bots for doing things.
  654. 45:03Uh, but yeah, when you're working on them and it's failing, fix the system.
  655. 45:07Don't fix the code itself.
  656. 45:09Fix the system that generated the code so that next time it can be fixed automatically, otherwise it would n't have done it at all.
  657. 45:15And that's what my retro idea does.
  658. 45:20And speaking of which, this is how you prevent it or prevent it from going off the rails.
  659. 45:25You give it ways to verify and you trust those gates and you believe that they will prevent the model from developing any further until those gates are met.
  660. 45:35So you have more confidence in letting him act on his own because he will understand what "done" means.
  661. 45:41And " done" for me doesn't mean what "done" means for a bot.
  662. 45:44So we need to come to an agreement on what that actually means, and that's what this verification is all about.
  663. 45:49Yes.
  664. 45:50This may vary depending on the project.
  665. 45:52This might happen if you have TypeScript, it will be obvious commands like lint and build, and do you know if it compiles safely?
  666. 45:58When I write for my blog, I have a system that drives me crazy that is non-negotiable for PR requests, so all images have to return 200 from the CDN, the Open Graph image has to be perfectly built, there are other checks in the script , and a bad status code from that bash script causes the
  667. 46:15entire build process to fail, and the system detects this, retries, and fixes it all.
  668. 46:20But these problems are solved before it even reaches me, and the end result is still working PR.
  669. 46:25Yes.
  670. 46:25And that's what I meant when I said, just make the trusted path work the way the lazy path should work.
  671. 46:32And just make sure it has all these gates, and there are many ways to do that.
  672. 46:38You can force things like hooks on it.
  673. 46:41Hooks will prevent it from moving forward or performing actions, or cause it to trigger linting after saving updated files, and the like.
  674. 46:48And these are ways that simply cannot be avoided.
  675. 46:51It won't stop.
  676. 46:52He won't forget about it.
  677. 46:53One common thing we've seen when we do this is we tell him a lot of steps.
  678. 46:57We say, "Here are seven things you must do every time we work together ."
  679. 47:02"And if we're talking about 350, 400,000 tokens, the program will forget about steps four and five because they're so far out in the context that the program doesn't remember to do them, and it gets lost in the details of what we're actually doing.
  680. 47:14And so we've had a lot of success turning this into a kind of state machine in TypeScript that forces it to move forward.
  681. 47:23We've been building a lot on top of the pie to do that.
  682. 47:26Well, you can't avoid it.
  683. 47:29It has to move from one step to the next, and there's no way to get from step C, from step A to step C without going through step B. Yeah.
  684. 47:38Well, we've talked about some of them.
  685. 47:40We can skip some of them.
  686. 47:41One of my favorite things that's been bearing the most fruit, I'd say, in the last three or four weeks since I started experimenting with this is adding a hook to Claude.
  687. 47:49And the coolest thing about that...
  688. 47:50Hooks are also that you can just tell Claude to add a hook, right?
  689. 47:53And it will do it correctly.
  690. 47:54So add a hook, So whenever you're working on something very sensitive or urgent , because it's going to trigger a production issue or has a certain complexity (and you can even evaluate it), you have to push your diff code to Codex 55 for CLI verification.
  691. 48:11You install Codex yourself or you get Claude to help you.
  692. 48:15You go into Codex, log in with your OpenAI account, and go through Google, you know, OAuth , and from there Claude builds with his, you know, constant and always cheerful self, saying, "Oh, I hit the jackpot.
  693. 48:27It's perfect .
  694. 48:28Don't worry.
  695. 48:29Don't look at it."
  696. 48:30"Okay?
  697. 48:30And then he's supposed to open a PR, but the catch is you have to go through and get an adversarial review.
  698. 48:36That means finding significant issues regularly for me, and it cuts down on time because it doesn't start expensive Reptile builds and reviews that we have to pay credits for.
  699. 48:45There's something about it that finds issues locally, locally, before it even gets to that point.
  700. 48:50And so, we found that we even experimented with this a few weeks ago, and we were finishing a pretty major system migration, and we found that four different agents from different companies reviewing the same code, each finding different things, which isn't surprising, considering they have slightly different training, you know, processes, workloads, and so on.
  701. 49:11Codex actually provides some skills.
  702. 49:13They provide review skills, adversarial review, rescue, all these easy ways for Claude to trigger it, get advice from Codex, and then immediately restore it so Claude can work on it.
  703. 49:25In different ways.
  704. 49:26It's a great way to get them working together when it happens, and it's a really good way to just get a multimodal perspective."
  705. 49:33on the code you write and how it works.
  706. 49:36Yes.
  707. 49:37Because, you know, that one model will say, especially if it's still in the same session, that it will adapt back to the context it's still in , where it told you that you told it, "I want you to achieve this goal."
  708. 49:46"And it will try to keep confabulating helpfully and pleasantly and telling you it achieved this goal.
  709. 49:52Whereas a model that doesn't necessarily have the same context, but has a very concise message: here's the input, here's the output, here's the goal, can immediately start finding problems in the code.
  710. 50:04So, yeah, it's basically about not trusting the model's confidence as much as you'd like, or feeling biased towards Claude or any other agent.
  711. 50:12You trust the gates you 've set up that prevent the system from lying to you.
  712. 50:17And when you discover it's lied to you or found a way around it, fix it.
  713. 50:22Don't fix the code.
  714. 50:23So we can try this now with the verification gates.
  715. 50:28We can do something about it to prove that it actually works.
  716. 50:32So in this repository, you can just add a hook that runs a lint, a type check, and a test on every change and fix those, well, flags.
  717. 50:40Fix anything it reports.
  718. 50:42Yeah, I can't talk.
  719. 50:44That's fine.
  720. 50:46You can also ask it for a second opinion.
  721. 50:50So ask it to just He spread out and did an adversarial analysis, and then fixed everything he found.
  722. 50:54That way, the two models work together.
  723. 50:57One fixes it, the other reviews it, and they can go back and forth on it .
  724. 51:00And get the job done.
  725. 51:02Yes.
  726. 51:03You can also, by the way, it doesn't always have to be an adversarial analysis.
  727. 51:07You can say, you know, define the scope of this work until it 's very clear, and then divide the elements where it's like a clean seam, so you can both work on it faster and spend more of my money faster.
  728. 51:18Yes.
  729. 51:19The important thing I do is that we have access to all these models.
  730. 51:23And yes, I often use that adversarial part in the planning phase.
  731. 51:28So Claude and I create a plan, uh, using different skills and ways I would create it.
  732. 51:35And then, once I have those artifacts, I'll hand it over to Claude for Codex, uh, and have him review and evaluate what makes sense and what does n't.
  733. 51:42And often he'll reduce it.
  734. 51:43It'll be something like, "Well, well, that doesn't make much sense."
  735. 51:46"And he'll immediately relay this to Claude, and Claude will say, 'Yes, Codex is right.
  736. 51:49I understand what they're getting at.
  737. 51:51And we often agree.
  738. 51:53It's very easy to influence things that way.
  739. 51:56But in the end, it turns out to be a better plan.
  740. 51:59And I always do it well, like in HTML plans.
  741. 52:03And that's something I stick to .
  742. 52:05I feel like a dog at the computer, like I'm contributing to something .
  743. 52:09But we manage to get something done.
  744. 52:14Okay, schedule a task.
  745. 52:15So we have a few minutes.
  746. 52:17We'll go through it.
  747. 52:18One of my favorite things when you're just starting to set it up.
  748. 52:21What are tasks in the schedule and why do you care about them?
  749. 52:23It's that thing you still dread doing on Slack every Monday.
  750. 52:26It's like collecting reports and I have to dig through them.
  751. 52:28The idea.
  752. 52:29I have to find the Slack with the information about how they wanted the formatting.
  753. 52:32Then I have to download the CSV file and review it like clients.
  754. 52:35All of this can, you know, be worked out in a session first, until it's done.
  755. 52:39And then, once it's working, you can say, " Schedule this for every Monday at 9:30 because I never want to do this again ."
  756. 52:46This will actually launch and save a persistent, you know, cloud session on a virtual machine that, um, can do this work for you on a repetitive basis, regardless of your schedule.
  757. 52:57There are tools in the schedule command, such as list, that let you see what is running and can modify or remove it.
  758. 53:04But think of it like anything you have to do at work or for yourself that's tedious and you want to automate, the schedule command is what you reach for.
  759. 53:12Yes .
  760. 53:13And what next?
  761. 53:14Well, yes.
  762. 53:15So quickly, since we've already introduced a lot of these things.
  763. 53:17Primitives for you, just a quick word on when to think about them .
  764. 53:20Hooks are things that will be run automatically in certain cases.
  765. 53:23Maybe you could write a hook that will always run, but will only continue if this case exists, right?
  766. 53:28Because it's a bash script, for example.
  767. 53:30But these are the things you want Claude or any other agent to always remember, do, never forget, and never be able to forget.
  768. 53:36The point is , if you have a clear goal, like I do, you simply have to push that demand through.
  769. 53:40So you need to fix those 13 tests and then refactor those authentications.
  770. 53:44And when you do that and you can prove it, I want you to stop wasting money.
  771. 53:49The loop is that I want you to oversee this request.
  772. 53:51You know , I want you to constantly ping here and then inform me of changes to this website, and this is the data that will return.
  773. 53:58I want you to go through Slack and collect all the customer complaints and send them all out .
  774. 54:03as linear tasks.
  775. 54:04Keep going until I tell you to stop, or according to the timer I gave you.
  776. 54:07And then the schedule is something that should be permanent.
  777. 54:09I want you to run it every 2 weeks, every Monday, every Wednesday, every Friday, every day, like a cron job, right?
  778. 54:15I want you to do some work, and you might have some secrets too.
  779. 54:19But these are the building blocks that, when combined, can create fully autonomous systems and automate many tedious tasks in your workflow.
  780. 54:28So each of them is a great candidate.
  781. 54:34If you've done any of these tasks , like dependency changes, status reports, evaluation runs, you know if they occur and you have access to Claude and the scheduled tasks, you shouldn't have to do it manually.
  782. 54:43Yes.
  783. 54:43The most important thing for me is always what I like, what do I need to prepare for meetings?
  784. 54:48That's the most important thing for me, often just getting things ready for that and on a specific schedule.
  785. 54:53There are other triggers too, like Well.
  786. 54:55It doesn't necessarily have to be time-based.
  787. 54:58Various things may come up , such as new pull requests .
  788. 55:02How do you deal with this?
  789. 55:03This can automatically trigger the process.
  790. 55:05It could be Slack messages that just take over.
  791. 55:07For example, we have an entire Knitting channel for our documents, and anything posted there immediately gets a linear ticket, and then the bot automatically tries to fix it.
  792. 55:19And 90% of the time it can fix it on its own because these are minor, nitty-gritty issues and it will automatically fix it.
  793. 55:24Otherwise, they would just sit in a queue that we would never prioritize.
  794. 55:29But this way, as soon as they are reported, they are fixed, which is just an amazing feeling.
  795. 55:34I also have one that runs weekly, uh, on my website, and it's a cool visualization that shows me what I've done that week, uh, specifically in terms of token usage, and it shows me trends...
  796. 55:48How I'm using tokens over time, the code that I'm changing.
  797. 55:51It's not just his personal expenses.
  798. 55:53It's not mine.
  799. 55:53These are not my personal expenses.
  800. 55:55I was terrified.
  801. 55:57But that also sums up my PR week.
  802. 56:00These are all the PRs that I've opened, and this is a summary of the work that we've been particularly focused on in a given week, and it just goes back, uh, I think the last 4 months is how far.
  803. 56:10I thought it was brilliant when Nick told me that, because you don't want to rush into an interim review and overthink it and then, you know, dig in every time , especially if you know it's going to be twice a year.
  804. 56:19So it 's just easier to set it up once and then automate it.
  805. 56:23Bright.
  806. 56:26Okay, so a little tip that I think works really well.
  807. 56:29If you're trying to set up your first scheduled task, one thing that really works is providing all the context necessary to connect, connect MCP connectors, add skills, right?
  808. 56:39Get that first work session going on your own by just talking and seeing how it works, and then once it works, say, "Now I want you to schedule this for whatever ."
  809. 56:48"And just keep doing it.
  810. 56:50That way you make sure there are no secrets missing, no connectors missing, you have all the data you need.
  811. 56:59And that allows you to really let go.
  812. 57:01For example, once you've set it up, setting up things that can loop, you think about how I can loop it, how I can make it think about what that means, or what phases it needs to go through to give me confidence that everything will be done the way I expect it to be; then you can schedule it
  813. 57:18.
  814. 57:19You can schedule it and trigger it, for example, when a new bug appears, for example, a new GitHub issue.
  815. 57:24That will trigger everything and not bother you.
  816. 57:26You can be out walking in the woods and, um, only get notified when a fix is ​​ ready, or only when it actually needs your help with something it can't handle on its own.
  817. 57:36Because it 's new and novel, and then you can go in and fix that loop so it does n't have to ask you again, and you can move on.
  818. 57:44Yeah.
  819. 57:44And so, as I mentioned earlier , there's a whole An interface built around a schedule.
  820. 57:50You can list what you have in progress and, um, you can see it.
  821. 57:52You can modify schedules or customize them as needed, then find if they're running and delete them.
  822. 57:57Um, yeah.
  823. 57:58And you can mix and match them too .
  824. 58:00You can say, for example, "Schedule this goal every week."
  825. 58:03"And it will run this goal, uh , and this.
  826. 58:06So now you can try this .
  827. 58:10In the last two minutes we have, we'll go through this, but uh, you could try scheduling it with different loops.
  828. 58:17For example, if you run this report every minute, uh, schedule every workday at 9:00 a.m. You'll run this report and summarize the list you get from it, and then list all the scheduled tasks.
  829. 58:27Uh, and then you can close the loop we had from the beginning.
  830. 58:30So if you just say, 'Run my final summary.'
  831. 58:33" Well, hopefully we'll see the visualization.
  832. 58:36If...yes, if we get enough submissions, we'll see.
  833. 58:39And as long as it works, then...
  834. 58:41that's it.
  835. 58:42Yes.
  836. 58:42Yes.
  837. 58:43We'll check it out.
  838. 58:45So, save all this...content.
  839. 58:48It's all on GitHub.
  840. 58:50These slides are there too.
  841. 58:52We'll also be downstairs at the Work OS booth all week.
  842. 58:55So if you have any questions or want to talk about it, we live and breathe to talk about it.
  843. 59:00We'd love to, but we'd love to ...well, we have one more minute for the last questions.
  844. 59:08Yes.
  845. 59:09Sure.
  846. 59:09Yes, we'll give you the link.
  847. 59:10And in the "Read Me" section at the top, there are links to the slides, a link to the glossary, which will be live, and you can use it to ask questions."
  848. 59:18And here I am…Continuation at check-in for the entire application.
  849. 59:21Incredible.
  850. 59:22Oh, great.
  851. 59:23Thank you.
  852. 59:24We appreciate it.
  853. 59:24We will give you our recognition .
  854. 59:26Yes.
  855. 59:27Yes.
  856. 59:28Yes, sorry.
  857. 59:29I'm a little deaf.
  858. 59:30I think the question was about, in my company, PR, especially for agents, being on volume, it's really difficult because sometimes you can get a stack of 30 PRs and then the code is bad or there's just a huge amount, like 50,000 files in one pull request and you have to review that.
  859. 59:45We think about it in terms of applying the same concepts as we do now, it's a problem of high system complexity.
  860. 59:51And some of the tools that we've had success with are, you know, Devin, Greptile, getting some kind of first warning analysis.
  861. 59:58That doesn't mean Greptile, because sometimes they're wrong, because we know why, for example , if they say five out of five are ready to merge, it's just ...
  862. 60:07The first kind of signal.
  863. 60:08And then you can throw out all the twos and ones and tell people it's not ready for me to care and honestly throw it out to them.
  864. 60:17And then, if it's fours and fives and it looks good, there will be comments and conclusions.
  865. 60:21I will ask the agents in the loop to oversee the PR and correct this until there are no more comments.
  866. 60:27And then, as I get closer to the end of this loop, I'll start using my human brain for the first time in three years, and I'll open up the differential file and say, "OK, what files do I remember that were once sensitive?"
  867. 60:38Let's take a look at the configuration.
  868. 60:40Let 's look at authentication, let's look at published routes, right ?
  869. 60:44And then let's just randomly check it out.
  870. 60:46Well, that's what works for us now too.
  871. 60:48But this guy handles a much larger volume of incoming PR than I do.
  872. 60:52Yes, let me just say that since we don't have time, come talk to us at the stand.
  873. 60:56I have a tool called Data that I'm building that tells you the story of your PRs so you can stay up to date and keep track of all the context.
  874. 61:03I'd love to talk to you about it.
  875. 61:05Yes, great question.
  876. 61:06All right.
  877. 61:07Thank you all.
  878. 61:08We don't have time anymore.
  879. 61:09Thank you very much.
  880. 61:10I appreciate it.
  881. 61:27Great!