Full Workshop: Setting Yourself Up for Success —Jason Liu, OpenAI Codex

AI Engineer · 75 min · 918 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:01[music]
  2. 00:13All right, let's just uh kick things off.
  3. 00:15How many people here already saw the keynote that I gave?
  4. 00:19Okay, not everyone.
  5. 00:21That's good.
  6. 00:21This talk is effectively going to be a stretched version of what I had given in the main stage except two things.
  7. 00:28One, I want to give you some time to try to set things up yourself, you know, Wi-Fi gods permitting.
  8. 00:33And then also two, be a little more interactive.
  9. 00:35I I had to be very high level when I was talking about what I use Codex for, but here if you have any questions, we have like 70 minutes.
  10. 00:42If you have any questions, just raise your hand and we can start answering some of these things, especially because not a lot of the workflows have been really well documented.
  11. 00:48And so if you if you're very curious on how things work, I'm really happy to uh answer any questions we have here.
  12. 00:55Um, yep.
  13. 00:57I'm Jason.
  14. 00:57I work at OpenAI.
  15. 00:58I don't really know what my job is anymore.
  16. 01:00We do a lot of things.
  17. 01:01Um, clearly the slides are already in the wrong order as well.
  18. 01:05But generally, you know, I've done things like doing a lot of prototyping work and just writing lots of code and just having a goal run for two days to build a game or some kind of web application, right?
  19. 01:16We've also looked at things like running evals and hill climbing things.
  20. 01:19Uh I also use computer use to edit iMovies and make little videos.
  21. 01:24I do partnerships and education and operations by taking my meeting notes, turning them into documents, working with other vendors, and also working with different foundations and programs to get funding.
  22. 01:36And all of this work is done effectively in the Codeex app.
  23. 01:40Uh I don't know if you can tell, but right now the slide is being served on local host in the inapp browser of the Codeex application.
  24. 01:46And so anytime I find something I don't like, I might just hit the annotate tool, give a comment, and have codeex clean up these slides.
  25. 01:53Um, I'm not the biggest token token maxer.
  26. 01:58Uh, I I think I'm doing all right.
  27. 02:00I see some folks doing like, you know, a couple couple billion every day.
  28. 02:04Uh, the goal of this talk isn't just to like waste all your tokens, but really help you avoid wasting your tokens by telling you what has actually worked and in particular sort of the tricks I use to make these things productive.
  29. 02:16Um, and again, the goal, like I said earlier today, was to catch you up on what's changed in the Codeex app, uh, give you some time to set things up.
  30. 02:23So, if you have time right now and you haven't downloaded Codex, just go ahead and do that.
  31. 02:27And then we can go a little bit deeper into setting things up.
  32. 02:29And so, I've actually prepared a little monor repo that you can use to clone in, get all the skills that you need, get all the setup that you need, and go from there.
  33. 02:39So, let's go a little bit deeper.
  34. 02:40Um, if you're new, feel free to set things up.
  35. 02:42If you're pretty experienced, just like chill out, try some of the things I'm talking about and then, you know, I think every 15 minutes we'll have some time for questions and we can go into the more like
  36. 02:53what feels like AI psychosis but maybe actually works kind of uh domain of using these systems, right?
  37. 03:00A lot of the work in knowledge work now because the coding is solved, because a lot of this operations work is solved is really just understanding what you can do.
  38. 03:07Right?
  39. 03:07In a world without AI, maybe I have 10 teammates.
  40. 03:11Each teammate is working on one thing.
  41. 03:13So, I need to have like 10 things I'm keeping track of.
  42. 03:16Now, we live in a world where like everyone I'm working with has 10 projects.
  43. 03:20I now have to keep track of like 200 things and I don't know what's important.
  44. 03:24There's definitely a Slack message I've missed.
  45. 03:25There's probably some email I've missed somewhere by some foundation.
  46. 03:28And Codeex helped me organize all of this stuff.
  47. 03:31And so, again, the things I really want you to take away from this workshop is the fact that compaction works really, really well.
  48. 03:37Like I have threads now that are like five weeks old that have you know 400 sub aents in them and they generally just know what they need to do.
  49. 03:48They know what their job is.
  50. 03:49I also want you to become really comfortable with talking your computer.
  51. 03:53Uh earlier today I said that Tony Stark is not texting Jarvis, right?
  52. 03:58And there's really no future when text input is the thing that matters.
  53. 04:04I basically use a foot pedal.
  54. 04:05So I like I have a button that is transcribe and a button that says enter.
  55. 04:08And so I'll just come by my desk with my hands behind my back and I just go like you know fix this make this change also like message this guy on Slack
  56. 04:17and then I just go back to you know talking to my co-workers and trying to figure out what is like the human side of actually working at OpenAI rather than just like monitoring Slack all day.
  57. 04:26Uh appshots is my favorite feature of all time.
  58. 04:29It's like very satisfying.
  59. 04:31If any of you just like are on Codex right now, just press the command button side by side.
  60. 04:36You're gonna get this real nice animation or you're gonna get a modal to tell you to install computer use, just do that.
  61. 04:42It's amazing.
  62. 04:43Uh, invest in your personal memory, right?
  63. 04:47I at this point when someone asks me what I'm doing, I don't even have any idea.
  64. 04:51I kind of have to like look at my threads and look at the conversations to figure out how much I've delegated away and how much has been automated.
  65. 04:57And if you can then invest not only in skills for yourself but also plugins for your entire team, you can become the superhero that actually sort of augments the rest of your company.
  66. 05:07It's one thing to say, "Oh man, like I can use all these tokens and look how many tokens I'm using."
  67. 05:11But actually, if you're rewarded by how often the plugins you've built are being used by your teammates, that's a huge win, right?
  68. 05:19How are we doing like implementation?
  69. 05:20Like one of the most popular skills is just the like finalize the codeex app skill.
  70. 05:25and anyone who makes a pull request basically triggers the skill before review and basically everyone at the company uses it and it's always been able to find things that I've done wrong or is against the style guide.
  71. 05:36Um, one of the skills I have is just like reviewing docs and it's basically just copying the poll requests of uh the PR reviews of our friend Charlie over here.
  72. 05:46And so I just have like a review my code like Charlie based off the past year of like feedback he's given on poll requests.
  73. 05:52Review my code like Dominic.
  74. 05:53And these things are incredibly valuable.
  75. 05:56And then lastly, once you get more comfortable with all those first four things, your pinned threads with automations, these things that wake up these threads over time, they're going to feel like teammates.
  76. 06:08And more interestingly, now that threads can talk to each other, so every thread has the ability to list other pin threads, has the ability to rename threads, and it has the ability to send messages to each other.
  77. 06:18Not only can you have teammates, but you can have teammates that work together and you can effectively start having managers, right?
  78. 06:25And so you you went from an IC enabled by an IDE, then you have pin threads that feel like a team where you're the manager.
  79. 06:31And very quickly in the future, as models get better, this is where the the puck is going to skate to.
  80. 06:37You're going to start having your like manager threads and then your IC threads.
  81. 06:40And I'm sure in the future there's going to be some other crazy orchestration, right?
  82. 06:46And all of this really is due to the fact that compaction works.
  83. 06:50Even six months ago, I don't know like how many people here have been told this, but you were always told if a conversation goes very long, start a new thread,
  84. 06:59right?
  85. 07:00After 20 messages, it's it's not going to be that good.
  86. 07:03Um, every feature should be its own its own uh conversation.
  87. 07:07If you do a code review, start a new session.
  88. 07:09Those things basically aren't true anymore.
  89. 07:11And a lot of it has to do with compaction.
  90. 07:13just pin the thread, rename it to the project ID, and that project thread should be able to delegate to sub agents, create new threads, and have conversations, and then write to your memory vault, which will allow you to
  91. 07:25just log what's happening.
  92. 07:27And then with automations, you can just wake them up.
  93. 07:32And so there's really three acts of working with AI, right?
  94. 07:35Working in codecs.
  95. 07:36You bring the context in, and I'll talk about how you do that and what are the ways you can bring context in.
  96. 07:40Then you work on it, right?
  97. 07:42For example, the slide deck is just in the Codex app and then you take actions out in the real world.
  98. 07:47[snorts]
  99. 07:49So I asked this during the keynote, but I'm also curious what the audience here is doing, but how many people use dictation when they interact with an AI?
  100. 07:59Nice.
  101. 08:00How many use dictation even at work?
  102. 08:04Yeah, I think we I think we should all be a little bit more shameless, you know, uh in doing these kinds of things.
  103. 08:10uh you generally talk about three times faster than you type and it's just incredibly productive to be able to give the messy version of what you're thinking about to the AI and take that extra time and just try to be even more thoughtful to
  104. 08:24the people you work with, right?
  105. 08:26It's like now like I don't want to send my coworker like a 15-minute voice memo, but I should be I should feel very comfortable sending an AI a 15minute voice memo because you're going to include some random tangents.
  106. 08:37You might just say, "I'm pretty sure I had a meeting with Charlie sometime last week about the agents SDK and it will go and read like 35 meeting messages to figure out which one it was and make it relevant."
  107. 08:51And now all of a sudden like whatever memo you're going to write or some project tracker that you're trying to do uh is going to work, right?
  108. 08:57But I would never do that with with um AI.
  109. 09:02So as you guys are doing and uh just listening to this talk like try to just sort of set up codecs the way that I've been describing these things right.
  110. 09:11So once you have your ability to just have you input into the machine a lot more effectively you can start thinking about using things like skills and plugins.
  111. 09:20Um skills is a very simple construct.
  112. 09:22It's just a couple of files and some scripts.
  113. 09:24A plug-in is a library of these things.
  114. 09:26And as you are just doing things many many times, you can start thinking about creating your own skills.
  115. 09:31And as you package a bunch of skills, you might start thinking about building out a plugin.
  116. 09:37If you want to install the plugins, we have a pretty good ecosystem now.
  117. 09:40Something I'm really proud of.
  118. 09:41If you just go in the sidebar, click plugins.
  119. 09:44You can just search whatever plugins make sense for you.
  120. 09:46So if you use Slack, you can install Slack.
  121. 09:47If you use Gmail, Teams, most of these things are pretty built out.
  122. 09:52If there's something that you feel like you are missing, just at me on Twitter and I'm sure one of my Twitter monitors will pick it up and send a message to someone in the connectors team.
  123. 10:03If you're already actually looking at the plugins panel, I also really recommend just starting the process of setting up the Chrome extension as well as computer use.
  124. 10:11We'll talk about this a little bit later, but like computer use was the first time in a long time I really sort of felt the AGI of being at work, right?
  125. 10:21Right.
  126. 10:21I was in iMovie for the first time.
  127. 10:22I didn't know how to use it and it was just teaching me how to like export the movie.
  128. 10:27It was able to like figure out where the sound effects were and it placed it in the right timestamps.
  129. 10:31Really small things like this that really make using a computer very fun again, right?
  130. 10:35Like I don't really have the time to like learn new software.
  131. 10:37But if code if Codex can just show me what's going on, it's it's it's pretty awesome.
  132. 10:41And as you can see the cursor move oftent times we're like cheering for it to to do the right action.
  133. 10:46If you don't have a link for the Chrome extension, you can just click this button here.
  134. 10:50The difference between computer use is computer use can work behind the scenes to control any application, right?
  135. 10:56So whether it's Slack or some, you know, trading software, god forbid, uh it can control all of those things.
  136. 11:03With the Chrome extension, it just controls everything in the Chrome app.
  137. 11:08But the cool thing here too is again it doesn't take over your screen, right?
  138. 11:12Sometimes I'll just be working on my computer.
  139. 11:14I'll go to the Chrome browser and I'll realize that like Codex has just opened up three tabs to just look at my Twitter DMs and then just closes them back up as I'm just, you know, responding to some other email.
  140. 11:25It's really cool to watch these things work in the background.
  141. 11:29You can connect a bunch of other plugins.
  142. 11:30I use things like Notion Linear.
  143. 11:32I also use Obsidian.
  144. 11:33It's just a good time.
  145. 11:36Once you do this, what you're going to find is just by asking really vague questions about your day and just tagging the right plugins, you're going to realize the AI can learn a lot about you, right?
  146. 11:47The AI does not a system now where it does like one search request and try to come up with an answer, right?
  147. 11:53It might check your emails and find a loose thread.
  148. 11:55It might check Slack or some meetings and figure out what's actually going on.
  149. 11:58Who are these people?
  150. 12:00I had I had one of my loops basically realize I was meeting with somebody look at their LinkedIn and realize that uh we went to the same university at the same time
  151. 12:09and so the moment I jumped on my call I was like hey you were also from water you know do you remember this this and this person and immediately
  152. 12:16we had a we had a connection right and obviously I didn't tell him it was AI but that's kind of some of the small things that you can do by just improving your automation you know it can make you closer to people
  153. 12:26um as you build out your memory system right as as you build up the codeex memory system and your your ability to trigger plugins, maybe day one, you have to tag everything,
  154. 12:36but I've become like a worse and worse manager over time.
  155. 12:39Right now, I'll just open up the composer and just say like what has changed about the launch and it'll be able to do a good job, right?
  156. 12:46And it it's that's possible because you have this long history, you have all these pin threads, you have these memories, right?
  157. 12:53It's the same thing with an employee.
  158. 12:55day one you have to show them every standard operating procedure you know but at some point you have an employee that has been here for seven years and you can just say hey I think you should make the company more money
  159. 13:05and they can figure it out but it's only because they have this context um one thing you can just do for example if you want to try it out is you can just say hey check out the schedule
  160. 13:14find all the sessions organize them in a markdown file put them in a spread spreadsheet and you'll just realize that we can do these things and maybe it'll do it with web search maybe you can do it with Chrome home.
  161. 13:25Um, lots of fun things here.
  162. 13:29If you want to get inspired by just looking at what kind of skills exist, we have two really great sources.
  163. 13:35One is if you just run the skill installer skill, it will actually list out all of the OpenAI curated skills.
  164. 13:41These include ones for things like GitHub, uh, best practices when writing playright code, you know, reotion for example.
  165. 13:49But you can also check out websites like skill set.sh SH or use uh I think this is like Versel's skills uh tool and then you just find other skills, right?
  166. 13:58So if I'm thinking about doing some more motion design or web design or I know that like someone told me I shouldn't do like use memo in React but I don't really know what that means.
  167. 14:07I can now go install the React best practices skill, right?
  168. 14:11But again, internally, one of the highest impact things I think I think you can do as like the AI champion in your company is to figure out what the team needs and build out those skills,
  169. 14:22right?
  170. 14:22I have a lot of skills on doing things like triage and how how do you do comms, right?
  171. 14:26If there's an outage on Twitter, how do you like convert that?
  172. 14:29Figure out who who needs to hear this, how do you like start the SEV, what stat sig gates do you need to check?
  173. 14:35All these things are now just automated.
  174. 14:39And that's exactly what I just said in this slide.
  175. 14:42Um, we also have a really good plug-in creator and a skills creator skill.
  176. 14:45So if you just ask Codex to trigger it, it will try to interview you to figure out what's going on.
  177. 14:50And even in a more useful way, you can also just do it yourself once, document everything, and just tell Codex to make a skill from what you've learned, right?
  178. 14:59And as long as you tell it, hey, by the way, every time you run this skill, you're allowed to edit yourself.
  179. 15:03If you learn something new, you can edit the skill file.
  180. 15:06These things will also improve over time.
  181. 15:09And a big theme that's happening over this talk really is just you kind of have to just get really comfortable with asking.
  182. 15:17Like we'll obviously try to make more of these things like more slash commands, but more and more like I'm just not touching a computer.
  183. 15:23So it doesn't even make sense for me to like run a slash command.
  184. 15:26I just want to say what's launching this week, check Twitter, you know, look at what I'm seeing in the browser.
  185. 15:34Um the example I've been developing internally has just been this like developer experience triage skills, right?
  186. 15:39So again this skill just documents like every Slack channel that should be uh you should be aware of.
  187. 15:46It knows which engineers have worked on what projects.
  188. 15:49It knows what Slack channels are taking in feedback.
  189. 15:52I know that if you DM me on Slack and you tell me that some regression has happened, I need to ask for a feedback ID.
  190. 15:58Right now, the agent does this automatically and it does it automatically with appshots.
  191. 16:06So again, I don't know how many times I'm going to say this, but appshots is one of my favorite features.
  192. 16:10How many people here have just like sent a screenshot to Slack to Codeex, right?
  193. 16:17Like almost everybody.
  194. 16:18But the issue is the screenshot does not have that much information, right?
  195. 16:22The model has to then do OCR.
  196. 16:23And if you send a screenshot of like a Slack thread, the model has to like read the Slack thread and then do a list Slack channels function and then realize like there's a guy named Charlie and then do like a list persons.
  197. 16:36It takes a lot of hops.
  198. 16:38But with appshots, it takes not only the image but the entire accessibility tree of the app.
  199. 16:44And so when I give it an appshot of a Slack channel, it knows the channel ID.
  200. 16:48So it knows exactly what function to call to post there.
  201. 16:51It has the user ids of every single person in that channel.
  202. 16:54So if I take an appshot and say do some research and reply, it's only one function call.
  203. 16:59It knows to send the send a message to channel like U12725 and then because you know it knows that Charlie is like U425, it can do that in a very fast hop.
  204. 17:12So, not only is it a very quick way of getting context into your system, it just gives so much more context that these subsequent tool calls do a really good job.
  205. 17:22I have not like filled out a form in like two weeks because I just now tell Codeex to fill out this form, right?
  206. 17:28It it knows all the fields.
  207. 17:30It then figures out that it's in Chrome and so it'll use the browser extension.
  208. 17:33If it's in Safari, it'll use computer use.
  209. 17:36The model has become really really intelligent.
  210. 17:39And so just like you might have a manager that gets an email and they forward the email to me with like three question marks and is your job to figure out what's going on.
  211. 17:47You can kind of start doing that with your AI as you start investing in these skills.
  212. 17:51And most of this is because of the fact that you've built out your memory system.
  213. 17:56So if if you guys are are taking a look at these these slides, JXML/personal monorreo template that is actually the template I use on my personal computer.
  214. 18:08It's basically just a directory tree and a bunch of skills that I use to sort of grow out my memory.
  215. 18:14I'll also make one call out which is uh if you open this in your browser just press appshots and tell Codus to set this up for you and then you can pay attention to the rest of the talk.
  216. 18:30Yeah.
  217. 18:30Yeah.
  218. 18:30Yeah.
  219. 18:33You can just tell Codeex, Jason has written a personal monorreo template on GitHub.
  220. 18:38Please find it and install it.
  221. 18:43All right.
  222. 18:44Yep.
  223. 18:51Yeah.
  224. 18:51So, this is this is a really good point.
  225. 18:54So like for example on the DX team I make a lot of demos and so I have like 16 repos like you know real time demo one like real time demo two like funny right you have all these demos
  226. 19:05I don't create new projects for them right the only project that exists on my sidebar is the like personal monor repo sidebar but codeex is able to still manage files outside of that project directory
  227. 19:20and so in my agents MD file I just say, don't save any of the code in the monor repo.
  228. 19:26Save it in like slashdev.
  229. 19:28And just by that one line, if I tell it to clone a new project, it saves it in /dev.
  230. 19:33If I tell it that I want to work in my slides, it knows that there's like a slashdev slides directory.
  231. 19:38But it's just an easier way of managing everything, right?
  232. 19:40Like I want to start all my projects from my personal vault and then it can touch the file system in any way that it wants to.
  233. 19:48um one column it kind of breaks uh like git review sometimes in the sidebar but generally it's been a pretty good experience for me because I just re I just review my code in in GitHub.
  234. 19:59Sweet.
  235. 20:01Um these are some of the skills I have just installed there.
  236. 20:04Uh there's no need to take a photo just ask codeex afterwards.
  237. 20:08But um the assistant plugin basically has the ability to uh onboard you.
  238. 20:14It will interview you.
  239. 20:15It will figure out what plugins you need to install and then it will actually go create the threads it thinks it needs.
  240. 20:20It'll create the automations.
  241. 20:21It's a pretty fun one.
  242. 20:22I have a bunch of skills on like auditing AI code and AI writing.
  243. 20:27Um, I don't include this, but one of the my favorite skills of all time is called write like me.
  244. 20:33And if you want to make one like that, all you had to tell Codeex is, "Hey, Codex, I want you to read all the emails I've written in the past six months, all the Slack messages I've written the past six months, and write a style guide for how to message just like me."
  245. 20:47And then that's it.
  246. 20:48And then anytime I tell it to send a Slack message or write an email, it'll go, "Okay, this is an email.
  247. 20:53Clearly, this is just like a customer support form, so I will be much more stern in my messaging.
  248. 20:58Let me go draft this email."
  249. 20:59Hasn't failed me yet.
  250. 21:01Um, one thing I've also added that I think are are really valuable to call out is I've made my own loop skill just because I I do like having a slash command every once in a while.
  251. 21:10I'll talk about this in in part two.
  252. 21:13And I also have a skill called simple HTML artifact that just designs artifacts the way I like them.
  253. 21:18I want my backgrounds to be white.
  254. 21:20I want some uh certain style guides.
  255. 21:23And an Ultra Goal, which is like a super version of Goal that we'll also talk a little bit more about.
  256. 21:28Um, and then if anyone's curious, like new person, new project.
  257. 21:31That's just a way of like writing a script to bootstrap a new person.
  258. 21:35I kind of have like a palunt here for my personal life now.
  259. 21:37It's just like a CRM.
  260. 21:39And basically, anytime my AI agent like finds a new person that's emailed me or messages me on Slack or on iMessage, I just keep track of these things.
  261. 21:47And the new project is the same way.
  262. 21:51Let me just double check.
  263. 21:52Yeah, cool.
  264. 21:53Um, and so I'll give you maybe like 10 minutes to try to try to set this up and we can go in a little bit of a Q&A.
  265. 22:00I'm happy to answer any questions about like how we bring context into our systems, how I've organized my personal memory vault and uh, you know, some other crazy uses of app shots if anyone has any questions.
  266. 22:11Yeah.
  267. 22:12What's your question?
  268. 22:12Does this get rid of the need for like an
  269. 22:16[snorts]
  270. 22:20Yeah.
  271. 22:20Um, so the question was, do I still basically use Obsidian Brain?
  272. 22:24The answer is yes, because I still want to sort of like keep track of everything.
  273. 22:28Um, one thing I actually really like doing is I make my monor repo vault like a git repo.
  274. 22:35And so maybe it'll work on it for like a couple of days and I'll come back and I'll just run git diff.
  275. 22:41And by running git diff, I can just see like what the model has updated and what the model has not updated.
  276. 22:46And I can just confidently review that over time and just realize that, oh yeah, like I guess Charlie did respond to this person and close the loop.
  277. 22:54And I didn't realize that, but now I know, right?
  278. 22:56And and oftentimes that's relevant in another conversation.
  279. 23:00More than that, it's also very helpful for when other people are asking me questions, right?
  280. 23:05Codeexes will feel very good about reading my memory vault, drafting a response, and then ask me for permission to send that message off.
  281. 23:12And so if someone messaged me on Slack a question that the AI could have answered, the AI will just try to answer it.
  282. 23:19Um, and it might be simple things like, oh, like who should I talk to about this project, right?
  283. 23:24And the model knows because it's it's in the memory vault.
  284. 23:27Um, one thing I also call is if you want to use more tokens, you can also have like custom automations where the job is to maintain and manage and garden your memory vault.
  285. 23:37Um, but generally that has not been a big issue for me.
  286. 23:42One question over there.
  287. 23:55Yeah,
  288. 23:59honestly I generally go down the path of like yolo one shot only because
  289. 24:04Oh, that's scared the hell out of me.
  290. 24:08Um, only because I know that like the way I build my skills is that they they self-improve all the time, right?
  291. 24:16Like I think the difference would be if I make a skill that I share with my team, I think about that a little bit more, right?
  292. 24:25Because it's like okay, do do does the does the triage plugin know that like who is working on what feature?
  293. 24:30Can it route correctly?
  294. 24:32For my personal work, I generally just build a skill as quickly as possible and every time it makes a mistake, I just correct it and I tell it to to move on.
  295. 24:40And then generally what happens is if I've used a skill for two months, I just generally feel pretty good about sharing with my team because I I've just experienced it working.
  296. 24:48Um yeah, most of my skills connect to so many other plugins and connectors that uh I just don't know how to eval because I can't like snapshot my Slack
  297. 25:00at any given time.
  298. 25:02There's a question over there.
  299. 25:09Sometimes I'll use the codec cli every once in a while if I want to like like be a little bit faster, but generally the desktop app has pretty good has a pretty good experience primarily because everything I do is an appshot.
  300. 25:28Like if I'm watching a video, I'll just like appshot like summarize this and I'll continue to watch the video.
  301. 25:33I'll just watch a video with like the LLM like summary, right?
  302. 25:36Or it's like if I see some kind of form or someone asked me to sign something, I go like appshot use docuign like sign this and save it to my desktop.
  303. 25:43Uh like last week it like docu signed something then like found a faxing service and like fax my medical records.
  304. 25:50Like that's awesome, but the CLI can't really do that.
  305. 25:53Um yeah, any other questions?
  306. 26:00Yep.
  307. 26:02Why?
  308. 26:04Like I'm like learning to juggle.
  309. 26:06I'm like I'm like learning to play the drums.
  310. 26:08Um well, I think it's like two things.
  311. 26:10Like in the office, what I'm trying to do is I'm trying to like talk to more people, right?
  312. 26:15It's like it's like I'm just the AI's assistant to get more context that the AI can't get.
  313. 26:20Uh no, I think my job when I'm at work really is just when the AI is running, I should be talking to somebody.
  314. 26:25I should be like learning about what they're working on, try to make connections and then figure out what are the, you know, points of connection, right?
  315. 26:33It's like I should be talking to more people in real life as AI works.
  316. 26:38Like someone had a question over there.
  317. 26:40Oh,
  318. 26:51[snorts]
  319. 26:53yeah.
  320. 26:54So the question is like when do I think 55 is overkill versus 53 spark I think this is colored by two things like because I have unlimited tokens I I don't really make those decisions and then secondly because I'm not watching my AI work like most of these things are automations that run in the background the latency
  321. 27:12has not really affected me the times where I use Spark is primarily when there's a very simple uh there's a really simple computer use task Like I just wanted to like click all the buttons and fill out this form
  322. 27:27like uh I think I think I have a thing that just checks me into flights and that is like a spark agent.
  323. 27:33And so now like anytime there's an email that's that's like a flight check in my agent will check me in, download the boarding pass and then send the boarding pass to myself on iMessage,
  324. 27:44right?
  325. 27:44And like I just never do that kind of stuff anymore.
  326. 27:48Uh again, it is really weird when you're just like working all of a sudden you check your you check your Chrome desktop and it's just like you know JetBlue is just like the first page but again it's it speaks to the fact that like
  327. 27:59having access to your computer is uniquely powerful because it has your O and your credentials and and your your um your file system.
  328. 28:08Yeah.
  329. 28:09Over there.
  330. 28:16Yeah.
  331. 28:20Sorry, you're you're really quiet.
  332. 28:21Do you mind just speaking up?
  333. 28:41Yeah.
  334. 28:41So I have we'll talk about this basically in the next section but um with remote control I can control both my local computer and like a remote computer.
  335. 28:53I think the difference is computer use is tricky because it has to be on your computer.
  336. 28:57That's one thing.
  337. 28:58The second thing too is if you go into settings computer use there's a flag called locked use.
  338. 29:06And if you enable that, as long as your laptop is plugged in, even if the monitor is closed, you can still trigger computer use commands through your phone.
  339. 29:15Um, that also gets really weird because like go on.
  340. 29:22Yeah,
  341. 29:31you're you're really quiet.
  342. 29:32I can't hear you.
  343. 29:34Okay.
  344. 29:34Okay, cool.
  345. 29:35Um, yeah, like if you have multiple computers, you can still connect both your your your iPhone to both those devices, right?
  346. 29:42Like some people just have a Mac mini.
  347. 29:44Uh, I think the difference is do you want to control your codecs or specifically computer use because that that requires like an operating system with a guey.
  348. 29:51Um, but we can talk about this in in a little bit.
  349. 29:54Cool.
  350. 29:58Like I said before, were there any more questions?
  351. 30:00I don't know if someone just raised their hand.
  352. 30:02Go ahead.
  353. 30:05risk and how do you control that?
  354. 30:08Yeah, I mean there's always some kind of risk like like I think earlier versions might like edit a document a little too eagerly, right?
  355. 30:20Um, but realistically I think we've done I I think these models have done a really good job of being very precautious and more often than not it's me going like no please just just do it like just just like just sign the document like this is like please just like send this message.
  356. 30:37Um I found that uh the 55 models are pretty reluctant to take these like destructive actions.
  357. 30:43That's one thing.
  358. 30:44The second thing is if you look at the sidebar here you have the ability to change your permissions.
  359. 30:55And so as a show of hands, like how many people use like uh ask me for every permission?
  360. 31:02Yeah.
  361. 31:03Yeah.
  362. 31:04Exactly.
  363. 31:04Okay.
  364. 31:05How many people use like full auto full permissions yolo mode?
  365. 31:11Okay.
  366. 31:12I don't like that.
  367. 31:13Um and then how many people have used auto review?
  368. 31:17Yeah.
  369. 31:18So, I think auto review is actually has been really really great.
  370. 31:21And again, I'm usually annoyed by the fact that my models won't do more than I want them to do.
  371. 31:26Um, and so generally has not been as big of an issue.
  372. 31:30The only times the only examples where I'm really annoyed is it will like edit documents it shouldn't be editing, but I just like add something with the HSMD and it's never really
  373. 31:41messed with me too much.
  374. 31:43Yeah, I know that's not a real answer, but I think with a combination of auto review and agents MD file, I have generally felt pretty safe.
  375. 31:51Yeah.
  376. 31:52And then if you're also at an organization, there's different admin settings that you can have.
  377. 31:56So, for example, at OpenAI, uh you can't use an MCP server to send an email if any of the people in that email is a non OpenAI email, right?
  378. 32:06Or like you can't uh send a Slack message to external Slack channels.
  379. 32:11Um, that's when things get dangerous, right?
  380. 32:14But these are some things that you can control.
  381. 32:19One question over there.
  382. 32:20Yeah.
  383. 32:29Do you have any concerns regarding either security and or privacy?
  384. 32:36That's not because I work here.
  385. 32:38Um, I think that I think that's hard to answer because I don't really know what are like data retention policies for like individual versus enterprise.
  386. 32:48But Charlie, do you have any thoughts there?
  387. 32:51I'm just gonna throw over to you.
  388. 32:59Privacy.
  389. 33:00Let's say you get a I don't know email offering you a job at a competitor company.
  390. 33:07[snorts]
  391. 33:09Uh I mean I I think a lot of it goes back to we do want to make like the models are fallible, right?
  392. 33:15I don't think anybody in this room would be shocked to understand that like you can still jailbreak a model for example.
  393. 33:20Uh but both the the models themselves are getting smarter and better at not you know doing silly things and at the same time we're figuring out like Jason mentioned what are those bigger
  394. 33:31you know limitations around the sandbox.
  395. 33:33We we started with very simple sandboxes where it was like you can just run this command and nothing else.
  396. 33:38And slowly the sandbox has grown to the entire computer and I think we're figuring out what are the like computer level or organization level you know edges to that sandbox that we need to build.
  397. 33:52This is me using the delegate skill.
  398. 33:57Great answer.
  399. 33:58Any more questions before we jump into act two?
  400. 34:02Sweet.
  401. 34:03Awesome.
  402. 34:03So, we just talked about a bunch of different ways of bringing context into the system, right?
  403. 34:07You can use your voice, you have plugins, you can use appshots, and then you can also design different skills and plugins to figure out how to do more system systematic work.
  404. 34:18So, now we can talk a little bit more about the work itself.
  405. 34:21So, like I said before, like every pinned thread effectively is a teammate in my mind.
  406. 34:26I have my chief of staff thread.
  407. 34:28I have uh you know, Swix prefers to call it like, you know, the god thread.
  408. 34:31uh I I have a thread to manage the agency SDK whether that's implementation and documentation.
  409. 34:37It has two sub agents that it delegates to uh the CLI, the open source program and uh Twitter.
  410. 34:49But if you want to make it wake up, right, all you have to say is keep an eye on this until sometime, you know, keep an eye on this every 30 minutes.
  411. 34:58If you remember those like secret words, you can effectively automate like about everything in your life at this point.
  412. 35:03Um, and what this does, it will trigger a heartbeat automation, a thread automation.
  413. 35:09You should think of it as a way of scheduling a message back into the thread.
  414. 35:14Right?
  415. 35:15In the beginning when we set up automations, it was very much the case that a automation would create a new thread every time.
  416. 35:21So it might be give me a morning brief and it would create a new thread and then it would do this kind of work.
  417. 35:25But as these models got better, I think the right design is scheduling these messages into the same thread.
  418. 35:31So for example, because if you if you download the monor repo, we have a loop skill.
  419. 35:36If you just do loop, right, this is the equivalent of just saying keep an eye on this.
  420. 35:40Uh keep an eye on this pull request anytime there's feedback fix it.
  421. 35:45Make sure it's always mergeable.
  422. 35:46Make sure it's always rebased on master.
  423. 35:48Make sure that CI is always passing.
  424. 35:50And it'll just do that.
  425. 35:52And then maybe you make a pull request in in on a Monday, you get really busy.
  426. 35:56Thursday afternoon, all the feedback has been integrated.
  427. 36:00You know, CI is passing and uh you know, you're not like 4,000 commits behind uh the main thread.
  428. 36:08[snorts]
  429. 36:08I also do this with support, right?
  430. 36:10Again, if if someone is like dealing with some issues on on Twitter, on Slack, Appshot, right?
  431. 36:16you know like at like at developer experience skill figure this out and it'll say okay this was an issue on the browser side like James is the one that works in a browser the channel is called like browser feedback
  432. 36:30I'm going to post in that channel DM James and then I'll use computer use to open up Twitter to let them know that I've like escalated this internally and then I will check every hour to figure out if James on that channel has responded
  433. 36:45and then let the user No. And then sometime later in the future, you're like checking your computer and all of a sudden like Twitter opens up and it's just like hey so and so this has been resolved
  434. 36:56and then you just you you just hit enter and then you check your thread it's like oh yeah pull request has been made it will get merged by next Thursday
  435. 37:02and like this is a crazy experience to witness right this is this actually allows us to do way more support without making it like the worst part of my job.
  436. 37:16And then with a chief of staff thread, oh this is this is I remember this is where my slides get really messed up thanks to a codeex.
  437. 37:22So I can't do everything just yet.
  438. 37:23Um you can also just do a loop that says check all my connectors and give me an update as to what is the most important thing I should be thinking about.
  439. 37:32You know give it give it to me in a nice format.
  440. 37:34You know make sure you have links to every email that you read.
  441. 37:38Make sure you have a Slack link so you can deep link into the application.
  442. 37:41And now just it's been really really helpful to track these random things.
  443. 37:44And again I think in my chief of staff thread I had that line that says check into all my flights if you can like send me the book the uh the boarding pass on iMessage
  444. 37:56and it just works.
  445. 37:58And again it's really weird when you start seeing your computer doing doing stuff while you're while you're working because again most of these tasks run in the background and because I've had my permissions on
  446. 38:08it's not like it needs to tell me that it's using Safari.
  447. 38:10I just like find out that it's using Safari that might be scared to some people, but you know, it's pretty great.
  448. 38:17And then lastly, this is something that happens really a lot at at OpenAI, which is I don't even know what's what what's launching, right?
  449. 38:26Is something delayed?
  450. 38:28Has something landed?
  451. 38:29Is it, you know, going to happen at 11:00 a.m.?
  452. 38:31Is it happen at 4 p.m.?
  453. 38:33I have no idea.
  454. 38:35We've actually made a ton of progress on this and I'm pretty sure this is also just automation and AI, but I actually used to have codeex make me a PowerPoint
  455. 38:43every Wednesday night of what's shipping for the rest of the week, right?
  456. 38:48And that's useful.
  457. 38:48And then you just say, "Great, it's useful for me.
  458. 38:50Now, let me make sure I can just post this on the channel as a Slack message."
  459. 38:54And now you're again using your skills not only to benefit yourself, but benefit your entire team.
  460. 38:59That's kind of like the plug-in hero mindset.
  461. 39:03And this one's pretty funny too.
  462. 39:04Uh I have an example.
  463. 39:05Let me just double check the slide.
  464. 39:07Yeah, I have another example which has also been pretty crazy which is the one I gave during the keynote which is I had been editing like a short film
  465. 39:16in iMovie and on my bike ride someone gave me feedback about the video.
  466. 39:21So I just went on my phone and I just like sent told the like, "Okay, there's a file somewhere.
  467. 39:25Can you just like find it in iMovie?
  468. 39:27There's only one iMovie project.
  469. 39:28Read the Slack message.
  470. 39:29Export the video.
  471. 39:31If you can't, if the Slack MCP server does not allow file upload, use computer use to upload the file and then watch that thread every hour and if they have any feedback, reexport the video and reshare it
  472. 39:43and then like bite home.
  473. 39:45And by the time I got home, it was like, oh, by the way, it was actually way easier to use a Google Drive connector.
  474. 39:50So, I've just been uploading the same file on Google Drive instead.
  475. 39:53So, they only have like one URL to manage.
  476. 39:56And uh yeah, we like fixed a bunch of stuff in the typography and then we shipped it again like kind of mind-blowing stuff.
  477. 40:03It's very simple but mind-blowing, right?
  478. 40:06But that's what a heartbeat is, right?
  479. 40:07A heartbeat is just a way of waking up your thread over time to take some actions.
  480. 40:11The other thing you can do is set goals.
  481. 40:14So slash goal is pretty amazing.
  482. 40:16It basically defines a verification step and says, "Okay, as long as this is running, check this verification step.
  483. 40:25if if it's not done, keep going, right?
  484. 40:28Very simple idea.
  485. 40:29Um, and as long as there's a verifier, it does really really well.
  486. 40:35For example, I've just been uh migrating a bunch of software into Rust, right?
  487. 40:41It's like if this is a Python project that is amendable to be rewritten in Rustgo, migrate the back end to Rust, make sure all the unit tests pass.
  488. 40:52And I was able to not only rewrite the rich terminal library in Rust, I also rewrote UV and TypeScript um just to see if I could and uh we're like 100%
  489. 41:03test coverage.
  490. 41:04It's it's pretty amazing.
  491. 41:05Obviously, you should not be doing this in your work, but it's it's very helpful to just understand that as these systems have better verification, you can make a lot of progress.
  492. 41:14In the in the monor repo, I've also included a skill called ultra goal.
  493. 41:19And all it does is instead of setting the goal in the app, we set it in a file, right?
  494. 41:25So we have a goal MD file.
  495. 41:27And what that means is you can edit the goal while it's being run.
  496. 41:30So you can add more scope uh just like many many real projects do.
  497. 41:35We also define a plan again that we can reference.
  498. 41:38But the benefit of this is as you're learning more about the projects and you're changing the plan and the goal as these as these models are just like looping
  499. 41:46uh it can update it understanding of the system and then sometimes I have like a state MD file or a work log just to track these like longer running tasks like if if things are running for like a day or two I want to know what's going on and I'm never going to read this like 4 gigabyte
  500. 42:01like you know session JSON object but I can look at the work log have it have another model summarize it using like a side chat and go from there.
  501. 42:11This meant to say remote control, but again, if folks are just like on their computers, I also recommend trying that out.
  502. 42:18In the sidebar, there should be a button that says remote control, especially if you have the iOS app.
  503. 42:22This is the thing where we talked about being able to control your phone.
  504. 42:26So, control your computer through your phone, right?
  505. 42:28So if you go on the iOS app, you enter codecs, you can you can do this like flow where you can scan a QR code and all of a sudden your your your uh touchp
  506. 42:38can message and cue any thread in the in the application including I think remote threads.
  507. 42:45Uh this is super powerful because again oftent times you know every time I try to leave the house someone's like asking me for something and now I can just ask codecs.
  508. 42:54Uh really I should just be having something that monitors Slack and just does it.
  509. 42:58But like I you know I'm not there yet.
  510. 43:01But again this is one of those big uh you know feel the AGI moments, right?
  511. 43:09Yeah.
  512. 43:10And again like I said the chief of staff threat effectively is like the single source of truth for basically what's going on in in in my life on my personal computer.
  513. 43:17I have a different one.
  514. 43:19Uh lots of good stuff.
  515. 43:22This is all I wrote for mine.
  516. 43:24Create a create and pin a cha staff thread every day at 8 am.
  517. 43:28Check all these connectors, figure out what's going on, and then uh you know, do a good job.
  518. 43:34And again, you'll just start you'll start editing it over time, right?
  519. 43:37Maybe you don't like the formatting or you wish you included links.
  520. 43:40Uh for a while, what I made it do was if it found all the emails, not only to ask it to draft the responses, but I would make it open a Chrome tab for every email I need to reply to in Chrome.
  521. 43:51And so I'll open my computer, I'll take a meeting, I come back and on my computer is just like seven Chrome tabs and I can just review the drafts and send each one like small things like this just to prepare your computer
  522. 44:02while your meetings are happening.
  523. 44:03Uh really productive.
  524. 44:05So now we talked a little bit more about like just like doing the work itself, right?
  525. 44:08We haven't really gone into things like artifacts just yet, but I'm also curious if anyone has any questions on how I've been doing things so far
  526. 44:15over here.
  527. 44:16So I notic so I do a lot of
  528. 44:19Yeah.
  529. 44:20I notic.
  530. 44:27I get a lot of stuff.
  531. 44:30So, what's your advice?
  532. 44:32I generally always prefer to have the model write the goal or write the write the prompt itself.
  533. 44:39Like more and more of these models are just getting better at doing that.
  534. 44:43And it's like more in distribution to what they want.
  535. 44:46Um, in reality, I just I will send like a 10-minute voice memo, right?
  536. 44:50I'm just like like, you know, this is some issue that's happening.
  537. 44:54I think there's a project about some thread.
  538. 44:57Uh, I don't know if they got back to me.
  539. 44:58I think their name is Dylan.
  540. 45:00Uh, please look in Gmail.
  541. 45:03Like, take it's like a really really messy.
  542. 45:07And then it's like, okay, well, do I want to make a set of gold?
  543. 45:09Do I want it to create a new thread?
  544. 45:11Do I want to make a new skill?
  545. 45:13Like that's really sort of where my taste lies now.
  546. 45:16It's like, okay, how do I want to organize what the work product is, you know?
  547. 45:20Do I tell it to then do all this work and put it into a index HTML to share it?
  548. 45:25Do I want to make a word doc?
  549. 45:28That's basically it.
  550. 45:29But I generally just send very long messages.
  551. 45:32Yeah, like for example here, like this is not the prompt for the chief of staff thread.
  552. 45:37This is the prompt so the model can make the cha staff thread, right?
  553. 45:42And that model will be much that output will be much better at determining how verbose the the automation is or how often it should check or what connectors.
  554. 45:51Um and then because I have this monor repo, one of the things I do is every project file has a link to every slack channel this project is relevant for.
  555. 46:03And the model will just see that and you go read those slack channels, right?
  556. 46:06every like person file has their email address, their Slack connector, multiple work addresses, and they'll read all those things.
  557. 46:15I would never prompt that myself.
  558. 46:17Yeah.
  559. 46:20Yeah.
  560. 46:20Yeah.
  561. 46:21So, I think in that example, what I realized was when I mentioned the Slack channels that are relevant for a project, the results were better.
  562. 46:27So, I just added that in the front matter of like the markdown file.
  563. 46:30But these are the things that you grow over time.
  564. 46:46the only
  565. 46:51I mean I have not tried like the deepest.
  566. 46:54So the question is like what is the difference between this and like an open claw and a Hermes agent?
  567. 46:58Think I think I'm sure there's differences I'm not aware of right now, but I can imagine a world where it's going to get much like very very similar very quickly,
  568. 47:07right?
  569. 47:07Like most of my work and I'll talk about this later on is like my threads manage themselves, right?
  570. 47:12It could be the same thing as a sub agent.
  571. 47:14Um I don't know how many people use Hermes agents and open claw for like very wide work, right?
  572. 47:22Like I think I like I think I tried openclaw to do some like house automation stuff.
  573. 47:29Yeah.
  574. 47:36Yeah.
  575. 47:36So the qu the comments like yeah with with these models there's only one thread.
  576. 47:40I think that's very reasonable just in in reality there's just there's just like so many things that we work on that like I need the organization right it's like
  577. 47:48you know it's like is there a world where like my banker and my therapist and my personal trainer and my girlfriend is the same person like maybe but like
  578. 47:56my like you know tiny brain can't figure that out and like it's easy for me to understand what the work is by making folders right it's like the computer doesn't know the folders is but the folders are for me in some ways if that makes sense
  579. 48:10Yeah, cool.
  580. 48:13Great.
  581. 48:15No more.
  582. 48:15Yeah, one question that one.
  583. 48:30Yeah.
  584. 48:38Yeah.
  585. 48:39skills, but like I didn't know that it just automatically.
  586. 48:46Um I think it depends.
  587. 48:47I think it depends on how proactive you are in like telling the AI to remember these things.
  588. 48:53So for example, like once I realized I should include Slack channel IDs in like project documents, the model's like, "Oh, there's a Slack channel.
  589. 49:00I should read the Slack channel."
  590. 49:01And that became really obvious.
  591. 49:03Now I I basically never tag things.
  592. 49:06I don't really know when that happened.
  593. 49:08Um but yeah, again a a lot of it is just getting the habit of remember this for next time, update the skill for next time, update the agent dot for next time.
  594. 49:18And that is the
  595. 49:21uh sometimes
  596. 49:25I mean I think it would be hard for me to like prove it.
  597. 49:27Like there's also a memory system that's like outside of the docs.
  598. 49:31Generally, I'm like pretty happy with the memory.
  599. 49:33I I don't know if if I can like give you like a time estimate, but I think I'll just try it out, right?
  600. 49:38Like use it for a couple weeks, have make sure memory is turned on, by the way.
  601. 49:42Like it's also in your settings.
  602. 49:44I know our settings panel is like pretty crazy right now, but um yeah, I think I would just turn memories on and just see whether or not these changes happen over time because like I basically never I don't remember a single time I've like you like at mentioned something.
  603. 49:58Um yeah,
  604. 50:00and the last part is pretty pretty fast, right?
  605. 50:02So we talked about reading context, we talked about working with the context.
  606. 50:06Now that the last thing to do is just like write context.
  607. 50:09Um most of this has been pretty pretty uh simple, right?
  608. 50:13You can draft emails, you can draft Slack updates.
  609. 50:15If you feel very brave, you can send them, but like be please be respectful.
  610. 50:19Like um I'm sure Charlie has gotten hundreds of sent from chatbt uh Slack messages from me over time.
  611. 50:26Uh but I hope they're they sound like me more now.
  612. 50:29Uh building one pages is also pretty good.
  613. 50:32Like this slide deck was made with uh codecs and soon we'll be able to do things like serve applications and now I think you know at least internally so much of the work that we do has just been sharing like
  614. 50:44apps rather than like full documents.
  615. 50:49I don't know how many people know about this but we also have a really good like artifacts ecosystem.
  616. 50:52like more and more uh codeex has become a tool for all of the work that I do.
  617. 50:56It can open and render like Excel spreadsheets, word documents, uh PDFs, slides and with the annotation tool uh editing things is like pretty fun.
  618. 51:07So even with these slides, this is this is actually served on the inapp browser of the Codex app.
  619. 51:12And what I'll do is I'll just give my talk and I'll press next and press next.
  620. 51:16And then when I don't like something, I just select it.
  621. 51:17It's like, hey, like fix this.
  622. 51:18I don't like the white space.
  623. 51:19Like these two slides need to be broken up two more slides.
  624. 51:21I hit enter.
  625. 51:23As Codex is working to clean this up, I'm just going down to get down the slides.
  626. 51:27And it's it's generally been a pretty natural way of working.
  627. 51:30Like this this deck really came from me reading my own blog post out loud and then generating the material as we go along.
  628. 51:39But, you know, it's still it's still like two or three skills to make the slides look this way.
  629. 51:45Yeah.
  630. 51:47Um Yeah.
  631. 51:49And then once you do that, you can do again again.
  632. 51:51It's the same concept over and over again, right?
  633. 51:53You can you can build out these loops that just touch other parts of the system.
  634. 51:56Like most of our project tractors are just Google Sheets updated with loops.
  635. 51:59Uh the slides are loops and the annotations.
  636. 52:02Yeah, I just said everything.
  637. 52:03My bad.
  638. 52:05Um and then one thing that's also very helpful is like as the company gets more like context dense, right?
  639. 52:12Maybe it's all OA agents, just the ability to like summarize things like over Slack has been incredibly useful.
  640. 52:21That was clearly like a slop slide that uh Chad GBT edited in.
  641. 52:28So once you can take actions on these D artifacts, I think the the biggest thing and the thing I really want people to try out is just computer use.
  642. 52:35Um again, it's like it's like the first time I had that like feel the AGI moment, right?
  643. 52:40Like a cursor is like trying to do some action.
  644. 52:43you can see like move across the screen and like when it does it really well like you really gain a lot of faith in the system.
  645. 52:50So we we obviously have like plugins, right?
  646. 52:52And this is for sending stack messages, but then the inapp browser is going to get even more powerful soon, right?
  647. 52:58We're going to be able to basically treat this like the browser that you use.
  648. 53:03Like I now try to use the in browser as much as possible.
  649. 53:05And then for everything else, use computer use.
  650. 53:08Um how many people have used computer use by the way?
  651. 53:11Very.
  652. 53:12Oh, it's like 10% of you.
  653. 53:14What's a crazy thing that you've done?
  654. 53:16What's like the craziest thing anyone's done?
  655. 53:19Any any volunteers?
  656. 53:21Managing my home.
  657. 53:23Home lab.
  658. 53:24Yes.
  659. 53:24What's a home lab?
  660. 53:33Nice.
  661. 53:46Wow.
  662. 53:47Any other like crazy computer use stories?
  663. 53:53Got to get AGI pill and try out computer use.
  664. 53:57Um yeah, one thing I'll say is like because computers are so powerful like you get like you mentioned before there is some like safety component.
  665. 54:05I am now remembering this example where like again because the slack connector was not able to upload files.
  666. 54:10If this model is really determined, right?
  667. 54:13Uh it could be like the one wish willow.
  668. 54:15it just says, "Okay, great.
  669. 54:16Well, if I can't add a Slack uh file, let me like go on computer use and press file upload and do that, right?"
  670. 54:22There will be some times where the model based on how you prompted becomes really determined and say, "Oh, like I I it seems like I can't email someone using the Gmail connector.
  671. 54:32Let me open up Chrome and hit the send button."
  672. 54:35So, those are the kind of things that you should be really wary about, right?
  673. 54:38These are like real security issues.
  674. 54:39And again, um, more than not, like having things in the agent MD file has been really, really helpful, especially if you do things like, uh, guardian mode or auto mode, excuse me.
  675. 54:51Um, and it has also really changed the way I think about uh doing work.
  676. 54:55I feel like now when I'm doing things like building an application, if it's a native application, most of my testing is just done by codeex using computer use.
  677. 55:04If it's a website, again, it's just using the inapp browser.
  678. 55:08Um, I think we talked a lot about this already, like handle service work, like it's kind of awesome just like being on a checkout page, app shot, you know, find me a coupon.
  679. 55:18Like, it's made me more money than than I I would have uh expected earlier.
  680. 55:22Uh, filling out forms, testing applications.
  681. 55:25Um, I haven't really had a good use of this just yet, but it can also control the iPhone through screen screen mirroring.
  682. 55:31So, uh, do with that what you will.
  683. 55:35And this is kind of sort of the escalation of the talk, right?
  684. 55:38Then the question is like what can the computer not do and like why can't it do it?
  685. 55:42And one of the things that like you can't do is control codecs.
  686. 55:45But you don't really need to because these codeex threads can already talk to each other.
  687. 55:51Uh you it can already control itself in very powerful ways, right?
  688. 55:57If you think of the example, this is not the slide I want.
  689. 56:02Damn.
  690. 56:02Okay.
  691. 56:03I think I think I messed up some slides.
  692. 56:06Earlier in the talk, I talked about this idea that if there was some kind of support issue, I could take an appshot and it'll call this like DX triage skill and I will it will rename the skill.
  693. 56:16It'll do the loop and it communicates like Slack and Twitter and it's very nice, but I still need to be the person that triggers these things, right?
  694. 56:23That's the same thing as me seeing an issue and making a polar request in the future.
  695. 56:29Well, I I mean the future for you it's happening already now here.
  696. 56:31But now what I just have is I just have a single like monitor thread and anytime it identifies any of these issues, it will go off and create a new thread and the thread's job is to do all this triage.
  697. 56:45And what might happen is maybe this triage is waiting on someone on Slack to acknowledge this issue.
  698. 56:50Maybe a pull request has been created but it has not merged.
  699. 56:53If someone else complains again in the future, the monitor thread just goes, "Oh, yeah.
  700. 56:57I think this is the same issue."
  701. 56:59Not only is the same issue, let me send a message to that, you know, downstream thread so that it's aware that this issue is recurring.
  702. 57:06Maybe that thread will send a Slack message, but in the main thread, I'll still see a message that says, "Hey, it's been like the third day this issue has been live
  703. 57:13uh based on Twitter feedback.
  704. 57:16Should we do something?"
  705. 57:18And like those are the things I'm I'm it's very hard for me to keep track of these things because I'm just like on Twitter all the time.
  706. 57:25But by the agent being able to just manage and consume all this information, it makes these things much more tractable.
  707. 57:32Um I definitely messed up the slides over here, so I'm going to skip a couple slides.
  708. 57:39Yeah, this is this is the example.
  709. 57:42Um and and I really want you I really really want you to play around with this idea.
  710. 57:47I don't think it's been fully baked yet, right?
  711. 57:51Most of my work is about just having monitors create subthreads.
  712. 57:54These threads are then managed and pinned onto the sidebar.
  713. 57:57Like that's also one of the best things, right?
  714. 57:59Like with a sub agent, the thread just has these like shapeless entities floating in the background.
  715. 58:06Uh like you know the JSON thread or like the Galileo thread or whatever.
  716. 58:12The difference with sub aents versus these threads is because they show up in the sidebar, you can kind of just notice that something has changed.
  717. 58:17You know, there's a new issue that's come up, right?
  718. 58:19And a lot of it too is like just using the sidebar as effectively like the hub of understanding like what are the ongoing like work streams.
  719. 58:28And this has been like super powerful for me.
  720. 58:30But it all just starts from pinning a thread, taking actions, having this monor repo to sort of manage all of your context and then having different ways of waking systems up.
  721. 58:39And earlier these systems wake up because you messaged it or you set up a heartbeat.
  722. 58:44And now these things can be woken up by another thread running somewhere else.
  723. 58:49And so more often than not like most of my automations just happen on the monitor level of threads and they trigger and create new threads and then manage themselves.
  724. 58:59Yeah.
  725. 59:00So I think we talked about a bunch of things, right?
  726. 59:02tal about computer use structured plugins and uh yeah, I think that's basically it.
  727. 59:11Try out some computer use stuff.
  728. 59:12I I know I can't really I'm like really worried about the Wi-Fi here.
  729. 59:16I'm going to try something too bad, but um yeah, try out app shots, try out computer use, and really play around with what uh these models are capable of.
  730. 59:25There's a question over there.
  731. 59:25I'll do that one next.
  732. 59:35I don't think I can show you on this computer.
  733. 59:37They're going to like take me away.
  734. 59:40Um I mean I I can talk about a little bit.
  735. 59:43So So basically my monor repo is set up so that it looks very much like the one over there.
  736. 59:49I have a projects directory and in that projects directory is a named directory for every workstream I'm working on.
  737. 59:55So maybe it is the you know voice launch video or it is the um like the the agents SDK right or it is uh the codeex open source grant program.
  738. 60:07So those are sort of some of the projects.
  739. 60:10Then I have a people directory which is just like every single person that's ever DM me, right?
  740. 60:16I know what they're working on.
  741. 60:18I know what kind of problems they're thinking about.
  742. 60:19I know what other like uh like Slack channels they're part of.
  743. 60:24And then I have a bunch of different like loose notes, agent summaries.
  744. 60:28I have like daily summaries of what I've done.
  745. 60:29This is mostly just to test the limits of AI.
  746. 60:31I don't think it's very useful for that kind of stuff.
  747. 60:34Mostly it's just the projects and the people, right?
  748. 60:39Um, and then I have a to-do list that an agent just maintains.
  749. 60:42So I have a single thread just like checks that to-do list.
  750. 60:44Oh, this says it was undone.
  751. 60:46Let me have a sub agent verify that like no one has done this task.
  752. 60:50Those things are pretty token token expensive.
  753. 60:52I don't really know if it would be worth people do setting this up for themselves unless they just like don't use that many credits.
  754. 60:59Um, but I think the biggest ones is just like the Shiva staff thread.
  755. 61:049:00 a.m. tell me what's happening today.
  756. 61:05Tell me what's happening this week.
  757. 61:07And I think if you just do that, you're going to get a lot of uh a lot of juice out of that.
  758. 61:19It's like honestly I just I've never experienced it.
  759. 61:24I think like I think I think the compaction is just like really really good.
  760. 61:30Um I know this not a very satisfactory answer, but like if I could improve parts of the model, I would rather improve like its writing tone rather than like its ability to search the context.
  761. 61:43Um, generally it's been it's been a pretty good I think it might have to do with like the way the codeex memories have been set up, but I have not dug in too much into those details.
  762. 61:52I think you had a question over here.
  763. 61:56Yeah.
  764. 62:16Yeah.
  765. 62:22Yeah.
  766. 62:22So I think the question was just like how do you deal with memories like bleeding across different projects?
  767. 62:29I don't think I have a good answer to that primarily because it's unclear to me what the downstream side effect would be.
  768. 62:36Maybe like one project uses like npm, another project uses like yarn or something.
  769. 62:42But I mean generally I just clean those up in the agent MD files for those specific projects, right?
  770. 62:47So another thing to mention is like every project directory has its own readme and its own agent MD files.
  771. 62:53And so yeah, like I think sometimes like one project I was working on was like npm and one was like PMP.
  772. 62:59I just clean that up in the agents MD.
  773. 63:01And I was I would be curious like how much of that can be cleaned up just by using those simple tools.
  774. 63:05But talk to me afterwards.
  775. 63:06I'm really curious like what's so bad about the the bleeding.
  776. 63:12Yeah, maybe that that would be a good example of like having project level scoping versus a single thread.
  777. 63:18But um yeah, I might have to look into more details about how the memory part is actually running.
  778. 63:25Any other questions over here?
  779. 63:36No. So I think um honestly at this point it just kind of knows.
  780. 63:42It's like such a it's such a bad answer but it's almost like an AGI like I feel like to me it feels like a very AGI pil.
  781. 63:48But I think generally in the beginning when I was working with it I would have a skill called like check notes and it's like hey if you feel like you need to check your notes like check the notes.
  782. 63:57All right.
  783. 63:58And so if I if I had a hunch or some intuition that the model might not be able to do this, I would just mention like, hey, check the notes.
  784. 64:06Um, but even like if you look at my like uh open profile, I think like my like check notes skill has been used like a 150,000 times, but I've like never mentioned it
  785. 64:19in the past like two three months.
  786. 64:20It's because again, I think this the memory system has been uh doing a lot of the heavy lifting.
  787. 64:24But again it's it's like onboarding an employee in the in the first you know two three months of onboarding an employee.
  788. 64:30You have to give a lot of instruction.
  789. 64:32You have to give a lot of you have to give them a lot of context and you have to let them fail and when they fail you have to give them an opportunity like write down what they've learned and like clean up itself.
  790. 64:42Um I would be very surprised if you got good results in the first like day of setting this up.
  791. 64:47But I mean just yet maybe not to your surprise but now it's like I don't it kind of just works right.
  792. 64:53And that that's a great feeling to have, right?
  793. 64:55Because it it it lets me go back in the flow of just like doing my job.
  794. 64:59You a question?
  795. 65:00Yeah.
  796. 65:00Thanks so much for asking CSS and
  797. 65:09given that it's rapidly changing
  798. 65:15develop like just so not productivity but just like coding in general.
  799. 65:18Yeah.
  800. 65:22Yeah,
  801. 65:28I'm so old.
  802. 65:29That's so long ago.
  803. 65:31Um, like I have a line that's like if you want to have good taste, you kind of have to eat, right?
  804. 65:43Like I think your job is to like consume a little bit more like try out different applications like you know like do you know what a good onboarding flow looks like?
  805. 65:51Do you know what that feels like?
  806. 65:52Have you built an app that makes you frustrated?
  807. 65:55And just like your ability just like consume more things and develop your vocabulary on like how to complain about things that are bad.
  808. 66:02How do you like complain about like the slop?
  809. 66:05Those are the things those are the skills I think will be very valuable.
  810. 66:08And specifically around vocabulary, right?
  811. 66:10like you can't really describe things that you don't really understand.
  812. 66:13Um, and this happens in like both like things like cooking, right?
  813. 66:18Like if you just don't know what the ingredients are, you can't describe that something is too salty or it could be it's like it lacks some acid.
  814. 66:24Like you just need to like consume a little bit more and then think critically about why you like something and why you don't like something.
  815. 66:31Um because I think I think taste is the is a big issue but I think a lot of it is because we are not consuming the right like stuff
  816. 66:41right like try out all the different apps if you're thinking about apps for example.
  817. 66:46Yeah.
  818. 66:49Any other questions?
  819. 66:55Yep.
  820. 66:57[snorts]
  821. 66:59Yeah.
  822. 67:13Uh which part?
  823. 67:14So, so I understand aast
  824. 67:40Yeah.
  825. 67:40I mean like unironically that slide just says like threads can talk to each other just ask.
  826. 67:45Basically there is a list thread tool and a send message to thread tool that Codex has available.
  827. 67:51And so typing that is kind of awkward but really I just say like sometime last week I was working on slides.
  828. 68:02Can you go find that thread, rename it and pin it for next time?
  829. 68:06Right?
  830. 68:06Like I would never type that but again voice input makes it really easy.
  831. 68:10Um, sometimes I'll say for for example, this talk I I I think I messed up some slides.
  832. 68:14Basically, I had a main slide writing system.
  833. 68:17Uh, and I said, "Okay, like this slide could be split up in three acts.
  834. 68:21Make up a thread for each act.
  835. 68:23Pin it, rename it act one, two, and three, and uh, review each section.
  836. 68:28And then once it's done, review the whole slide."
  837. 68:30Um, and yeah, I generally like to just ask it.
  838. 68:35I mean, I think I'm going to try to add more like commands to be a little bit more explicit around thread control, but I think that will just come as the models get better and and as the model is like more aware of what codecs can do.
  839. 68:46Um, but yeah, it really is just like like I think one thing you should try to do is just have a thread and just say read all the other threads that are pinned and like rename them and like use an emoji to color code its like readiness.
  840. 69:01And just seeing it do that will be like a first step um to to just like understand how this thread control works.
  841. 69:06This feature is like very new, but it already has been pretty wild to just sort of have a single thread.
  842. 69:11You're just talking to it for like the entire afternoon.
  843. 69:15Yeah.
  844. 69:17Sweet.
  845. 69:17If there's no more questions.
  846. 69:19Yeah, one more.
  847. 69:28Don't say that name to me.
  848. 69:30No. No. Uh so the question was like do other coding systems um have these kind of tools?
  849. 69:34Not yet.
  850. 69:35I think I'm sure you know I'm sure they will soon.
  851. 69:40Uh but as far as I know, this goes back to the eating thing like I just have not tried like the latest versions of other tools like I don't know if like conductor has these skills but definitely
  852. 69:51um I'm sure again like very competitive space the these features will probably propagate very quickly.
  853. 69:59threads run on old models.
  854. 70:01Will you have the new threads go back and pull that in as context?
  855. 70:05It depends.
  856. 70:05Sometimes just say find the old thread and like update the model and it'll just update the model.
  857. 70:11Like for for example, I I I like so when when uh 45 54 came out and then we moved to 55, I realized all my old automations were just on
  858. 70:2254 and it really bugged me and I was like, "Hey guys, like can we add a feature that like has like GPT latest so we can avoid this issue and someone was just like
  859. 70:33tell a thread to update the other threads and I was like yeah like so much of it is just ask you just have to like develop the language anguage to figure out what you want to do.
  860. 70:42But I think that's again like the most important most important thing.
  861. 70:45Yeah.
  862. 70:45So just ask it and then it'll automatically do it rather than you having to go through it.
  863. 70:49Yeah.
  864. 70:49Like I think I have an automation that just like cleans up old threads and like by like model ids.
  865. 70:54Um but uh yeah.
  866. 70:59What's up?
  867. 71:01Uh not yet.
  868. 71:04I'm not happy with the slides enough that I I'll publish them, but I have a blog post.
  869. 71:08just search if you just Google codeex maxing it's all there uh you know three x's I think uh but I I'll I'll share some more stuff on on Twitter go on
  870. 71:19yeah so you said that you work
  871. 71:21yeah
  872. 71:22[snorts]
  873. 71:24so do you have any loops you
  874. 71:30[snorts]
  875. 71:30get a lot into the limit advice share how we can limit.
  876. 71:47Yeah, I think the biggest misconception here is that like X high will give me the best results.
  877. 71:54And so there's like there's a lot of like X high maximalists here.
  878. 71:57I want everything on X high, right?
  879. 71:59Like uh like I was demoing the like getting a coupon thing and my friend like does an app shot and like hits enter.
  880. 72:05I'm like why did you turn it on X high?
  881. 72:07right?
  882. 72:07It ran for like two minutes like searching like every website available for like coupons.
  883. 72:12Um, obviously I I can't give that much advice because it's something like on my personal computer I don't really run into limits but really get comfortable with like low and medium thinking.
  884. 72:24Like these models are still very very smart, right?
  885. 72:26Like low thinking on five like five five is like still so much better than prior models.
  886. 72:33And for a lot of this work that's not just like you know make me a video game from scratch.
  887. 72:37You just don't need that kind of work.
  888. 72:39Like my my chief of staff I think is like default medium.
  889. 72:43That's very helpful advice.
  890. 72:44I agree.
  891. 72:44Just as a follow I guess in a few words how do I tell the thread to shut up same thing and
  892. 72:52oh uh I mean the answer is really bad but I just like I just say like if there's no updates just reply with like one word no updates.
  893. 73:04Uh that's one thing and the second thing too is um I would also play around with like how often these heartbeats are happening.
  894. 73:09You know are you running this thing every 30 minutes?
  895. 73:12are you running this every 9:00 am?
  896. 73:14That's one thing.
  897. 73:14And the second thing, too, it's like, do you have some kind of stopping criteria?
  898. 73:18Right?
  899. 73:18So, for example, like I was I had some argument with like Amazon and I'm just like great.
  900. 73:23Like they put me in a 75m minute wait list.
  901. 73:26Like check every five minutes if the queue is like better.
  902. 73:31And once you get to five minute wait time, check every one minute and keep replying until you get my money back.
  903. 73:37And I took a shower and when I came back, I had like $400 in my credit card.
  904. 73:46I've not tried too much there, but it's all possible, right?
  905. 73:51Because to create a heartbeat is just edit a text file.
  906. 73:53And so you should definitely be able to create your own heartbeat and then change how frequent or infrequent.
  907. 73:59Um, I might try to do that actually.
  908. 74:01I might just say like, "Hey, like during the weekdays like change your heartbeat to be more active and during weekends or in the afternoons change them to um be less active."
  909. 74:13Yeah.
  910. 74:13Like I've tried versions of the chief of staff thread where I just set a goal that says like never stop and you're only allowed to like set sleep.
  911. 74:20And that has also worked pretty well.
  912. 74:22It'll be like, "Hey, it's like 900 p.m. Jason like I haven't seen Jason post a Slack message in like two hours.
  913. 74:26I'm going to sleep for like five hours."
  914. 74:28Uh and that also kind of works.
  915. 74:32Yeah.
  916. 74:32But um I I think the biggest one honestly is just like people should not be afraid of low reasoning.
  917. 74:39Like X high is not like X high results.
  918. 74:41It's just like think more