Develop at Idea Velocity - Jeffrey Lee-Chan, Snapchat

AI Engineer · 15 min · 239 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:01workshop setup, so um there are some people who kind of have like never installed open claw, and I want to kind of get them set up.
  2. 00:09And then the other people like want to try an experimental thing, which I haven't even fully gotten working, but I think for more advanced people that'll be cool um to get like a sort of staging environment
  3. 00:20and run two open claws.
  4. 00:21So, I think if I'm not, you know, too busy helping new people set up, then I'll try to help people like finish debugging the last part, or maybe someone else will get it.
  5. 00:31And I think uh 10 minutes Q&A, 10 minutes some more advanced topics, and then just more Q&A.
  6. 00:39And uh probably after every 5 minutes or so, we can do a bit of a few minutes of discussion.
  7. 00:44So, I'll just set some timers to make sure I don't go off track.
  8. 00:49All right.
  9. 00:50Let me start.
  10. 01:02All right.
  11. 01:02Can everyone see this presentation?
  12. 01:07Yes.
  13. 01:08All right, cool.
  14. 01:09Yeah.
  15. 01:11So, this kind of describes like my setup.
  16. 01:16Um I'll just keep it really high level for this part where when everything works, you know, maybe like 70% of the time, you describe your task.
  17. 01:25And then what's nice is open claw will have like context and memory about what you're talking about.
  18. 01:31So, for example, like let's say I'm like, "All right, fix this thing."
  19. 01:35It's a very brief Slack message, but open claw can remember like what I've asked it to do before.
  20. 01:41So, for example, I'm working on this thing I'm calling a skeptic agent, and it's a custom code review thing.
  21. 01:47But if I'm like, "All right, fix the skeptic agent."
  22. 01:49or whatever, right?
  23. 01:50It kind of knows what that is, which is nice.
  24. 01:53You don't have to re-explain a lot.
  25. 01:54Um I use multiple agents with work trees, that's very key for parallelization.
  26. 02:00Uh, CI code good integration.
  27. 02:07But you can kind of see you can talk to it and be pretty brief and then, you know, usually get reasonable outcomes, but sometimes you'll have to debug it.
  28. 02:16So, then it's like, all right, there's some time out thing.
  29. 02:19I'm like, give it a longer time out, right?
  30. 02:24And then I asked that here like like uh, you know, what are the priorities and what we're working on.
  31. 02:28And I'm reading it here.
  32. 02:31And then I might have just said like, uh, I just made code for it, right?
  33. 02:38So, another interesting thing about this is you can kind of see from like, um,
  34. 02:43[clears throat]
  35. 02:43the types of responses I give, they're not that like, um, special.
  36. 02:48As in I I actually think an agent could replace me.
  37. 02:51So, that's kind of what I'm working on right now to like get things even more autonomous.
  38. 02:56You know, and you and you'll see this with LLMs like you're working with them and they'll be like, "Hey, I've got this thing ready."
  39. 03:02And I'm like, "Okay, we'll run the test, right?"
  40. 03:06Uh, all right.
  41. 03:07So, I'll share the next slide now.
  42. 03:10Oops.
  43. 03:16Okay.
  44. 03:17We're going to some concepts right now.
  45. 03:19Like, what is Open Claw?
  46. 03:20So, I think what's cool about Open Claw is it's not just about a particular repo or the code, but more of the concept.
  47. 03:28So, you know, I've got this cool super diagram, but first concept is frictionless communication.
  48. 03:36So, I know we have other things like Claude co-work or whatever, but I think they still have a similar concept where it's like, can you like easily talk to your AI versus like you've got a,
  49. 03:47um, remote desktop into your computer or you have to go sit down.
  50. 03:51So, that kind of enables certain really higher level of velocity.
  51. 03:56Um then I think the central axis is very important.
  52. 03:59So, you know, there's a spectrum like I kind of go like a little more cowboy and I give it act.
  53. 04:12Open Claw has these agent orchestrator managers that manage your workers for you.
  54. 04:16Uh or sometimes when I want more personal control, I use tmux terminals.
  55. 04:21So tmux is like a terminal program, but it's pretty good for parallelization and AI development.
  56. 04:27Then I use the open source framework that I forked agent orchestrator to do the workers.
  57. 04:33And then once it gets to here, this part is like not really controlled by me as much anymore.
  58. 04:37Um so, I'm calling them managed agents, but basically like you have a worker that runs Claude code and Claude code itself can run agents and those can have sub agents.
  59. 04:49So once you get to this Claude code part, right?
  60. 04:51This part is not exactly my stack, just stuff I use, but this part of the stack I'm changing a lot more.
  61. 04:58Um I'll pause here to see if anyone has any questions or thoughts.
  62. 05:06Um can you hear me?
  63. 05:07So I I have a question.
  64. 05:08So the the real question is, you know, why why use Open Claw versus directly use Claude?
  65. 05:17So yeah, that's that's definitely a good question.
  66. 05:20Um
  67. 05:20[clears throat]
  68. 05:21The reason I use Open Claw as specialization.
  69. 05:26So when Open Claw makes a decision, I want that context to be more about like the spec or the goals or like the history of what I want in the task
  70. 05:35rather than the code.
  71. 05:37And you can just imagine this, right?
  72. 05:39As soon as you open up Claude, it reads Claude MDs, it reads skills, it reads um MCPs.
  73. 05:46A lot of those things are sort of independent of like the actual task.
  74. 05:49It's more about how to do the task.
  75. 05:51Um so imagine like 25% of your context already taken up by implementation.
  76. 05:57Versus like when you think about open claw, you're like, okay.
  77. 06:00I want to think about exactly what I want to do and how it relates to all the other slacks Jeffrey sent me in the last 2 weeks and put it all together and give me like a reasonable spec.
  78. 06:16Does every single thing a player does, but it does it without a browser, right?
  79. 06:19So you could use MCP, which usually is JSON or whatever or HTTP.
  80. 06:24Doesn't really matter actually.
  81. 06:26And then the final thing is the browser test.
  82. 06:28So some things like are visual.
  83. 06:30I mean, maybe you could even have CSS tests or JS tests, right?
  84. 06:33And that kind of goes here.
  85. 06:35But then the final thing is visual.
  86. 06:37So I would recommend you like try um similar approaches to this and be like, okay, like which problems truly needed that human or not.
  87. 06:45Um and I think what's kind of cool is like every quarter or you maybe even every month like it improves.
  88. 06:53So before like I had to really manually test a lot because like you know, this worked fine, right?
  89. 06:58But these things didn't work that well with agents like last year but like a year ago even 6 months ago.
  90. 07:03But now like um agents are pretty good at like nailing down a lot of browser tests for me.
  91. 07:08Versus like when I started my agent would have a lot of problems, you know, finding a pop-up and entering a password.
  92. 07:14Now no problem.
  93. 07:16Um that was a little long, but does that kind of address the theme of your question?
  94. 07:21Yeah, I think this is good.
  95. 07:22Good.
  96. 07:22Thank you.
  97. 07:23Okay.
  98. 07:23Thank you very
  99. 07:24Just just remind me for all these slash commands.
  100. 07:26I'll put um links here instead.
  101. 07:29All right, cool.
  102. 07:30Anyone have any other questions related to some of that stuff?
  103. 07:37All right.
  104. 07:38If not, I will go here.
  105. 07:43All right.
  106. 07:43So, here's a two websites that I've built.
  107. 07:48[clears throat]
  108. 07:53Yeah, while that loads, I'll show this one.
  109. 07:56So, this is like a AI RPG.
  110. 07:58Um what's cool about this is you can I've got a default campaign, for example.
  111. 08:04Um
  112. 08:04[clears throat]
  113. 08:05Yeah, pick some options.
  114. 08:07You've got a avatar.
  115. 08:09And uh what's cool about this is it builds a custom world for you that reacts to you.
  116. 08:14So, you can like kind of say whatever you want, do whatever you want, you'll get a story back.
  117. 08:18Um the main difference between playing this versus like um you know, you could just always go into the Gemini chat or or chat GPT or whatever is that um I have like a D&D system.
  118. 08:30So, if you play your own like games or novels, like you kind of just win too much.
  119. 08:35Versus like with this, you'll actually do like dice rolls and be like, you know, did the person actually um did you actually succeed in your action or not?
  120. 08:45All right.
  121. 08:45So, while this is going, I'll also um show this thing.
  122. 08:49Yeah, so this is funny.
  123. 08:51Um this website is like a multi-AI analysis website.
  124. 08:55Um so, what I found was like whenever I was doing research or or whatever, I would go to multiple models and I'd be like, what's the answer?
  125. 09:03Then I copy and paste them all.
  126. 09:05I put them into one model.
  127. 09:06Um so, pretty simple concept, but this does it for you.
  128. 09:09So, usually I like these answers better than like asking one model.
  129. 09:14And I'm like, okay, how do I do this workshop?
  130. 09:21So, I can be like, okay, what's the status?
  131. 09:30Uh what's the latest PR now?
  132. 09:34And um what I like about this is it has that sort of vertical tab type of thing.
  133. 09:40Cuz when you go horizontal, it's really easy to lose track of your tabs.
  134. 09:43Um but we've got notifications, too.
  135. 09:45So, when this thing is finished or this is finished and I need to look, it'll give me a notification.
  136. 09:51And I can just focus on clearing the notifications.
  137. 09:54So, here um the way of using these terminals, usually it's more like a manager rather than a coder.
  138. 10:01Um and it's kind of interesting, but it gives you a certain benefit where I feel I feel like the work is not biased anymore.
  139. 10:08So, when I when I code with these directly, um I usually feel like there's a bias, where it wants to say things are really working or whatever, right?
  140. 10:16Versus like, you know, um I'll show this.
  141. 10:21Let's look.
  142. 10:24If this had been working on PR 294 by itself, I think it would have been like, "This PR is amazing.
  143. 10:29Like, we got to merge it, right?"
  144. 10:31But then this one was like, "No, like, there's another PR that should supersede it, and probably we should just close this PR."
  145. 10:37Right?
  146. 10:37So, that's kind of the benefit you get, where the manager has a different context um than the workers.
  147. 10:44[snorts]
  148. 10:45Uh let's see.
  149. 10:48All right, so this is compacting whatever.
  150. 10:50So, it will take some time to complete.
  151. 10:53Don't want to wait for it, but when it completes, you see these notifications, and I think my efficiency has improved a lot with tmux.
  152. 10:59Um I think is Austin or someone from tmux here?
  153. 11:05Yeah, here.
  154. 11:05What's up?
  155. 11:06Go.
  156. 11:07Yeah, so Austin is uh
  157. 11:09[clears throat]
  158. 11:09one of the guys behind tmux.
  159. 11:11Um you know, big fan of it, and I'm not I'm not paid to say this or anything, but big fan of tmux.
  160. 11:17Uh been using it a lot.
  161. 11:19I don't know, you can spend a minute if you want to talk about anything cool about tmux.
  162. 11:24Yeah, so yeah, one of the creators of tmux.
  163. 11:26Really happy to have Jeff as one of our power users.
  164. 11:29And also like kind of endorsing it for free.
  165. 11:33But yeah, lots of uh yeah, lots of things that we're shipping.
  166. 11:35We shipped a Cloud Code Teams integration.
  167. 11:38So, if anyone is using Cloud Code Teams and wants to actually see what the salvations are doing, we'll automatically spawn terminals for that.
  168. 11:46We also shipped Cmux SSH.
  169. 11:49So, if you do any SSH work, you can use our native Cmux SSH to you know, do your tailscale and etc. To other computers.
  170. 11:57You can even use it to run your own open claw in your Mac minis.
  171. 12:03But yeah, just one of the creators.
  172. 12:05Please feel free to email me if you have anything that you think is a bug or any feedback, feature requests.
  173. 12:12I'm always available and yeah, just love seeing people using Cmux and always want to take as much feedback as I can to like get them as get them
  174. 12:32Okay, so just download that.
  175. 12:33Yeah.
  176. 12:35They actually have an app too.
  177. 12:36So, the app might be nice.
  178. 12:37Yeah.
  179. 12:38Okay.
  180. 12:40And then use that to help me get back in and set set up.
  181. 12:44Exactly.
  182. 12:44Yeah.
  183. 12:45Okay.
  184. 12:45Okay, cool.
  185. 12:46Thanks.
  186. 12:47All right.
  187. 12:48Is using the sandbox, it should not.
  188. 12:50Where would you Have you had Have you run into any troubles Aaron about sandbox token usage?
  189. 12:57Uh no. I'm just I'm not technical at all.
  190. 13:01I have open claw installed, so I'm
  191. 13:03Okay.
  192. 13:04very newbie questions.
  193. 13:06No no no bad questions.
  194. 13:07All good.
  195. 13:08Yeah.
  196. 13:09Uh they're not the sandbox wouldn't do that.
  197. 13:12I think if if everything's working well for you, you know, keep using the sandbox.
  198. 13:16For me, I just you know, don't use a sandbox and so so far nothing bad has happened.
  199. 13:21But I think if I had an external bot, then I would use a sandbox too.
  200. 13:27You changing the model testing a new model like it'll go down then I have to figure out how to get it back up again.
  201. 13:32So I think I want to set up this sandbox because
  202. 13:35Oh, sorry.
  203. 13:35Sorry.
  204. 13:35Um okay, wait.
  205. 13:36There's two kinds of things.
  206. 13:37Okay.
  207. 13:38The sandbox that most people talk about means like they run it in Docker or they like have it running in isolated part of their system.
  208. 13:47But the other one was the one where I made the staging instructions where I was like oh
  209. 13:50Yeah.
  210. 13:51Okay.
  211. 13:52So that could double your token usage if you send the same work to both of them.
  212. 13:58If you have two of them, right?
  213. 13:59Um but I think the way I would use it is not like that.
  214. 14:02Like I think I would do local development and then I would run integration tests on the um sandbox or staging one, right?
  215. 14:09So I have two of them.
  216. 14:10And then once like everything's good, then I would merge the code and deploy to the production one.
  217. 14:15I see.
  218. 14:16It wouldn't double your usage, but it would definitely increase it.
  219. 14:19But then the benefit is you might have some more reliability.
  220. 14:22So
  221. 14:22Yeah.
  222. 14:23I I made some instructions and I was kind of in the middle of setting it up, but then, you know, it's like competing with the other one, so I got to debug it.
  223. 14:30Yeah.
  224. 14:30But I think, you know, it's it's worth trying out though.
  225. 14:32So I had that for more advanced people to give it a give it a try.
  226. 14:37Cool.
  227. 14:38Thanks.
  228. 14:38What
  229. 14:38Or if you're not as advanced, if you want it, you know, you can try it.
  230. 14:41Yeah, I'm definitely going to play around.
  231. 14:43But what a what model do you use?
  232. 14:46Like do you have a something that like a default orchestrator model and then you say for this you use
  233. 14:53use Codex 53.
  234. 14:55Um I found I found uh GPT 54 to just use more tokens.
  235. 15:00Um and even 53 like like I I just get destroyed all the time, right?
  236. 15:06So I might have to adjust that.
  237. 15:08Um so basically I use this until like this is getting low and then I just switch to MiniMax, which is not as good, but it kind of gets the job done.
  238. 15:17And then this is more about money than like preference, but some some work I can still kind of just do um um here.
  239. 15:25So, I don't always have to