fighting slop with slop — Vaibhav Gupta, Boundary

AI Engineer · 21 min · 288 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:01[music]
  2. 00:13Fighting slop with slop.
  3. 00:16My name is Vaibhav and I'm going to talk about something that is a little I would say maybe a little silly at first.
  4. 00:23I'm going to show you our team's engineering practices really quickly.
  5. 00:26We do no code reviews.
  6. 00:28We require every engineer to work on things in parallel.
  7. 00:31And we have no standardization on how people do AI.
  8. 00:35And I know immediately what almost all of you are thinking.
  9. 00:38We're probably a Zoomer YC startup.
  10. 00:41And I can guarantee you I'm clearly a millennial.
  11. 00:44So what do we actually do at our company without code reviews?
  12. 00:47Well, we built about 3 years ago we decided to build a programming language.
  13. 00:52That's something that has absolutely no room for slop.
  14. 00:55It's something that has to work every single time exactly the same way.
  15. 01:00It's something that you can't just change a year later or a month later because you made a bad design decision.
  16. 01:05You have to be correct.
  17. 01:06And for last 3 years we've been in an onslaught of war against slop.
  18. 01:11And when I first met this enemy, I went to my great mentor.
  19. 01:16Slop Sue.
  20. 01:17And he taught me something.
  21. 01:19To to defeat the slop, we must become the slop.
  22. 01:23So we began and we prepared and then we started winning.
  23. 01:27So when we think about it, what is slop?
  24. 01:30Slop is just any code you don't read.
  25. 01:32And whether any of you admit it or not, this is the least amount of slop that your code base will ever have.
  26. 01:39Cherish it.
  27. 01:41So we started fighting back against the slop and we started fighting back with slop.
  28. 01:46So how do we go ship a stable programming language with these engineering practices?
  29. 01:51Well, the first skirmish we ever had was a skirmish of standards.
  30. 01:55The hard part about hiring great engineers is you sadly can't tell them what to do.
  31. 01:58Some of them want to use Claude, some of them want to use Codex, some of them want want to use the latest thing that they just found on Hacker News.
  32. 02:04So, instead of trying to hold standards in our codebase, we did something that is an invariant.
  33. 02:10We built an architecture.md file.
  34. 02:12Instead of using Claude.md, just pick something that every model can just understand.
  35. 02:17This file has to be incredibly small, and it can only have things that will not change for months or for years.
  36. 02:24In our case, it's the layers of the compiler.
  37. 02:26You go deeper into the compiler, tell the agent to just talk to at least one other person.
  38. 02:31That slows it down a little bit.
  39. 02:33So, now we have standards, so anyone can use whatever they want.
  40. 02:36But, the real foe we faced was actually the battle of design.
  41. 02:40Everyone here knows that you have to write perfect design docs.
  42. 02:43And we have a very simple rule in our team.
  43. 02:46Code can be slop, writing cannot.
  44. 02:48And of course, if I tell every engineer this, they write beautiful writing, and they handwrite everything, they don't use AI.
  45. 02:52Well, sadly not.
  46. 02:54So, we built a design tool design doc tool.
  47. 02:56What this design doc tool does, it's a replacement for both Notion and GitHub effectively for design docs.
  48. 03:01It will have versioning, commenting, all the stuff you want.
  49. 03:03And obviously, we do this, people use this.
  50. 03:05Well, sadly not.
  51. 03:06We built another tool on top of that.
  52. 03:09And this tool was a Slack integration for that tool.
  53. 03:11Every time a design doc got updated, this channel got notifications.
  54. 03:15And what ended up happening is this channel became the most popular channel in our company really fast.
  55. 03:18At 2:00 a.m., someone shipped a new design doc, three people started reading it right away cuz it's just interesting.
  56. 03:23Then, most interesting stuff is design docs that are not going to change.
  57. 03:27But, this wasn't enough.
  58. 03:28All of this is actually backed by Markdown files and simple CLI scripts that make it treat like GitHub without being GitHub itself.
  59. 03:35So, now agents can go do this.
  60. 03:37But, the real problem with all this is I built this, and I hit a little bit of AI psychosis, and I started shipping 10 design docs a day, and soon the team was fighting my slop.
  61. 03:48So, we had to go and back at the last rule.
  62. 03:49This last rule was, if you're going to ship a design doc, you require people to actually go read it.
  63. 03:54And with this last standard, we suddenly had design docs that were incredibly high quality.
  64. 04:00But what about the battle of architecture?
  65. 04:02You how do you have your code base converge?
  66. 04:04We built another tool.
  67. 04:05This tool basically visualizes our dependency graph internally with some external dependencies as well, and allows us to watch the code base change.
  68. 04:11It has semantic boundaries, individual packages, but what's more interesting is we can go build CLI tools that guarantee that certain invariants can't be broken.
  69. 04:20And what this does is when Claude builds a new package or adds a dependency that's leaky, we now have CI/CD changing or a simple Git commit history that tells us exactly where things break.
  70. 04:31And by this, we're actually able to make our architecture change.
  71. 04:33We haven't changed our architecture in the last three or four months.
  72. 04:38But as much as we might do design docs, and as much as we might have stable code, would you genuinely ship code without reading it?
  73. 04:45Would you trust your team to go do that?
  74. 04:47And think about a programming language.
  75. 04:49A programming language just so many invariants.
  76. 04:51You have generics, you have closures, you have memory allocation, you have FFI boundaries.
  77. 04:57Could you trust that system?
  78. 04:58Python has bugs 25 years later.
  79. 05:01Well, here's where we did something slightly different.
  80. 05:05What we did was we built a system that actually has agents constantly running and creating BAML programs.
  81. 05:12We take these BAML programs.
  82. 05:151 second.
  83. 05:17And we have agents try and spin something up from scratch.
  84. 05:19We then look at the entire Claude transcript, see what tools it used, see what happened, and obviously we as humans can inspect them.
  85. 05:25But more importantly, we can have agents go inspect them.
  86. 05:29And agents find what was good, what was bad.
  87. 05:31And not just what was bad in terms of what was incorrect in the language, but what took three tool calls when it should have only taken one.
  88. 05:39And then we can go ahead and find issues, and we can have humans collaborate with these issues to figure out which ones are real, which ones are hallucinations, which ones aren't don't have taste, as much as I hate to use that word.
  89. 05:48And then we can have agents go ahead and create fixes to these problems and go to address them.
  90. 05:54And most importantly, instead of trying to just detect these issues, we can go one step further.
  91. 06:00What if you could find language features and instead of guessing what was good, guessing what skill was good, you could go and AB test it.
  92. 06:06You could figure out which ones took less tool calls, which one took uh which one made less errors, which one produced the correct outcome and deterministically know what's going on.
  93. 06:17The point is you can start building data-driven systems without ever writing a single line of code.
  94. 06:22And the thing that really I care about the most over here is not that any one of these tools is specifically what you should go build.
  95. 06:30But the fact of the matter is in order to build a programming language, it wouldn't have taken eight people.
  96. 06:35It wouldn't have taken less than two years.
  97. 06:37It would have taken hundreds and thousands and tens of thousands of man-hours and then you would still have a broken system.
  98. 06:43And today we can just spend millions of tokens and make it work.
  99. 06:46And we can make it stable.
  100. 06:48And you could you two can go home and build these internal tools and these sloppy tools to make sure that your code bases can ship without really having to read necessarily every single line of code.
  101. 06:58Cuz your engineers aren't going to.
  102. 07:01And I think we can start winning this battle against slop.
  103. 07:04And as we win this battle, slop can be defeated.
  104. 07:07But sadly, I have a sad thing to say.
  105. 07:10I think we're still going to lose the war.
  106. 07:13I think the reason that we're going to lose this war is because some of the foundational stuff that we try and go use itself is broken.
  107. 07:22How many of you have used TypeScript?
  108. 07:25Probably most of you hopefully at this point or at least your agents have, so something around there.
  109. 07:30Um did you know that TypeScript's main design goal is to strike a balance between correctness and productivity?
  110. 07:36And there's an asterisk here because what they really mean is human productivity.
  111. 07:41And if you think about it, there are things you would never do in a programming language at the very core layer if you were designing for in a world where humans never wrote a single line of code.
  112. 07:50Let me show you what that really means.
  113. 07:56I'm going to write something and try and guess what this code does.
  114. 08:01Pretty safe.
  115. 08:05What about this one?
  116. 08:08Or even more so, this one.
  117. 08:11Why do we turn things to strings when we sort them?
  118. 08:14This is just slop baked into the language, whether you like it or not.
  119. 08:19What about this?
  120. 08:21I love this part of TypeScript.
  121. 08:24And you know what my agent loves?
  122. 08:26This part of TypeScript.
  123. 08:30This is slop baked into the language.
  124. 08:33And whether you like it or not, the systems will have slop if you build using these tools.
  125. 08:39Oh, I'm sorry, wrong talk.
  126. 08:42But, if you think about what JavaScript does, JavaScript exists, and then after JavaScript existed, we started building systems to layer it on.
  127. 08:54We built CoffeeScript, then TypeScript, and now we're trying to build a fact.
  128. 08:58But thing is, the thing underneath is already broken.
  129. 09:01And more so, the way we write code is also different now.
  130. 09:06So, why are we trying to patch something like this?
  131. 09:08Why don't we just try and do something a little different?
  132. 09:10And I think what we might need if we try and go do that is basically going to be a made-up language.
  133. 09:16So, let me show you what BAML really can do.
  134. 09:19And when you start thinking from first principles, how you can try and combat slop from the very foundational layer itself.
  135. 09:26I keep talking about not reading code.
  136. 09:29Does it even matter?
  137. 09:31Well, let me show you a new way to think about code.
  138. 09:33And this isn't to say we all have to go do this right away.
  139. 09:37But, what if every single time I look at code, whoops.
  140. 09:47What if every single time I look at code, what I really saw was not the code itself, but a quick little thing that could actually visualize all the code for me.
  141. 09:58As I clicked around, it took me to exact lines of code that was linked to.
  142. 10:01If I wanted to have a slightly broader view, I could zoom in and click around and have it expand.
  143. 10:07I could navigate my code bases more interestingly.
  144. 10:10I'm going to let this run really quickly, but while it runs, I'll show you a different pipeline.
  145. 10:14Without any of you ever reading the code, you know I'm setting up stuff and I have an agent loop because the semantic boundaries in there.
  146. 10:19I can expand this.
  147. 10:21I can keep expanding this and I can say, "Nope, that's too much slop.
  148. 10:23Let's let that be slop."
  149. 10:25And walk away.
  150. 10:26So, instead of having to understand all the code, I can opt into what parts of the code I want to read and understand and go to the exact lines when I really care about them.
  151. 10:34But, if we go back to the previous pipeline that was running, what if while it's running, I can actually get a full execution trace?
  152. 10:40In a world where we don't read all the code, the only way to understand the code is actually by the execution trace.
  153. 10:46And actually by seeing exactly how much time was spent on what parts of my program at any given time.
  154. 10:52If you want to go and actually track your program through, think about how slow your program would be if you have to go trace everything in Python or TypeScript.
  155. 10:59It's untenable.
  156. 11:01And the best part here is if you start from first principles, you can make this effectively zero performance cost.
  157. 11:07Not only can we make it great for humans, but because it's all built for agents anyway, you can go ahead and make it so that every single file has a tracing system that Claude can navigate through.
  158. 11:17So, Claude can find what were bugs, what were errors, and what were inefficiencies and start optimizing your code without you having to do it yourself.
  159. 11:25And I think if we go start thinking about it from this way, it's not so much about reading all all code, but it's more so about us human understanding the system that you're working with.
  160. 11:34And the tools that you can build can give you information about the system that you're working with.
  161. 11:39But, I think there's another layer to it.
  162. 11:43We've spent decades building IDE tooling.
  163. 11:46And that Think about how long it took before someone like me who does not know how to escape Vim to this day can finally start using VS Code.
  164. 11:54It was a beautiful day when that happened.
  165. 11:56I became a real programmer.
  166. 11:58Well, according to some people, I'm still not cuz I can't write Vim code.
  167. 12:02But, what does agent-first tooling look like?
  168. 12:07I think we're all familiar with grep, so I'm not going to go and talk about it.
  169. 12:12But, I will talk about Ripgrep cuz grep should not be used anywhere.
  170. 12:16If I want to grep through my code base and understand what it was, I would Ripgrep say something like calculate, and it'd give me a bunch of code where everything was being used.
  171. 12:25And maybe it'd be somewhat useful.
  172. 12:28But, what if you could instead start describing code and say, "Can you describe calculate for me?"
  173. 12:35What if it came with all the docstrings?
  174. 12:37What if it came with the actual source code?
  175. 12:39And what if it also told you everywhere it was actually used under the hood?
  176. 12:43We can make something that used to be multiple tool calls a single tool call all of a sudden.
  177. 12:49What if the way you want to learn about libraries that you're using, instead of having to go to a web search, you just said you did you could just ask for any external library as well.
  178. 13:00And it would just give it to you.
  179. 13:01Because when I first started learning how to code, one valuable lesson I had was the code is always the source of truth.
  180. 13:07Don't read anything but the code itself.
  181. 13:09The docs may lie, the um the actual description or architecture file or readme file will definitely lie, but the code cannot lie.
  182. 13:18Except if you're working on some weird architectures.
  183. 13:21And then when you go down this road, you go from not reading the code to understand the architecture.
  184. 13:25You go from not searching the code to understanding exactly what you're getting in every one tool call.
  185. 13:30But, what's the next thing you do?
  186. 13:32Well, the last thing I do to truly understand code is I run the code.
  187. 13:36So, what if every single thing you ran, every single function you ever wrote was immediately available And I'll pull this code over here.
  188. 13:47Was immediately available as a simple CLI command.
  189. 13:51So, if I run add, add becomes a CLI command that has A and B parameters attached to it.
  190. 13:56And I can just run it really quickly and see what happens.
  191. 13:59What if every single CLI tool I had could be packed into a live into CLI that's completely standalone.
  192. 14:09Multiply.
  193. 14:12That I can just run without ever having to actually execute any of the code.
  194. 14:18And it's now a total CLI binary that has functions just bundled in.
  195. 14:21Suddenly, we can build really quick tooling where agents don't have to go grep through what's happening.
  196. 14:26Everything is type safe, everything is deterministic, and everything is actually guessable.
  197. 14:31And the best part is imagine you could build on any system, and your agents don't have to worry about deployments across Windows, Mac, and Linux.
  198. 14:39And you can just target any layer you want, and it builds for any architecture, including WASM systems.
  199. 14:45All of a sudden, as an engineer, you're super charged.
  200. 14:47You're no longer bottlenecked by what you can do in the systems underneath you are preventing.
  201. 14:51You can just move very fast.
  202. 14:53You can move at agent speed.
  203. 14:56But, a lot of the stuff that I've been talking about to this date has been about tooling.
  204. 15:01What if we try to fix some of the real sins of JavaScript?
  205. 15:04Some of the stuff that is deep in the language.
  206. 15:07Not the sort stuff, but I mean more important stuff, like errors.
  207. 15:12Have you seen error handling be beautiful ever, other than Rust?
  208. 15:17Um what I see agents do over here is you do try catch, and then they keep nesting try catch after try catch after try catch.
  209. 15:23And eventually they give up and say console.log some error happened and deal with it.
  210. 15:28What if we could do error handling from very first principles?
  211. 15:30What happens in that world?
  212. 15:34Well, I showed you add, multiply, subtract.
  213. 15:37I didn't show you divide.
  214. 15:38Divide is dangerous.
  215. 15:40It's spooky.
  216. 15:42So, let's go look at divide.
  217. 15:44You can see over here, divide throws a division by zero error.
  218. 15:49But, what else happens?
  219. 15:51The function actually knows that it throws division by zero error.
  220. 15:55Without you having to write any any code.
  221. 15:57If I go up to the calculate function, which at some point calls divide, this function also knows it throws division by zero error.
  222. 16:06So, error types now get inferred without you ever having to do any guesswork.
  223. 16:10That means if you catch or handle errors, we can do exhaustive guarantees, and the compiler can prove that you have handled the error or not handled the error.
  224. 16:19It's no more guessing.
  225. 16:20There's no unknowns.
  226. 16:21It's guaranteed to be proven.
  227. 16:23So, if you wanted to ship an API where it guarantees that it never throws, well, this system is broken because it doesn't meet the constraints.
  228. 16:32It has two errors that you're not throwing.
  229. 16:35If you wanted to go catch that, well, I can write the code for that in a second.
  230. 16:41But, you can start catching certain errors.
  231. 16:49Uh I mean, I'm just going to return a sentinel value for now.
  232. 16:53And now this parse thing, which previously threw division by zero error, is now guaranteed to no longer throw the division by zero error because if I catch any exceptions in here, I return a zero value every single time.
  233. 17:06The compiler and the tooling can do a lot of work for us.
  234. 17:08And we're already used to this in our codebases.
  235. 17:11We many of us probably don't know how compilers work under the hood, and we trust them.
  236. 17:15Code is a matter of trust.
  237. 17:16The reason that we don't use ML code blindly is because we don't trust it yet, cuz the systems underneath them don't have enough rigidity.
  238. 17:27One more thing.
  239. 17:28But before I tell you all to go write a bunch of ML code, cuz I've been there and I can tell you what someone tell me if I said, "Hey, use this new programming language.
  240. 17:38It's going to solve all your problems."
  241. 17:39It's just going to become come with a whole slew of new problems.
  242. 17:44So, we said, "I think we'll lose the war on slop if we try to ask everyone to rewrite all their code in the world into this new system."
  243. 17:51So, what does a solution like that look like where you don't have to rewrite all your code?
  244. 17:56Well, we What we started to do was we started to think about that about 2 years ago.
  245. 18:02And we said, "What if you could use ML not just stand-alone like I showed today, but from within any existing language of your choice?
  246. 18:10From Python to TypeScript to Rust to Go to Ruby to Java to anything new that comes up even after it."
  247. 18:17What if every function in ML is immediately accessible in the language of your choice?
  248. 18:22So, in this case, I'm calling the ML calculate function directly from Python, and it's completely type safe.
  249. 18:28Not only do we get calculate, we get calculate async in case some of us want to write async code.
  250. 18:32So, ML while it has no function coloring, it does give you the benefit of having to do whatever you want across your code.
  251. 18:39But what if you went a little bit sillier?
  252. 18:44What if you started passing around lambdas across language boundaries?
  253. 18:49I have a function here called with timeout.
  254. 18:51This function times out after a certain number of milliseconds, and if this work doesn't complete.
  255. 18:56And it's guaranteed to no matter how long it takes.
  256. 19:00Well, in that world, you can even pass Python lambdas across the bridge.
  257. 19:03You can pass generics across the bridge.
  258. 19:05You can pass closures.
  259. 19:07It should just work, so engineers don't have to go fuss with it.
  260. 19:10And more importantly, so when the agent does something, the type system never lies.
  261. 19:14The type system becomes the absolute center of truth that prevents invariants from entering your codebase.
  262. 19:22And what I really wanted to talk about today was not any one specific thing.
  263. 19:27But it's this general concept.
  264. 19:29You can build incredibly complex systems without traditional systems like code reviews.
  265. 19:33You don't You can work in things in parallel, and you can use AI however you want without requiring any sort of standardization.
  266. 19:40But the most important part is you have to be incredibly thoughtful about how your engineering team actually uses the systems under the hood.
  267. 19:47When we started building BAML, I didn't think it would be possible to build some of the software we did.
  268. 19:52And just yesterday one of our engineers built a partial C compiler purely in BAML.
  269. 19:58So, when I start pushing the boundaries of these systems and you stop reading the code, in some ways in my mind, it releases the floodgates for your engineering team to actually cover the gaps
  270. 20:08of what existed in your old process.
  271. 20:10Have you ever worked at a company that had no CI/CD?
  272. 20:13They said adding CI/CD would slow us down.
  273. 20:15They They do slow down for 3 months while they add it, but after that, they move a lot faster.
  274. 20:20Our processes have to evolve if we're going to ship at agent speed.
  275. 20:25And remember, this is the least amount of slop your codebase will ever have to this day.
  276. 20:30So, just embrace it and start fighting it back.
  277. 20:34I fell in love with software about 15 years ago, and it was the first thing that truly changed the way I perceive the world.
  278. 20:41And I really genuinely don't want slop to win.
  279. 20:43And I think we can all build a world of beautiful software.
  280. 20:47And I think what it takes is I want each of you to go home today and build these sloppy tools.
  281. 20:51Make your systems more robust.
  282. 20:53Make your processes more robust.
  283. 20:55And then for the bravest of you, I want you to go back and think about these core foundation layer systems.
  284. 21:01Think about how they're broken and see if you can imagine a way to fix them.
  285. 21:04I think we do need a new Git.
  286. 21:06I think we do need a new database, and yes, I think we need a new programming language.
  287. 21:11I'm Vaibhav, and I work on Bemo.
  288. 21:13Thank you.