Don't Vibe Code Your Agent Infrastructure - Akshat Bubna, Modal

AI Engineer · 17 min · 168 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:12Hello everyone.
2. 00:13I'm Akshat, CTO of Modal.
3. 00:17And I'll talk about what agents really need from their infrastructure.
4. 00:25I was told that this audience is closely related to finance.
5. 00:30So I thought it would be worth discussing what if every cell in a spreadsheet was supported by an agent.
6. 00:39Imagine you are a financial analyst at a fund.
7. 00:45You want to conduct a complex analysis of each company in your portfolio.
8. 00:51Normally in the past this would have taken days.
9. 00:53You would have to connect to each portfolio company, examine their data sources, and run Python code for each of them.
10. 01:02And now, obviously, it should be much easier with agents.
11. 01:06In fact, one of our clients, Heavy AI , has created just such a solution.
12. 01:14You can right now in their product open a table and run a query where literally every cell is served by a separate agent.
13. 01:26And I wanted to talk about what creating such a product actually looks like .
14. 01:34First of all, each agent runs in a Modal sandbox.
15. 01:41A sandbox is a safe environment for executing untrusted code.
16. 01:48Previously, each sandbox corresponded to a single container or microvirtual machine.
17. 01:55In fact, that is no longer the case.
18. 01:57Today, Modal has a feature that allows you to run multiple containers or microvirtual machines inside a single sandbox.
19. 02:07And this is actually needed for tasks like this where you run a control mechanism in this architecture.
20. 02:14This mechanism is an agent workflow that performs actions such as calling the model API for inference, launching tools, and repeating them cyclically.
21. 02:24You want to separate this from where the tool calls are actually executed, for security, isolation, and many other purposes.
22. 02:33So, you have a sandbox that runs multiple containers, including a sidecar.
23. 02:39You also need a shared repository so your agents can communicate with each other.
24. 02:44And you decide how to structure it.
25. 02:47For example, perhaps each row gets the same amount of memory, and agents in that row of the table can collaborate.
26. 02:55You may also want to share storage between the primary container and the secondary container in some cases.
27. 03:01And finally, you need this to run very quickly, because there can be thousands of cells in a table.
28. 03:07If someone launches a request, you want to instantly activate thousands of agents and then deactivate them when everything is complete.
29. 03:15So before I talk more about it, I wanted to take a little detour and talk about what Modal is.
30. 03:23Modal is a primitives platform built for AI applications.
31. 03:29We actually built this platform from scratch.
32. 03:32We have built our own file systems, scheduler, and runtime to provide you with high performance and development convenience for tasks such as inference (including LLM), sandboxes, training, and batch processing functions, etc.
33. 03:48Today I want to talk in more detail about sandboxes: what kind of primitive it is and how it develops.
34. 03:53So, if you want to build something like this or even more complex—we'll talk about what that means—what primitives do you need to do that ?
35. 04:03The first thing you need if you are creating an agent is a workspace.
36. 04:06I have a few examples from our current clients.
37. 04:10You may have heard of Ramp Inspect.
38. 04:13Ramp Inspect is Ramp's internal code writing agent that seems to handle almost every PR in Ramp at this point .
39. 04:22If you are creating an agent for code, you need your own repository, for example on GitHub.
40. 04:28You need your dependencies.
41. 04:31You need all your development services in one workspace for each agent.
42. 04:37Another example is Basis.
43. 04:40This is an accounting agent.
44. 04:41If you are creating such an agent, you need your documents.
45. 04:45You need your accounting working papers.
46. 04:47You need the scripts you wrote, all in one workspace.
47. 04:51We have another very interesting company.
48. 04:53This is a game development company.
49. 04:55If you are developing games using agents, you need a game engine.
50. 04:59You need your assets, all in one place .
51. 05:02The next step is to make these workspaces permanent.
52. 05:10Returning to the Basis example: imagine your documents arrive.
53. 05:16Your agent is activated to work with them.
54. 05:18But then he expects a response from the accountant.
55. 05:21And you don't know how long it might take.
56. 05:24This can take many days.
57. 05:26When the accountant responds, you want the agent to start again and continue where it left off.
58. 05:35That is, you want to preserve state over time, potentially indefinitely.
59. 05:41But you don't want to waste computing resources all this time because it's expensive.
60. 05:46Therefore, calculations should only work when the agent is directly engaged in work.
61. 05:51This is what we support at Modal, and we have realized that there are different types of persistence that users need .
62. 05:57Sometimes you may need a snapshot of the entire root file system.
63. 06:02And sometimes you need to take a snapshot of just a single directory.
64. 06:05And maybe it's like your node_modules folder.
65. 06:08But over time, you will want to update your basic look.
66. 06:13And sometimes you might need something similar to a shared file system, like a volume.
67. 06:17A very interesting thing that emerged is that agents actually need a lot of elasticity in their resource consumption.
68. 06:29We received permission from Ramp to share some of the data with Ramp Inspect.
69. 06:35And if you look at random sandboxes running in their workload, you'll see huge variability—both between them and within a single sandbox over time—in how much CPU and memory the agent is using.
70. 06:51And this is quite logical.
71. 06:53That is, if you are compiling something very resource-intensive or performing analysis that requires a lot of memory, your usage will change.
72. 07:02And in the second graph you can see the peak usage for each sandbox .
73. 07:06And, of course, there are some consistent trends, but overall the figures vary greatly.
74. 07:11So if you were to allocate resources for peak loads, you would have to provide about 15 CPU cores and 15 GB of RAM for each sandbox , which is simply not economically feasible.
75. 07:26At Modal, we've worked hard to ensure that users don't have to request these resources in advance, but instead receive them reliably when the sandbox needs them , and only pay for the time they use
76. 07:38them.
77. 07:41Another interesting trend is that agents require more complex computations.
78. 07:46They need GPUs, later they may need TPUs, AMD, who knows.
79. 07:54A great example of this is Cloud Science , which comes with a ready-made Modal integration.
80. 08:03It works like this: when Cloud needs to perform intensive computations, such as preprocessing data on the CPU or running a GPU for a scientific model, it hands off that task to the Modal
81. 08:15sandbox.
82. 08:17We see this emerging in game design, video editing, 3D development , CAD—all of these things require GPUs for your agent.
83. 08:28And, of course, what everyone is really excited about right now is RSI, or recursive self-improvement.
84. 08:35This is something that GPU sandboxes do well, and agents are very adept at using them.
85. 08:42We have a blog post about what happens when you give an agent a sandbox to optimize model training.
86. 08:49It turns out they are very effective.
87. 08:51They can change GPUs, run ablation studies in parallel, conduct many experiments, and achieve their goals very quickly.
88. 08:59The next trend is that agents have started to collaborate much more with each other.
89. 09:09This means that you share a file system with a sub-agent, and file systems and files with a human when you are all working on the same task.
90. 09:21This requires fast file systems with data integrity and development tools that can be managed.
91. 09:27This is what we are currently working on within the model volumes.
92. 09:30A related thing—we see a lot more examples of self-improvement.
93. 09:35Agents store and update memory, performing background tasks now called "dreams" to update that memory over time.
94. 09:44In fact, it also looks like regular files in the file system.
95. 09:48But again , this file system requires very high performance.
96. 09:59The worst thing about agents is that they can make a mistake and not realize that they are transferring your confidential data, for example, to a German forum.
97. 10:10Therefore, you need to have tight control over what they can and cannot do.
98. 10:15You need to control which IP addresses can interact with the agent, and how to authenticate it.
99. 10:21What domains can it connect to ?
100. 10:25Additionally, if you use credentials to access services, you want to transmit them in a way that an agent cannot see and later steal them somewhere .
101. 10:39An interesting development of this problem is that we have realized that whitelists alone are not enough.
102. 10:47Many of our users want very specific control, such as which paths can be accessed in DataDog or GitHub.
103. 10:56Maybe I want to understand the specific operations that the agent performs.
104. 11:02Returning to sidecars, we noticed a pattern emerging where users write their own lightweight proxies with specific rules about what exactly can be done.
105. 11:12Ramp is another example of a company that is successfully using this.
106. 11:20And the last underestimated thing is the extreme importance of qualitative observability.
107. 11:26Returning to the example with auto-learning and the agent performing the training: what happens when the training run fails?
108. 11:34You want your agent to be able to easily review error logs, analyze what happened, and run another iteration with a different set of parameters.
109. 11:44While the training is ongoing, you want it to be able to monitor metrics.
110. 11:48You want to see the GPU load and make sure the process hasn't stalled.
111. 11:52You may want to keep an eye on your budget so you don't spend too much in your sandbox.
112. 11:57And of course, all of this is very important for humans, but because agents use much more computing resources, we spend a lot of time giving agents access to these tools so they can analyze them.
113. 12:12OK.
114. 12:12I think we know what we need.
115. 12:14Why don't we commission Astra to create an agent platform for us?
116. 12:19Well, I don't think that's a good idea.
117. 12:21You shouldn't create a sandbox platform "based on intuition.
118. 12:24" There are three main reasons for this .
119. 12:26The first is reliability.
120. 12:28I think we are all painfully aware of how hard it is to get it right.
121. 12:33Certain websites that we use every day are constantly down.
122. 12:39The reason is that even with AI, it is very difficult to analyze the feedback loop , for example, what will happen in emergency cases when the system breaks down.
123. 12:49That is why you need to be very careful when creating systems.
124. 12:53Security is another aspect.
125. 12:54One could create a "sandbox" based on intuition, but who would do all the work of constantly analyzing security boundaries and ensuring security in times when everything is changing very quickly?
126. 13:10And finally—scaling.
127. 13:11We have a blog post about how we are scaling to a million sandboxes, which involves a lot of interesting engineering decisions and systems analysis.
128. 13:19And this is again the point that you shouldn't trust a platform built "on intuition," because when it comes to scale, you want everything to work.
129. 13:30By the way, we are indeed seeing massive scaling for many of the things I mentioned.
130. 13:36For example, the number of system snapshots I mentioned is growing quite rapidly.
131. 13:41We have 20 million images per week.
132. 13:45We process about 8 petabytes of data per day for each of these snapshots.
133. 13:51And our users create almost 2.6 million volumes per week.
134. 13:56And all this is growing extremely quickly.
135. 14:00Let's take just the client workload for a personal assistant: you need to create 10,000 sandboxes per second.
136. 14:08You need to run almost a million sandboxes.
137. 14:11And they won't last very long.
138. 14:14They will only work for 2 or 3 minutes.
139. 14:16But when they launch, you want them to become interactive in less than a second.
140. 14:21This is exactly what this workload requires.
141. 14:26Another interesting point about scaling is that you can't get that kind of scale from a single cloud platform.
142. 14:33The world is actually experiencing a shortage not only of GPUs, but also of CPUs.
143. 14:38Therefore, we spent a lot of time aggregating cloud capacity from a number of hyperscalers and neoclouds.
144. 14:44So, our users have access to processor power around the world when they need it.
145. 14:49But we can't rely on just one cloud provider for this.
146. 14:54Okay, so where is all this going?
147. 14:56I mentioned this before, but harnesses are great at taking advantage of dynamic environments.
148. 15:05And this leads to the separation of the “brain,” i.e. the binding, from the “hands,” i.e. the sandbox or runtime environment where everything works.
149. 15:17This means—I have an example here—that the bindings will be able to deploy environments based on what they need right now.
150. 15:26They are good at taking snapshots and saving them as artifacts that will be needed later.
151. 15:33And again, adjust all of this as needed.
152. 15:38I think this points to an interesting evolution of the "pets versus livestock " analogy we're all familiar with.
153. 15:47Let me remind you: in the past, people treated services like pets.
154. 15:51Each one was unique and needed to be cared for.
155. 15:55You had to take care of each individual deployment individually.
156. 16:02Over time, thanks to containerization and Kubernetes, people started to see this as a uniform pattern—the servers are all the same and can be scaled.
157. 16:14If one of them fails, it's not a problem because the system can recover itself.
158. 16:18And I believe that this is where many of those who are deploying agents are today.
159. 16:23They leverage these environments and scale their agents.
160. 16:28But this is very limiting, as models are great at working with dynamic environments: taking snapshots, creating multiple environments as needed, and interacting between them.
161. 16:39So what does this new world look like?
162. 16:43One analogy: it resembles an ocean of microbes, where all these organisms self-develop and cooperate with each other.
163. 16:52Thanks to Six for this analogy.
164. 16:58But I'm really inspired by the fact that when you give agents robust primitives for compute, storage, and networking, they can do a lot more, doing what I was talking about: running multiple environments, working together, and remaining
165. 17:12very dynamic.
166. 17:17So, I and the entire Model team believe that great primitives allow your agents to do more.
167. 17:24And if you are interested in discussing this in more detail, please contact us.
168. 17:28Thank you.