Don't Vibe Code Your Agent Infrastructure - Akshat Bubna, Modal
https://www.youtube.com/watch?v=_0SQh8a1Osk
AI Engineer · 17 min · 168 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:12] Hello everyone.
[00:13] I'm Akshat, CTO of Modal.
[00:17] And I'll talk about what agents really need from their infrastructure.
[00:25] I was told that this audience is closely related to finance.
[00:30] So I thought it would be worth discussing what if every cell in a spreadsheet was supported by an agent.
[00:39] Imagine you are a financial analyst at a fund.
[00:45] You want to conduct a complex analysis of each company in your portfolio.
[00:51] Normally in the past this would have taken days.
[00:53] You would have to connect to each portfolio company, examine their data sources, and run Python code for each of them.
[01:02] And now, obviously, it should be much easier with agents.
[01:06] In fact, one of our clients, Heavy AI , has created just such a solution.
[01:14] You can right now in their product open a table and run a query where literally every cell is served by a separate agent.
[01:26] And I wanted to talk about what creating such a product actually looks like .
[01:34] First of all, each agent runs in a Modal sandbox.
[01:41] A sandbox is a safe environment for executing untrusted code.
[01:48] Previously, each sandbox corresponded to a single container or microvirtual machine.
[01:55] In fact, that is no longer the case.
[01:57] Today, Modal has a feature that allows you to run multiple containers or microvirtual machines inside a single sandbox.
[02:07] And this is actually needed for tasks like this where you run a control mechanism in this architecture.
[02:14] This mechanism is an agent workflow that performs actions such as calling the model API for inference, launching tools, and repeating them cyclically.
[02:24] You want to separate this from where the tool calls are actually executed, for security, isolation, and many other purposes.
[02:33] So, you have a sandbox that runs multiple containers, including a sidecar.
[02:39] You also need a shared repository so your agents can communicate with each other.
[02:44] And you decide how to structure it.
[02:47] For example, perhaps each row gets the same amount of memory, and agents in that row of the table can collaborate.
[02:55] You may also want to share storage between the primary container and the secondary container in some cases.
[03:01] And finally, you need this to run very quickly, because there can be thousands of cells in a table.
[03:07] If someone launches a request, you want to instantly activate thousands of agents and then deactivate them when everything is complete.
[03:15] So before I talk more about it, I wanted to take a little detour and talk about what Modal is.
[03:23] Modal is a primitives platform built for AI applications.
[03:29] We actually built this platform from scratch.
[03:32] We have built our own file systems, scheduler, and runtime to provide you with high performance and development convenience for tasks such as inference (including LLM), sandboxes, training, and batch processing functions, etc.
[03:48] Today I want to talk in more detail about sandboxes: what kind of primitive it is and how it develops.
[03:53] So, if you want to build something like this or even more complex—we'll talk about what that means—what primitives do you need to do that ?
[04:03] The first thing you need if you are creating an agent is a workspace.
[04:06] I have a few examples from our current clients.
[04:10] You may have heard of Ramp Inspect.
[04:13] Ramp Inspect is Ramp's internal code writing agent that seems to handle almost every PR in Ramp at this point .
[04:22] If you are creating an agent for code, you need your own repository, for example on GitHub.
[04:28] You need your dependencies.
[04:31] You need all your development services in one workspace for each agent.
[04:37] Another example is Basis.
[04:40] This is an accounting agent.
[04:41] If you are creating such an agent, you need your documents.
[04:45] You need your accounting working papers.
[04:47] You need the scripts you wrote, all in one workspace.
[04:51] We have another very interesting company.
[04:53] This is a game development company.
[04:55] If you are developing games using agents, you need a game engine.
[04:59] You need your assets, all in one place .
[05:02] The next step is to make these workspaces permanent.
[05:10] Returning to the Basis example: imagine your documents arrive.
[05:16] Your agent is activated to work with them.
[05:18] But then he expects a response from the accountant.
[05:21] And you don't know how long it might take.
[05:24] This can take many days.
[05:26] When the accountant responds, you want the agent to start again and continue where it left off.
[05:35] That is, you want to preserve state over time, potentially indefinitely.
[05:41] But you don't want to waste computing resources all this time because it's expensive.
[05:46] Therefore, calculations should only work when the agent is directly engaged in work.
[05:51] This is what we support at Modal, and we have realized that there are different types of persistence that users need .
[05:57] Sometimes you may need a snapshot of the entire root file system.
[06:02] And sometimes you need to take a snapshot of just a single directory.
[06:05] And maybe it's like your node_modules folder.
[06:08] But over time, you will want to update your basic look.
[06:13] And sometimes you might need something similar to a shared file system, like a volume.
[06:17] A very interesting thing that emerged is that agents actually need a lot of elasticity in their resource consumption.
[06:29] We received permission from Ramp to share some of the data with Ramp Inspect.
[06:35] And if you look at random sandboxes running in their workload, you'll see huge variability—both between them and within a single sandbox over time—in how much CPU and memory the agent is using.
[06:51] And this is quite logical.
[06:53] That is, if you are compiling something very resource-intensive or performing analysis that requires a lot of memory, your usage will change.
[07:02] And in the second graph you can see the peak usage for each sandbox .
[07:06] And, of course, there are some consistent trends, but overall the figures vary greatly.
[07:11] So if you were to allocate resources for peak loads, you would have to provide about 15 CPU cores and 15 GB of RAM for each sandbox , which is simply not economically feasible.
[07:26] At Modal, we've worked hard to ensure that users don't have to request these resources in advance, but instead receive them reliably when the sandbox needs them , and only pay for the time they use
[07:38] them.
[07:41] Another interesting trend is that agents require more complex computations.
[07:46] They need GPUs, later they may need TPUs, AMD, who knows.
[07:54] A great example of this is Cloud Science , which comes with a ready-made Modal integration.
[08:03] It works like this: when Cloud needs to perform intensive computations, such as preprocessing data on the CPU or running a GPU for a scientific model, it hands off that task to the Modal
[08:15] sandbox.
[08:17] We see this emerging in game design, video editing, 3D development , CAD—all of these things require GPUs for your agent.
[08:28] And, of course, what everyone is really excited about right now is RSI, or recursive self-improvement.
[08:35] This is something that GPU sandboxes do well, and agents are very adept at using them.
[08:42] We have a blog post about what happens when you give an agent a sandbox to optimize model training.
[08:49] It turns out they are very effective.
[08:51] They can change GPUs, run ablation studies in parallel, conduct many experiments, and achieve their goals very quickly.
[08:59] The next trend is that agents have started to collaborate much more with each other.
[09:09] This means that you share a file system with a sub-agent, and file systems and files with a human when you are all working on the same task.
[09:21] This requires fast file systems with data integrity and development tools that can be managed.
[09:27] This is what we are currently working on within the model volumes.
[09:30] A related thing—we see a lot more examples of self-improvement.
[09:35] Agents store and update memory, performing background tasks now called "dreams" to update that memory over time.
[09:44] In fact, it also looks like regular files in the file system.
[09:48] But again , this file system requires very high performance.
[09:59] The worst thing about agents is that they can make a mistake and not realize that they are transferring your confidential data, for example, to a German forum.
[10:10] Therefore, you need to have tight control over what they can and cannot do.
[10:15] You need to control which IP addresses can interact with the agent, and how to authenticate it.
[10:21] What domains can it connect to ?
[10:25] Additionally, if you use credentials to access services, you want to transmit them in a way that an agent cannot see and later steal them somewhere .
[10:39] An interesting development of this problem is that we have realized that whitelists alone are not enough.
[10:47] Many of our users want very specific control, such as which paths can be accessed in DataDog or GitHub.
[10:56] Maybe I want to understand the specific operations that the agent performs.
[11:02] Returning to sidecars, we noticed a pattern emerging where users write their own lightweight proxies with specific rules about what exactly can be done.
[11:12] Ramp is another example of a company that is successfully using this.
[11:20] And the last underestimated thing is the extreme importance of qualitative observability.
[11:26] Returning to the example with auto-learning and the agent performing the training: what happens when the training run fails?
[11:34] You want your agent to be able to easily review error logs, analyze what happened, and run another iteration with a different set of parameters.
[11:44] While the training is ongoing, you want it to be able to monitor metrics.
[11:48] You want to see the GPU load and make sure the process hasn't stalled.
[11:52] You may want to keep an eye on your budget so you don't spend too much in your sandbox.
[11:57] And of course, all of this is very important for humans, but because agents use much more computing resources, we spend a lot of time giving agents access to these tools so they can analyze them.
[12:12] OK.
[12:12] I think we know what we need.
[12:14] Why don't we commission Astra to create an agent platform for us?
[12:19] Well, I don't think that's a good idea.
[12:21] You shouldn't create a sandbox platform "based on intuition.
[12:24] " There are three main reasons for this .
[12:26] The first is reliability.
[12:28] I think we are all painfully aware of how hard it is to get it right.
[12:33] Certain websites that we use every day are constantly down.
[12:39] The reason is that even with AI, it is very difficult to analyze the feedback loop , for example, what will happen in emergency cases when the system breaks down.
[12:49] That is why you need to be very careful when creating systems.
[12:53] Security is another aspect.
[12:54] One could create a "sandbox" based on intuition, but who would do all the work of constantly analyzing security boundaries and ensuring security in times when everything is changing very quickly?
[13:10] And finally—scaling.
[13:11] We have a blog post about how we are scaling to a million sandboxes, which involves a lot of interesting engineering decisions and systems analysis.
[13:19] And this is again the point that you shouldn't trust a platform built "on intuition," because when it comes to scale, you want everything to work.
[13:30] By the way, we are indeed seeing massive scaling for many of the things I mentioned.
[13:36] For example, the number of system snapshots I mentioned is growing quite rapidly.
[13:41] We have 20 million images per week.
[13:45] We process about 8 petabytes of data per day for each of these snapshots.
[13:51] And our users create almost 2.6 million volumes per week.
[13:56] And all this is growing extremely quickly.
[14:00] Let's take just the client workload for a personal assistant: you need to create 10,000 sandboxes per second.
[14:08] You need to run almost a million sandboxes.
[14:11] And they won't last very long.
[14:14] They will only work for 2 or 3 minutes.
[14:16] But when they launch, you want them to become interactive in less than a second.
[14:21] This is exactly what this workload requires.
[14:26] Another interesting point about scaling is that you can't get that kind of scale from a single cloud platform.
[14:33] The world is actually experiencing a shortage not only of GPUs, but also of CPUs.
[14:38] Therefore, we spent a lot of time aggregating cloud capacity from a number of hyperscalers and neoclouds.
[14:44] So, our users have access to processor power around the world when they need it.
[14:49] But we can't rely on just one cloud provider for this.
[14:54] Okay, so where is all this going?
[14:56] I mentioned this before, but harnesses are great at taking advantage of dynamic environments.
[15:05] And this leads to the separation of the “brain,” i.e. the binding, from the “hands,” i.e. the sandbox or runtime environment where everything works.
[15:17] This means—I have an example here—that the bindings will be able to deploy environments based on what they need right now.
[15:26] They are good at taking snapshots and saving them as artifacts that will be needed later.
[15:33] And again, adjust all of this as needed.
[15:38] I think this points to an interesting evolution of the "pets versus livestock " analogy we're all familiar with.
[15:47] Let me remind you: in the past, people treated services like pets.
[15:51] Each one was unique and needed to be cared for.
[15:55] You had to take care of each individual deployment individually.
[16:02] Over time, thanks to containerization and Kubernetes, people started to see this as a uniform pattern—the servers are all the same and can be scaled.
[16:14] If one of them fails, it's not a problem because the system can recover itself.
[16:18] And I believe that this is where many of those who are deploying agents are today.
[16:23] They leverage these environments and scale their agents.
[16:28] But this is very limiting, as models are great at working with dynamic environments: taking snapshots, creating multiple environments as needed, and interacting between them.
[16:39] So what does this new world look like?
[16:43] One analogy: it resembles an ocean of microbes, where all these organisms self-develop and cooperate with each other.
[16:52] Thanks to Six for this analogy.
[16:58] But I'm really inspired by the fact that when you give agents robust primitives for compute, storage, and networking, they can do a lot more, doing what I was talking about: running multiple environments, working together, and remaining
[17:12] very dynamic.
[17:17] So, I and the entire Model team believe that great primitives allow your agents to do more.
[17:24] And if you are interested in discussing this in more detail, please contact us.
[17:28] Thank you.