WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=_0SQh8a1Osk NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/_0SQh8a1Osk.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.463 --> 00:00:13.447 Hello everyone. s2 00:00:13.447 --> 00:00:17.171 I'm Akshat, CTO of Modal. s3 00:00:17.171 --> 00:00:22.837 And I'll talk about what agents really need from their infrastructure. s4 00:00:25.209 --> 00:00:30.934 I was told that this audience is closely related to finance. s5 00:00:30.934 --> 00:00:39.112 So I thought it would be worth discussing what if every cell in a spreadsheet was supported by an agent. s6 00:00:39.893 --> 00:00:44.298 Imagine you are a financial analyst at a fund. s7 00:00:45.376 --> 00:00:51.234 You want to conduct a complex analysis of each company in your portfolio. s8 00:00:51.234 --> 00:00:53.905 Normally in the past this would have taken days. s9 00:00:53.905 --> 00:01:02.056 You would have to connect to each portfolio company, examine their data sources, and run Python code for each of them. s10 00:01:02.056 --> 00:01:06.276 And now, obviously, it should be much easier with agents. s11 00:01:06.276 --> 00:01:12.917 In fact, one of our clients, Heavy AI , has created just such a solution. s12 00:01:14.370 --> 00:01:24.157 You can right now in their product open a table and run a query where literally every cell is served by a separate agent. s13 00:01:26.866 --> 00:01:32.972 And I wanted to talk about what creating such a product actually looks like . s14 00:01:34.476 --> 00:01:40.680 First of all, each agent runs in a Modal sandbox. s15 00:01:41.419 --> 00:01:47.544 A sandbox is a safe environment for executing untrusted code. s16 00:01:48.909 --> 00:01:55.389 Previously, each sandbox corresponded to a single container or microvirtual machine. s17 00:01:55.389 --> 00:01:57.773 In fact, that is no longer the case. s18 00:01:57.773 --> 00:02:06.897 Today, Modal has a feature that allows you to run multiple containers or microvirtual machines inside a single sandbox. s19 00:02:07.353 --> 00:02:14.802 And this is actually needed for tasks like this where you run a control mechanism in this architecture. s20 00:02:14.802 --> 00:02:24.998 This mechanism is an agent workflow that performs actions such as calling the model API for inference, launching tools, and repeating them cyclically. s21 00:02:24.998 --> 00:02:33.340 You want to separate this from where the tool calls are actually executed, for security, isolation, and many other purposes. s22 00:02:33.340 --> 00:02:39.229 So, you have a sandbox that runs multiple containers, including a sidecar. s23 00:02:39.229 --> 00:02:44.306 You also need a shared repository so your agents can communicate with each other. s24 00:02:44.306 --> 00:02:47.080 And you decide how to structure it. s25 00:02:47.080 --> 00:02:55.007 For example, perhaps each row gets the same amount of memory, and agents in that row of the table can collaborate. s26 00:02:55.007 --> 00:03:01.591 You may also want to share storage between the primary container and the secondary container in some cases. s27 00:03:01.591 --> 00:03:07.498 And finally, you need this to run very quickly, because there can be thousands of cells in a table. s28 00:03:07.498 --> 00:03:15.269 If someone launches a request, you want to instantly activate thousands of agents and then deactivate them when everything is complete. s29 00:03:15.269 --> 00:03:23.400 So before I talk more about it, I wanted to take a little detour and talk about what Modal is. s30 00:03:23.785 --> 00:03:29.362 Modal is a primitives platform built for AI applications. s31 00:03:29.382 --> 00:03:32.907 We actually built this platform from scratch. s32 00:03:32.907 --> 00:03:48.797 We have built our own file systems, scheduler, and runtime to provide you with high performance and development convenience for tasks such as inference (including LLM), sandboxes, training, and batch processing functions, etc. s33 00:03:48.797 --> 00:03:53.828 Today I want to talk in more detail about sandboxes: what kind of primitive it is and how it develops. s34 00:03:53.828 --> 00:04:03.188 So, if you want to build something like this or even more complex—we'll talk about what that means—what primitives do you need to do that ? s35 00:04:03.188 --> 00:04:06.907 The first thing you need if you are creating an agent is a workspace. s36 00:04:06.907 --> 00:04:10.321 I have a few examples from our current clients. s37 00:04:10.321 --> 00:04:13.626 You may have heard of Ramp Inspect. s38 00:04:13.626 --> 00:04:22.876 Ramp Inspect is Ramp's internal code writing agent that seems to handle almost every PR in Ramp at this point . s39 00:04:22.876 --> 00:04:28.681 If you are creating an agent for code, you need your own repository, for example on GitHub. s40 00:04:28.681 --> 00:04:31.071 You need your dependencies. s41 00:04:31.071 --> 00:04:37.539 You need all your development services in one workspace for each agent. s42 00:04:37.539 --> 00:04:40.102 Another example is Basis. s43 00:04:40.102 --> 00:04:41.977 This is an accounting agent. s44 00:04:41.977 --> 00:04:45.397 If you are creating such an agent, you need your documents. s45 00:04:45.397 --> 00:04:47.917 You need your accounting working papers. s46 00:04:47.917 --> 00:04:51.621 You need the scripts you wrote, all in one workspace. s47 00:04:51.621 --> 00:04:53.529 We have another very interesting company. s48 00:04:53.529 --> 00:04:55.715 This is a game development company. s49 00:04:55.715 --> 00:04:59.917 If you are developing games using agents, you need a game engine. s50 00:04:59.917 --> 00:05:02.457 You need your assets, all in one place . s51 00:05:02.457 --> 00:05:08.596 The next step is to make these workspaces permanent. s52 00:05:10.140 --> 00:05:16.080 Returning to the Basis example: imagine your documents arrive. s53 00:05:16.080 --> 00:05:18.944 Your agent is activated to work with them. s54 00:05:18.944 --> 00:05:21.890 But then he expects a response from the accountant. s55 00:05:21.890 --> 00:05:24.651 And you don't know how long it might take. s56 00:05:24.651 --> 00:05:26.559 This can take many days. s57 00:05:26.559 --> 00:05:35.502 When the accountant responds, you want the agent to start again and continue where it left off. s58 00:05:35.855 --> 00:05:41.959 That is, you want to preserve state over time, potentially indefinitely. s59 00:05:41.959 --> 00:05:46.325 But you don't want to waste computing resources all this time because it's expensive. s60 00:05:46.325 --> 00:05:51.025 Therefore, calculations should only work when the agent is directly engaged in work. s61 00:05:51.025 --> 00:05:57.988 This is what we support at Modal, and we have realized that there are different types of persistence that users need . s62 00:05:57.988 --> 00:06:02.310 Sometimes you may need a snapshot of the entire root file system. s63 00:06:02.310 --> 00:06:05.708 And sometimes you need to take a snapshot of just a single directory. s64 00:06:05.708 --> 00:06:08.949 And maybe it's like your node_modules folder. s65 00:06:08.949 --> 00:06:13.285 But over time, you will want to update your basic look. s66 00:06:13.285 --> 00:06:17.788 And sometimes you might need something similar to a shared file system, like a volume. s67 00:06:17.788 --> 00:06:29.614 A very interesting thing that emerged is that agents actually need a lot of elasticity in their resource consumption. s68 00:06:29.614 --> 00:06:35.609 We received permission from Ramp to share some of the data with Ramp Inspect. s69 00:06:35.609 --> 00:06:51.476 And if you look at random sandboxes running in their workload, you'll see huge variability—both between them and within a single sandbox over time—in how much CPU and memory the agent is using. s70 00:06:51.476 --> 00:06:53.006 And this is quite logical. s71 00:06:53.006 --> 00:07:02.145 That is, if you are compiling something very resource-intensive or performing analysis that requires a lot of memory, your usage will change. s72 00:07:02.145 --> 00:07:06.546 And in the second graph you can see the peak usage for each sandbox . s73 00:07:06.546 --> 00:07:11.970 And, of course, there are some consistent trends, but overall the figures vary greatly. s74 00:07:11.970 --> 00:07:26.619 So if you were to allocate resources for peak loads, you would have to provide about 15 CPU cores and 15 GB of RAM for each sandbox , which is simply not economically feasible. s75 00:07:26.629 --> 00:07:38.528 At Modal, we've worked hard to ensure that users don't have to request these resources in advance, but instead receive them reliably when the sandbox needs them , and only pay for the time they use s76 00:07:38.528 --> 00:07:40.528 them. s77 00:07:41.709 --> 00:07:46.687 Another interesting trend is that agents require more complex computations. s78 00:07:46.687 --> 00:07:53.498 They need GPUs, later they may need TPUs, AMD, who knows. s79 00:07:54.234 --> 00:08:01.717 A great example of this is Cloud Science , which comes with a ready-made Modal integration. s80 00:08:03.145 --> 00:08:15.450 It works like this: when Cloud needs to perform intensive computations, such as preprocessing data on the CPU or running a GPU for a scientific model, it hands off that task to the Modal s81 00:08:15.450 --> 00:08:17.450 sandbox. s82 00:08:17.463 --> 00:08:27.970 We see this emerging in game design, video editing, 3D development , CAD—all of these things require GPUs for your agent. s83 00:08:28.865 --> 00:08:35.378 And, of course, what everyone is really excited about right now is RSI, or recursive self-improvement. s84 00:08:35.935 --> 00:08:42.164 This is something that GPU sandboxes do well, and agents are very adept at using them. s85 00:08:42.164 --> 00:08:49.008 We have a blog post about what happens when you give an agent a sandbox to optimize model training. s86 00:08:49.008 --> 00:08:51.318 It turns out they are very effective. s87 00:08:51.318 --> 00:08:59.678 They can change GPUs, run ablation studies in parallel, conduct many experiments, and achieve their goals very quickly. s88 00:08:59.678 --> 00:09:08.830 The next trend is that agents have started to collaborate much more with each other. s89 00:09:09.200 --> 00:09:20.515 This means that you share a file system with a sub-agent, and file systems and files with a human when you are all working on the same task. s90 00:09:21.221 --> 00:09:27.287 This requires fast file systems with data integrity and development tools that can be managed. s91 00:09:27.287 --> 00:09:30.659 This is what we are currently working on within the model volumes. s92 00:09:30.659 --> 00:09:35.541 A related thing—we see a lot more examples of self-improvement. s93 00:09:35.541 --> 00:09:44.523 Agents store and update memory, performing background tasks now called "dreams" to update that memory over time. s94 00:09:44.523 --> 00:09:48.813 In fact, it also looks like regular files in the file system. s95 00:09:48.813 --> 00:09:55.184 But again , this file system requires very high performance. s96 00:09:59.736 --> 00:10:10.101 The worst thing about agents is that they can make a mistake and not realize that they are transferring your confidential data, for example, to a German forum. s97 00:10:10.827 --> 00:10:15.384 Therefore, you need to have tight control over what they can and cannot do. s98 00:10:15.384 --> 00:10:21.964 You need to control which IP addresses can interact with the agent, and how to authenticate it. s99 00:10:21.964 --> 00:10:25.640 What domains can it connect to ? s100 00:10:25.640 --> 00:10:36.283 Additionally, if you use credentials to access services, you want to transmit them in a way that an agent cannot see and later steal them somewhere . s101 00:10:39.817 --> 00:10:47.760 An interesting development of this problem is that we have realized that whitelists alone are not enough. s102 00:10:47.760 --> 00:10:56.951 Many of our users want very specific control, such as which paths can be accessed in DataDog or GitHub. s103 00:10:56.951 --> 00:11:02.376 Maybe I want to understand the specific operations that the agent performs. s104 00:11:02.376 --> 00:11:12.523 Returning to sidecars, we noticed a pattern emerging where users write their own lightweight proxies with specific rules about what exactly can be done. s105 00:11:12.523 --> 00:11:18.674 Ramp is another example of a company that is successfully using this. s106 00:11:20.349 --> 00:11:26.182 And the last underestimated thing is the extreme importance of qualitative observability. s107 00:11:26.182 --> 00:11:34.720 Returning to the example with auto-learning and the agent performing the training: what happens when the training run fails? s108 00:11:34.720 --> 00:11:44.894 You want your agent to be able to easily review error logs, analyze what happened, and run another iteration with a different set of parameters. s109 00:11:44.894 --> 00:11:48.613 While the training is ongoing, you want it to be able to monitor metrics. s110 00:11:48.613 --> 00:11:52.992 You want to see the GPU load and make sure the process hasn't stalled. s111 00:11:52.992 --> 00:11:57.575 You may want to keep an eye on your budget so you don't spend too much in your sandbox. s112 00:11:57.575 --> 00:12:12.358 And of course, all of this is very important for humans, but because agents use much more computing resources, we spend a lot of time giving agents access to these tools so they can analyze them. s113 00:12:12.368 --> 00:12:12.693 OK. s114 00:12:12.693 --> 00:12:14.975 I think we know what we need. s115 00:12:14.975 --> 00:12:19.358 Why don't we commission Astra to create an agent platform for us? s116 00:12:19.358 --> 00:12:21.388 Well, I don't think that's a good idea. s117 00:12:21.388 --> 00:12:24.031 You shouldn't create a sandbox platform "based on intuition. s118 00:12:24.031 --> 00:12:26.989 " There are three main reasons for this . s119 00:12:26.989 --> 00:12:28.976 The first is reliability. s120 00:12:28.976 --> 00:12:33.408 I think we are all painfully aware of how hard it is to get it right. s121 00:12:33.408 --> 00:12:39.369 Certain websites that we use every day are constantly down. s122 00:12:39.369 --> 00:12:49.574 The reason is that even with AI, it is very difficult to analyze the feedback loop , for example, what will happen in emergency cases when the system breaks down. s123 00:12:49.574 --> 00:12:53.183 That is why you need to be very careful when creating systems. s124 00:12:53.183 --> 00:12:54.958 Security is another aspect. s125 00:12:54.958 --> 00:13:09.520 One could create a "sandbox" based on intuition, but who would do all the work of constantly analyzing security boundaries and ensuring security in times when everything is changing very quickly? s126 00:13:10.124 --> 00:13:11.418 And finally—scaling. s127 00:13:11.418 --> 00:13:19.936 We have a blog post about how we are scaling to a million sandboxes, which involves a lot of interesting engineering decisions and systems analysis. s128 00:13:19.936 --> 00:13:30.471 And this is again the point that you shouldn't trust a platform built "on intuition," because when it comes to scale, you want everything to work. s129 00:13:30.584 --> 00:13:36.425 By the way, we are indeed seeing massive scaling for many of the things I mentioned. s130 00:13:36.425 --> 00:13:41.105 For example, the number of system snapshots I mentioned is growing quite rapidly. s131 00:13:41.105 --> 00:13:45.295 We have 20 million images per week. s132 00:13:45.332 --> 00:13:51.830 We process about 8 petabytes of data per day for each of these snapshots. s133 00:13:51.866 --> 00:13:56.215 And our users create almost 2.6 million volumes per week. s134 00:13:56.215 --> 00:14:00.811 And all this is growing extremely quickly. s135 00:14:00.811 --> 00:14:08.813 Let's take just the client workload for a personal assistant: you need to create 10,000 sandboxes per second. s136 00:14:08.813 --> 00:14:11.516 You need to run almost a million sandboxes. s137 00:14:11.516 --> 00:14:14.120 And they won't last very long. s138 00:14:14.120 --> 00:14:16.605 They will only work for 2 or 3 minutes. s139 00:14:16.605 --> 00:14:21.120 But when they launch, you want them to become interactive in less than a second. s140 00:14:21.120 --> 00:14:26.092 This is exactly what this workload requires. s141 00:14:26.282 --> 00:14:33.985 Another interesting point about scaling is that you can't get that kind of scale from a single cloud platform. s142 00:14:33.985 --> 00:14:38.439 The world is actually experiencing a shortage not only of GPUs, but also of CPUs. s143 00:14:38.439 --> 00:14:44.805 Therefore, we spent a lot of time aggregating cloud capacity from a number of hyperscalers and neoclouds. s144 00:14:44.805 --> 00:14:49.642 So, our users have access to processor power around the world when they need it. s145 00:14:49.642 --> 00:14:54.478 But we can't rely on just one cloud provider for this. s146 00:14:54.478 --> 00:14:56.760 Okay, so where is all this going? s147 00:14:56.760 --> 00:15:05.110 I mentioned this before, but harnesses are great at taking advantage of dynamic environments. s148 00:15:05.855 --> 00:15:15.272 And this leads to the separation of the “brain,” i.e. the binding, from the “hands,” i.e. the sandbox or runtime environment where everything works. s149 00:15:17.171 --> 00:15:26.467 This means—I have an example here—that the bindings will be able to deploy environments based on what they need right now. s150 00:15:26.467 --> 00:15:33.020 They are good at taking snapshots and saving them as artifacts that will be needed later. s151 00:15:33.020 --> 00:15:37.296 And again, adjust all of this as needed. s152 00:15:38.284 --> 00:15:46.877 I think this points to an interesting evolution of the "pets versus livestock " analogy we're all familiar with. s153 00:15:47.068 --> 00:15:51.969 Let me remind you: in the past, people treated services like pets. s154 00:15:51.969 --> 00:15:55.814 Each one was unique and needed to be cared for. s155 00:15:55.814 --> 00:16:00.795 You had to take care of each individual deployment individually. s156 00:16:02.858 --> 00:16:13.428 Over time, thanks to containerization and Kubernetes, people started to see this as a uniform pattern—the servers are all the same and can be scaled. s157 00:16:14.266 --> 00:16:18.592 If one of them fails, it's not a problem because the system can recover itself. s158 00:16:18.592 --> 00:16:23.518 And I believe that this is where many of those who are deploying agents are today. s159 00:16:23.518 --> 00:16:28.087 They leverage these environments and scale their agents. s160 00:16:28.087 --> 00:16:39.240 But this is very limiting, as models are great at working with dynamic environments: taking snapshots, creating multiple environments as needed, and interacting between them. s161 00:16:39.240 --> 00:16:43.207 So what does this new world look like? s162 00:16:43.952 --> 00:16:52.285 One analogy: it resembles an ocean of microbes, where all these organisms self-develop and cooperate with each other. s163 00:16:52.677 --> 00:16:56.717 Thanks to Six for this analogy. s164 00:16:58.319 --> 00:17:12.448 But I'm really inspired by the fact that when you give agents robust primitives for compute, storage, and networking, they can do a lot more, doing what I was talking about: running multiple environments, working together, and remaining s165 00:17:12.448 --> 00:17:15.490 very dynamic. s166 00:17:17.352 --> 00:17:24.016 So, I and the entire Model team believe that great primitives allow your agents to do more. s167 00:17:24.016 --> 00:17:28.216 And if you are interested in discussing this in more detail, please contact us. s168 00:17:28.216 --> 00:17:30.599 Thank you.