WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=zkX03APVj0M

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/zkX03APVj0M.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:14.000 --> 00:00:15.720
So, I appreciate the intro.

s2
00:00:15.720 --> 00:00:19.320
Uh my name is Joseph and this is my co-founder Sid.

s3
00:00:19.320 --> 00:00:26.320
Emulated is a data lab focused on increasing the reliability and autonomy of AI agents.

s4
00:00:26.320 --> 00:00:36.320
And if you've been a AI engineer and you've watched the talks, seen the tracks, then there's probably one takeaway that all the talks have in common.

s5
00:00:36.320 --> 00:00:45.840
And it's that we're headed towards a future where agents are able to perform useful work over longer and longer horizons with little to no supervision.

s6
00:00:45.840 --> 00:00:52.720
So, today we're going to answer some of the questions of what this means for the data and model layers.

s7
00:00:52.720 --> 00:00:55.160
Uh we're going to touch on some pretty cool things.

s8
00:00:55.160 --> 00:00:56.920
Uh so, look out for them.

s9
00:00:56.920 --> 00:01:04.440
Um like how to simulate a company within a sandbox or sandboxes for multi-node systems and distributed clusters.

s10
00:01:04.440 --> 00:01:12.520
Um and if we have a little bit of time, we'll also go into some of the work that we're doing with post-training pipelines and how these new types of sandboxes

s11
00:01:12.520 --> 00:01:17.160
are affecting post-training infra as well.

s12
00:01:17.800 --> 00:01:25.520
So, where Sid and I come from, um our backgrounds are in network infra, distributed databases, and sandbox infra.

s13
00:01:25.520 --> 00:01:30.280
And these are all areas where the workloads are mission critical.

s14
00:01:30.280 --> 00:01:38.880
We all saw a couple months ago that uh when something like DynamoDB goes down, so does US East 1 and half the internet.

s15
00:01:38.880 --> 00:01:50.800
Um and working on these systems, we saw model capability gap when it came to operating and building these systems at scale at scale and thinking about uh the consequences

s16
00:01:50.800 --> 00:01:55.760
of architecture and system system design over the course of years.

s17
00:01:55.760 --> 00:01:58.880
Yeah, so it led to a pretty uh natural question, right?

s18
00:01:58.880 --> 00:02:10.800
For such mission-critical services, why is it that my model or my agent is so proficient at handling the application layer, but is struggles when it comes to reasoning through

s19
00:02:10.800 --> 00:02:12.320
infrastructure complexities?

s20
00:02:12.320 --> 00:02:23.880
For example, things like MVCC on a database engine, which can lead to corruption issues, which is one of which was one of the roots of the DynamoDB failure a few months ago.

s21
00:02:26.160 --> 00:02:31.720
So, like with everything in NML, uh the gap in models is usually a gap in data.

s22
00:02:31.720 --> 00:02:36.720
Models typically are only as good at as data is.

s23
00:02:36.720 --> 00:02:39.160
Um and to really highlight this point, right?

s24
00:02:39.160 --> 00:02:45.800
Model capability has never uh regressed whenever you introduce more high-quality data.

s25
00:02:45.800 --> 00:02:48.640
Um so, with that being said, what is the data gap then?

s26
00:02:48.640 --> 00:02:50.959
What does data look like right now?

s27
00:02:50.959 --> 00:02:55.440
And how is this influencing the model capability gap here?

s28
00:02:55.440 --> 00:03:09.040
So, if you look at any of the frontier or recent benchmarks, like SweBench Pro, Terminal Bench, or something like Frontier Code and Deep Sweep, um the tasks only operate within the code base.

s29
00:03:09.040 --> 00:03:21.880
Uh the agent is given a pretty large uh task uh and over the course of 50 to 100 turns produces a couple thousand-line PR.

s30
00:03:21.880 --> 00:03:25.400
Um but it doesn't do all of the work that a human does.

s31
00:03:25.400 --> 00:03:39.600
It doesn't do uh what a PM does with talking to customers, understanding their problems, what an engineer does with trying out different approaches, performing performance testing them, um and owning the underlying infra

s32
00:03:39.600 --> 00:03:44.560
for the code base over the course of not just months, but years.

s33
00:03:44.560 --> 00:03:47.160
And this is really the gap that we're closing.

s34
00:03:47.160 --> 00:03:52.320
We've taken software engineering companies and we've put them into containerized environments.

s35
00:03:52.320 --> 00:04:01.160
so this includes uh include like organizational contexts like projects, incidents, customer conversations.

s36
00:04:01.160 --> 00:04:10.680
Uh the agent also has to deal with issues that only appear at scale like network failures between distributed nodes, data corruption, and clock skew.

s37
00:04:10.680 --> 00:04:21.640
And through all this, we also want the agents to reason about orchestrating through distributed clusters and also thinking about things like operational blast radius while solving live traffic.

s38
00:04:21.640 --> 00:04:31.360
And the result is that the task that these agents have to complete or we want the agents to learn is that environments are far more complex and long horizon

s39
00:04:31.360 --> 00:04:34.400
than a simple code diff.

s40
00:04:35.320 --> 00:04:42.160
So, let's just let's bring a picture into the mix because it tends to make things more interesting.

s41
00:04:42.160 --> 00:04:49.640
Uh here's an example we've built of an SCD consensus cluster that a typical production service might rely on.

s42
00:04:49.640 --> 00:05:00.840
So, an old environment uh might tended to operate and work primarily on that little blue square entitled SCD source code in the bottom right there.

s43
00:05:00.840 --> 00:05:07.919
But, a lot of the fun and the model capability gap that results from it is really in everything that surrounds it.

s44
00:05:07.919 --> 00:05:12.720
So, you you start with the tickets, projects, postmortems.

s45
00:05:12.720 --> 00:05:14.000
What are the train wrecks?

s46
00:05:14.000 --> 00:05:15.320
Why did they happen?

s47
00:05:15.320 --> 00:05:17.400
How did customers feel about them?

s48
00:05:17.400 --> 00:05:21.080
And often times those aren't necessarily up to date.

s49
00:05:21.080 --> 00:05:30.240
Um the agent has to incorporate all that when it's reasoning through the actual change that current environments have it make.

s50
00:05:30.240 --> 00:05:35.520
After it makes that change, uh you need a kick kick off rolling deployments.

s51
00:05:35.520 --> 00:05:41.320
Those deployment systems can often times be complicated, have conflicts, may not work.

s52
00:05:41.320 --> 00:05:58.720
Um and all through that, when you're finally migrating off of from old hard onto new hardware, um you run into unforeseen problems which you did not sort of the the the the the the agent has to reason through in real time, just like a human would, right?

s53
00:05:58.720 --> 00:06:01.840
You have um failing nodes.

s54
00:06:01.840 --> 00:06:04.760
You have stale deprecated nodes.

s55
00:06:04.760 --> 00:06:12.480
And while all of this is happening, the service can't go down because there is a blast radius to serving live traffic.

s56
00:06:12.480 --> 00:06:15.080
You have to observe and monitor your service.

s57
00:06:15.080 --> 00:06:24.560
All of these components in in in the system is really uh what sort of exemplifies like a full end-to-end infrastructure task.

s58
00:06:24.680 --> 00:06:39.680
So, what Sid is describing here is an environment in a single node sandbox where we're simulating uh distributed cluster with multiple nodes, flapping nodes, lagging learners um in a single sandbox.

s59
00:06:39.680 --> 00:06:41.360
And you can get pretty far with this, right?

s60
00:06:41.360 --> 00:06:47.320
Like you can see that there's live traffic, there's a lot of operational issues that a real engineer would have to deal with.

s61
00:06:47.320 --> 00:06:54.520
And you can make this pretty long horizon by just say doing multiple deployments instead of just one.

s62
00:06:54.520 --> 00:06:58.520
But really what we're seeing is that this is not enough.

s63
00:06:58.520 --> 00:07:04.840
Uh this fits into standard post-training pipelines in the sense that a standard post-training pipeline is kind of boring.

s64
00:07:04.840 --> 00:07:06.160
Uh it's kind of homogeneous.

s65
00:07:06.160 --> 00:07:10.760
You know, everything just runs harbor, everything is a single sandbox, containerized.

s66
00:07:10.760 --> 00:07:15.600
But real infrastructure uh doesn't work like this.

s67
00:07:15.600 --> 00:07:18.520
Uh this isn't how real companies run.

s68
00:07:18.520 --> 00:07:32.480
And uh even though you can use something like deterministic simulation to simulate network failures, it doesn't represent what you might run into if you're building an AWS-scale service.

s69
00:07:32.480 --> 00:07:35.160
So, I did see I think a couple people at AWS.

s70
00:07:35.160 --> 00:07:37.440
Somebody had Viceroy open on their laptop.

s71
00:07:37.440 --> 00:07:38.880
Um fun times.

s72
00:07:38.880 --> 00:07:45.560
Um but let's imagine here that we are all AWS engineers or GCP engineers.

s73
00:07:45.560 --> 00:07:46.280
Azure, too.

s74
00:07:46.280 --> 00:07:47.400
No shade, right?

s75
00:07:47.400 --> 00:07:51.720
Um, and we are building a cloud service.

s76
00:07:51.720 --> 00:07:57.200
Um, it can also be some infrastructure service like DataDog, Vercel, Superbase.

s77
00:07:57.200 --> 00:08:00.120
Uh, all of these services run into the same problems.

s78
00:08:00.120 --> 00:08:02.720
You start off with a shiny piece of software.

s79
00:08:02.720 --> 00:08:07.080
And this piece of software can service a single customer pretty well.

s80
00:08:07.080 --> 00:08:08.800
Um, maybe it's running on your machine.

s81
00:08:08.800 --> 00:08:11.600
If you're working for NLB, it should be a load balancer, right?

s82
00:08:11.600 --> 00:08:15.560
If you're working for AWS Lambda, it'd be some sort of serverless runtime.

s83
00:08:15.560 --> 00:08:19.400
But, uh, it needs to actually run somewhere.

s84
00:08:19.400 --> 00:08:25.360
So, if you're infrastructure engineer, next step is you get into resource provisioning.

s85
00:08:25.360 --> 00:08:30.080
Um, and this is already where the single node sandbox starts breaking down.

s86
00:08:30.080 --> 00:08:33.479
How do you provision resources within a single sandbox?

s87
00:08:33.479 --> 00:08:37.680
You can't exactly simulate something like EC2 or Cloud Run, right?

s88
00:08:37.680 --> 00:08:40.320
Um, so you get into this host provisioning.

s89
00:08:40.320 --> 00:08:46.640
Uh, it also includes provisioning of other resources like VPCs, subnets, security groups.

s90
00:08:46.640 --> 00:08:55.640
Um, and you need to expose this through some sort of API because your customers are going to want to do things like, "Oh, give me this shiny piece of software."

s91
00:08:55.640 --> 00:08:57.200
Or, "I don't want it anymore.

s92
00:08:57.200 --> 00:08:58.200
It cost too much.

s93
00:08:58.200 --> 00:08:59.200
I'm going bankrupt.

s94
00:08:59.200 --> 00:09:00.800
Delete it, please."

s95
00:09:00.800 --> 00:09:03.880
Um, and so you're going to need some sort of front-end API.

s96
00:09:03.880 --> 00:09:15.120
And if you have enterprise grade customers who really care about quality, then you're going to have to meet certain bars like throttling, authentication, authorization.

s97
00:09:15.120 --> 00:09:17.880
You can't really like go without these things, right?

s98
00:09:17.880 --> 00:09:20.880
Uh, if you're AWS, then that's CloudTrail, too.

s99
00:09:20.880 --> 00:09:25.640
Um, and then beyond this, uh, software is living.

s100
00:09:25.640 --> 00:09:29.920
People forget this all the time, especially like investors, right?

s101
00:09:29.920 --> 00:09:31.200
Like they'll be like, "Oh, you wrote it.

s102
00:09:31.200 --> 00:09:31.920
You're done."

s103
00:09:31.920 --> 00:09:40.680
Um, but software is living and you probably need some sort of software deployment component as well.

s104
00:09:40.680 --> 00:09:45.480
Uh something whenever you have an update to roll out roll it out.

s105
00:09:45.480 --> 00:09:48.760
Um and God forbid something goes wrong, roll it back.

s106
00:09:48.760 --> 00:09:54.440
Uh you need to manage all the different versions and make sure your deployments are gradual to limit your blast radius.

s107
00:09:54.440 --> 00:09:56.800
And we're just kind of getting started with this.

s108
00:09:56.800 --> 00:10:03.400
There's all sorts of things that you need to think about like health monitoring with awareness for network partitions.

s109
00:10:03.400 --> 00:10:08.480
Uh and then how do you communicate with your host so you can change configs on the fly.

s110
00:10:08.480 --> 00:10:13.800
Um maybe your customer actually wants to call your endpoint, so you need DNS and cert management.

s111
00:10:13.800 --> 00:10:23.080
And then, you know, your our service grows a bunch, you need to keep track of all your resources, what's going on, fraud and stuff, then you need admin consoles,

s112
00:10:23.080 --> 00:10:28.880
uh telemetry, billing if you're making money, um all sorts of things.

s113
00:10:28.880 --> 00:10:33.880
And with all of this, I think there's like one more slide for is it scheduling?

s114
00:10:33.880 --> 00:10:34.600
Yeah.

s115
00:10:34.600 --> 00:10:36.600
Um

s116
00:10:36.800 --> 00:10:39.800
I think the point is fairly clear at this point.

s117
00:10:39.800 --> 00:10:49.800
Beyond beyond a certain threshold, there is a critical mass at which sandboxing on a single node uh can only get you so far.

s118
00:10:49.800 --> 00:10:58.480
And that's why we envision the the future being going towards a world where environments do provision real infrastructure.

s119
00:10:58.480 --> 00:11:08.160
Yeah, so what this is is um a multi-node sandbox with access to real infra, real cloud resources.

s120
00:11:08.160 --> 00:11:12.600
Uh we kind of put a cloud in box, so cloud box could be another name for this.

s121
00:11:12.600 --> 00:11:26.240
Um and as you can imagine, changing the sandbox type so drastically here affects post-training pipelines as well, which um I think we might be running a little bit low on time, so we won't get like too much into it.

s122
00:11:26.240 --> 00:11:38.720
Um but yeah, like uh one really cool thing, too, is like you can put a post-training pipeline in the sandbox, um and there's some cool stuff with model training and RSI that you can get into there.

s123
00:11:38.720 --> 00:11:44.800
Um so, you know, then this begs a question, uh this is all cool stuff, Joseph.

s124
00:11:44.800 --> 00:11:46.680
Uh thank you, Sid, for speaking.

s125
00:11:46.680 --> 00:11:50.360
Why are you leaking all of this alpha, right?

s126
00:11:50.360 --> 00:11:57.280
Why are you like telling all your organizational secrets and telling everybody like, "Oh, okay, you know, how do you build a system like this?"

s127
00:11:57.280 --> 00:12:01.960
Um it's because uh we're really interested in these challenges here.

s128
00:12:01.960 --> 00:12:03.760
We think they're very fun.

s129
00:12:03.760 --> 00:12:05.720
Uh you know, we think they're really cool.

s130
00:12:05.720 --> 00:12:09.560
We think that you guys are cool people, uh or maybe I'm just lying, who knows.

s131
00:12:09.560 --> 00:12:16.360
Uh and we want to share these challenges with you uh in case you're interested in working on them as well.

s132
00:12:16.360 --> 00:12:20.360
As you can imagine, there's a lot of different problems that we haven't touched on here.

s133
00:12:20.360 --> 00:12:26.560
Like, for example, spinning up the entire stack for something like AWS Lambda takes hours.

s134
00:12:26.560 --> 00:12:29.520
Um how do you fit that into a post training rollout?

s135
00:12:29.520 --> 00:12:31.480
Uh and then there's cost as well.

s136
00:12:31.480 --> 00:12:33.280
How do you efficiently manage this?

s137
00:12:33.280 --> 00:12:38.520
How do you make sure the sim-to-real gap, even with real resources, it still exists, right?

s138
00:12:38.520 --> 00:12:40.920
You still have to have live customer traffic.

s139
00:12:40.920 --> 00:12:45.640
You still have to have uh problems that only appear at a certain scale.

s140
00:12:45.640 --> 00:12:56.360
So, you know, if you're a distributed systems engineer, um you know, if you know, this stuff if you train models before, um if you think that this stuff is cool,

s141
00:12:56.360 --> 00:12:58.440
uh then you know, we'd love to talk.

s142
00:12:58.440 --> 00:13:07.040
Uh we'd love to talk, uh kind of like see where your opinions are, uh hear what you've worked on, maybe that's like, "Oh, Kubernetes."

s143
00:13:07.040 --> 00:13:08.000
And you have like opinions.

s144
00:13:08.000 --> 00:13:13.880
Well, everybody has like opinions on like auto scaling and rolling deployments and whatever, but like really niche opinions, right?

s145
00:13:13.880 --> 00:13:15.839
Like, at CDO.

s146
00:13:15.839 --> 00:13:21.280
Um yeah, we'd love to talk to you and hear what you have.

s147
00:13:21.280 --> 00:13:21.920
Thank you.

s148
00:13:21.920 --> 00:13:22.520
Yep.

s149
00:13:22.520 --> 00:13:26.360
Um what's your like primary goal with emulator?

s150
00:13:26.360 --> 00:13:31.440
Yeah, um that touches into why it's called emulate in the first place, right?

s151
00:13:31.440 --> 00:13:41.560
Uh, the real world is very, very complex, um, and how we as a industry emulate the real world is incredibly contrived and low fidelity.

s152
00:13:41.560 --> 00:13:56.560
So, emulated goal is really how do you make these agents own systems like this, uh, maybe beyond systems, entire companies, by emulating the real world with full fidelity.

s153
00:13:56.800 --> 00:13:57.600
Yeah, go ahead.

s154
00:13:57.600 --> 00:14:17.360
Next question is, are you predominantly focused on like infra and you have a lot of containers, actually hardware kind of related stuff, or are you also like full RL environments

s155
00:14:17.360 --> 00:14:19.560
like a digital twin?

s156
00:14:19.560 --> 00:14:20.000
Yeah, of course.

s157
00:14:20.000 --> 00:14:23.280
Like, the question is like, you know, infra is really cool.

s158
00:14:23.280 --> 00:14:26.920
Um, in 2026, all of us are on an infra's comeback, and it's very sexy.

s159
00:14:26.920 --> 00:14:28.880
Everybody wants to work on it, right?

s160
00:14:28.880 --> 00:14:31.880
But, there's other types of RL environments as well.

s161
00:14:31.880 --> 00:14:41.480
Um, there is, uh, other workflows that you really want to capture that aren't necessarily infra related.

s162
00:14:41.480 --> 00:14:45.200
Uh, so the reason why we're starting with infra is, um, there's a couple.

s163
00:14:45.200 --> 00:14:48.640
Well, the first most important one is it speaks to our background the most.

s164
00:14:48.640 --> 00:14:54.839
Um, we think that domain expertise is something that informs how high quality your data can be.

s165
00:14:54.839 --> 00:14:58.200
Uh, especially with boutique nature of data nowadays.

s166
00:14:58.200 --> 00:15:04.680
Um, and the second is that when we're simulating full companies, infra is the easiest to approach.

s167
00:15:04.680 --> 00:15:14.960
Uh, if you think about like any infra company out there, whether it be Superbase or Modal, uh, or any dev tools company, the problem statement is pretty clear.

s168
00:15:14.960 --> 00:15:17.120
Uh, engineers kind of know what they want.

s169
00:15:17.120 --> 00:15:25.880
If you are working for Modal, you know that your users want GPU sandbox, very low latency, very low cost.

s170
00:15:25.880 --> 00:15:28.960
You don't want to fail halfway through your training run.

s171
00:15:28.960 --> 00:15:30.840
Um that's what you care about.

s172
00:15:30.840 --> 00:15:33.480
So, the problem statement becomes much easier.

s173
00:15:33.480 --> 00:15:41.400
Whereas, if you are, you know, a a company in the YC summer 2026 batch, you're probably still trying to find product market fit, right?

s174
00:15:41.400 --> 00:15:41.800
Yeah.

s175
00:15:41.800 --> 00:15:52.200
At the same time, there's also lessons learned that going really vertical on a single domain like infrastructure do translate into other horizontal domains.

s176
00:15:52.200 --> 00:15:59.440
So, we're also um exploring going deep into one and scaling out that way.

s177
00:15:59.960 --> 00:16:00.320
All right.

s178
00:16:00.320 --> 00:16:02.040
Uh really appreciate it.

s179
00:16:02.040 --> 00:16:03.440
Uh appreciate the questions.

s180
00:16:03.440 --> 00:16:10.280
Um we'll probably step out and we can like take a couple more uh outside just to make sure the next speaker has room.

s181
00:16:10.280 --> 00:16:11.760
Um yeah.

s182
00:16:11.760 --> 00:16:13.160
Uh I thank you guys for listening.

s183
00:16:13.160 --> 00:16:15.520
Appreciate it.

s184
00:16:30.371 --> 00:16:32.371
[music]