Emulated: The Data for Fully Autonomous Software Engineers and Companies — Joseph Wang

AI Engineer · 16 min · 184 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:14So, I appreciate the intro.
00:15Uh my name is Joseph and this is my co-founder Sid.
00:19Emulated is a data lab focused on increasing the reliability and autonomy of AI agents.
00:26And if you've been a AI engineer and you've watched the talks, seen the tracks, then there's probably one takeaway that all the talks have in common.
00:36And it's that we're headed towards a future where agents are able to perform useful work over longer and longer horizons with little to no supervision.
00:45So, today we're going to answer some of the questions of what this means for the data and model layers.
00:52Uh we're going to touch on some pretty cool things.
00:55Uh so, look out for them.
00:56Um like how to simulate a company within a sandbox or sandboxes for multi-node systems and distributed clusters.
01:04Um and if we have a little bit of time, we'll also go into some of the work that we're doing with post-training pipelines and how these new types of sandboxes
01:12are affecting post-training infra as well.
01:17So, where Sid and I come from, um our backgrounds are in network infra, distributed databases, and sandbox infra.
01:25And these are all areas where the workloads are mission critical.
01:30We all saw a couple months ago that uh when something like DynamoDB goes down, so does US East 1 and half the internet.
01:38Um and working on these systems, we saw model capability gap when it came to operating and building these systems at scale at scale and thinking about uh the consequences
01:50of architecture and system system design over the course of years.
01:55Yeah, so it led to a pretty uh natural question, right?
01:58For such mission-critical services, why is it that my model or my agent is so proficient at handling the application layer, but is struggles when it comes to reasoning through
02:10infrastructure complexities?
02:12For example, things like MVCC on a database engine, which can lead to corruption issues, which is one of which was one of the roots of the DynamoDB failure a few months ago.
02:26So, like with everything in NML, uh the gap in models is usually a gap in data.
02:31Models typically are only as good at as data is.
02:36Um and to really highlight this point, right?
02:39Model capability has never uh regressed whenever you introduce more high-quality data.
02:45Um so, with that being said, what is the data gap then?
02:48What does data look like right now?
02:50And how is this influencing the model capability gap here?
02:55So, if you look at any of the frontier or recent benchmarks, like SweBench Pro, Terminal Bench, or something like Frontier Code and Deep Sweep, um the tasks only operate within the code base.
03:09Uh the agent is given a pretty large uh task uh and over the course of 50 to 100 turns produces a couple thousand-line PR.
03:21Um but it doesn't do all of the work that a human does.
03:25It doesn't do uh what a PM does with talking to customers, understanding their problems, what an engineer does with trying out different approaches, performing performance testing them, um and owning the underlying infra
03:39for the code base over the course of not just months, but years.
03:44And this is really the gap that we're closing.
03:47We've taken software engineering companies and we've put them into containerized environments.
03:52so this includes uh include like organizational contexts like projects, incidents, customer conversations.
04:01Uh the agent also has to deal with issues that only appear at scale like network failures between distributed nodes, data corruption, and clock skew.
04:10And through all this, we also want the agents to reason about orchestrating through distributed clusters and also thinking about things like operational blast radius while solving live traffic.
04:21And the result is that the task that these agents have to complete or we want the agents to learn is that environments are far more complex and long horizon
04:31than a simple code diff.
04:35So, let's just let's bring a picture into the mix because it tends to make things more interesting.
04:42Uh here's an example we've built of an SCD consensus cluster that a typical production service might rely on.
04:49So, an old environment uh might tended to operate and work primarily on that little blue square entitled SCD source code in the bottom right there.
05:00But, a lot of the fun and the model capability gap that results from it is really in everything that surrounds it.
05:07So, you you start with the tickets, projects, postmortems.
05:12What are the train wrecks?
05:14Why did they happen?
05:15How did customers feel about them?
05:17And often times those aren't necessarily up to date.
05:21Um the agent has to incorporate all that when it's reasoning through the actual change that current environments have it make.
05:30After it makes that change, uh you need a kick kick off rolling deployments.
05:35Those deployment systems can often times be complicated, have conflicts, may not work.
05:41Um and all through that, when you're finally migrating off of from old hard onto new hardware, um you run into unforeseen problems which you did not sort of the the the the the the agent has to reason through in real time, just like a human would, right?
05:58You have um failing nodes.
06:01You have stale deprecated nodes.
06:04And while all of this is happening, the service can't go down because there is a blast radius to serving live traffic.
06:12You have to observe and monitor your service.
06:15All of these components in in in the system is really uh what sort of exemplifies like a full end-to-end infrastructure task.
06:24So, what Sid is describing here is an environment in a single node sandbox where we're simulating uh distributed cluster with multiple nodes, flapping nodes, lagging learners um in a single sandbox.
06:39And you can get pretty far with this, right?
06:41Like you can see that there's live traffic, there's a lot of operational issues that a real engineer would have to deal with.
06:47And you can make this pretty long horizon by just say doing multiple deployments instead of just one.
06:54But really what we're seeing is that this is not enough.
06:58Uh this fits into standard post-training pipelines in the sense that a standard post-training pipeline is kind of boring.
07:04Uh it's kind of homogeneous.
07:06You know, everything just runs harbor, everything is a single sandbox, containerized.
07:10But real infrastructure uh doesn't work like this.
07:15Uh this isn't how real companies run.
07:18And uh even though you can use something like deterministic simulation to simulate network failures, it doesn't represent what you might run into if you're building an AWS-scale service.
07:32So, I did see I think a couple people at AWS.
07:35Somebody had Viceroy open on their laptop.
07:37Um fun times.
07:38Um but let's imagine here that we are all AWS engineers or GCP engineers.
07:45Azure, too.
07:46No shade, right?
07:47Um, and we are building a cloud service.
07:51Um, it can also be some infrastructure service like DataDog, Vercel, Superbase.
07:57Uh, all of these services run into the same problems.
08:00You start off with a shiny piece of software.
08:02And this piece of software can service a single customer pretty well.
08:07Um, maybe it's running on your machine.
08:08If you're working for NLB, it should be a load balancer, right?
08:11If you're working for AWS Lambda, it'd be some sort of serverless runtime.
08:15But, uh, it needs to actually run somewhere.
08:19So, if you're infrastructure engineer, next step is you get into resource provisioning.
08:25Um, and this is already where the single node sandbox starts breaking down.
08:30How do you provision resources within a single sandbox?
08:33You can't exactly simulate something like EC2 or Cloud Run, right?
08:37Um, so you get into this host provisioning.
08:40Uh, it also includes provisioning of other resources like VPCs, subnets, security groups.
08:46Um, and you need to expose this through some sort of API because your customers are going to want to do things like, "Oh, give me this shiny piece of software."
08:55Or, "I don't want it anymore.
08:57It cost too much.
08:58I'm going bankrupt.
08:59Delete it, please."
09:00Um, and so you're going to need some sort of front-end API.
09:03And if you have enterprise grade customers who really care about quality, then you're going to have to meet certain bars like throttling, authentication, authorization.
09:15You can't really like go without these things, right?
09:17Uh, if you're AWS, then that's CloudTrail, too.
09:20Um, and then beyond this, uh, software is living.
09:25People forget this all the time, especially like investors, right?
09:29Like they'll be like, "Oh, you wrote it.
09:31You're done."
09:31Um, but software is living and you probably need some sort of software deployment component as well.
09:40Uh something whenever you have an update to roll out roll it out.
09:45Um and God forbid something goes wrong, roll it back.
09:48Uh you need to manage all the different versions and make sure your deployments are gradual to limit your blast radius.
09:54And we're just kind of getting started with this.
09:56There's all sorts of things that you need to think about like health monitoring with awareness for network partitions.
10:03Uh and then how do you communicate with your host so you can change configs on the fly.
10:08Um maybe your customer actually wants to call your endpoint, so you need DNS and cert management.
10:13And then, you know, your our service grows a bunch, you need to keep track of all your resources, what's going on, fraud and stuff, then you need admin consoles,
10:23uh telemetry, billing if you're making money, um all sorts of things.
10:28And with all of this, I think there's like one more slide for is it scheduling?
10:33Yeah.
10:34Um
10:36I think the point is fairly clear at this point.
10:39Beyond beyond a certain threshold, there is a critical mass at which sandboxing on a single node uh can only get you so far.
10:49And that's why we envision the the future being going towards a world where environments do provision real infrastructure.
10:58Yeah, so what this is is um a multi-node sandbox with access to real infra, real cloud resources.
11:08Uh we kind of put a cloud in box, so cloud box could be another name for this.
11:12Um and as you can imagine, changing the sandbox type so drastically here affects post-training pipelines as well, which um I think we might be running a little bit low on time, so we won't get like too much into it.
11:26Um but yeah, like uh one really cool thing, too, is like you can put a post-training pipeline in the sandbox, um and there's some cool stuff with model training and RSI that you can get into there.
11:38Um so, you know, then this begs a question, uh this is all cool stuff, Joseph.
11:44Uh thank you, Sid, for speaking.
11:46Why are you leaking all of this alpha, right?
11:50Why are you like telling all your organizational secrets and telling everybody like, "Oh, okay, you know, how do you build a system like this?"
11:57Um it's because uh we're really interested in these challenges here.
12:01We think they're very fun.
12:03Uh you know, we think they're really cool.
12:05We think that you guys are cool people, uh or maybe I'm just lying, who knows.
12:09Uh and we want to share these challenges with you uh in case you're interested in working on them as well.
12:16As you can imagine, there's a lot of different problems that we haven't touched on here.
12:20Like, for example, spinning up the entire stack for something like AWS Lambda takes hours.
12:26Um how do you fit that into a post training rollout?
12:29Uh and then there's cost as well.
12:31How do you efficiently manage this?
12:33How do you make sure the sim-to-real gap, even with real resources, it still exists, right?
12:38You still have to have live customer traffic.
12:40You still have to have uh problems that only appear at a certain scale.
12:45So, you know, if you're a distributed systems engineer, um you know, if you know, this stuff if you train models before, um if you think that this stuff is cool,
12:56uh then you know, we'd love to talk.
12:58Uh we'd love to talk, uh kind of like see where your opinions are, uh hear what you've worked on, maybe that's like, "Oh, Kubernetes."
13:07And you have like opinions.
13:08Well, everybody has like opinions on like auto scaling and rolling deployments and whatever, but like really niche opinions, right?
13:13Like, at CDO.
13:15Um yeah, we'd love to talk to you and hear what you have.
13:21Thank you.
13:21Yep.
13:22Um what's your like primary goal with emulator?
13:26Yeah, um that touches into why it's called emulate in the first place, right?
13:31Uh, the real world is very, very complex, um, and how we as a industry emulate the real world is incredibly contrived and low fidelity.
13:41So, emulated goal is really how do you make these agents own systems like this, uh, maybe beyond systems, entire companies, by emulating the real world with full fidelity.
13:56Yeah, go ahead.
13:57Next question is, are you predominantly focused on like infra and you have a lot of containers, actually hardware kind of related stuff, or are you also like full RL environments
14:17like a digital twin?
14:19Yeah, of course.
14:20Like, the question is like, you know, infra is really cool.
14:23Um, in 2026, all of us are on an infra's comeback, and it's very sexy.
14:26Everybody wants to work on it, right?
14:28But, there's other types of RL environments as well.
14:31Um, there is, uh, other workflows that you really want to capture that aren't necessarily infra related.
14:41Uh, so the reason why we're starting with infra is, um, there's a couple.
14:45Well, the first most important one is it speaks to our background the most.
14:48Um, we think that domain expertise is something that informs how high quality your data can be.
14:54Uh, especially with boutique nature of data nowadays.
14:58Um, and the second is that when we're simulating full companies, infra is the easiest to approach.
15:04Uh, if you think about like any infra company out there, whether it be Superbase or Modal, uh, or any dev tools company, the problem statement is pretty clear.
15:14Uh, engineers kind of know what they want.
15:17If you are working for Modal, you know that your users want GPU sandbox, very low latency, very low cost.
15:25You don't want to fail halfway through your training run.
15:28Um that's what you care about.
15:30So, the problem statement becomes much easier.
15:33Whereas, if you are, you know, a a company in the YC summer 2026 batch, you're probably still trying to find product market fit, right?
15:41Yeah.
15:41At the same time, there's also lessons learned that going really vertical on a single domain like infrastructure do translate into other horizontal domains.
15:52So, we're also um exploring going deep into one and scaling out that way.
15:59All right.
16:00Uh really appreciate it.
16:02Uh appreciate the questions.
16:03Um we'll probably step out and we can like take a couple more uh outside just to make sure the next speaker has room.
16:10Um yeah.
16:11Uh I thank you guys for listening.
16:13Appreciate it.
16:30[music]