{ "video": { "id": "zkX03APVj0M", "title": "Emulated: The Data for Fully Autonomous Software Engineers and Companies — Joseph Wang", "duration": 993, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "zkX03APVj0M", "title": "Emulated: The Data for Fully Autonomous Software Engineers and Companies — Joseph Wang", "one_liner": "Emulated is a data lab that containerizes entire software-engineering companies — org context, incidents, customer conversations and real distributed infra — because benchmarks that stop at the code base are why models can write the application layer but can't reason about MVCC, blast radius or rolling deployments.", "summary": "Joseph Wang and co-founder Sid argue that the model capability gap in infrastructure is a data gap: frontier benchmarks like SWE-bench Pro, Terminal-Bench and Frontier Code/Deep SWE only operate inside the code base, giving an agent 50–100 turns to produce a couple-thousand-line PR, while ignoring everything a real engineer and PM do around it over months and years. Emulated puts whole companies into containerized environments — tickets, projects, postmortems, customer conversations, plus scale-only failures like network partitions, data corruption and clock skew — and walks through an etcd-style consensus cluster where the source code is one small blue square and the rest of the diagram is the actual work. They then show that a single-node sandbox breaks down once you try to simulate a real cloud service (resource provisioning, VPCs, front-end APIs, throttling/authn/authz, deployment and rollback, health monitoring, DNS and certs, billing), so the future is multi-node sandboxes that provision real infrastructure — a 'cloud in a box'. The talk is explicitly a recruiting pitch: they're leaking the alpha because they want distributed systems engineers with niche opinions to come work on the open problems.", "key_points": [ "The framing: 'the gap in models is usually a gap in data' — model capability has never regressed from adding more high-quality data, so the infra reasoning gap is a data problem, not an architecture one.", "Current benchmarks (SWE-bench Pro, Terminal-Bench, Frontier Code, Deep SWE) confine the agent to the code base: a large task, 50–100 turns, a couple-thousand-line PR — and none of the PM work of talking to customers, trying different approaches, performance testing, or owning the underlying infra for years.", "Emulated puts software engineering companies into containerized environments including organizational context — projects, incidents, customer conversations — where the postmortems and tickets are often not up to date, so the agent must reason through stale documentation like a human.", "Environments inject failures that only appear at scale: network failures between distributed nodes, data corruption, and clock skew; the agent must reason about operational blast radius while serving live traffic.", "Worked example: an etcd-style consensus cluster (rendered as 'SCD' in the transcript) where the source code is one small blue square in the corner, surrounded by tickets/postmortems, rolling deployments that conflict or fail, migration from old to new hardware, failing and stale deprecated nodes, and monitoring — the full end-to-end infrastructure task.", "Single-node sandboxes with simulated flapping nodes and lagging learners get you far, and can be made long-horizon by doing multiple deployments — but standard post-training pipelines are 'kind of boring' and homogeneous, and deterministic simulation of network failures doesn't represent an AWS-scale service.", "The breaking point is resource provisioning: you can't simulate EC2 or Cloud Run inside one sandbox, nor VPCs, subnets, security groups, a front-end API with throttling/authentication/authorization (CloudTrail if you're AWS), gradual deployment with rollback and version management, health monitoring aware of network partitions, config push, DNS and cert management, admin consoles, telemetry and billing.", "Their answer: a multi-node sandbox with access to real cloud resources — 'cloud in a box' — which in turn changes post-training infra; you can even put a post-training pipeline inside the sandbox, with implications for model training and RSI.", "Open problems they name: spinning up the full stack for something like AWS Lambda takes hours, which doesn't fit a post-training rollout; cost efficiency; and a sim-to-real gap that persists even with real resources because you still need live customer traffic and scale-only problems.", "They start with infra because it matches their background (network infra, distributed databases, sandbox infra) and because the problem statement is clearest — a Modal user wants a GPU sandbox with low latency, low cost, and no failure halfway through a training run — versus a YC Summer 2026 company still hunting product-market fit." ], "takeaways": [ "Stop treating a merged PR as the unit of software engineering evaluation — if your agent is only scored on a code diff, you are not measuring the provisioning, deployment, migration and incident reasoning that dominates real infra work.", "Build environments that include the organizational layer — tickets, projects, postmortems, customer conversations — and deliberately let some of it be stale, since reconciling out-of-date context is part of the task.", "Inject the failures that only appear at scale (network partitions between nodes, data corruption, clock skew) and require the agent to keep live traffic up, so blast radius becomes a learned constraint rather than a stated one.", "Recognize the single-node sandbox ceiling: once the task involves provisioning EC2-like hosts, VPCs, subnets and security groups, containerizing on one node stops being a fidelity shortcut and starts being the wrong abstraction — move to multi-node sandboxes backed by real cloud resources.", "If you're designing post-training around these environments, budget for the real constraints they name: hours-long stack spin-up versus rollout latency, cloud cost per rollout, and the residual sim-to-real gap that real resources alone don't close." ], "topics": [ "agents", "rl-environments", "post-training", "sandboxes", "distributed-systems", "benchmarks", "training-data", "infrastructure" ], "tools": [ "Emulated", "DynamoDB", "SWE-bench Pro", "Terminal-Bench", "Frontier Code", "Deep SWE", "etcd", "AWS", "GCP", "Azure", "AWS Lambda", "EC2", "Cloud Run", "CloudTrail", "Datadog", "Vercel", "Supabase", "Modal", "Kubernetes", "Viceroy", "Y Combinator" ], "quotes": [ { "text": "Like with everything in ML, the gap in models is usually a gap in data. Models typically are only as good as data is.", "at": "02:26", "url": "https://www.youtube.com/watch?v=zkX03APVj0M&t=146s" }, { "text": "Why is it that my model or my agent is so proficient at handling the application layer, but struggles when it comes to reasoning through infrastructure complexities?", "at": "01:55", "url": "https://www.youtube.com/watch?v=zkX03APVj0M&t=115s" }, { "text": "Beyond a certain threshold, there is a critical mass at which sandboxing on a single node can only get you so far. And that's why we envision the future being going towards a world where environments do provision real infrastructure.", "at": "10:36", "url": "https://www.youtube.com/watch?v=zkX03APVj0M&t=636s" }, { "text": "The real world is very, very complex, and how we as an industry emulate the real world is incredibly contrived and low fidelity.", "at": "13:29", "url": "https://www.youtube.com/watch?v=zkX03APVj0M&t=809s" } ], "words": 2986 }, "summary_url": "/#zkX03APVj0M", "transcript": { "html": "/transcripts/zkX03APVj0M.html", "txt": "/transcripts/zkX03APVj0M.txt", "vtt": "/transcripts/zkX03APVj0M.vtt" } }