{ "video": { "id": "k72eLcYpVys", "title": "1,000 Agent Tasks in a Sandbox: What Breaks When LLMs Run Code — Kevin Orellana, AWS", "duration": 919, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "k72eLcYpVys", "title": "1,000 Agent Tasks in a Sandbox: What Breaks When LLMs Run Code — Kevin Orellana, AWS", "one_liner": "Kevin Orellana had a coding agent run 1,000 generated workflows across 10 domains against AWS's sandbox platform through an MCP server. The failures grouped around sandbox primitives like file-system permissions, memory, seccomp and package locations, so he argues for fuzz-testing agent sandboxes remotely with broad domain tests, not just locally.", "summary": "Orellana works on Amazon Bedrock AgentCore. He pointed a coding agent at his team's cloud sandbox hosting environment overnight and told it to try to break it. The agent reached the sandbox through a quickly built MCP server, ran about 1,000 arbitrary tests across 10 domains, and logged results to SQLite. Failures fell into two layers: the interface (MCP encoding bugs, null bytes) and the sandbox itself (files that vanished, browser sessions running out of memory). His argument is that agent workloads are arbitrary rather than shaped like a REST API, so low-level sandbox constraints decide whether agents can finish their goals. Broad fuzz testing in the remote sandbox is how you find those sharp edges.", "key_points": [ "Frontier models already know MCP well, so the team spent \"a day or two\" building an MCP server that wraps the cloud sandbox. Coding agents and agent apps could then use the platform natively.", "They asked the agent to write 1,000 tests across 10 domains. The number was deliberately arbitrary, \"to cast a wide net\". Runs took hours and used a lot of tokens, and the agent itself handled credential injection and worked out whether a failure came from the service or the MCP layer.", "Results went to a SQLite database instead of markdown files, which don't scale to 1,000 tests. This also let multiple local subagents run tests concurrently.", "Test domains included sandbox lifecycle under stress, file-system IO (upload/download), performance, and SDLC tasks like building Python and Java packages. Individual tests ranged from end-to-end ML training and PDF report generation to simply opening a web page.", "Three example failures: an uploaded image didn't round-trip because of wrong encoding in the MCP layer; a null byte threw off tests; and files written to a specific sandbox directory couldn't be retrieved.", "Comparing the code sandbox with the cloud browser showed different failure profiles. Browser tests (many tabs, scraping, screenshots, form filling, login automation) often killed the sandbox by running out of memory, which the code interpreter didn't see.", "Failures cluster around sandbox primitives: file-system permissions (read/write/execute), mounted versus persistent or ephemeral storage, memory and CPU, seccomp limits on spawning processes, eBPF kernel interceptors that control egress, and whether Python packages or Rust binaries sit where the runtime expects them.", "He cited two studies. UC Berkeley researchers used an exploit agent with the right environment controls to reach up to 100% on SWE-bench, and another study found that giving agents broken tools greatly reduced their effectiveness. AgentCore recently added SSH into sandboxes so people, or more often coding agents, can debug them." ], "takeaways": [ "Before hosting agents in sandboxes, fuzz-test the remote sandbox with a broad set of agent-generated domain workflows, not only local tests.", "Put an MCP server in front of your platform so coding agents can drive it natively, and send test results to a structured store like SQLite instead of markdown files so subagents can run in parallel.", "When triaging failures, separate interface-layer bugs (encoding, null bytes in the MCP layer) from sandbox-layer bugs (permissions, persistence, memory).", "Size and configure each sandbox for its workload: browser workloads need far more memory than code interpreters. Check process-spawn permissions, seccomp/eBPF egress rules, and package and binary paths.", "Expect your local agent setup (MCP tools, local apps) to differ from the remote sandbox, and give yourself or your agent a way, such as SSH, to get in and investigate." ], "topics": [ "agents", "sandboxes", "fuzz-testing", "mcp", "agent-infrastructure", "evals", "coding-agents" ], "tools": [ "Amazon Bedrock AgentCore", "AWS", "MCP (Model Context Protocol)", "SQLite", "eBPF", "seccomp", "SSH", "SWE-bench", "UC Berkeley", "Kiro", "Claude Code", "Codex" ], "quotes": [ { "text": "I took a coding agent. I pointed it at my team's uh cloud sandbox hosting environment and it said try to break it and I let that run overnight.", "at": "00:23", "url": "https://www.youtube.com/watch?v=k72eLcYpVys&t=23s" }, { "text": "agents are pretty prone to write markdown files but that doesn't really scale when you have a thousand tests that you want to run", "at": "03:30", "url": "https://www.youtube.com/watch?v=k72eLcYpVys&t=210s" }, { "text": "I don't know if anyone writes unit tests by hand anymore.", "at": "11:09", "url": "https://www.youtube.com/watch?v=k72eLcYpVys&t=669s" }, { "text": "if you're looking to host your agents on sandboxes, sure test it locally, but also fuzzy test it with a wide set of domain tests um remotely.", "at": "14:44", "url": "https://www.youtube.com/watch?v=k72eLcYpVys&t=884s" } ], "words": 2992 }, "summary_url": "/#k72eLcYpVys", "transcript": { "html": "/transcripts/k72eLcYpVys.html", "txt": "/transcripts/k72eLcYpVys.txt", "vtt": "/transcripts/k72eLcYpVys.vtt" } }