{
  "video": {
    "id": "dAoOYFx7ZYw",
    "title": "Stop Giving Your Agent Tools. Give It a Sandbox - Mikyo King, Arize AI",
    "duration": 878,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "dAoOYFx7ZYw",
    "title": "Stop Giving Your Agent Tools. Give It a Sandbox - Mikyo King, Arize AI",
    "one_liner": "Instead of giving an agent a huge catalog of tools, Arize's \"code mode\" lets it search for the tools it needs and call them from code in a sandbox (Bash, Python, SQL). On their Harbor eval runs this cut tokens by 43% and cost by 31%, while latency stayed roughly flat.",
    "summary": "Mikyo King, who leads Arize AI's open source group, argues that coding agents remove the old line between software built for engineers and software for everyone else, because the agent can write the software it needs on the fly. Agents should be built the same way. In traditional tool calling, every tool definition and output lands in the context window, which leads to bloat, context rot, many turns and compaction. In code mode the agent searches for tools and runs them in a programming environment, so it can chain them, fan them out in parallel and wrap them in conditionals in one turn, and return only what it needs. He shows how Arize built this into the Phoenix agent, lists its costs and open problems, and explains how they measure it with Harbor.",
    "key_points": [
      "Traditional tool calling puts every tool definition and output into the context window. As tasks grow longer this causes bloat, context rot, more turns and compaction, which raise latency and cost and slow the user's time to value.",
      "He gives four reasons code mode works: (1) tools are disclosed progressively, so the agent searches for only what it needs; (2) tools can be composed in sequence, in parallel, as a fan-out or as a chain; (3) token efficiency, because the agent trims tool output before it reaches the context; (4) emergent behavior, because the agent can build its own tools. His example: stripping PII from records in code, so credit card numbers never pass through the context window.",
      "Arize's Phoenix agent started with Bash, composing sed, grep and Arize's own CLI and saving results to files. They then added a Python sandbox with all their REST CRUD endpoints exposed inside it (for example fetching traces or changing datasets). Finally they added SQL for aggregate queries, such as finding LLM spans with a given P95 latency, so the agent doesn't have to page through every trace.",
      "Arize also built a programming interface bound to their UI. It runs in a web worker with two-way data binding into the browser, so the agent can change the time range or read the frontend data store. He says this gives 'computer use'-like emergent behavior for complex UI interactions.",
      "The costs: the code is LLM-generated; hosted sandboxes need managing (CPU, memory, how long they live); network egress has to be controlled; secrets must never come back into the context; and permissions are usually too broad, sometimes only once operations are combined. Operations get harder too: sandboxes don't always boot, you debug code rather than individual tools, the tool parameter is now code so its accuracy matters more, and tool discovery needs a handshake.",
      "When code mode exists in both MCP and the harness, you get a 'double sandboxing' problem: one sandbox's output is serialized to JSON, then parsed and fed into another code mode.",
      "Open problems: a long script that fails partway can't resume from that step; the choice of language matters if you need deterministic math (LLMs are good at JavaScript, but it's weak for math); getting images or bytes out of the sandbox needs something like a signed handoff so the binary doesn't end up in the context; and smaller, cheaper models may not handle code mode as well as frontier models.",
      "Evaluation: Arize dogfoods Arize Phoenix and runs its agents in the Harbor framework. Agents run in sandboxes with network access off, over multiple trials to cancel out flakiness. Harbor verifiers score each task 1 or 0, Arize tracks pass@k per task across many permutations, and they compare surfaces (Claude Code, MCP, CLI). Before vs after code mode: tokens down 43%, cost down 31%, latency roughly constant despite sandbox overhead."
    ],
    "takeaways": [
      "Don't load a huge tool catalog up front. Give the agent tool search plus a code environment where it can chain, parallelize and put conditionals around tool calls, and return only what it needs to the context.",
      "Expose your existing REST CRUD endpoints inside a sandbox, and add SQL for aggregate queries that tools for create, read, update and delete can't express.",
      "Plan for sandbox risks before shipping: network egress, keeping secrets out of the context, permissions that are too broad (especially once operations are combined), boot failures, and getting binary output out via signed handoffs.",
      "Measure it properly: run agents in isolated sandboxes over repeated trials with binary verifiers and pass@k, and track tokens, cost and latency before and after code mode.",
      "Record trajectories alongside eval results, then give them to a coding agent to find which parts of the instructions mislead it and to hill-climb the evals."
    ],
    "topics": [
      "agents",
      "code mode",
      "sandboxing",
      "tool calling",
      "context engineering",
      "evals",
      "observability",
      "mcp"
    ],
    "tools": [
      "Arize AI",
      "Arize Phoenix",
      "Harbor framework",
      "Claude Code",
      "Claude dynamic workflows",
      "Anthropic",
      "MCP",
      "SWE-bench",
      "Bash",
      "sed",
      "grep",
      "Python",
      "SQL",
      "JavaScript"
    ],
    "quotes": [
      {
        "text": "So, there's a progressive disclosure of tools.",
        "at": "04:52",
        "url": "https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=292s"
      },
      {
        "text": "tool parameter accuracy becomes a much bigger issue because the tool parameter now is code.",
        "at": "09:52",
        "url": "https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=592s"
      },
      {
        "text": "JavaScript is something LLMs are really good at doing, but maybe not great for math.",
        "at": "11:05",
        "url": "https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=665s"
      },
      {
        "text": "I hope this talk convinces you to give your agent a sandbox and then use one of our tools to evaluate it.",
        "at": "14:22",
        "url": "https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=862s"
      }
    ],
    "words": 3013
  },
  "summary_url": "/#dAoOYFx7ZYw",
  "transcript": {
    "html": "/transcripts/dAoOYFx7ZYw.html",
    "txt": "/transcripts/dAoOYFx7ZYw.txt",
    "vtt": "/transcripts/dAoOYFx7ZYw.vtt"
  }
}