{ "video": { "id": "QSeaZ6qHyXs", "title": "Claude Managed Agents Q&A: Evals, CI/CD and Multi-Tenancy (Part 4 of 4) — Anthropic", "duration": 1138, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "QSeaZ6qHyXs", "title": "Claude Managed Agents Q&A: Evals, CI/CD and Multi-Tenancy (Part 4 of 4) — Anthropic", "one_liner": "Anthropic engineers answer audience questions on Claude Managed Agents (CMA): manage agents as YAML in CI through the CLI, use a separate workspace per customer when data must be segregated, build evals from high-quality data such as historical sessions, and let a coordinator agent stay responsive while sub-agents wait on custom tools.", "summary": "This is a Q&A after a Claude Managed Agents workshop. The speakers explain how to run CMA resources through CI/CD with a Terraform-like CLI driven by YAML. They also cover tenancy options (an agent, workspace or org per customer) and how Anthropic runs evals on its own harness at scale. Other answers cover agent-driven PR review and QA, custom compute through self-hosted sandboxes such as Modal, Slack-style multiplayer agents, long-running sessions built on the outcome system, and using sub-agents so the main agent is never blocked waiting on a custom tool.", "key_points": [ "CI/CD: a CLI (transcribed as 'Ancho CLI') wraps the API and works in a Terraform-like way with YAML. You check long-lived resources into your code base as YAML and have CI push updates whenever they change. Anthropic does the same internally for durable resources.", "You can queue messages while the model is working, but not while it is waiting for a custom tool result. Execution pauses, and because of ordering, sending a message at that point returns a 400.", "Internal evals: Anthropic gets new models early and updates its assumptions about the harness as soon as possible, removing things that are no longer needed. Its internal tooling runs datasets as sessions, for example 10,000 parallel sessions in the cloud, and the results are used to tune prompts and assumptions. The evals cover general purposes, not just coding.", "Eval advice for builders: get high-quality data. Claude Code is good at building an eval, but 'garbage in, garbage out'. Some of Anthropic's own early evals were ill-defined and scored 100% without saying anything about real performance. Historical sessions are a good starting point for a dataset.", "Access control: resources are bound to a workspace, and any API key in that workspace can reach them. The org admin sets up workspaces and decides who joins each one. There are no fine-grained per-resource sharing controls, but integrators can build partial sharing for their own end users.", "Multi-tenancy: you can have an agent per customer, a workspace per customer, or an org per customer for true isolation. A single shared agent can tag sessions with a customer ID in the session metadata field. Legal data-segregation requirements call for separate workspaces, with tooling to sync one agent definition from code to many workspaces through CI. On-prem is not supported.", "Code review bottleneck: a CMA integration can start many sessions and sub-agents on each PR to find bugs. Internally this has cut per-PR review time and bug counts. Claude can also run the app in the sandbox and act as a QA agent, driving a browser and possibly returning a video. There is no tunneling for live UI access today, and the speakers called keeping up with the explosion in PRs 'partially an open question'.", "Compute: all provided sandboxes are currently identical. For more memory or a GPU, use self-hosted sandboxes such as Modal; there are cookbooks for Modal, E2B (transcribed 'EB2') and Daytona. Long runs (6+ hours) need nothing special, and the outcome system with an exit criterion or rubric keeps Claude iterating until it is met. Progress can be watched through the session events API or a custom 'update status' tool." ], "takeaways": [ "Keep agent definitions in YAML in your repo and deploy them through CI with the CLI, including syncing one definition across many workspaces.", "Choose isolation to match your requirements: session metadata (a customer ID) on a shared agent for light separation, a workspace per customer for legal data segregation, and an org per customer for true isolation.", "Build evals from high-quality data, such as scored historical sessions, and check that a 100% score actually means good performance.", "Put custom tools (such as human-in-the-loop) on a sub-agent so the main coordinator agent can always receive messages instead of returning a 400.", "For multiplayer chat agents, map each thread or channel to a session, label each message with the sender, and give the agent an explicit post tool rather than mirroring every reply." ], "topics": [ "managed agents", "evals", "ci/cd", "multi-tenancy", "code review", "sandboxes", "sub-agents", "long-running agents" ], "tools": [ "Claude Managed Agents (CMA)", "Claude Code", "Anchor CLI (transcribed 'Ancho CLI')", "Terraform", "YAML", "Modal", "E2B (transcribed 'EB2')", "Daytona", "Slack", "Claude Tag (transcribed 'Cloud Tag')", "Code with Claude conference", "Anthropic" ], "quotes": [ { "text": "You'll get a 400.", "at": "01:40", "url": "https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=100s" }, { "text": "It's because it's all in the cloud it's easy to say like spin up 10,000 sessions to do this all in parallel", "at": "03:25", "url": "https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=205s" }, { "text": "You don't necessarily want to just mirror whatever the agent is responding. You want it to be explicit.", "at": "14:12", "url": "https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=852s" }, { "text": "and it will just mercilessly iterate until it matches the the rubric.", "at": "15:46", "url": "https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=946s" } ], "words": 4055 }, "summary_url": "/#QSeaZ6qHyXs", "transcript": { "html": "/transcripts/QSeaZ6qHyXs.html", "txt": "/transcripts/QSeaZ6qHyXs.txt", "vtt": "/transcripts/QSeaZ6qHyXs.vtt" } }