{
  "video": {
    "id": "kRJjSOYqW0I",
    "title": "How to Test Your Agents on Real Tasks With Harbor - Xander Song, Arize AI",
    "duration": 3361,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "kRJjSOYqW0I",
    "title": "How to Test Your Agents on Real Tasks With Harbor - Xander Song, Arize AI",
    "one_liner": "Harbor runs each natural-language task in a fresh Docker sandbox seeded with real data, then grades the agent with verifier scripts that check only the side effects it leaves behind. Because the tasks and verifiers stay fixed, you can swap prompts, tools, skills or harnesses and quickly learn whether a change helped or hurt, instead of shipping on vibes.",
    "summary": "Xander Song, who works on Arize's Phoenix and calls himself a power user of Harbor rather than a maintainer, explains Harbor's building blocks: tasks (an instruction markdown file, a Dockerfile environment and verifier tests), datasets, trials, jobs, rewards, ATIF trajectories and oracle solutions. He argues verifiers should work like integration tests, ignoring how the agent is built and judging only its side effects, so you can 'refactor under green.' He then shows how his team runs the whole single-node Phoenix server inside Harbor to test three agent surfaces: a coding agent with their CLI, a coding agent with their MCP server, and their server-side agent Pixie. The worked example measures a one-line prompt change that pushes agents to use a new read-only SQL tool for aggregate questions, with results compared in Phoenix.",
    "key_points": [
      "A task has three parts: an instruction markdown file given to the agent, an environment (usually a Dockerfile seeded with data such as a SQLite file or a git repo at a given commit), and a tests directory of verifier scripts. Verifiers write rewards, such as 0/1 or pass/fail, to a reward.txt file. A set of tasks is a dataset, and a full run of all tasks, trials, verifiers and trajectory collection is a job.",
      "Each trial runs in a brand-new container to account for agents being stochastic, so a pass@k metric (e.g. k=3) can be computed per task or across the dataset. Because every trial is independent, the work scales horizontally on sandbox providers such as Daytona, Modal or E2B. His team uses Daytona with about 200 concurrent workers.",
      "Oracles are scripts that solve the task. Running them first shakes out problems with the data, the environment and the verifiers before any money is spent on tokens, and gives confidence in verifiers without reviewing them line by line.",
      "In local Docker runs, verifiers and solutions are mounted only after the agent finishes, so the agent cannot cheat. Canonical verifiers treat the agent as a black box, like well-written integration tests. Verifiers that peek into the ATIF trajectory are possible and sometimes worth it (for example when optimizing tool-call counts or testing whether a skill fires), at the cost of maintainability: rename a skill and the test flakes.",
      "Harbor is a CLI orchestrator that runs on a laptop or a GitHub Actions runner. Its contract is essentially 'did I put my scripts in the right place' (task start, verifier and oracle scripts). There are no linters for this, so the best help is downloading Harbor's own skills.",
      "Because Phoenix runs on a single node ('Grafana-like'), Arize puts the whole server, a realistic SQLite file, the CLI and skills into one Docker image. Verifiers can then hit a REST endpoint or query the SQLite file directly to check, for example, that a datasets row was written. Distributed systems that depend on things like S3 are harder, and mocking those calls leads back into the world of mocking that Harbor is meant to escape.",
      "Case study: users asked aggregate questions (busiest day, most expensive model) that REST and GraphQL could only answer by paginating. Arize added a read-only SQL tool to its MCP server plus a prompt line telling agents to prefer SQL for counts and group-bys. Claude generated synthetic aggregate and non-aggregate queries with oracle solutions, some verifiers being LLM-as-judge. Comparing Opus 5 before and after the change, reward stayed the same, SQL calls went up, and GraphQL calls, tool calls and turns went down.",
      "Phoenix maps Harbor concepts onto its own: datasets map to datasets, jobs to experiments, and rewards to evaluations. ATIF trajectories are converted to OpenInference, a set of semantic conventions on top of OpenTelemetry that was recently donated to the OTel working group / CNCF. Phoenix adds versioned datasets, side-by-side comparison of jobs task by task, natural-language filtering through an in-browser local model (which failed in the live demo), and skills, a CLI and an MCP server so agents can read the results."
    ],
    "takeaways": [
      "Write verifiers that check only end state (files, database rows, API responses), not the agent's internals, so you can change prompts, tools, skills or the harness without the tests breaking. Only peek at trajectories when the metric you are optimizing is the agent's behaviour itself.",
      "Write and run an oracle solution for every task before spending tokens on agents, so you know the data, environment and verifiers are sound.",
      "Treat the system under test as the coding agent plus everything you maintain around it (MCP server, CLI, skills, APIs), and package all of that together in the Docker environment so it is evaluated as one unit.",
      "Use several trials per task and pass@k metrics, and run them in parallel on a sandbox provider so a large suite finishes fast.",
      "Put most of the effort into realistic data, which he calls the hard part. Use a coding agent to brainstorm tasks and oracles from your product's domains, but spot-check them, and trim large data files down to what each task needs."
    ],
    "topics": [
      "agents",
      "evals",
      "sandboxing",
      "observability",
      "coding agents",
      "mcp",
      "regression testing",
      "tool use"
    ],
    "tools": [
      "Harbor",
      "Arize AI",
      "Phoenix",
      "Pixie",
      "ATIF (Agent Trajectory Interchange Format)",
      "OpenInference",
      "OpenTelemetry",
      "OTLP",
      "CNCF",
      "Docker",
      "Daytona",
      "Modal",
      "E2B",
      "Claude Code",
      "Claude",
      "Opus 5",
      "Codex",
      "GPT-5",
      "Cursor",
      "MCP",
      "SQLite",
      "Postgres",
      "GraphQL",
      "REST",
      "GitHub Actions",
      "Playwright",
      "Grafana",
      "S3"
    ],
    "quotes": [
      {
        "text": "The theme of today's talk is shipping based on evidence rather than vibes.",
        "at": "00:38",
        "url": "https://www.youtube.com/watch?v=kRJjSOYqW0I&t=38s"
      },
      {
        "text": "They actually don't know anything about the agent at all. All they care about is what the agent leaves behind.",
        "at": "10:57",
        "url": "https://www.youtube.com/watch?v=kRJjSOYqW0I&t=657s"
      },
      {
        "text": "once you start mocking out S3 calls, you're kind of in a world of pain because now you're in the world of mocking stuff again",
        "at": "30:52",
        "url": "https://www.youtube.com/watch?v=kRJjSOYqW0I&t=1852s"
      },
      {
        "text": "the data is the hard part, right? I think the data getting the data is the hard part",
        "at": "55:37",
        "url": "https://www.youtube.com/watch?v=kRJjSOYqW0I&t=3337s"
      }
    ],
    "words": 11978
  },
  "summary_url": "/#kRJjSOYqW0I",
  "transcript": {
    "html": "/transcripts/kRJjSOYqW0I.html",
    "txt": "/transcripts/kRJjSOYqW0I.txt",
    "vtt": "/transcripts/kRJjSOYqW0I.vtt"
  }
}