{
  "video": {
    "id": "Ib5t2RLtxvM",
    "title": "From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI",
    "duration": 1224,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "Ib5t2RLtxvM",
    "title": "From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI",
    "one_liner": "Snorkel AI's Rustem Feyzkhanov argues every company needs its own private, production-shaped agent benchmark — built by turning production traces into repeatable offline simulations with Dockerized environments, Oracle solutions and verifiers — and to treat that benchmark as software with its own CI.",
    "summary": "Traces tell you what happened in production but can't be replayed apples-to-apples, because database state and tool versions drift; offline simulation turns those traces into repeatable experiments you can run in parallel across agent configurations, scored not just on pass rate but on cost, latency and retries. The talk walks through the anatomy of a benchmark task in the Harbor format (instruction.md, a Dockerfile/Compose environment, plus a hidden Oracle solution and verifiers), how to build the environment as a \"mini production\" with snapshot databases, sidecar containers, mocked APIs and LLM-simulated users, and how to verify against final environment state, trace and artifacts. Feyzkhanov insists the benchmark is code — it needs a CI pipeline that pins dependencies, runs the Oracle, checks the task is solvable but hard, and tags difficulty — and that it becomes a trifecta: evaluation, release gate and training set. Snorkel runs millions of agent simulations per month and treats benchmark construction as an engineering discipline.",
    "key_points": [
      "Traces find production failures but can't be compared apples-to-apples across variants — even A/B testing gives you different database state and tool versions; offline simulation makes runs repeatable and parallel.",
      "Public benchmarks are domain-locked (SWE-bench = GitHub issues, Terminal-Bench = terminal agents, Co-Bench = computer use) and fixated on pass rate; a private benchmark adds your tools, policies and the metrics that matter in production — cost, latency, retries.",
      "Simulation lets you test the full stack, not just model-vs-model: thinking level, prompt, harness, skills and available tools, while holding environment and evaluators fixed between runs.",
      "Harbor format (from the Terminal-Bench team) is the reference task layout: what the agent sees (instruction.md + Dockerfile/Docker Compose environment), what it doesn't (Oracle solution + verifiers), plus metadata.",
      "An Oracle solution is mandatory — it runs the whole sequence in place of the agent to prove the task is solvable before an agent is ever asked to solve it.",
      "The environment is \"mini production\": snapshot database rather than the live one, sidecar containers for API services/databases/MCP tools, mocked APIs, LLM-simulated users, and multi-step checkpoints with per-step prompts and verifiers so long-horizon tasks can be cut short when the agent fails.",
      "Verifiers score final environment state, trace and artifacts — deterministic checks for output and tool calls, LLM/harness/agent-as-judge for trace quality and planning, subject-matter experts only where agent and verifier disagree.",
      "Benchmarks have their own failure modes to catch in development: agents reward-hacking the simulation, verifiers too broad (always pass), verifiers incorrect (always fail), and high run-to-run variance.",
      "A benchmark CI pipeline pins dependencies, checks the base image and fixtures, runs the Oracle (and confirms verifiers fail without it), runs several agent attempts to confirm the task is solvable but hard, tags it simple/medium/hard by success rate, then approves it into the set.",
      "Two connected loops in production: observability (e.g. Arize) feeds recorded failures into benchmark expansion, and a simulation runner tests new agent configs against the expanded benchmark as a release gate.",
      "Snorkel used simulation environments to fine-tune a small planning model to match a large planning model's performance on specific tasks."
    ],
    "takeaways": [
      "Build a private benchmark from your own production traces instead of relying on public leaderboards — use public ones to orient and set priors, your own to decide what ships.",
      "Score agents on cost, latency and retries alongside success rate, and keep environment and evaluators identical between runs so configuration changes are the only variable.",
      "Always write an Oracle solution and put the benchmark under its own CI: pinned dependencies, Oracle passes, verifiers fail without it, multiple agent runs to confirm the task is solvable but hard, then difficulty-tag it.",
      "Stop pushing every fix into the prompt (\"never do this, only do that\") — with a full-stack simulation you can put the fix where it belongs: the harness for context overload, a skill for a missing procedure, structured output for a schema requirement.",
      "Cover both the bread-and-butter happy path and edge cases (tool failures, database problems), hold out a test split (~80/20) the agent never saw during experimentation, and wire observability and experimentation together as two sides of the same coin."
    ],
    "topics": [
      "agents",
      "evals",
      "benchmarks",
      "simulation",
      "agent-ops",
      "verifiers",
      "observability",
      "fine-tuning"
    ],
    "tools": [
      "Snorkel AI",
      "SWE-bench",
      "Terminal-Bench",
      "Co-Bench",
      "Harbor",
      "Docker",
      "Docker Compose",
      "MCP",
      "Arize"
    ],
    "quotes": [
      {
        "text": "Basically, public benchmark is useful to orient and build your prior, but your private benchmark is useful to ship.",
        "at": "03:41",
        "url": "https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=221s"
      },
      {
        "text": "because in your production, you don't care about the model, you care about the full system.",
        "at": "04:55",
        "url": "https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=295s"
      },
      {
        "text": "you don't want your agent to know that it's running within simulation. So, it has to be real.",
        "at": "08:46",
        "url": "https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=526s"
      },
      {
        "text": "benchmark is software. It's code. It's files. You need to treat it as such. You need to have a separate CI pipeline for it.",
        "at": "13:09",
        "url": "https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=789s"
      }
    ],
    "words": 3740
  },
  "summary_url": "/#Ib5t2RLtxvM",
  "transcript": {
    "html": "/transcripts/Ib5t2RLtxvM.html",
    "txt": "/transcripts/Ib5t2RLtxvM.txt",
    "vtt": "/transcripts/Ib5t2RLtxvM.vtt"
  }
}