{
  "video": {
    "id": "_zUp6y9bDU8",
    "title": "Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI",
    "duration": 2164,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "_zUp6y9bDU8",
    "title": "Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI",
    "one_liner": "Give your coding agent (Claude Code, Cursor, Codex) a CLI and skills that pull down production traces, evals and annotations, and it can automate the whole debug loop: find failures, build datasets, write evals, make fixes, and run experiments that check them.",
    "summary": "Duggal argues that shipping agents 'based off vibes' doesn't hold up in production. Evals need traces underneath them, because an agent's code is basically just `agent.run(query)`, so the only way to see what it did is the trace. The standard improvement loop is to identify failures, collect a sample, diagnose, build evals, make changes, experiment and ship, and it is 'pretty long, pretty manual, and pretty tedious.' He shows how Arize's CLI and skills let a coding agent bring traces, evals, annotations, APM data and the codebase together into one place, cutting that loop from days to minutes, using a demo financial agent that gave unlicensed investment advice.",
    "key_points": [
      "Traditional APM traces are easy to debug because the flow is deterministic. An agent is just `agent.run(query)` and you can't know whether it calls a tool, does analysis or thinks internally until you see the trace.",
      "Example trace failure: a user asks where their refund is, and the agent calls `get_order_status` (the wrong tool) and replies 'your refund was issued' instead of explaining why it hasn't arrived.",
      "Setup takes a few steps: install the Arize CLI with one line, create a profile that connects it to where your data is stored, then install the skills (through the Claude Code marketplace in the demo). Underneath, the skills drive the CLI, which can download traces, make API requests and create evals, datasets and experiments on the platform.",
      "Claude Code analysed the demo financial agent's traces and found that about 44% of interactions produced a bad summary. It also found fabricated answers, a fairly high rate of redundant tool calls, a numeric groundedness problem, silent tool failures (reported as success but returning nothing) and hallucination when tools fail instead of disclosing the failure.",
      "Non-US stock example (South Korean stocks): Claude found the failing spans, built a dataset and ran experiments. The buggy baseline passed 0%. Normalising by adding the country name still failed because the tool was outdated. A third iteration that updated the tool to the stable version reached about 90% success.",
      "Claude built two evaluators on the Arize platform, one code-based and one LLM-as-a-judge. The code-based one failed to set up properly, so the LLM judge was the final version. Experiments ran on his local machine and evaluations ran on Arize.",
      "Eval models are fully configurable, including local open-source models. Some teams use the highest-end models for evals even if they don't use them in production, because 'their agent's only going to be as good as their evals.' Eval traces show cost, and you can sample a percentage of traffic instead of running on 100% of traces.",
      "Phoenix, Arize's open-source version, also has skills and MCP. The enterprise edition adds easier online evals, more dashboarding and a few other features."
    ],
    "takeaways": [
      "Instrument your agent so every LLM call, retrieval and tool call is traced before you try to build evals. Evals sit on top of traces.",
      "Give your coding agent trace, eval, annotation, APM, ticket and codebase context so it fixes 'here's where it failed, here's why' instead of guessing from the output alone. Duggal says this also keeps it focused on real issues rather than linting.",
      "Turn production failures into datasets and run experiments that compare the baseline with each fix iteration before shipping. Then, as the human, check the fix actually improved the agent and didn't just tidy linting.",
      "Keep evals even after fixing prompts or skills. Model regressions or users asking questions in new ways can degrade performance months later, and evals are how you monitor that.",
      "Build custom evals and choose eval models for your own use case. Out-of-the-box evals and public benchmarks give you out-of-the-box results."
    ],
    "topics": [
      "evals",
      "agents",
      "observability",
      "tracing",
      "coding-agents",
      "agent-skills",
      "llm-as-a-judge",
      "experimentation"
    ],
    "tools": [
      "Arize AI",
      "Arize Phoenix",
      "Claude Code",
      "Cursor",
      "Codex",
      "Claude",
      "OpenAI",
      "Jira",
      "MCP"
    ],
    "quotes": [
      {
        "text": "It's pretty long, pretty manual, and pretty tedious.",
        "at": "05:43",
        "url": "https://www.youtube.com/watch?v=_zUp6y9bDU8&t=343s"
      },
      {
        "text": "about 44% of our agent interactions are resulting in a bad summary.",
        "at": "16:56",
        "url": "https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1016s"
      },
      {
        "text": "they know their agent's only going to be as good as their evals so they use the highest end models for their evals",
        "at": "24:11",
        "url": "https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1451s"
      },
      {
        "text": "if you're going to use out of the box eval to judge your agent you're going to get out of the box results",
        "at": "34:57",
        "url": "https://www.youtube.com/watch?v=_zUp6y9bDU8&t=2097s"
      }
    ],
    "words": 7608
  },
  "summary_url": "/#_zUp6y9bDU8",
  "transcript": {
    "html": "/transcripts/_zUp6y9bDU8.html",
    "txt": "/transcripts/_zUp6y9bDU8.txt",
    "vtt": "/transcripts/_zUp6y9bDU8.vtt"
  }
}