{
  "video": {
    "id": "fnuNVFgLjHA",
    "title": "Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI",
    "duration": 2335,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "fnuNVFgLjHA",
    "title": "Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI",
    "one_liner": "Arize argues observability has to move from humans reading dashboards to programmatic interfaces that agents consume. Its loop is detect, investigate, fix, verify: an always-on agent (Signal) or a coding agent with Arize skills reads production traces alongside the repo, opens a PR, and checks the fix against a dataset of real failed traces before a human reviews it.",
    "summary": "Fuad Ali, who previously worked on backend distributed systems at SpaceX, compares self-improving agents to a rocket that course-corrects by consuming its own telemetry. He describes three phases: software 1.0, where humans read Datadog and write code; 2.0, where humans read telemetry and write prompts; and 3.0, a fully autonomous loop where a coding agent reads the telemetry. The core argument is that the failures that cost you are invisible at launch, so you can't write evals for them in advance. Using a demo shopping agent with a planted price-filter bug, he shows Arize Signal clustering production failures and opening PRs, Arize skills letting Claude Code query traces, 'harness as a judge' for adaptive evaluation, and agent experiments that re-run a fix against real failed traces.",
    "key_points": [
      "Three phases of software: 1.0 means an app traced to Datadog that a human reads before writing code. 2.0 means a human reads the telemetry and writes a prompt. 3.0 means a coding agent reads the telemetry itself. With thousands of agents it becomes 'pretty much impossible for a human being to actually read those things.'",
      "The demo is the Wonder Toys shopping assistant: OpenAI Agents SDK on a FastAPI backend, vector search with a text fallback, traced to Arize. Its tools are search products, get product details, purchase, check order status and cancel order. The planted bug: a request for a 'birthday gift under $30' returns items over $30, and the agent confidently buys the wrong item because it never verifies the product.",
      "Signal is an out-of-the-box agent that runs in a sandbox on the Arize platform and is turned on with one toggle in project settings. It reads every production trace, clusters failures by root cause and de-dupes them across runs. Connected to GitHub and to sources like Datadog, it writes an investigation (severity, hypothesis, evidence, proposed fix) and can open a PR. It can also be triggered by alerts, such as an eval going above a 1% trigger rate or a latency or cost spike.",
      "Arize skills (installed with `npx skills add arize-ai/arize-skills`) let Cursor, Claude Code, Codex, OpenCode and '40 plus other coding agents' query Arize traces, curate datasets from failures, and run evaluators. He frames Claude Code with skills as the on-demand path and Signal as the always-on path.",
      "'Harness as a judge' replaces LLM-as-a-judge, which is a fixed rubric that only catches issues you already know about. With harness as a judge, the criteria emerge from the trajectory: was this the right tool, were the arguments valid, was the answer grounded in context handed off by another agent? The rubric is one plain-language sentence, it reads the trace tree directly, and it runs in a sandbox. Because it can call external systems, it can do things like access checks: is this user allowed to do this?",
      "Agent experiments: deploy the fixed agent to dev behind an endpoint, stream in a dataset of real failed traces (e.g. 25 failed budget-constrained searches), and score with the same judge rubric used in production. Compare before and after the fix, including latency and cost.",
      "The full autonomous loop: the agent collects its own failures, builds a dataset, opens a fix PR, redeploys itself, checks the PR against those failures, and reports results in the PR before asking for human review. His example reported a 36% improvement on the evaluator. In the demo, code review was still left to a human, which he recommends.",
      "Scale claims for Arize: over 1 trillion spans processed, over 60 million evaluations a month, over 3 million downloads a month. Customers named include Spotify, Reddit, Uber and TripAdvisor. Ramp Labs is cited as a non-customer moving toward self-healing software."
    ],
    "takeaways": [
      "Instrument tracing first. 'If you're not collecting tracing, nothing else works,' and he says it takes under 10 minutes on your own agent.",
      "Give your analysis agent both the production traces (what the agent actually does) and the repo (what the agent should do), so it can tie a failure to the exact code path, such as the price filter.",
      "Use agent-based judges that can call tools and adapt their criteria to the trajectory, so you catch unknown unknowns. Write the rubric as a plain sentence, e.g. 'pass if every returned product's price is within the requested min/max range.'",
      "Before shipping a fix, turn real failed traces into a dataset and run the fixed dev agent against it with the production judge. Compare before and after on quality, latency and cost.",
      "Trigger automated investigations from alerts (eval rate thresholds, latency or cost spikes), and keep a human reviewing the code on agent-authored PRs."
    ],
    "topics": [
      "agents",
      "observability",
      "evals",
      "self-improving agents",
      "llm-as-a-judge",
      "tracing",
      "coding agents",
      "experimentation"
    ],
    "tools": [
      "Arize AI",
      "Arize Signal",
      "Arize skills",
      "Alex (Arize agent)",
      "OpenInference",
      "ADB (Arize AI data store)",
      "Arize agent experiments",
      "Arize agent playground",
      "OpenAI Agents SDK",
      "FastAPI",
      "Datadog",
      "Datadog CLI",
      "Cursor",
      "Claude Code",
      "Codex",
      "OpenCode",
      "GitHub",
      "SpaceX",
      "Spotify",
      "Reddit",
      "Uber",
      "TripAdvisor",
      "Ramp Labs"
    ],
    "quotes": [
      {
        "text": "the failures that cost you are invisible at launch. And that's sort of by definition because if you had caught them, they wouldn't have cost you.",
        "at": "05:06",
        "url": "https://www.youtube.com/watch?v=fnuNVFgLjHA&t=306s"
      },
      {
        "text": "observability has to evolve from humans reading dashboards to really programmatic interfaces that agents consume, agents can operate on, and agents can self-reflect on as well.",
        "at": "03:38",
        "url": "https://www.youtube.com/watch?v=fnuNVFgLjHA&t=218s"
      },
      {
        "text": "How do you actually write an eval for an edge case you haven't seen yet?",
        "at": "09:12",
        "url": "https://www.youtube.com/watch?v=fnuNVFgLjHA&t=552s"
      },
      {
        "text": "If you're not collecting tracing, nothing else works.",
        "at": "35:39",
        "url": "https://www.youtube.com/watch?v=fnuNVFgLjHA&t=2139s"
      }
    ],
    "words": 7499
  },
  "summary_url": "/#fnuNVFgLjHA",
  "transcript": {
    "html": "/transcripts/fnuNVFgLjHA.html",
    "txt": "/transcripts/fnuNVFgLjHA.txt",
    "vtt": "/transcripts/fnuNVFgLjHA.vtt"
  }
}