{
  "video": {
    "id": "XyV6bSMyq-I",
    "title": "How We Built an Agent That Improves Itself — Zubin Aysola, Weights & Biases",
    "duration": 1026,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "XyV6bSMyq-I",
    "title": "How We Built an Agent That Improves Itself — Zubin Aysola, Weights & Biases",
    "one_liner": "W&B builds its Arya agent with a flywheel. Production and offline runs of a byte-identical agent are both traced in Weave, every production miss (or win) becomes a YAML eval task, and Arya now does the hill-climbing on itself: it turns production traces into regression tasks and tests prompt/skill variants against the production agent.",
    "summary": "Zubin Aysola argues that benchmarks, evals and agent configurations are all covariant. So you need identical measurement in production and in an offline simulation environment, closing the 'sim-to-real gap' familiar from RL. W&B runs a byte-wise identical Arya agent in both places, logs both to Weave in the same format, and syncs production to research every 4 hours to prevent drift. Tasks are defined in YAML and the harness makes it cheap to spin up many parallel variants. In a live demo, Arya took a production trace, turned it into a regression task, found an SDK-call bug and fixed it with a small prompt/skill injection in a candidate variant, benchmarked that variant against the production agent, and wrote up a W&B Report.",
    "key_points": [
      "Arya, the W&B agent, went GA on Monday. It is built with mostly prompt engineering and skills, not reinforcement learning, because frontier models are already good at W&B tasks. The team still borrows RL's simulation-environment rigour.",
      "Production and research code are byte-wise identical. A 4-hour sync job copies production into the research environment so there is no drift while researchers cut new variants and skills.",
      "Weave logs production traces and offline traces in the exact same format, so production traces can be pulled straight into offline environments for hill-climbing or error fixing.",
      "The eval pipeline is an agnostic DAG: YAML config → hydrate (load live data) → set up the environment (expensive; may simulate GPU jobs, so parallelise it) → rehydrate (hot-patch runtime config that YAML can't express) → run the agent → score → tear down, so parallel runs don't clobber teammates' work.",
      "Scoring works two ways. Normative scoring asks whether the task passed. Relativistic scoring compares variants, e.g. one that asks the user clarifying questions vs one that doesn't. Aysola spends most of his time on scoring robustness and on the drift between evals and production.",
      "There are 886 tasks, categorised by level and exposed to the product team for review. Tasks are YAML start and end conditions, simulated either as a single text instruction or as a multi-turn conversation with an LLM playing a user persona.",
      "Nightly CI evaluates the production agent and candidate variants over time (7 weeks of traces shown, about 66% on some tasks). When CI broke, Arya fixed it itself. The agent is strong at conceptual guidance and weaker at error analysis.",
      "Live demo: Arya turned a production trace into a WBAF (Weights & Biases Agent Factory) regression task and found that weave.log was not being called properly in the sandbox. It added a hill-climb target, ran the prod and candidate variants (the candidate had a small prompt injection in the system prompt or a skill), and wrote a W&B Report."
    ],
    "takeaways": [
      "Run a byte-identical agent in production and offline, log both in the same trace format, and sync them automatically so offline evals don't drift from production.",
      "Turn every production failure, and every production success, into an eval task so you hill-climb in both directions.",
      "Make agent configurations YAML-defined and cheap to mutate, then run many parallel variants: 'it's just better to run more experiments than fewer.'",
      "Put serious time into scoring robustness and eval health, using both pass/fail scoring and relative comparisons between variants.",
      "Give the agent an unconstrained sandbox and let it review its own rollouts and run its own evals. Your job becomes guiding how the system improves, not writing the benchmarks by hand."
    ],
    "topics": [
      "agents",
      "evals",
      "observability",
      "self-improving agents",
      "agent harnesses",
      "simulation environments",
      "prompt engineering"
    ],
    "tools": [
      "Weights & Biases",
      "W&B Weave",
      "Arya (W&B agent)",
      "W&B Reports",
      "W&B Artifacts",
      "WBAF (Weights & Biases Agent Factory)",
      "CoreWeave",
      "Claude",
      "Claude Code",
      "NVIDIA H200",
      "nanochat (Karpathy)"
    ],
    "quotes": [
      {
        "text": "benchmarks, evaluations, the agents, and how you configure them are all covariant.",
        "at": "01:38",
        "url": "https://www.youtube.com/watch?v=XyV6bSMyq-I&t=98s"
      },
      {
        "text": "it's just better to run more experiments than fewer.",
        "at": "07:55",
        "url": "https://www.youtube.com/watch?v=XyV6bSMyq-I&t=475s"
      },
      {
        "text": "every production miss or every production goodness as well because I think it's useful to hill climb in a positive direction uh becomes a task for our agent framework.",
        "at": "12:21",
        "url": "https://www.youtube.com/watch?v=XyV6bSMyq-I&t=741s"
      },
      {
        "text": "I haven't written a line of code in maybe eight months because I just tell Claude to write all my code for me. And that's a really nice pattern, but it doesn't absolve yourself of the thought behind how we want to make improvements.",
        "at": "14:48",
        "url": "https://www.youtube.com/watch?v=XyV6bSMyq-I&t=888s"
      }
    ],
    "words": 4531
  },
  "summary_url": "/#XyV6bSMyq-I",
  "transcript": {
    "html": "/transcripts/XyV6bSMyq-I.html",
    "txt": "/transcripts/XyV6bSMyq-I.txt",
    "vtt": "/transcripts/XyV6bSMyq-I.vtt"
  }
}