{
  "video": {
    "id": "aHhB3sjGjkI",
    "title": "Agents Building Agents - Alfonso Graziano, Nearform",
    "duration": 1814,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "Online"
  },
  "analysis": {
    "video_id": "aHhB3sjGjkI",
    "title": "Agents Building Agents - Alfonso Graziano, Nearform",
    "one_liner": "Nearform's Alfonso Graziano shows \"AutoAgent\" — a Claude Code loop that branches, hypothesises, edits an agent's prompt/tools and re-runs evals — taking a toy agent from 18% to 83% and a real production agent from 67% to 86% in ~10 iterations.",
    "summary": "The talk treats an AI agent as just software (an LLM in a loop with tools and context) and therefore something a coding agent can build and optimise. It attacks two failure classes: bad performance on evals, fixed by AutoAgent — a Karpathy-\"auto research\"-inspired optimisation loop that generates hypotheses, branches per iteration, runs the eval suite against a golden dataset and scorers, keeps improvements and rolls back regressions; and bad performance on live data, fixed by a pipeline that collects traces plus thumbs-up/down feedback and SME annotations, clusters failure modes with a purpose-built coding-agent skill, triages with subject matter experts, then has the coding agent implement fixes that become new golden-dataset entries. Both rest on \"Harness Engineering\": building the environment — specs, quality gates, context engineering, observability — so the coding agent can validate its own changes.",
    "key_points": [
      "An agent, from first principles, is \"an LLM inside an agentic loop, which is connected to tools and can retrieve context\" — one type of software, so AI can build it.",
      "A golden dataset is built with subject matter experts and acts as a test suite for a non-deterministic system; expected outputs go beyond text/numbers to \"call this tool\", \"call this tool with this parameter\", or a chain of tool calls. Paired scorers turn it into an accuracy number for baselining and regression detection.",
      "Demo baseline: a Mastra \"math agent\" with no tools and a naive contains-the-ground-truth evaluator passes 18% — the questions the LLM's weights already answer (additions, multiplications); the rest need tools or context.",
      "AutoAgent, inspired by Karpathy's \"auto research\" loop for ML hyperparameters/code, applies the same idea to agents: Claude Code writes the target agent, reads its evals and thinking traces, and iterates. It lifted the naive agent from 18% to 83% in ~10 iterations.",
      "On a real production agent that was already humanly optimised, AutoAgent found improvements humans hadn't: 67% → 86% in ~10 iterations, plus 10% on internal benchmarks — by finding edge cases, improving the system prompt and tool descriptions, and fixing tool logic, \"without actually cheating\".",
      "Loop mechanics: a markdown optimisation job (objective, target repo, metrics); a baseline report; then per iteration — new git branch, one hypothesis tackling one failure class, code change, eval run, reports.md, update a global memory file across runs; continue from the branch on improvement, roll back on regression. The result is a full changelog of hypotheses, so humans can read a promising-but-failed hypothesis and steer the next attempt.",
      "Humans must explicitly forbid the coding agent from editing the golden dataset or scorers just to make evals pass.",
      "Live-data pipeline: collect traces (tool usage, latency, tokens) + thumbs up/down with comment, or SME trace annotations; export ~114 traces to JSON; a custom skill clusters failure modes, runs an adversarial review and root-cause analysis against the agent's actual code, and emits a markdown report with clusters, trace IDs, user feedback, root causes and fixes (e.g. \"URLs are not hyperlinked\"). Reports once per sprint was found reasonable.",
      "Clusters must be triaged and validated with SMEs — some are false positives or intended behaviour — then fixed by the coding agent, and every confirmed failure mode is added to the golden dataset so it becomes a permanent regression test."
    ],
    "takeaways": [
      "Build the golden dataset with subject matter experts before optimising, and express expected outputs as tool calls and tool-call chains, not just final text — then write scorers so you have one accuracy number to move.",
      "Wrap your coding agent in an optimisation loop rather than hand-tuning prompts: one hypothesis per iteration, a git branch per iteration, evals as the gate, roll back regressions, and keep a cross-run memory file plus per-hypothesis reports so failed-but-promising ideas can be steered by a human.",
      "Explicitly instruct the coding agent not to touch the golden dataset or scorers — otherwise it will optimise the test instead of the system.",
      "Instrument production: collect traces with thumbs up/down and comments, let SMEs annotate traces when real feedback is thin, then cluster failure modes with a coding-agent skill that also has access to the agent's source for root-cause analysis. Run it roughly once per sprint, scaled to trace volume.",
      "Triage clusters with SMEs before fixing, and promote every validated failure mode into the golden dataset so the fix is protected by a regression test.",
      "Invest in the harness — specs per failure mode, linting, unit tests, evals, LLM code review, context engineering and observability — because a coding agent that can validate its own changes fixed entire issue suites from a single prompt."
    ],
    "topics": [
      "agents",
      "evals",
      "coding-agents",
      "observability",
      "spec-driven-development",
      "prompt-optimization",
      "traces",
      "harness-engineering"
    ],
    "tools": [
      "Mastra",
      "Claude Code",
      "AutoAgent",
      "Nearform",
      "O'Reilly",
      "LinkedIn",
      "Python"
    ],
    "quotes": [
      {
        "text": "And given that AI agents is just one type of software, as you may guess, we are using AI to build AI.",
        "at": "01:17",
        "url": "https://www.youtube.com/watch?v=aHhB3sjGjkI&t=77s"
      },
      {
        "text": "We also improved some evals by 10% on a production agent that was already humanly optimized. So, actually the agents, the coding agent found new ways that humans didn't find to improve the agent.",
        "at": "10:59",
        "url": "https://www.youtube.com/watch?v=aHhB3sjGjkI&t=659s"
      },
      {
        "text": "Updating the golden data sets or the scorers just to let the evals pass is not a good idea, so we want to enforce, we want to tell the AI agent to not do that.",
        "at": "12:43",
        "url": "https://www.youtube.com/watch?v=aHhB3sjGjkI&t=763s"
      },
      {
        "text": "Harness Engineering is the idea of building the environment around our coding agent so that they can work reliably.",
        "at": "28:30",
        "url": "https://www.youtube.com/watch?v=aHhB3sjGjkI&t=1710s"
      }
    ],
    "words": 5486
  },
  "summary_url": "/#aHhB3sjGjkI",
  "transcript": {
    "html": "/transcripts/aHhB3sjGjkI.html",
    "txt": "/transcripts/aHhB3sjGjkI.txt",
    "vtt": "/transcripts/aHhB3sjGjkI.vtt"
  }
}