{
  "video": {
    "id": "2e9ANoOEn28",
    "title": "What if the harness mattered more than the model? - Aditya Bhargava, Etsy",
    "duration": 1924,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "Online"
  },
  "analysis": {
    "video_id": "2e9ANoOEn28",
    "title": "What if the harness mattered more than the model? - Aditya Bhargava, Etsy",
    "one_liner": "An Etsy staff engineer argues that the harness — not the model — is where the leverage is, and walks a coding agent up a seven-step harness ladder (tools → handlers → partial function application → ReAct loop → sub-agents → self-optimization) in Agency, a new language he built for writing agents.",
    "summary": "The speaker pushes back on the industry wisdom that \"models are so good you can keep the harness simple,\" arguing that it makes us dependent on a handful of companies' proprietary models that can't be run locally; the better bet is a harness good enough to get cutting-edge performance out of a local open-source model. He cites a HarnessBench paper — 106 tasks, same model and same evaluation, only the harness varying — where scores range from 52.4% to 76.2%, and where the harness matters more for weaker models. His second controversial claim is that a good harness needs language-level support, which is why he spent ~6 months building Agency (agencylang.com), a TypeScript-flavoured language for agents. The rest is a live demo: the same bug (a median function that doesn't average the middle two elements on even-length lists), the same model, seven successive harnesses.",
    "key_points": [
      "HarnessBench: 106 tasks, same model, same evaluation, only the harness changes — scores span 52.4% to 76.2%, a more-than-20-point swing, and the weaker the model the more the harness matters.",
      "If the harness matters more than the model, that's good news: anyone can build a harness, so you're not dependent on the handful of companies that can train frontier models.",
      "Terminology used throughout: an agent = harness + model. Claude Code is the agent, Opus is the model, everything else is the harness.",
      "In Agency every function is automatically usable as a tool — it generates the JSON schema from the signature and uses the docstring as the tool description; no registration step.",
      "Every standard-library function that mutates state, does something destructive, or reads sensitive data raises an *interrupt* first; unhandled, the program crashes with a pointer to the handler guide, so unsafe file access is off by default.",
      "Partial function application (borrowed from FP) locks arguments the LLM can't see or change — `read.partial(directory='demo')` means the model only sees a `filename` parameter and can only touch that directory: safety without human-in-the-loop latency.",
      "The ReAct loop (reason → act → observe → reason) is what finally gets the agent to *fix* the code: read the files, run the tests, read the failure, edit, re-run, confirm green. Before it, the agent produced a correct patch but only offered to apply it.",
      "Sub-agents in Agency are just functions called as tools — a main agent gets a coding sub-agent and a Wikipedia sub-agent (each with their own LLM call and tool set) and runs them in parallel, which adds capability without bloating context or confusing tool selection.",
      "Built-in optimizers: mark prompt variables with an `optimize` modifier, give a goal (\"fix the bug in median.py using TDD\"), and the optimizer establishes a baseline objective (0.2 in the demo) and rewrites the prompts — in his recorded run it hit the objective on the first iteration.",
      "Agency supports true pause-and-resume: an interrupt inside a for loop inside a tool call inside a sub-agent pauses at exactly that point, serializes, returns control to the user, and resumes a week later with no code changes."
    ],
    "takeaways": [
      "Treat harness quality as an engineering discipline with its own ladder: tools → make it safe → make it autonomous → make it reason → give it sub-agents → have it self-optimize. Climb it deliberately rather than dumping tools on a model.",
      "Prefer constraining capability by construction over asking the human: lock dangerous parameters (directory, scope, credentials) with partial application so the model never sees them, instead of approving every call.",
      "Don't stop at an agent that proposes a correct answer — give it the ability to run the tests and loop until they pass, so it verifies its own change.",
      "When an agent has many unrelated tools, group them behind sub-agents so the top-level agent only picks a sub-agent; confused tool choice is usually too many unrelated concepts in one context.",
      "Replace guess-and-check prompt tweaking with an optimizer against a measurable goal, so harness improvements are systematically measured rather than tried."
    ],
    "topics": [
      "agents",
      "harness",
      "agent-safety",
      "local-models",
      "tool-use",
      "react",
      "sub-agents",
      "prompt-optimization"
    ],
    "tools": [
      "Agency (agencylang.com)",
      "HarnessBench",
      "Claude Code",
      "Opus",
      "Etsy",
      "TypeScript",
      "Python",
      "npx",
      "Wikipedia search (Agency stdlib)",
      "Grokking Algorithms",
      "ducktyped.org",
      "Bluesky"
    ],
    "quotes": [
      {
        "text": "Yes, but that's moving in the wrong direction because that is making us reliant on fancy proprietary models that can't be run locally.",
        "at": "01:07",
        "url": "https://www.youtube.com/watch?v=2e9ANoOEn28&t=67s"
      },
      {
        "text": "So, more than a 20-point difference, and only the harness changed.",
        "at": "02:31",
        "url": "https://www.youtube.com/watch?v=2e9ANoOEn28&t=151s"
      },
      {
        "text": "I think building a good harness actually requires language-level support.",
        "at": "03:46",
        "url": "https://www.youtube.com/watch?v=2e9ANoOEn28&t=226s"
      },
      {
        "text": "We're no longer doing trial and error. We're no longer just trying stuff to see what works. Instead, now we have a way to systematically measure and improve performance.",
        "at": "26:20",
        "url": "https://www.youtube.com/watch?v=2e9ANoOEn28&t=1580s"
      }
    ],
    "words": 5193
  },
  "summary_url": "/#2e9ANoOEn28",
  "transcript": {
    "html": "/transcripts/2e9ANoOEn28.html",
    "txt": "/transcripts/2e9ANoOEn28.txt",
    "vtt": "/transcripts/2e9ANoOEn28.vtt"
  }
}