{
  "video": {
    "id": "Ve2bAarPA2I",
    "title": "Better AI Models Won't Fix Your Engineering Team — Nicholas Arcolano, Jellyfish",
    "duration": 999,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "Ve2bAarPA2I",
    "title": "Better AI Models Won't Fix Your Engineering Team — Nicholas Arcolano, Jellyfish",
    "one_liner": "Across roughly 50,000 engineers at 447 companies, Jellyfish found the tool and model explain less than 1% of the change in merged-PR throughput, while the individual engineer explains about three quarters and the company and team about 12%: if AI isn't getting you the outcomes you want, \"maybe it's not the model, you know, maybe it's you.\"",
    "summary": "Nicholas Arcolano, head of AI and research at Jellyfish, uses a golf analogy (clubs vs. fundamentals vs. the course) to break down what drives the change in merged PRs per engineer per week. He uses Jellyfish data, which joins AI coding tool and agent telemetry with engineering workflow data across hundreds of thousands of developers at about 1,300 companies. His statistical model splits it this way: tool (harness plus model) under 1%, how well you use it (skills) about 3%, how much you use it (tokens) about 9%, where you work about 12%, and the individual about three quarters. His argument is that leaders can't control the models, but they can control people and organisation. In his words: \"unlike golf, you can change the course.\"",
    "key_points": [
      "Natural experiment: around the time SpaceX announced its acquisition of Cursor, Cursor promoted Grok in its UI and changed how auto mode worked. Jellyfish saw the fastest model change it has ever recorded, but there was no statistical correlation between an engineer's shift to Grok (0–100% of their model share) and their PR throughput relative to colleagues at the same company.",
      "Engineers compared with themselves showed no meaningful throughput change when going all in on Opus vs Sonnet, switching to ChatGPT, moving from Cursor to Claude Code, or moving from autocomplete to agentic mode. Arcolano called it \"statistically, it's kind of a nothingburger.\"",
      "The curated dataset was about 50,000 engineers at 447 companies, using this year's data from Cursor and Claude Code users. The metric was merged PRs per engineer per week, with no quality metric in this analysis.",
      "Skills: as of September, almost half of engineer-weeks had essentially no skill use, and only 12% used skills at a level he considers modern AI-engineering practice. At 30+ skill runs per engineer per week there is a 30% throughput bump with other factors held constant, which moves output far more than model choice.",
      "Consumption: across about 500,000 developer-weeks, more tokens meant more throughput, with diminishing returns along a saturating curve he likened to the rocket equation. Going from the median to the top 10% roughly doubles output but costs about 10x the tokens.",
      "Where you work: about a third of this factor is the team and two thirds is the company (codebase, review process, culture). Holding consumption fixed, there is a 2x spread between the top-10% and bottom-10% companies.",
      "The individual: the spread between bottom-10% and top-10% engineers is 3.5x, versus 2x for companies.",
      "AI isn't reshuffling who is best. Across about 10,000 engineers, roughly 44% cycle out of the top quartile every half year before AI and 41% after, so the best engineers before AI are largely still the best."
    ],
    "takeaways": [
      "Stop expecting model or harness switches to fix throughput. Leaders can't control the tools, and switching them showed almost no effect.",
      "Push real skill adoption (skills, cursor rules, harness customisation). Most engineer-weeks show little or none, and heavy use (30+ runs a week) is linked to a 30% throughput gain.",
      "Treat token spend as fuel with diminishing returns: more tokens means more output, but doubling output from the median costs about 10x the tokens.",
      "Invest in the environment leaders control, meaning codebase, review process, team and culture. At fixed AI spend, top-10% companies get 2x the output of bottom-10% companies.",
      "Keep valuing individual engineering craft. AI isn't levelling the playing field, and the individual is the largest factor."
    ],
    "topics": [
      "ai coding tools",
      "developer productivity",
      "engineering leadership",
      "skills",
      "token spend",
      "engineering culture",
      "natural experiments"
    ],
    "tools": [
      "Jellyfish",
      "Cursor",
      "Claude Code",
      "Grok",
      "Opus",
      "Sonnet",
      "ChatGPT",
      "Artificial Analysis",
      "SpaceX",
      "Substack"
    ],
    "quotes": [
      {
        "text": "chances are it's actually not the model, the the problem is you.",
        "at": "00:26",
        "url": "https://www.youtube.com/watch?v=Ve2bAarPA2I&t=26s"
      },
      {
        "text": "skill use when it's prevalent moves output way more than model choice.",
        "at": "09:23",
        "url": "https://www.youtube.com/watch?v=Ve2bAarPA2I&t=563s"
      },
      {
        "text": "Your your best engineers before AI by and large are still your best engineers.",
        "at": "14:45",
        "url": "https://www.youtube.com/watch?v=Ve2bAarPA2I&t=885s"
      },
      {
        "text": "unlike golf, you can change the course.",
        "at": "15:47",
        "url": "https://www.youtube.com/watch?v=Ve2bAarPA2I&t=947s"
      }
    ],
    "words": 3371
  },
  "summary_url": "/#Ve2bAarPA2I",
  "transcript": {
    "html": "/transcripts/Ve2bAarPA2I.html",
    "txt": "/transcripts/Ve2bAarPA2I.txt",
    "vtt": "/transcripts/Ve2bAarPA2I.vtt"
  }
}