{
  "video": {
    "id": "gDRCpTMCqEU",
    "title": "What Technical Due Diligence Finds When the Deck Says “70% Automated” — Soukaïna Abelhad",
    "duration": 389,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "gDRCpTMCqEU",
    "title": "What Technical Due Diligence Finds When the Deck Says “70% Automated” — Soukaïna Abelhad",
    "one_liner": "An automation number is often a real measurement stretched to a conclusion it can't support, so run three checks (arithmetic, definition, evaluation), then turn only the claims that survive into unit economics, ideally cost per accepted autonomous task.",
    "summary": "Speaker Soukaïna Abelhad, an AI tech lead who has built AI systems for investment professionals and now leads AI evaluation in production, argues that a dangerous number is not always fabricated. It is often a genuine measurement carried into a conclusion it cannot support. She gives three diligence checks. Arithmetic: does human labour fall as workload grows? Definition: AI-attributed is not autonomous, so ask for the straight-through rate inside a named boundary. Evaluation: do experts agree with the automated grader? Claims that survive all three are then translated into unit economics. She illustrates the checks with GitHub-style productivity stats, Faros telemetry, Uber's 70% AI-attributed PRs, a Meta grader-vs-maintainer study and the collapse of builder.ai.",
    "key_points": [
      "The often-cited '55.8% faster' figure comes from a controlled experiment with 95 professional developers, but the task was one JavaScript HTTP server built from a template against 12 visible tests.",
      "Faros observational data covered more than 10,000 developers across 1,255 teams. High-AI-adoption teams merged 98% more pull requests, PR review time rose 91% and the average PR became 154% larger. This is observational telemetry, not a controlled labour study, but it shows that downstream review burden grew along with the extra work.",
      "Arithmetic check: put workload volume next to fully loaded human labour, counting operators, reviewers, trainers, escalation and rework. If labour tracks volume, the deck describes assistance, not labour removal.",
      "Uber says 70% of its pull requests are AI-attributed, but that doesn't give the fully autonomous rate, and the period and total count were not disclosed.",
      "Automation ladder: AI drafted → edited by humans → approved by humans → no human intervention. The last rung is what finance calls straight-through processing, defined as complete end-to-end with no human intervention inside a boundary you name (code drafted, PR approved, or change reaches production).",
      "Meta found a 24-percentage-point gap between an automated grader and merge decisions, after adjusting to the human baseline. Four maintainers from three repositories reviewed 296 AI-generated PRs blind to whether a patch came from AI or a human, and the grader was substantially more optimistic.",
      "From her own project: 10 out of 10 outputs passed automated validation, but once experts reviewed them, barely a few survived.",
      "builder.ai: the Wall Street Journal publicly questioned its AI-capability claims in 2019. In May 2025 the Financial Times reported its 2024 revenue estimate revised from $220M to roughly $55M, and it filed Chapter 7 on June 2nd. Its insolvency involved revenue and financing issues beyond the AI claims. The lesson: the warning existed, but the process did not turn it into a gate."
    ],
    "takeaways": [
      "Ask for the company's own 18-month time series of workload volume against fully loaded human labour, including everyone whose work exists because of AI.",
      "Treat 'AI-attributed' and 'autonomous' as separate claims. Demand the straight-through rate inside a named workflow and boundary.",
      "Judge evaluation maturity first: who labelled ground truth, whether experts agree and whether anyone measures that agreement, what the automated judge was validated against, and what bad output escaped.",
      "Model economics as eligible workflow value × actual adoption × straight-through success × accepted quality, minus the cost of AI, human review, escalation and rework. Track cost per accepted autonomous task, and count failed, escalated or reworked tasks as cost.",
      "Turn warning signs into hard gates in the process: believe it once it's tested, or walk away."
    ],
    "topics": [
      "due diligence",
      "ai productivity metrics",
      "evals",
      "llm-as-judge",
      "automation claims",
      "unit economics",
      "code review"
    ],
    "tools": [
      "Faros",
      "Uber",
      "Meta",
      "builder.ai",
      "Wall Street Journal",
      "Financial Times",
      "JavaScript"
    ],
    "quotes": [
      {
        "text": "The dangerous number is not always fabricated. It's often a genuine measurement carried into a conclusion it cannot support.",
        "at": "00:49",
        "url": "https://www.youtube.com/watch?v=gDRCpTMCqEU&t=49s"
      },
      {
        "text": "If human labor tracks workload volume, the deck has described assistance, not labor removal.",
        "at": "02:04",
        "url": "https://www.youtube.com/watch?v=gDRCpTMCqEU&t=124s"
      },
      {
        "text": "So, remember, AI attributed and autonomous are two different claims.",
        "at": "03:01",
        "url": "https://www.youtube.com/watch?v=gDRCpTMCqEU&t=181s"
      },
      {
        "text": "the job of diligence is not to ask whether the number is impressive. It's to ask what must be true for the number to matter.",
        "at": "06:18",
        "url": "https://www.youtube.com/watch?v=gDRCpTMCqEU&t=378s"
      }
    ],
    "words": 1121
  },
  "summary_url": "/#gDRCpTMCqEU",
  "transcript": {
    "html": "/transcripts/gDRCpTMCqEU.html",
    "txt": "/transcripts/gDRCpTMCqEU.txt",
    "vtt": "/transcripts/gDRCpTMCqEU.vtt"
  }
}