What Technical Due Diligence Finds When the Deck Says “70% Automated” — Soukaïna Abelhad
https://www.youtube.com/watch?v=gDRCpTMCqEU
AI Engineer · 6 min · 79 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:00] My name is Sukaina, and I'm an AI tech lead, and today I'm here to talk about what a technical due diligence finds when the deck says 70% automated.
[00:09] Someone has probably shown you a number like this.
[00:11] 55.8% faster.
[00:14] And you've been wondering whether to believe it or not.
[00:17] Well, this one comes from a controlled experiment with 95 professional developers.
[00:21] But the task was one JavaScript HTTP server built from a template against 12 visible tests.
[00:28] Today, I'll give you three checks for what technical due diligence finds in the AI era.
[00:35] I've actually spent more than 80 years on both sides of the question.
[00:39] On the front side, I built AI system that investment professional use.
[00:43] On a build side, I now lead AI evaluation in production and help create the numbers that reach executives.
[00:49] The dangerous number is not always fabricated.
[00:52] It's often a genuine measurement carried into a conclusion it cannot support.
[00:56] Chun Chan described the buyer side in a nice way.
[01:00] Built for the memo, not for the demo.
[01:03] I want to extend that standard to operational metrics.
[01:06] And the first check requires no knowledge of models.
[01:09] It's pure arithmetic.
[01:13] In Faros observational data, high AI adoption teams merged 98% more pull pull requests.
[01:20] Pull request review time increased by 91% and the average pull request became 154% larger.
[01:28] And the sample covered more than 10,000 developers across 1,255 teams.
[01:34] This does not prove that AI increased head count.
[01:37] It's an observational telemetry, not a controlled labor study.
[01:40] But it still give us a warning.
[01:42] More work entered the system and the downstream review burden grew with it as well.
[01:49] So in diligence, ask for the company's own 18-month theory.
[01:53] Put workload volume next to the fully loaded human labor.
[01:56] Count everyone whose work exists because of AI, for example, operators, reviewer reviewers, trainers, escalation and rework.
[02:04] If human labor tracks workload volume, the deck has described assistance, not labor removal.
[02:11] But when the arithmetic survives, the definition call still can fail.
[02:18] Uber, for example, says that 70% of its pull requests are AI attributed.
[02:24] This is impressive.
[02:25] It still does not tell us the full autonomous rate.
[02:28] The period and total count were not disclosed, either.
[02:32] So, every automation claim sits somewhere in this ladder.
[02:35] It's either AI drafted it, or edited by humans, or approved by humans, or no human intervention.
[02:43] Finance already has a useful term for the last one.
[02:46] It's called straight-through processing.
[02:49] Define it as a complete end-to-end without human intervention inside a boundary you name.
[02:54] So, does the boundary here end when the code was drafted, or when the pull request is approved, or when the change reaches production?
[03:01] So, remember, AI attributed and autonomous are two different claims.
[03:06] Okay, now suppose the definition is precise.
[03:09] Can we trust the measurement?
[03:12] Meta found 24% point gap between automated grader and merge decision after adjustment to the human baseline.
[03:20] Four maintainers from three repositories reviewed 296 AI generated pull requests across different unique issues.
[03:29] They did not know whether the patch came from AI or human.
[03:33] The automated grader was substantially more optimistic than the people responsible for accepting the work.
[03:37] Not a surprise.
[03:39] I have seen the same thing from the other side.
[03:41] One of the projects I worked on, 10 out of 10 output would pass the automated validation.
[03:46] Once you bring the the experts on the table, barely few survive.
[03:51] This is why evaluation maturity is first credibility signal I would look for.
[03:55] So, always ask who labeled ground truth, and whether the expert agrees, and whether anyone measures that agreement.
[04:02] Ask what the automated judge was validated against, and ask what bad output escaped.
[04:08] So, when arithmetic's definition and evaluation all fell, a word it still has to become a decision, right?
[04:16] This is not a claim that weak AI positioning caused builder.ai collapse.
[04:21] Its insolvency involved revenue financing issues beyond AI claims.
[04:27] The diligent lesson started way back in 2019, when the Wall Street Journal publicly questioned the claim of the AI capability.
[04:36] In May 2025, the The Financial Times also reported that its 2024 revenue estimated had been revised from $220 million to roughly $55 million.
[04:47] On June 2nd, the company filed Chapter 7. So, the warning existed, but the process did not turn it into a gate.
[04:56] So, even when a claim survives those check, it still has to survive the economics.
[05:04] So, start with eligible workflow value.
[05:07] Apply actual adoption, straight-through process success, and accepted quality.
[05:13] Then, subtract the cost of AI, human review, escalation, and rework.
[05:18] The metric I would want is cost for accepted autonomous task.
[05:23] If the task fails, escalates, or need rework, this belongs to the cost.
[05:28] So, always put those checks together, and this is the card I would probably take with me to the next meeting.
[05:36] To sum up, three main checks.
[05:38] The arithmetic's, compare workload volume with fully loaded human labor.
[05:42] Two, definition, identify the wrong and and the straight-through rate inside a name workflow.
[05:48] Three, the tell.
[05:50] Ask who defines quality.
[05:52] Whether expert agrees in what that output escapes.
[05:55] Then translate only the surviving claims into a unit economics.
[06:00] And of course, believe it tested or walk away.
[06:03] The printable card in all the source notes are in the website and this QR code will lead you to it.
[06:10] So, take a card, scan it, print it, and take it to your next vendor meeting or investment discussion.
[06:17] And always remember, the job of diligence is not to ask whether the number is impressive.
[06:22] It's to ask what must be true for the number to matter.
[06:25] Thank you.