What Technical Due Diligence Finds When the Deck Says “70% Automated” — Soukaïna Abelhad

AI Engineer · 6 min · 79 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:00My name is Sukaina, and I'm an AI tech lead, and today I'm here to talk about what a technical due diligence finds when the deck says 70% automated.
2. 00:09Someone has probably shown you a number like this.
3. 00:1155.8% faster.
4. 00:14And you've been wondering whether to believe it or not.
5. 00:17Well, this one comes from a controlled experiment with 95 professional developers.
6. 00:21But the task was one JavaScript HTTP server built from a template against 12 visible tests.
7. 00:28Today, I'll give you three checks for what technical due diligence finds in the AI era.
8. 00:35I've actually spent more than 80 years on both sides of the question.
9. 00:39On the front side, I built AI system that investment professional use.
10. 00:43On a build side, I now lead AI evaluation in production and help create the numbers that reach executives.
11. 00:49The dangerous number is not always fabricated.
12. 00:52It's often a genuine measurement carried into a conclusion it cannot support.
13. 00:56Chun Chan described the buyer side in a nice way.
14. 01:00Built for the memo, not for the demo.
15. 01:03I want to extend that standard to operational metrics.
16. 01:06And the first check requires no knowledge of models.
17. 01:09It's pure arithmetic.
18. 01:13In Faros observational data, high AI adoption teams merged 98% more pull pull requests.
19. 01:20Pull request review time increased by 91% and the average pull request became 154% larger.
20. 01:28And the sample covered more than 10,000 developers across 1,255 teams.
21. 01:34This does not prove that AI increased head count.
22. 01:37It's an observational telemetry, not a controlled labor study.
23. 01:40But it still give us a warning.
24. 01:42More work entered the system and the downstream review burden grew with it as well.
25. 01:49So in diligence, ask for the company's own 18-month theory.
26. 01:53Put workload volume next to the fully loaded human labor.
27. 01:56Count everyone whose work exists because of AI, for example, operators, reviewer reviewers, trainers, escalation and rework.
28. 02:04If human labor tracks workload volume, the deck has described assistance, not labor removal.
29. 02:11But when the arithmetic survives, the definition call still can fail.
30. 02:18Uber, for example, says that 70% of its pull requests are AI attributed.
31. 02:24This is impressive.
32. 02:25It still does not tell us the full autonomous rate.
33. 02:28The period and total count were not disclosed, either.
34. 02:32So, every automation claim sits somewhere in this ladder.
35. 02:35It's either AI drafted it, or edited by humans, or approved by humans, or no human intervention.
36. 02:43Finance already has a useful term for the last one.
37. 02:46It's called straight-through processing.
38. 02:49Define it as a complete end-to-end without human intervention inside a boundary you name.
39. 02:54So, does the boundary here end when the code was drafted, or when the pull request is approved, or when the change reaches production?
40. 03:01So, remember, AI attributed and autonomous are two different claims.
41. 03:06Okay, now suppose the definition is precise.
42. 03:09Can we trust the measurement?
43. 03:12Meta found 24% point gap between automated grader and merge decision after adjustment to the human baseline.
44. 03:20Four maintainers from three repositories reviewed 296 AI generated pull requests across different unique issues.
45. 03:29They did not know whether the patch came from AI or human.
46. 03:33The automated grader was substantially more optimistic than the people responsible for accepting the work.
47. 03:37Not a surprise.
48. 03:39I have seen the same thing from the other side.
49. 03:41One of the projects I worked on, 10 out of 10 output would pass the automated validation.
50. 03:46Once you bring the the experts on the table, barely few survive.
51. 03:51This is why evaluation maturity is first credibility signal I would look for.
52. 03:55So, always ask who labeled ground truth, and whether the expert agrees, and whether anyone measures that agreement.
53. 04:02Ask what the automated judge was validated against, and ask what bad output escaped.
54. 04:08So, when arithmetic's definition and evaluation all fell, a word it still has to become a decision, right?
55. 04:16This is not a claim that weak AI positioning caused builder.ai collapse.
56. 04:21Its insolvency involved revenue financing issues beyond AI claims.
57. 04:27The diligent lesson started way back in 2019, when the Wall Street Journal publicly questioned the claim of the AI capability.
58. 04:36In May 2025, the The Financial Times also reported that its 2024 revenue estimated had been revised from $220 million to roughly $55 million.
59. 04:47On June 2nd, the company filed Chapter 7. So, the warning existed, but the process did not turn it into a gate.
60. 04:56So, even when a claim survives those check, it still has to survive the economics.
61. 05:04So, start with eligible workflow value.
62. 05:07Apply actual adoption, straight-through process success, and accepted quality.
63. 05:13Then, subtract the cost of AI, human review, escalation, and rework.
64. 05:18The metric I would want is cost for accepted autonomous task.
65. 05:23If the task fails, escalates, or need rework, this belongs to the cost.
66. 05:28So, always put those checks together, and this is the card I would probably take with me to the next meeting.
67. 05:36To sum up, three main checks.
68. 05:38The arithmetic's, compare workload volume with fully loaded human labor.
69. 05:42Two, definition, identify the wrong and and the straight-through rate inside a name workflow.
70. 05:48Three, the tell.
71. 05:50Ask who defines quality.
72. 05:52Whether expert agrees in what that output escapes.
73. 05:55Then translate only the surviving claims into a unit economics.
74. 06:00And of course, believe it tested or walk away.
75. 06:03The printable card in all the source notes are in the website and this QR code will lead you to it.
76. 06:10So, take a card, scan it, print it, and take it to your next vendor meeting or investment discussion.
77. 06:17And always remember, the job of diligence is not to ask whether the number is impressive.
78. 06:22It's to ask what must be true for the number to matter.
79. 06:25Thank you.