WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=gDRCpTMCqEU NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/gDRCpTMCqEU.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.000 --> 00:00:09.080 My name is Sukaina, and I'm an AI tech lead, and today I'm here to talk about what a technical due diligence finds when the deck says 70% automated. s2 00:00:09.080 --> 00:00:11.680 Someone has probably shown you a number like this. s3 00:00:11.680 --> 00:00:14.200 55.8% faster. s4 00:00:14.200 --> 00:00:17.080 And you've been wondering whether to believe it or not. s5 00:00:17.080 --> 00:00:21.800 Well, this one comes from a controlled experiment with 95 professional developers. s6 00:00:21.800 --> 00:00:28.160 But the task was one JavaScript HTTP server built from a template against 12 visible tests. s7 00:00:28.160 --> 00:00:34.480 Today, I'll give you three checks for what technical due diligence finds in the AI era. s8 00:00:35.520 --> 00:00:39.440 I've actually spent more than 80 years on both sides of the question. s9 00:00:39.440 --> 00:00:43.800 On the front side, I built AI system that investment professional use. s10 00:00:43.800 --> 00:00:49.840 On a build side, I now lead AI evaluation in production and help create the numbers that reach executives. s11 00:00:49.840 --> 00:00:52.480 The dangerous number is not always fabricated. s12 00:00:52.480 --> 00:00:56.840 It's often a genuine measurement carried into a conclusion it cannot support. s13 00:00:56.840 --> 00:01:00.040 Chun Chan described the buyer side in a nice way. s14 00:01:00.040 --> 00:01:03.440 Built for the memo, not for the demo. s15 00:01:03.440 --> 00:01:06.440 I want to extend that standard to operational metrics. s16 00:01:06.440 --> 00:01:09.480 And the first check requires no knowledge of models. s17 00:01:09.480 --> 00:01:12.160 It's pure arithmetic. s18 00:01:13.400 --> 00:01:20.920 In Faros observational data, high AI adoption teams merged 98% more pull pull requests. s19 00:01:20.920 --> 00:01:28.520 Pull request review time increased by 91% and the average pull request became 154% larger. s20 00:01:28.520 --> 00:01:34.000 And the sample covered more than 10,000 developers across 1,255 teams. s21 00:01:34.000 --> 00:01:37.000 This does not prove that AI increased head count. s22 00:01:37.000 --> 00:01:40.800 It's an observational telemetry, not a controlled labor study. s23 00:01:40.800 --> 00:01:42.720 But it still give us a warning. s24 00:01:42.720 --> 00:01:49.160 More work entered the system and the downstream review burden grew with it as well. s25 00:01:49.160 --> 00:01:53.120 So in diligence, ask for the company's own 18-month theory. s26 00:01:53.120 --> 00:01:56.520 Put workload volume next to the fully loaded human labor. s27 00:01:56.520 --> 00:02:04.840 Count everyone whose work exists because of AI, for example, operators, reviewer reviewers, trainers, escalation and rework. s28 00:02:04.840 --> 00:02:11.200 If human labor tracks workload volume, the deck has described assistance, not labor removal. s29 00:02:11.360 --> 00:02:17.320 But when the arithmetic survives, the definition call still can fail. s30 00:02:18.959 --> 00:02:24.320 Uber, for example, says that 70% of its pull requests are AI attributed. s31 00:02:24.320 --> 00:02:25.920 This is impressive. s32 00:02:25.920 --> 00:02:28.840 It still does not tell us the full autonomous rate. s33 00:02:28.840 --> 00:02:32.240 The period and total count were not disclosed, either. s34 00:02:32.240 --> 00:02:35.680 So, every automation claim sits somewhere in this ladder. s35 00:02:35.680 --> 00:02:43.040 It's either AI drafted it, or edited by humans, or approved by humans, or no human intervention. s36 00:02:43.520 --> 00:02:46.000 Finance already has a useful term for the last one. s37 00:02:46.000 --> 00:02:49.000 It's called straight-through processing. s38 00:02:49.000 --> 00:02:54.320 Define it as a complete end-to-end without human intervention inside a boundary you name. s39 00:02:54.320 --> 00:03:01.519 So, does the boundary here end when the code was drafted, or when the pull request is approved, or when the change reaches production? s40 00:03:01.519 --> 00:03:06.840 So, remember, AI attributed and autonomous are two different claims. s41 00:03:06.840 --> 00:03:09.560 Okay, now suppose the definition is precise. s42 00:03:09.560 --> 00:03:12.280 Can we trust the measurement? s43 00:03:12.560 --> 00:03:20.920 Meta found 24% point gap between automated grader and merge decision after adjustment to the human baseline. s44 00:03:20.920 --> 00:03:29.160 Four maintainers from three repositories reviewed 296 AI generated pull requests across different unique issues. s45 00:03:29.160 --> 00:03:33.000 They did not know whether the patch came from AI or human. s46 00:03:33.000 --> 00:03:37.840 The automated grader was substantially more optimistic than the people responsible for accepting the work. s47 00:03:37.840 --> 00:03:39.280 Not a surprise. s48 00:03:39.280 --> 00:03:41.519 I have seen the same thing from the other side. s49 00:03:41.519 --> 00:03:46.200 One of the projects I worked on, 10 out of 10 output would pass the automated validation. s50 00:03:46.200 --> 00:03:51.120 Once you bring the the experts on the table, barely few survive. s51 00:03:51.120 --> 00:03:55.920 This is why evaluation maturity is first credibility signal I would look for. s52 00:03:55.920 --> 00:04:02.360 So, always ask who labeled ground truth, and whether the expert agrees, and whether anyone measures that agreement. s53 00:04:02.360 --> 00:04:08.600 Ask what the automated judge was validated against, and ask what bad output escaped. s54 00:04:08.959 --> 00:04:16.120 So, when arithmetic's definition and evaluation all fell, a word it still has to become a decision, right? s55 00:04:16.760 --> 00:04:21.560 This is not a claim that weak AI positioning caused builder.ai collapse. s56 00:04:21.560 --> 00:04:27.240 Its insolvency involved revenue financing issues beyond AI claims. s57 00:04:27.240 --> 00:04:36.160 The diligent lesson started way back in 2019, when the Wall Street Journal publicly questioned the claim of the AI capability. s58 00:04:36.160 --> 00:04:47.840 In May 2025, the The Financial Times also reported that its 2024 revenue estimated had been revised from $220 million to roughly $55 million. s59 00:04:47.840 --> 00:04:56.680 On June 2nd, the company filed Chapter 7. So, the warning existed, but the process did not turn it into a gate. s60 00:04:56.680 --> 00:05:02.520 So, even when a claim survives those check, it still has to survive the economics. s61 00:05:04.320 --> 00:05:07.160 So, start with eligible workflow value. s62 00:05:07.160 --> 00:05:13.120 Apply actual adoption, straight-through process success, and accepted quality. s63 00:05:13.120 --> 00:05:18.680 Then, subtract the cost of AI, human review, escalation, and rework. s64 00:05:18.680 --> 00:05:23.680 The metric I would want is cost for accepted autonomous task. s65 00:05:23.680 --> 00:05:28.800 If the task fails, escalates, or need rework, this belongs to the cost. s66 00:05:28.800 --> 00:05:35.400 So, always put those checks together, and this is the card I would probably take with me to the next meeting. s67 00:05:36.240 --> 00:05:38.440 To sum up, three main checks. s68 00:05:38.440 --> 00:05:42.520 The arithmetic's, compare workload volume with fully loaded human labor. s69 00:05:42.520 --> 00:05:48.720 Two, definition, identify the wrong and and the straight-through rate inside a name workflow. s70 00:05:48.720 --> 00:05:50.120 Three, the tell. s71 00:05:50.120 --> 00:05:52.360 Ask who defines quality. s72 00:05:52.360 --> 00:05:55.760 Whether expert agrees in what that output escapes. s73 00:05:55.760 --> 00:06:00.080 Then translate only the surviving claims into a unit economics. s74 00:06:00.080 --> 00:06:03.560 And of course, believe it tested or walk away. s75 00:06:03.560 --> 00:06:10.160 The printable card in all the source notes are in the website and this QR code will lead you to it. s76 00:06:10.160 --> 00:06:17.080 So, take a card, scan it, print it, and take it to your next vendor meeting or investment discussion. s77 00:06:17.080 --> 00:06:22.440 And always remember, the job of diligence is not to ask whether the number is impressive. s78 00:06:22.440 --> 00:06:25.760 It's to ask what must be true for the number to matter. s79 00:06:25.760 --> 00:06:27.960 Thank you.