WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=-a2oTS9J67U NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/-a2oTS9J67U.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:02.560 --> 00:00:10.280 Go-live proves that a system exists, but it doesn't prove the people behind the work will actually use it. s2 00:00:10.280 --> 00:00:15.040 Or that it will ever deliver a single dollar from the deal model. s3 00:00:15.040 --> 00:00:26.800 I am Michelle Grant, creator of Leading at Scale, and the question every enterprise is confronting right now is the agent works, does anyone trust it? s4 00:00:26.800 --> 00:00:39.720 I have spent over 20 years in enterprise transformation and more than a decade leading M&A integrations across media, tech, and consumer goods. s5 00:00:39.720 --> 00:00:46.720 On those deals, trust was not something we crossed our fingers and hoped for after launch. s6 00:00:46.720 --> 00:00:55.040 We engineered it before day one through early change management and risk management. s7 00:00:55.040 --> 00:01:02.880 So, when the platform went live, the teams on the ground actually relied on it. s8 00:01:02.880 --> 00:01:07.520 AI agents demand the exact same muscle. s9 00:01:07.520 --> 00:01:23.200 An agent earns trust when it delivers the value the business budgeted for while staying strictly within the company's approved risk appetite, policies, and regulatory guardrails. s10 00:01:23.440 --> 00:01:30.200 In my experience, leadership gets there by answering three straightforward questions. s11 00:01:30.200 --> 00:01:32.640 What value does it create? s12 00:01:32.640 --> 00:01:35.080 Who is on the hook for it? s13 00:01:35.080 --> 00:01:40.920 And what evidence gives it permission to run with greater autonomy? s14 00:01:41.440 --> 00:01:49.960 The moment an agent steps out of a sandbox and into a core business workflow, the entire test changes. s15 00:01:49.960 --> 00:02:02.280 A pilot tests whether the software functions, but daily operations is where we test whether it creates value within the controls set by leadership. s16 00:02:02.280 --> 00:02:07.040 Established AI risk frameworks give you strong principles. s17 00:02:07.040 --> 00:02:12.600 What they do not give you is how to apply these principles to each agent. s18 00:02:12.600 --> 00:02:17.160 For every agent, leadership has to decide four things. s19 00:02:17.160 --> 00:02:19.320 What is it permitted to do? s20 00:02:19.320 --> 00:02:21.040 Who owns it? s21 00:02:21.040 --> 00:02:27.320 What evidence it records and what triggers an escalation. s22 00:02:27.400 --> 00:02:30.680 That work is not an engineering task. s23 00:02:30.680 --> 00:02:34.640 We find it is an executive operating mandate. s24 00:02:34.640 --> 00:02:38.840 And today, I'll show you how to meet it. s25 00:02:38.840 --> 00:02:43.400 Let us ground this in a real-world scenario. s26 00:02:43.840 --> 00:02:51.160 Picture a $15 billion acquisition targeting 400 million in net synergies. s27 00:02:51.160 --> 00:02:56.760 The deal has already closed and the integration efforts are now underway. s28 00:02:56.760 --> 00:03:08.600 The integration management office, which we'll also refer to as the IMO today, is driving the synergy roadmap across 40 workstreams. s29 00:03:08.680 --> 00:03:15.160 Covering everything from rationalizing vendor contracts to consolidating ERP systems. s30 00:03:15.160 --> 00:03:22.680 And every single initiative has a business owner accountable for its bottom-line savings. s31 00:03:22.959 --> 00:03:32.239 Every week, analysts in the IMO are going to reconcile the numbers against the general ledger. s32 00:03:32.239 --> 00:03:45.920 As they track down status from all 40 leads, they're also comparing actuals against the plan, then finally assembling the executive deck for the steering committee. s33 00:03:45.920 --> 00:03:55.440 Which is going to include the CFO, the chief risk officer, business unit heads, and the integration lead. s34 00:03:55.440 --> 00:04:01.200 The engineering team builds an agent automate that first draft. s35 00:04:01.200 --> 00:04:13.600 The agent then ingest the weekly updates, reconciles the ledger, runs variance math against the plan, and flags the initiatives that are falling behind. s36 00:04:13.680 --> 00:04:18.040 Everything looks good, and they decide to run a four-week pilot. s37 00:04:18.040 --> 00:04:23.600 The agent's drafts match the analyst reports line for line. s38 00:04:23.600 --> 00:04:26.480 So, we can say that the tech works. s39 00:04:26.480 --> 00:04:37.000 Now, the question is, what value does that steering committee need before they can trust it to run the business? s40 00:04:37.000 --> 00:04:47.360 First, the steering committee need a written value case agreed upon before the agent ever touches production data. s41 00:04:47.360 --> 00:04:52.200 In our example, that comes down to three specific metrics. s42 00:04:52.200 --> 00:04:55.919 First, let's look at cycle time. s43 00:04:55.919 --> 00:05:05.440 If that weekly report is five business days to compile right now, you're going to want to set a far target such as two days. s44 00:05:05.440 --> 00:05:09.520 Your second metric will be analyst capacity. s45 00:05:09.520 --> 00:05:19.840 If four analysts spend half of their week building this deck, that is 80 hours a week the agent gives back. s46 00:05:19.840 --> 00:05:33.120 That time counts as value only when the integration lead assigns it to the workstreams that are behind on their synergy targets and need help getting back on track. s47 00:05:33.120 --> 00:05:44.680 Then, you only count that value after netting out the compute, licensing, and human review cost of running the agent. s48 00:05:44.680 --> 00:05:50.040 Your third metric is the most important one, detection time. s49 00:05:50.040 --> 00:05:57.160 That is how long it takes from the first sign of a slip in the data to the moment the steering committee sees it. s50 00:05:57.160 --> 00:06:05.760 Cutting the report from five days to two days does not guarantee that the committee sees that slip any sooner. s51 00:06:05.760 --> 00:06:12.560 The agent class to catch it and the value case measures that directly. s52 00:06:12.640 --> 00:06:17.160 Every single metric requires a pre-deployment baseline. s53 00:06:17.160 --> 00:06:22.440 If the IMO cannot state the baseline today, do not deploy. s54 00:06:22.440 --> 00:06:29.240 If you can't measure where you started, you can never prove the agent moved the needle. s55 00:06:29.240 --> 00:06:36.800 Next, the steering committee needs to know who owns the risk the agent creates. s56 00:06:36.800 --> 00:06:41.520 We are going to apply the three lines model to the agent. s57 00:06:41.520 --> 00:06:46.040 In the first line, management owns the controls. s58 00:06:46.040 --> 00:06:49.880 The integration lead is the accountable executive. s59 00:06:49.880 --> 00:06:56.360 She owns the agent's output exactly the way she owned the manual data. s60 00:06:56.360 --> 00:07:03.760 In the second line, risk and compliance oversee the agent and challenge how to run it. s61 00:07:03.760 --> 00:07:07.840 How deep they go depends on the agent's exposure. s62 00:07:07.840 --> 00:07:20.040 An internal draft scrubbed by a senior analyst warrants a very different review standard than an autonomous agent publishing figures straight to the board. s63 00:07:20.360 --> 00:07:32.880 In the third line, internal audit provides assurance deciding based on material risk when and how the agent enters the annual audit plan. s64 00:07:32.880 --> 00:07:39.640 The agent also needs one employee assigned to oversee it every day. s65 00:07:39.640 --> 00:07:43.000 In our example, that is the senior analyst. s66 00:07:43.000 --> 00:07:47.400 He reviews the draft before it reaches the committee. s67 00:07:47.400 --> 00:07:51.160 Her role requires four specific capabilities. s68 00:07:51.160 --> 00:07:55.520 She has to first understand the operational boundaries. s69 00:07:55.520 --> 00:07:59.920 She has to interpret its output with context. s70 00:07:59.920 --> 00:08:05.520 She must have the authority to override or reverse it. s71 00:08:05.520 --> 00:08:13.000 And she needs clear kill switch to shut it down safely this ticket it behaves unexpectedly. s72 00:08:14.200 --> 00:08:19.880 Accountability also means defining where the agent's authority stops. s73 00:08:19.880 --> 00:08:26.240 I document this in a one-page policy called the human work standard. s74 00:08:26.240 --> 00:08:34.120 For every fork in the process, it maps the decision, the human role authorized to make it. s75 00:08:34.120 --> 00:08:38.599 The required evidence and the escalation path. s76 00:08:38.599 --> 00:08:42.080 These decisions exist at every tier. s77 00:08:42.080 --> 00:08:50.320 The senior analyst decides whether a variance is an accounting timing difference or a real cash shortfall. s78 00:08:50.440 --> 00:08:55.080 The work stream lead decides how to remediate that shortfall. s79 00:08:55.080 --> 00:09:04.280 And the integration lead is going to decide whether to formally flag an initiative as at risk to the steering committee. s80 00:09:04.280 --> 00:09:12.600 The agent prepares the analysis behind every one of those decisions and it stops there. s81 00:09:12.600 --> 00:09:17.400 The person who makes the decision owns the consequence. s82 00:09:17.600 --> 00:09:22.920 So keeping these decisions with your team does not slow the agent down. s83 00:09:22.920 --> 00:09:28.200 It can go into use while each role keeps its authority. s84 00:09:28.200 --> 00:09:35.880 The one reason to delight delay deployment is a decision with no assigned owner. s85 00:09:36.200 --> 00:09:47.640 This means if leadership has not agreed who declares an initiative off track, the agent's flights lead to no action. s86 00:09:47.640 --> 00:09:54.120 The third question is where the engineers in this room come in. s87 00:09:54.120 --> 00:10:00.760 The evidence for greater autonomy comes from how the agent is built. s88 00:10:00.760 --> 00:10:03.480 I look for three artifacts. s89 00:10:03.480 --> 00:10:07.240 The first is an auditable decision log. s90 00:10:07.240 --> 00:10:19.120 For every workstream the agent evaluates, it records the raw data ingested, the tool calls executed, the prompt, and the mono versions. s91 00:10:19.120 --> 00:10:23.920 Output generated along with the human sign-off. s92 00:10:23.920 --> 00:10:29.160 Two engineering choices make that log airtight. s93 00:10:29.160 --> 00:10:43.480 First, your core math and reconciliation rules must run in deterministic, finance-approved code, not generated as conversational prose from an LLM. s94 00:10:43.480 --> 00:10:50.120 Second, treat the model's written summary as a helpful explanation, not proof. s95 00:10:50.120 --> 00:10:53.880 Your real proof lives in the telemetry. s96 00:10:53.880 --> 00:10:55.920 What records were pulled? s97 00:10:55.920 --> 00:10:58.520 What deterministic tools ran? s98 00:10:58.520 --> 00:11:01.920 And who signed the review? s99 00:11:02.800 --> 00:11:10.040 If reported savings do not match the general ledger, agent does not touch the number. s100 00:11:10.040 --> 00:11:14.080 It flags the discrepancy and roots it up. s101 00:11:14.080 --> 00:11:18.680 The second artifact is the model inventory record. s102 00:11:18.680 --> 00:11:35.280 This captures the accountable executive, data lineage, dependencies across source systems, validation results, escalation path, and the kill switch run book. s103 00:11:35.280 --> 00:11:50.360 Most importantly, it documents your regression triggers, which are any updates to our current base model or source API that require full revalidation before running in production again. s104 00:11:50.880 --> 00:11:56.440 Your third artifact, this is going to be the autonomy scorecard. s105 00:11:56.440 --> 00:12:00.508 The steering committee uses it to grant autonomy in four stages. s106 00:12:00.508 --> 00:12:05.120 [clears throat] Stage one is shadow mode. s107 00:12:05.120 --> 00:12:09.240 The analyst built the report as they do today. s108 00:12:09.240 --> 00:12:14.320 The agent runs right alongside them and the team compares the two. s109 00:12:14.320 --> 00:12:18.200 Stage two is the supervised draft. s110 00:12:18.200 --> 00:12:25.800 The agent compiles the first draft and the senior analyst reviews every single line item. s111 00:12:25.880 --> 00:12:37.480 Stage three is the exception ensemble, where the analyst reviews lag initiatives plus a random sample of the claims. s112 00:12:37.480 --> 00:12:42.640 Then we have stage four, where it's exceptions only. s113 00:12:42.640 --> 00:12:50.440 The analyst reviews only high-risk anomalies supported by periodic independent sampling. s114 00:12:50.440 --> 00:13:02.160 Pay close attention to stage four, however, because once humans only look at exceptions, your biggest vulnerability is what the agent failed to flag. s115 00:13:02.160 --> 00:13:14.080 So I advise tracking mis material issues through independent sampling, monitoring error severity and watching your override rate carefully. s116 00:13:14.080 --> 00:13:19.280 A falling override rate does not automatically mean the agent is getting smarter. s117 00:13:19.280 --> 00:13:23.280 It can also mean the depth of review has slipped. s118 00:13:23.280 --> 00:13:29.160 So independent sampling tells you which one of these you are seeing. s119 00:13:29.160 --> 00:13:40.560 The scorecard puts those control metrics side by side with your value metrics, cycle time, analyst hours deployed, and your detection time. s120 00:13:40.560 --> 00:13:55.040 And with that single page view, the steering committee can make clean, efficient, data-backed decisions to either promote the agent, hold it where it is, step it back, or down s121 00:13:55.040 --> 00:13:59.040 to supervised review stage. s122 00:13:59.040 --> 00:14:03.440 So, let's come back to the three questions. s123 00:14:03.440 --> 00:14:08.960 Ask them of every agent you build, fund, or govern. s124 00:14:08.960 --> 00:14:11.800 What value does it create? s125 00:14:11.800 --> 00:14:14.440 Who is accountable for it? s126 00:14:14.440 --> 00:14:19.440 And what evidence permits it to operate with autonomy? s127 00:14:19.640 --> 00:14:26.280 In our M&A example today, the agent did not magically create the 400 million in synergies. s128 00:14:26.280 --> 00:14:33.800 Procurement, supply chain, and operating leaders captured that value through the gritty work of integration. s129 00:14:33.800 --> 00:14:37.000 But what the agent delivers is clear. s130 00:14:37.000 --> 00:14:48.560 Verified evidence weeks ahead of schedule, giving leadership the runway intervene while the deal's synergies can still be captured. s131 00:14:48.640 --> 00:14:52.920 Building an agent that works is just the first step. s132 00:14:52.920 --> 00:15:00.360 Answering those three questions is what earns it the trust to run at scale. s133 00:15:00.360 --> 00:15:02.600 Thank you.