The Agent Works. Does Anyone Trust It? — Michele Grant, Leading at Scale

AI Engineer · 15 min · 133 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:02Go-live proves that a system exists, but it doesn't prove the people behind the work will actually use it.
2. 00:10Or that it will ever deliver a single dollar from the deal model.
3. 00:15I am Michelle Grant, creator of Leading at Scale, and the question every enterprise is confronting right now is the agent works, does anyone trust it?
4. 00:26I have spent over 20 years in enterprise transformation and more than a decade leading M&A integrations across media, tech, and consumer goods.
5. 00:39On those deals, trust was not something we crossed our fingers and hoped for after launch.
6. 00:46We engineered it before day one through early change management and risk management.
7. 00:55So, when the platform went live, the teams on the ground actually relied on it.
8. 01:02AI agents demand the exact same muscle.
9. 01:07An agent earns trust when it delivers the value the business budgeted for while staying strictly within the company's approved risk appetite, policies, and regulatory guardrails.
10. 01:23In my experience, leadership gets there by answering three straightforward questions.
11. 01:30What value does it create?
12. 01:32Who is on the hook for it?
13. 01:35And what evidence gives it permission to run with greater autonomy?
14. 01:41The moment an agent steps out of a sandbox and into a core business workflow, the entire test changes.
15. 01:49A pilot tests whether the software functions, but daily operations is where we test whether it creates value within the controls set by leadership.
16. 02:02Established AI risk frameworks give you strong principles.
17. 02:07What they do not give you is how to apply these principles to each agent.
18. 02:12For every agent, leadership has to decide four things.
19. 02:17What is it permitted to do?
20. 02:19Who owns it?
21. 02:21What evidence it records and what triggers an escalation.
22. 02:27That work is not an engineering task.
23. 02:30We find it is an executive operating mandate.
24. 02:34And today, I'll show you how to meet it.
25. 02:38Let us ground this in a real-world scenario.
26. 02:43Picture a $15 billion acquisition targeting 400 million in net synergies.
27. 02:51The deal has already closed and the integration efforts are now underway.
28. 02:56The integration management office, which we'll also refer to as the IMO today, is driving the synergy roadmap across 40 workstreams.
29. 03:08Covering everything from rationalizing vendor contracts to consolidating ERP systems.
30. 03:15And every single initiative has a business owner accountable for its bottom-line savings.
31. 03:22Every week, analysts in the IMO are going to reconcile the numbers against the general ledger.
32. 03:32As they track down status from all 40 leads, they're also comparing actuals against the plan, then finally assembling the executive deck for the steering committee.
33. 03:45Which is going to include the CFO, the chief risk officer, business unit heads, and the integration lead.
34. 03:55The engineering team builds an agent automate that first draft.
35. 04:01The agent then ingest the weekly updates, reconciles the ledger, runs variance math against the plan, and flags the initiatives that are falling behind.
36. 04:13Everything looks good, and they decide to run a four-week pilot.
37. 04:18The agent's drafts match the analyst reports line for line.
38. 04:23So, we can say that the tech works.
39. 04:26Now, the question is, what value does that steering committee need before they can trust it to run the business?
40. 04:37First, the steering committee need a written value case agreed upon before the agent ever touches production data.
41. 04:47In our example, that comes down to three specific metrics.
42. 04:52First, let's look at cycle time.
43. 04:55If that weekly report is five business days to compile right now, you're going to want to set a far target such as two days.
44. 05:05Your second metric will be analyst capacity.
45. 05:09If four analysts spend half of their week building this deck, that is 80 hours a week the agent gives back.
46. 05:19That time counts as value only when the integration lead assigns it to the workstreams that are behind on their synergy targets and need help getting back on track.
47. 05:33Then, you only count that value after netting out the compute, licensing, and human review cost of running the agent.
48. 05:44Your third metric is the most important one, detection time.
49. 05:50That is how long it takes from the first sign of a slip in the data to the moment the steering committee sees it.
50. 05:57Cutting the report from five days to two days does not guarantee that the committee sees that slip any sooner.
51. 06:05The agent class to catch it and the value case measures that directly.
52. 06:12Every single metric requires a pre-deployment baseline.
53. 06:17If the IMO cannot state the baseline today, do not deploy.
54. 06:22If you can't measure where you started, you can never prove the agent moved the needle.
55. 06:29Next, the steering committee needs to know who owns the risk the agent creates.
56. 06:36We are going to apply the three lines model to the agent.
57. 06:41In the first line, management owns the controls.
58. 06:46The integration lead is the accountable executive.
59. 06:49She owns the agent's output exactly the way she owned the manual data.
60. 06:56In the second line, risk and compliance oversee the agent and challenge how to run it.
61. 07:03How deep they go depends on the agent's exposure.
62. 07:07An internal draft scrubbed by a senior analyst warrants a very different review standard than an autonomous agent publishing figures straight to the board.
63. 07:20In the third line, internal audit provides assurance deciding based on material risk when and how the agent enters the annual audit plan.
64. 07:32The agent also needs one employee assigned to oversee it every day.
65. 07:39In our example, that is the senior analyst.
66. 07:43He reviews the draft before it reaches the committee.
67. 07:47Her role requires four specific capabilities.
68. 07:51She has to first understand the operational boundaries.
69. 07:55She has to interpret its output with context.
70. 07:59She must have the authority to override or reverse it.
71. 08:05And she needs clear kill switch to shut it down safely this ticket it behaves unexpectedly.
72. 08:14Accountability also means defining where the agent's authority stops.
73. 08:19I document this in a one-page policy called the human work standard.
74. 08:26For every fork in the process, it maps the decision, the human role authorized to make it.
75. 08:34The required evidence and the escalation path.
76. 08:38These decisions exist at every tier.
77. 08:42The senior analyst decides whether a variance is an accounting timing difference or a real cash shortfall.
78. 08:50The work stream lead decides how to remediate that shortfall.
79. 08:55And the integration lead is going to decide whether to formally flag an initiative as at risk to the steering committee.
80. 09:04The agent prepares the analysis behind every one of those decisions and it stops there.
81. 09:12The person who makes the decision owns the consequence.
82. 09:17So keeping these decisions with your team does not slow the agent down.
83. 09:22It can go into use while each role keeps its authority.
84. 09:28The one reason to delight delay deployment is a decision with no assigned owner.
85. 09:36This means if leadership has not agreed who declares an initiative off track, the agent's flights lead to no action.
86. 09:47The third question is where the engineers in this room come in.
87. 09:54The evidence for greater autonomy comes from how the agent is built.
88. 10:00I look for three artifacts.
89. 10:03The first is an auditable decision log.
90. 10:07For every workstream the agent evaluates, it records the raw data ingested, the tool calls executed, the prompt, and the mono versions.
91. 10:19Output generated along with the human sign-off.
92. 10:23Two engineering choices make that log airtight.
93. 10:29First, your core math and reconciliation rules must run in deterministic, finance-approved code, not generated as conversational prose from an LLM.
94. 10:43Second, treat the model's written summary as a helpful explanation, not proof.
95. 10:50Your real proof lives in the telemetry.
96. 10:53What records were pulled?
97. 10:55What deterministic tools ran?
98. 10:58And who signed the review?
99. 11:02If reported savings do not match the general ledger, agent does not touch the number.
100. 11:10It flags the discrepancy and roots it up.
101. 11:14The second artifact is the model inventory record.
102. 11:18This captures the accountable executive, data lineage, dependencies across source systems, validation results, escalation path, and the kill switch run book.
103. 11:35Most importantly, it documents your regression triggers, which are any updates to our current base model or source API that require full revalidation before running in production again.
104. 11:50Your third artifact, this is going to be the autonomy scorecard.
105. 11:56The steering committee uses it to grant autonomy in four stages.
106. 12:00[clears throat] Stage one is shadow mode.
107. 12:05The analyst built the report as they do today.
108. 12:09The agent runs right alongside them and the team compares the two.
109. 12:14Stage two is the supervised draft.
110. 12:18The agent compiles the first draft and the senior analyst reviews every single line item.
111. 12:25Stage three is the exception ensemble, where the analyst reviews lag initiatives plus a random sample of the claims.
112. 12:37Then we have stage four, where it's exceptions only.
113. 12:42The analyst reviews only high-risk anomalies supported by periodic independent sampling.
114. 12:50Pay close attention to stage four, however, because once humans only look at exceptions, your biggest vulnerability is what the agent failed to flag.
115. 13:02So I advise tracking mis material issues through independent sampling, monitoring error severity and watching your override rate carefully.
116. 13:14A falling override rate does not automatically mean the agent is getting smarter.
117. 13:19It can also mean the depth of review has slipped.
118. 13:23So independent sampling tells you which one of these you are seeing.
119. 13:29The scorecard puts those control metrics side by side with your value metrics, cycle time, analyst hours deployed, and your detection time.
120. 13:40And with that single page view, the steering committee can make clean, efficient, data-backed decisions to either promote the agent, hold it where it is, step it back, or down
121. 13:55to supervised review stage.
122. 13:59So, let's come back to the three questions.
123. 14:03Ask them of every agent you build, fund, or govern.
124. 14:08What value does it create?
125. 14:11Who is accountable for it?
126. 14:14And what evidence permits it to operate with autonomy?
127. 14:19In our M&A example today, the agent did not magically create the 400 million in synergies.
128. 14:26Procurement, supply chain, and operating leaders captured that value through the gritty work of integration.
129. 14:33But what the agent delivers is clear.
130. 14:37Verified evidence weeks ahead of schedule, giving leadership the runway intervene while the deal's synergies can still be captured.
131. 14:48Building an agent that works is just the first step.
132. 14:52Answering those three questions is what earns it the trust to run at scale.
133. 15:00Thank you.