The Agent Works. Does Anyone Trust It? — Michele Grant, Leading at Scale
https://www.youtube.com/watch?v=-a2oTS9J67U
AI Engineer · 15 min · 133 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:02] Go-live proves that a system exists, but it doesn't prove the people behind the work will actually use it.
[00:10] Or that it will ever deliver a single dollar from the deal model.
[00:15] I am Michelle Grant, creator of Leading at Scale, and the question every enterprise is confronting right now is the agent works, does anyone trust it?
[00:26] I have spent over 20 years in enterprise transformation and more than a decade leading M&A integrations across media, tech, and consumer goods.
[00:39] On those deals, trust was not something we crossed our fingers and hoped for after launch.
[00:46] We engineered it before day one through early change management and risk management.
[00:55] So, when the platform went live, the teams on the ground actually relied on it.
[01:02] AI agents demand the exact same muscle.
[01:07] An agent earns trust when it delivers the value the business budgeted for while staying strictly within the company's approved risk appetite, policies, and regulatory guardrails.
[01:23] In my experience, leadership gets there by answering three straightforward questions.
[01:30] What value does it create?
[01:32] Who is on the hook for it?
[01:35] And what evidence gives it permission to run with greater autonomy?
[01:41] The moment an agent steps out of a sandbox and into a core business workflow, the entire test changes.
[01:49] A pilot tests whether the software functions, but daily operations is where we test whether it creates value within the controls set by leadership.
[02:02] Established AI risk frameworks give you strong principles.
[02:07] What they do not give you is how to apply these principles to each agent.
[02:12] For every agent, leadership has to decide four things.
[02:17] What is it permitted to do?
[02:19] Who owns it?
[02:21] What evidence it records and what triggers an escalation.
[02:27] That work is not an engineering task.
[02:30] We find it is an executive operating mandate.
[02:34] And today, I'll show you how to meet it.
[02:38] Let us ground this in a real-world scenario.
[02:43] Picture a $15 billion acquisition targeting 400 million in net synergies.
[02:51] The deal has already closed and the integration efforts are now underway.
[02:56] The integration management office, which we'll also refer to as the IMO today, is driving the synergy roadmap across 40 workstreams.
[03:08] Covering everything from rationalizing vendor contracts to consolidating ERP systems.
[03:15] And every single initiative has a business owner accountable for its bottom-line savings.
[03:22] Every week, analysts in the IMO are going to reconcile the numbers against the general ledger.
[03:32] As they track down status from all 40 leads, they're also comparing actuals against the plan, then finally assembling the executive deck for the steering committee.
[03:45] Which is going to include the CFO, the chief risk officer, business unit heads, and the integration lead.
[03:55] The engineering team builds an agent automate that first draft.
[04:01] The agent then ingest the weekly updates, reconciles the ledger, runs variance math against the plan, and flags the initiatives that are falling behind.
[04:13] Everything looks good, and they decide to run a four-week pilot.
[04:18] The agent's drafts match the analyst reports line for line.
[04:23] So, we can say that the tech works.
[04:26] Now, the question is, what value does that steering committee need before they can trust it to run the business?
[04:37] First, the steering committee need a written value case agreed upon before the agent ever touches production data.
[04:47] In our example, that comes down to three specific metrics.
[04:52] First, let's look at cycle time.
[04:55] If that weekly report is five business days to compile right now, you're going to want to set a far target such as two days.
[05:05] Your second metric will be analyst capacity.
[05:09] If four analysts spend half of their week building this deck, that is 80 hours a week the agent gives back.
[05:19] That time counts as value only when the integration lead assigns it to the workstreams that are behind on their synergy targets and need help getting back on track.
[05:33] Then, you only count that value after netting out the compute, licensing, and human review cost of running the agent.
[05:44] Your third metric is the most important one, detection time.
[05:50] That is how long it takes from the first sign of a slip in the data to the moment the steering committee sees it.
[05:57] Cutting the report from five days to two days does not guarantee that the committee sees that slip any sooner.
[06:05] The agent class to catch it and the value case measures that directly.
[06:12] Every single metric requires a pre-deployment baseline.
[06:17] If the IMO cannot state the baseline today, do not deploy.
[06:22] If you can't measure where you started, you can never prove the agent moved the needle.
[06:29] Next, the steering committee needs to know who owns the risk the agent creates.
[06:36] We are going to apply the three lines model to the agent.
[06:41] In the first line, management owns the controls.
[06:46] The integration lead is the accountable executive.
[06:49] She owns the agent's output exactly the way she owned the manual data.
[06:56] In the second line, risk and compliance oversee the agent and challenge how to run it.
[07:03] How deep they go depends on the agent's exposure.
[07:07] An internal draft scrubbed by a senior analyst warrants a very different review standard than an autonomous agent publishing figures straight to the board.
[07:20] In the third line, internal audit provides assurance deciding based on material risk when and how the agent enters the annual audit plan.
[07:32] The agent also needs one employee assigned to oversee it every day.
[07:39] In our example, that is the senior analyst.
[07:43] He reviews the draft before it reaches the committee.
[07:47] Her role requires four specific capabilities.
[07:51] She has to first understand the operational boundaries.
[07:55] She has to interpret its output with context.
[07:59] She must have the authority to override or reverse it.
[08:05] And she needs clear kill switch to shut it down safely this ticket it behaves unexpectedly.
[08:14] Accountability also means defining where the agent's authority stops.
[08:19] I document this in a one-page policy called the human work standard.
[08:26] For every fork in the process, it maps the decision, the human role authorized to make it.
[08:34] The required evidence and the escalation path.
[08:38] These decisions exist at every tier.
[08:42] The senior analyst decides whether a variance is an accounting timing difference or a real cash shortfall.
[08:50] The work stream lead decides how to remediate that shortfall.
[08:55] And the integration lead is going to decide whether to formally flag an initiative as at risk to the steering committee.
[09:04] The agent prepares the analysis behind every one of those decisions and it stops there.
[09:12] The person who makes the decision owns the consequence.
[09:17] So keeping these decisions with your team does not slow the agent down.
[09:22] It can go into use while each role keeps its authority.
[09:28] The one reason to delight delay deployment is a decision with no assigned owner.
[09:36] This means if leadership has not agreed who declares an initiative off track, the agent's flights lead to no action.
[09:47] The third question is where the engineers in this room come in.
[09:54] The evidence for greater autonomy comes from how the agent is built.
[10:00] I look for three artifacts.
[10:03] The first is an auditable decision log.
[10:07] For every workstream the agent evaluates, it records the raw data ingested, the tool calls executed, the prompt, and the mono versions.
[10:19] Output generated along with the human sign-off.
[10:23] Two engineering choices make that log airtight.
[10:29] First, your core math and reconciliation rules must run in deterministic, finance-approved code, not generated as conversational prose from an LLM.
[10:43] Second, treat the model's written summary as a helpful explanation, not proof.
[10:50] Your real proof lives in the telemetry.
[10:53] What records were pulled?
[10:55] What deterministic tools ran?
[10:58] And who signed the review?
[11:02] If reported savings do not match the general ledger, agent does not touch the number.
[11:10] It flags the discrepancy and roots it up.
[11:14] The second artifact is the model inventory record.
[11:18] This captures the accountable executive, data lineage, dependencies across source systems, validation results, escalation path, and the kill switch run book.
[11:35] Most importantly, it documents your regression triggers, which are any updates to our current base model or source API that require full revalidation before running in production again.
[11:50] Your third artifact, this is going to be the autonomy scorecard.
[11:56] The steering committee uses it to grant autonomy in four stages.
[12:00] [clears throat] Stage one is shadow mode.
[12:05] The analyst built the report as they do today.
[12:09] The agent runs right alongside them and the team compares the two.
[12:14] Stage two is the supervised draft.
[12:18] The agent compiles the first draft and the senior analyst reviews every single line item.
[12:25] Stage three is the exception ensemble, where the analyst reviews lag initiatives plus a random sample of the claims.
[12:37] Then we have stage four, where it's exceptions only.
[12:42] The analyst reviews only high-risk anomalies supported by periodic independent sampling.
[12:50] Pay close attention to stage four, however, because once humans only look at exceptions, your biggest vulnerability is what the agent failed to flag.
[13:02] So I advise tracking mis material issues through independent sampling, monitoring error severity and watching your override rate carefully.
[13:14] A falling override rate does not automatically mean the agent is getting smarter.
[13:19] It can also mean the depth of review has slipped.
[13:23] So independent sampling tells you which one of these you are seeing.
[13:29] The scorecard puts those control metrics side by side with your value metrics, cycle time, analyst hours deployed, and your detection time.
[13:40] And with that single page view, the steering committee can make clean, efficient, data-backed decisions to either promote the agent, hold it where it is, step it back, or down
[13:55] to supervised review stage.
[13:59] So, let's come back to the three questions.
[14:03] Ask them of every agent you build, fund, or govern.
[14:08] What value does it create?
[14:11] Who is accountable for it?
[14:14] And what evidence permits it to operate with autonomy?
[14:19] In our M&A example today, the agent did not magically create the 400 million in synergies.
[14:26] Procurement, supply chain, and operating leaders captured that value through the gritty work of integration.
[14:33] But what the agent delivers is clear.
[14:37] Verified evidence weeks ahead of schedule, giving leadership the runway intervene while the deal's synergies can still be captured.
[14:48] Building an agent that works is just the first step.
[14:52] Answering those three questions is what earns it the trust to run at scale.
[15:00] Thank you.