Containment Is Lying to You: Measuring AI Agents Like Employees — Druid AI

AI Engineer · 22 min · 194 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Morning.
00:14I'm Michael, head of developer community at Druid AI, and this is Dan, our co-founder and chief product officer.
00:23Quick context on who we are.
00:25Uh, Druid is a conversation AI startup with hundreds of customers across EMIA and Americas and we've been around for a few years.
00:35And because we sit with so many customers putting agents into production, we hear the same question over and over again.
00:43How do we actually tell if our AI agent is effective?
00:48Which numbers matters?
00:50Which artifact should we inspect?
00:53There was no clean answer out there.
00:55So we built our own reference model and measurement framework and we use it to help our customers evaluate their agents.
01:05And one of the ways we explain it to our customers is a simple analogy uh interns, right?
01:12Who doesn't love interns?
01:14So that's exactly how we'll walk you through today on this session.
01:21So here's a scenario.
01:23Uh let's take a business scenario, right?
01:26You're a manager at a banking customer service center and hire two interns for the summer.
01:32Same team, same job description for both interns.
01:37By August, here's what their dashboard says, right?
01:41Intern A contains 80% of conversation, escalates 20% to their senior bankers.
01:48Intern B contains 96% and escalates four almost nothing and dashboard is green.
01:57Now you have one return offer to give.
02:00Who gets a full-time job offer?
02:02At first glance, intern B looks like the obvious hire here, right?
02:09Intern B hired handle almost everything alone.
02:13Didn't keep pulling their senior bankers into a conversation.
02:17But surprisingly, offer to intern B is the wrong offer.
02:22Here's what had actually happened with intern A as an example.
02:27The customer is authenticated at 7 a.m. on a mobile app, asked what it would take to uh ask what would it take to pay off a mortgage early.
02:38The agent can answer the math, pull the balance, run the calculation and give a complete response contained in every definition.
02:49But that question is a signal.
02:51The customer is rethinking their mortgage.
02:54That's a refinance.
02:55That's a real revenue on the table here.
02:57So the bank writes a rule, answer the customer, then route the context to senior banker.
03:04That design handoff is part of agent A's 20%.
03:09The metric doing this is a containment uh the percent of conversation handle without a human.
03:15It's easy to measure and it's on every dashboard.
03:19This is the exact gap our customer kept hitting and it's why we built a reference model and a scorecard and Dan's going to walk you through that framework.
03:32Thank you.
03:32Thank you, Michael.
03:34Um this is the framework that we build because our customers and prospects asked us how do you know that the agent our agent it's delivering to the job?
03:51How we know that the agent is effective?
03:55So we build this framework sitting on six pillars splitted in two groups with governance across all these pillars.
04:10The configured group, it's what the agent is built off, what exists before the agent runs.
04:21Under the runtime, it's what the agent is doing at the runtime.
04:27What tasks creates, what tasks execute, and what is the output across all these groups.
04:36It's governance.
04:38From the business perspective, what is the expected boundaries?
04:44And under governance, we observe and we monitor error tolerance, risk, very important, compliance, observability, accuracy and hallucination and tone and sentiment.
05:05all together 26 components that are observing and analyzing the agent behavior as a whole.
05:19So with that framework, you stop debating whether the agent feels ready and you move the discussion based on the facts and you level the field and you can take a decision
05:37based on mathematic rules.
05:40So let's dive into it.
05:45Each individual component it's scored on three levels.
05:51It's get zero if the respective component is missing doesn't exist.
05:59It it gets one point if we can measure it.
06:03We can absorb it, observe it and we can measure it not to that not necessary automatically.
06:12So I can observe it and measure by hand equally good like it is the risk defined what is the risk I can read it I know it and the best gets
06:26two points in score if it's managed managed means it's tracked it's observed and analyzed over time because everything it's in a dynamic What happens yesterday might be different today.
06:48And to accumulate all these different metrics, we wave them based on the importance.
06:56And you observe that the governance cluster is the most important because I am analyzing the effectiveness of the agent from the business per perspective.
07:09they produce the expected business outcome.
07:12So the governance group gets 39%.
07:17Further on the mandate and we will see what it means gets 26%.
07:24Platform 12 outputs 9 and so forth.
07:28And you will notice that the technical aspects are measured are contributed but are weighted lower because we focus on the business effectiveness of the agent.
07:44Based on this metrics, we will see that a green containment that at the beginning looks the winner, it's based on a fact and can be a very different outcome
08:00when we measure all these components.
08:05Let's look a little bit inside each cluster.
08:09governance super important and many times it's forgotten to be approached from the very beginning.
08:18We focus on the technical aspects and then we tend to observe the business effects afterwards with this framework.
08:27We start with that.
08:29It helps a lot in the envisioning phase as well.
08:33Choosing what functional area to approach with AI and I see in my projects literally departments competing who should come first because they have a mandate from a board you should absorb AI.
08:48So all of them are looking to do it and the framework helps them to decide.
08:54So anytime I'm defining an agent, I start setting his boundaries.
09:00What is accepted tolerance?
09:04And you might say zero.
09:07I want to be perfect.
09:10I want to be 90%, we know that AI is not always perfect, but sometimes 50% is good as well.
09:19In autonomous agents, I want to have a higher target because there is no human in the loop.
09:25Should be perfect or cancel the job.
09:28But in a humanto AI collaboration, I know that the final decision is to a human.
09:35So AI is creating content and then human decides.
09:39So 60% error tolerance might be good risk.
09:46It is the risk defined.
09:49I am in synergies with all the other departments.
09:53So I know that my agent should run under this risk uh uh blueprint.
10:01I need to look somewhere and have these risks defined.
10:05And uh if you look at the numbers the agents we measure in here are good 97% because there is risk compliance defined for this agent.
10:19Compliance it's good.
10:21We evaluated the agent is compliant with sock 2 and back policies.
10:26So got a good score.
10:28Observability it's also good.
10:31I look the agent is logging everything from input reasoning decisions and then call integrations everything is logged good score on there drift good score we are still in the boundaries in the base established for with the first go live date
10:52it is measured and tracked accuracy 3% hallucination part of them automatically determined other set manual annotation but altogether accuracy it's measured and tracked and tone and sentiment because these agents are conversational
11:16agents I observe and extract the tone and sentiment from the conversation so all together you see how many seven very important metrics measured agent mandate super important I know we have the system prompt which says the boundaries but it's
11:42just this is just a part what is the agent definition what this agent should do the other things unlimited scope should not be done by the agent so hallucination
11:55control what's the business impact Why we are doing this agent?
12:01We are not just experimenting technology.
12:04I expect this agent to achieve 70% contract renewal.
12:10This is an ROI.
12:11We will measure it at the end to judge the business success.
12:17What are the policies boundaries based on what facts my agent should reason?
12:24Isolated context.
12:26These are the documents.
12:27These are connected systems.
12:30security, what systems this agent is allowed to engage with.
12:36Super important.
12:37I want to sleep at night knowing that my agent is only accessing these three endpoints.
12:44I don't give give him access to an MCP server with 4,000 tools and I don't know what he's doing.
12:53Isolated context.
12:55And secondly, who is allowed to engage with my agent.
13:02And last but not least, I want to stay in the budget.
13:06I don't want to have the surprise that one week later, my agent consumed 20K with whatever resources he engaged with.
13:17So all these five pillars creates the agent mandate.
13:25platform capabilities.
13:29We want the agents to work together.
13:33We don't want the agents to reinvent the wheel.
13:37If one agent is performing good because he is managing right the memory, humans handoff and orchestration on his skills, I want all the agents to behave the same.
13:50So I am measuring that as well.
13:54very important handoff.
13:56I don't want each agent to handle his own handoff.
14:00I want to have a central action center where people are engaging with all agents in my organization.
14:09So I am evaluating if each particular agent is talking with this superior uh service.
14:19I'm getting more into technical topics which are contributors to the overall score but all of them are under governance and the business purpose.
14:30So yes there are skills defined but how granular, how discoverable they are, how orchestrated, how thin or how large it's the uh the prompt combining multiple skills are the skills coming from a central repository
14:48are built on the spot or are human coded.
14:53What tools are available?
14:54You will see at the final judgment that the performer agent is not the one with the largest number of tools.
15:02It is with the effective uses of tools.
15:07So very important this aspect as well.
15:11And with this I concluded the definition.
15:15But what happens at the runtime?
15:18I need to log.
15:20I need to create tasks based on the mandate and then execute the tasks.
15:26Are the tasks written?
15:29Is the input logged?
15:32Is the required context present?
15:35How do I know it's the success criteria created together with the task?
15:43Because if I look and I see that the agent renewed the contract, I want to measure if that renewal was successful or not.
15:52If I don't have the success criteria linked with that individual contract, it will be impossible to judge.
15:59So the agent should explain everything he's doing.
16:03This is what the task topic is.
16:07And at the end, I want to validate the output.
16:12there is one single score that evaluates if the agent was successful or not.
16:21So in summary, combining all these metrics, agent A scores 83%.
16:32If you remember at the beginning, his containment was 80%.
16:38Escalated 20 times.
16:40Agent B escalated just four times.
16:44However, observing and measuring the governance, the mandate, the final outcome, the final score, it's in favor of agent A. And if you look at the clusters, in each individual
17:01cluster, A was better than B except the skill.
17:07B had more skills but was lack in explanability in tracking success and compliance with the governance.
17:16So all together this framework gives the true winner Michael back to you.
17:24Thank you.
17:25Thank you Dan.
17:28So those seven pillars and the scorecard uh that's the framework and this is a framework that we've used uh with multiple of our customers and it's been working out really well for us
17:40and what customers typically tell us is that hey they finally have a some sort of framework that they can use to measure the effectness of the agents that deploying
17:49and be able to have a discussion intelligent discussion with their business stakeholders.
17:55So here's where going back to the inter uh intern metaphor breaks right in your favor.
18:02Uh when you hire a person, the interview ends, right?
18:05But when you hire an agent, it never ends.
18:07You rerun it every week on a live traffic against a human baseline instrument, evaluate, improve, retest.
18:15Again, going back to what Dan said earlier, the drift, right?
18:19The green line you see on the right here is the um the vanity dashboard as flat 92% all year.
18:26The teal line is the wheel score sliding from 83 to 60 because you're measuring it on a weekly basis.
18:32So same agent, same green dashboard but different metrics across the time.
18:39So now one of the things especially if you once you start to engage with the business stakeholders let's say your CFO or something you know the first first question comes out is the is the ROI
18:52right and the and the fear sets in the fear is that only measurable thing about an agent that you're working on you're deploying is like whose job is it does it take away from right
19:04and that fear survives because the we mostly haven't measured the alternative alternative.
19:09So here's a reframe that we try to typically work with our customers.
19:14A good intern lightens your workload.
19:16The routine works moves off to your desk and you get sharper at the work that's left.
19:22Right?
19:23Same test for an agent.
19:24What did it take off of your plate and what did that free the time to allow your business to do?
19:31Right?
19:31The agent hands a balance check at 7 and 9. So the banker spends next morning with the customer who's rethinking their mortgage.
19:39The agent made the banker better at banking and that's the column belongs on your scorecard and that's some of the things that we try to work with our customers as well.
19:50Now a a confession to make about the opening the intern was never hypothetical that 8020 split was a 2026 root AI adoption benchmark for financial services.
20:03So everything you just heard is backed by our anonymized aggregate AI agent production data.
20:09Uh the same metric containment four different industries four different right answers financial services at 80 healthcare at 87 HR and IT at 93 and higher education and 99 containment rate.
20:24Now when you look at the data higher ed isn't better than banking.
20:29banking just design more handoff in their agenting AI design that's what you wait the outcomes per agent against one universal number the numbers different across different industry this is the druid 2026
20:44AI adoption benchmark uh we gathered 15 months of the production AI agent data and uh it contains many useful data uh across different industry as well it's free to download
20:58and if you Look at the next slide here.
21:00this slide uh this is where you can um QR code you can scan it you can uh download the report and it's the framework that we developed is really about defining the mandate
21:13contract the boundaries score the whole agent run it weekly uh against the human baseline and that's a framework that we typically uh work with our customers to kind of so we wanted to
21:25present it you all kind of share the framework free to use and uh one Last thing is uh we're located at the booth LG11 right across from OpenAI Microsoft booth.
21:36So feel free to come by and I would love to have some conversation with you all.
21:41Thank you very much.
21:43Thank you.
21:57[music]