WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=PwCSYy-oGsM

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/PwCSYy-oGsM.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.719 --> 00:00:14.719
Morning.

s3
00:00:14.960 --> 00:00:23.439
I'm Michael, head of developer community at Druid AI, and this is Dan, our co-founder and chief product officer.

s4
00:00:23.439 --> 00:00:25.279
Quick context on who we are.

s5
00:00:25.279 --> 00:00:35.040
Uh, Druid is a conversation AI startup with hundreds of customers across EMIA and Americas and we've been around for a few years.

s6
00:00:35.040 --> 00:00:43.760
And because we sit with so many customers putting agents into production, we hear the same question over and over again.

s7
00:00:43.760 --> 00:00:48.480
How do we actually tell if our AI agent is effective?

s8
00:00:48.480 --> 00:00:50.320
Which numbers matters?

s9
00:00:50.320 --> 00:00:53.120
Which artifact should we inspect?

s10
00:00:53.120 --> 00:00:55.440
There was no clean answer out there.

s11
00:00:55.440 --> 00:01:05.040
So we built our own reference model and measurement framework and we use it to help our customers evaluate their agents.

s12
00:01:05.040 --> 00:01:12.560
And one of the ways we explain it to our customers is a simple analogy uh interns, right?

s13
00:01:12.560 --> 00:01:14.320
Who doesn't love interns?

s14
00:01:14.320 --> 00:01:19.040
So that's exactly how we'll walk you through today on this session.

s15
00:01:21.200 --> 00:01:23.200
So here's a scenario.

s16
00:01:23.200 --> 00:01:26.240
Uh let's take a business scenario, right?

s17
00:01:26.240 --> 00:01:32.960
You're a manager at a banking customer service center and hire two interns for the summer.

s18
00:01:32.960 --> 00:01:37.680
Same team, same job description for both interns.

s19
00:01:37.680 --> 00:01:41.520
By August, here's what their dashboard says, right?

s20
00:01:41.520 --> 00:01:48.240
Intern A contains 80% of conversation, escalates 20% to their senior bankers.

s21
00:01:48.240 --> 00:01:57.200
Intern B contains 96% and escalates four almost nothing and dashboard is green.

s22
00:01:57.200 --> 00:02:00.159
Now you have one return offer to give.

s23
00:02:00.159 --> 00:02:02.960
Who gets a full-time job offer?

s24
00:02:02.960 --> 00:02:09.360
At first glance, intern B looks like the obvious hire here, right?

s25
00:02:09.360 --> 00:02:13.280
Intern B hired handle almost everything alone.

s26
00:02:13.280 --> 00:02:17.520
Didn't keep pulling their senior bankers into a conversation.

s27
00:02:17.520 --> 00:02:22.239
But surprisingly, offer to intern B is the wrong offer.

s28
00:02:22.640 --> 00:02:27.520
Here's what had actually happened with intern A as an example.

s29
00:02:27.520 --> 00:02:38.160
The customer is authenticated at 7 a.m. on a mobile app, asked what it would take to uh ask what would it take to pay off a mortgage early.

s30
00:02:38.160 --> 00:02:49.840
The agent can answer the math, pull the balance, run the calculation and give a complete response contained in every definition.

s31
00:02:49.840 --> 00:02:51.680
But that question is a signal.

s32
00:02:51.680 --> 00:02:54.000
The customer is rethinking their mortgage.

s33
00:02:54.000 --> 00:02:55.200
That's a refinance.

s34
00:02:55.200 --> 00:02:57.840
That's a real revenue on the table here.

s35
00:02:57.840 --> 00:03:04.800
So the bank writes a rule, answer the customer, then route the context to senior banker.

s36
00:03:04.800 --> 00:03:09.120
That design handoff is part of agent A's 20%.

s37
00:03:09.120 --> 00:03:15.120
The metric doing this is a containment uh the percent of conversation handle without a human.

s38
00:03:15.120 --> 00:03:19.920
It's easy to measure and it's on every dashboard.

s39
00:03:19.920 --> 00:03:30.640
This is the exact gap our customer kept hitting and it's why we built a reference model and a scorecard and Dan's going to walk you through that framework.

s40
00:03:32.239 --> 00:03:32.879
Thank you.

s41
00:03:32.879 --> 00:03:34.159
Thank you, Michael.

s42
00:03:34.159 --> 00:03:51.360
Um this is the framework that we build because our customers and prospects asked us how do you know that the agent our agent it's delivering to the job?

s43
00:03:51.360 --> 00:03:55.200
How we know that the agent is effective?

s44
00:03:55.200 --> 00:04:10.239
So we build this framework sitting on six pillars splitted in two groups with governance across all these pillars.

s45
00:04:10.239 --> 00:04:20.959
The configured group, it's what the agent is built off, what exists before the agent runs.

s46
00:04:21.280 --> 00:04:27.280
Under the runtime, it's what the agent is doing at the runtime.

s47
00:04:27.280 --> 00:04:36.400
What tasks creates, what tasks execute, and what is the output across all these groups.

s48
00:04:36.400 --> 00:04:38.479
It's governance.

s49
00:04:38.479 --> 00:04:44.720
From the business perspective, what is the expected boundaries?

s50
00:04:44.720 --> 00:05:04.960
And under governance, we observe and we monitor error tolerance, risk, very important, compliance, observability, accuracy and hallucination and tone and sentiment.

s51
00:05:05.199 --> 00:05:18.880
all together 26 components that are observing and analyzing the agent behavior as a whole.

s52
00:05:19.600 --> 00:05:37.680
So with that framework, you stop debating whether the agent feels ready and you move the discussion based on the facts and you level the field and you can take a decision

s53
00:05:37.680 --> 00:05:40.160
based on mathematic rules.

s54
00:05:40.160 --> 00:05:43.919
So let's dive into it.

s55
00:05:45.199 --> 00:05:51.840
Each individual component it's scored on three levels.

s56
00:05:51.840 --> 00:05:59.120
It's get zero if the respective component is missing doesn't exist.

s57
00:05:59.120 --> 00:06:03.759
It it gets one point if we can measure it.

s58
00:06:03.759 --> 00:06:12.960
We can absorb it, observe it and we can measure it not to that not necessary automatically.

s59
00:06:12.960 --> 00:06:26.720
So I can observe it and measure by hand equally good like it is the risk defined what is the risk I can read it I know it and the best gets

s60
00:06:26.720 --> 00:06:47.360
two points in score if it's managed managed means it's tracked it's observed and analyzed over time because everything it's in a dynamic What happens yesterday might be different today.

s61
00:06:48.000 --> 00:06:56.000
And to accumulate all these different metrics, we wave them based on the importance.

s62
00:06:56.000 --> 00:07:09.199
And you observe that the governance cluster is the most important because I am analyzing the effectiveness of the agent from the business per perspective.

s63
00:07:09.199 --> 00:07:12.319
they produce the expected business outcome.

s64
00:07:12.319 --> 00:07:17.440
So the governance group gets 39%.

s65
00:07:17.599 --> 00:07:24.240
Further on the mandate and we will see what it means gets 26%.

s66
00:07:24.240 --> 00:07:28.639
Platform 12 outputs 9 and so forth.

s67
00:07:28.639 --> 00:07:44.319
And you will notice that the technical aspects are measured are contributed but are weighted lower because we focus on the business effectiveness of the agent.

s68
00:07:44.560 --> 00:08:00.800
Based on this metrics, we will see that a green containment that at the beginning looks the winner, it's based on a fact and can be a very different outcome

s69
00:08:00.800 --> 00:08:04.639
when we measure all these components.

s70
00:08:05.039 --> 00:08:09.919
Let's look a little bit inside each cluster.

s71
00:08:09.919 --> 00:08:18.560
governance super important and many times it's forgotten to be approached from the very beginning.

s72
00:08:18.560 --> 00:08:27.280
We focus on the technical aspects and then we tend to observe the business effects afterwards with this framework.

s73
00:08:27.280 --> 00:08:29.360
We start with that.

s74
00:08:29.360 --> 00:08:33.360
It helps a lot in the envisioning phase as well.

s75
00:08:33.360 --> 00:08:48.160
Choosing what functional area to approach with AI and I see in my projects literally departments competing who should come first because they have a mandate from a board you should absorb AI.

s76
00:08:48.160 --> 00:08:54.480
So all of them are looking to do it and the framework helps them to decide.

s77
00:08:54.480 --> 00:09:00.800
So anytime I'm defining an agent, I start setting his boundaries.

s78
00:09:00.800 --> 00:09:04.640
What is accepted tolerance?

s79
00:09:04.640 --> 00:09:07.920
And you might say zero.

s80
00:09:07.920 --> 00:09:10.240
I want to be perfect.

s81
00:09:10.240 --> 00:09:19.120
I want to be 90%, we know that AI is not always perfect, but sometimes 50% is good as well.

s82
00:09:19.120 --> 00:09:25.600
In autonomous agents, I want to have a higher target because there is no human in the loop.

s83
00:09:25.600 --> 00:09:28.240
Should be perfect or cancel the job.

s84
00:09:28.240 --> 00:09:35.519
But in a humanto AI collaboration, I know that the final decision is to a human.

s85
00:09:35.519 --> 00:09:39.680
So AI is creating content and then human decides.

s86
00:09:39.680 --> 00:09:46.880
So 60% error tolerance might be good risk.

s87
00:09:46.880 --> 00:09:49.600
It is the risk defined.

s88
00:09:49.600 --> 00:09:53.279
I am in synergies with all the other departments.

s89
00:09:53.279 --> 00:10:01.040
So I know that my agent should run under this risk uh uh blueprint.

s90
00:10:01.040 --> 00:10:05.760
I need to look somewhere and have these risks defined.

s91
00:10:05.760 --> 00:10:19.519
And uh if you look at the numbers the agents we measure in here are good 97% because there is risk compliance defined for this agent.

s92
00:10:19.519 --> 00:10:21.279
Compliance it's good.

s93
00:10:21.279 --> 00:10:26.240
We evaluated the agent is compliant with sock 2 and back policies.

s94
00:10:26.240 --> 00:10:28.800
So got a good score.

s95
00:10:28.800 --> 00:10:31.440
Observability it's also good.

s96
00:10:31.440 --> 00:10:52.640
I look the agent is logging everything from input reasoning decisions and then call integrations everything is logged good score on there drift good score we are still in the boundaries in the base established for with the first go live date

s97
00:10:52.640 --> 00:11:16.480
it is measured and tracked accuracy 3% hallucination part of them automatically determined other set manual annotation but altogether accuracy it's measured and tracked and tone and sentiment because these agents are conversational

s98
00:11:16.480 --> 00:11:42.000
agents I observe and extract the tone and sentiment from the conversation so all together you see how many seven very important metrics measured agent mandate super important I know we have the system prompt which says the boundaries but it's

s99
00:11:42.000 --> 00:11:55.920
just this is just a part what is the agent definition what this agent should do the other things unlimited scope should not be done by the agent so hallucination

s100
00:11:55.920 --> 00:12:01.279
control what's the business impact Why we are doing this agent?

s101
00:12:01.279 --> 00:12:04.160
We are not just experimenting technology.

s102
00:12:04.160 --> 00:12:10.639
I expect this agent to achieve 70% contract renewal.

s103
00:12:10.639 --> 00:12:11.839
This is an ROI.

s104
00:12:11.839 --> 00:12:17.279
We will measure it at the end to judge the business success.

s105
00:12:17.760 --> 00:12:24.800
What are the policies boundaries based on what facts my agent should reason?

s106
00:12:24.800 --> 00:12:26.079
Isolated context.

s107
00:12:26.079 --> 00:12:27.200
These are the documents.

s108
00:12:27.200 --> 00:12:30.000
These are connected systems.

s109
00:12:30.000 --> 00:12:36.639
security, what systems this agent is allowed to engage with.

s110
00:12:36.639 --> 00:12:37.519
Super important.

s111
00:12:37.519 --> 00:12:44.959
I want to sleep at night knowing that my agent is only accessing these three endpoints.

s112
00:12:44.959 --> 00:12:53.839
I don't give give him access to an MCP server with 4,000 tools and I don't know what he's doing.

s113
00:12:53.839 --> 00:12:55.200
Isolated context.

s114
00:12:55.200 --> 00:13:01.600
And secondly, who is allowed to engage with my agent.

s115
00:13:02.480 --> 00:13:06.480
And last but not least, I want to stay in the budget.

s116
00:13:06.480 --> 00:13:17.600
I don't want to have the surprise that one week later, my agent consumed 20K with whatever resources he engaged with.

s117
00:13:17.600 --> 00:13:24.399
So all these five pillars creates the agent mandate.

s118
00:13:25.760 --> 00:13:28.320
platform capabilities.

s119
00:13:29.760 --> 00:13:33.200
We want the agents to work together.

s120
00:13:33.200 --> 00:13:37.279
We don't want the agents to reinvent the wheel.

s121
00:13:37.279 --> 00:13:50.399
If one agent is performing good because he is managing right the memory, humans handoff and orchestration on his skills, I want all the agents to behave the same.

s122
00:13:50.399 --> 00:13:54.000
So I am measuring that as well.

s123
00:13:54.720 --> 00:13:56.800
very important handoff.

s124
00:13:56.800 --> 00:14:00.720
I don't want each agent to handle his own handoff.

s125
00:14:00.720 --> 00:14:09.040
I want to have a central action center where people are engaging with all agents in my organization.

s126
00:14:09.040 --> 00:14:18.079
So I am evaluating if each particular agent is talking with this superior uh service.

s127
00:14:19.519 --> 00:14:30.160
I'm getting more into technical topics which are contributors to the overall score but all of them are under governance and the business purpose.

s128
00:14:30.160 --> 00:14:48.720
So yes there are skills defined but how granular, how discoverable they are, how orchestrated, how thin or how large it's the uh the prompt combining multiple skills are the skills coming from a central repository

s129
00:14:48.720 --> 00:14:53.199
are built on the spot or are human coded.

s130
00:14:53.199 --> 00:14:54.639
What tools are available?

s131
00:14:54.639 --> 00:15:02.959
You will see at the final judgment that the performer agent is not the one with the largest number of tools.

s132
00:15:02.959 --> 00:15:07.120
It is with the effective uses of tools.

s133
00:15:07.120 --> 00:15:11.199
So very important this aspect as well.

s134
00:15:11.199 --> 00:15:15.279
And with this I concluded the definition.

s135
00:15:15.279 --> 00:15:18.160
But what happens at the runtime?

s136
00:15:18.160 --> 00:15:20.560
I need to log.

s137
00:15:20.560 --> 00:15:26.800
I need to create tasks based on the mandate and then execute the tasks.

s138
00:15:26.800 --> 00:15:29.519
Are the tasks written?

s139
00:15:29.519 --> 00:15:32.959
Is the input logged?

s140
00:15:32.959 --> 00:15:35.440
Is the required context present?

s141
00:15:35.440 --> 00:15:43.360
How do I know it's the success criteria created together with the task?

s142
00:15:43.360 --> 00:15:52.160
Because if I look and I see that the agent renewed the contract, I want to measure if that renewal was successful or not.

s143
00:15:52.160 --> 00:15:59.600
If I don't have the success criteria linked with that individual contract, it will be impossible to judge.

s144
00:15:59.600 --> 00:16:03.279
So the agent should explain everything he's doing.

s145
00:16:03.279 --> 00:16:07.680
This is what the task topic is.

s146
00:16:07.680 --> 00:16:12.079
And at the end, I want to validate the output.

s147
00:16:12.079 --> 00:16:19.920
there is one single score that evaluates if the agent was successful or not.

s148
00:16:21.279 --> 00:16:32.720
So in summary, combining all these metrics, agent A scores 83%.

s149
00:16:32.720 --> 00:16:38.320
If you remember at the beginning, his containment was 80%.

s150
00:16:38.320 --> 00:16:40.560
Escalated 20 times.

s151
00:16:40.560 --> 00:16:44.560
Agent B escalated just four times.

s152
00:16:44.560 --> 00:17:01.360
However, observing and measuring the governance, the mandate, the final outcome, the final score, it's in favor of agent A. And if you look at the clusters, in each individual

s153
00:17:01.360 --> 00:17:07.199
cluster, A was better than B except the skill.

s154
00:17:07.199 --> 00:17:16.160
B had more skills but was lack in explanability in tracking success and compliance with the governance.

s155
00:17:16.160 --> 00:17:24.079
So all together this framework gives the true winner Michael back to you.

s156
00:17:24.079 --> 00:17:25.600
Thank you.

s157
00:17:25.600 --> 00:17:27.919
Thank you Dan.

s158
00:17:28.319 --> 00:17:40.240
So those seven pillars and the scorecard uh that's the framework and this is a framework that we've used uh with multiple of our customers and it's been working out really well for us

s159
00:17:40.240 --> 00:17:49.840
and what customers typically tell us is that hey they finally have a some sort of framework that they can use to measure the effectness of the agents that deploying

s160
00:17:49.840 --> 00:17:55.440
and be able to have a discussion intelligent discussion with their business stakeholders.

s161
00:17:55.440 --> 00:18:02.400
So here's where going back to the inter uh intern metaphor breaks right in your favor.

s162
00:18:02.400 --> 00:18:05.600
Uh when you hire a person, the interview ends, right?

s163
00:18:05.600 --> 00:18:07.919
But when you hire an agent, it never ends.

s164
00:18:07.919 --> 00:18:15.760
You rerun it every week on a live traffic against a human baseline instrument, evaluate, improve, retest.

s165
00:18:15.760 --> 00:18:19.280
Again, going back to what Dan said earlier, the drift, right?

s166
00:18:19.280 --> 00:18:26.480
The green line you see on the right here is the um the vanity dashboard as flat 92% all year.

s167
00:18:26.480 --> 00:18:32.559
The teal line is the wheel score sliding from 83 to 60 because you're measuring it on a weekly basis.

s168
00:18:32.559 --> 00:18:39.440
So same agent, same green dashboard but different metrics across the time.

s169
00:18:39.440 --> 00:18:52.720
So now one of the things especially if you once you start to engage with the business stakeholders let's say your CFO or something you know the first first question comes out is the is the ROI

s170
00:18:52.720 --> 00:19:04.000
right and the and the fear sets in the fear is that only measurable thing about an agent that you're working on you're deploying is like whose job is it does it take away from right

s171
00:19:04.000 --> 00:19:09.360
and that fear survives because the we mostly haven't measured the alternative alternative.

s172
00:19:09.360 --> 00:19:14.240
So here's a reframe that we try to typically work with our customers.

s173
00:19:14.240 --> 00:19:16.960
A good intern lightens your workload.

s174
00:19:16.960 --> 00:19:22.640
The routine works moves off to your desk and you get sharper at the work that's left.

s175
00:19:22.640 --> 00:19:23.039
Right?

s176
00:19:23.039 --> 00:19:24.960
Same test for an agent.

s177
00:19:24.960 --> 00:19:31.200
What did it take off of your plate and what did that free the time to allow your business to do?

s178
00:19:31.200 --> 00:19:31.840
Right?

s179
00:19:31.840 --> 00:19:39.120
The agent hands a balance check at 7 and 9. So the banker spends next morning with the customer who's rethinking their mortgage.

s180
00:19:39.120 --> 00:19:50.000
The agent made the banker better at banking and that's the column belongs on your scorecard and that's some of the things that we try to work with our customers as well.

s181
00:19:50.080 --> 00:20:03.600
Now a a confession to make about the opening the intern was never hypothetical that 8020 split was a 2026 root AI adoption benchmark for financial services.

s182
00:20:03.600 --> 00:20:09.919
So everything you just heard is backed by our anonymized aggregate AI agent production data.

s183
00:20:09.919 --> 00:20:24.400
Uh the same metric containment four different industries four different right answers financial services at 80 healthcare at 87 HR and IT at 93 and higher education and 99 containment rate.

s184
00:20:24.400 --> 00:20:29.360
Now when you look at the data higher ed isn't better than banking.

s185
00:20:29.360 --> 00:20:44.720
banking just design more handoff in their agenting AI design that's what you wait the outcomes per agent against one universal number the numbers different across different industry this is the druid 2026

s186
00:20:44.720 --> 00:20:58.240
AI adoption benchmark uh we gathered 15 months of the production AI agent data and uh it contains many useful data uh across different industry as well it's free to download

s187
00:20:58.240 --> 00:21:00.799
and if you Look at the next slide here.

s188
00:21:00.799 --> 00:21:13.679
this slide uh this is where you can um QR code you can scan it you can uh download the report and it's the framework that we developed is really about defining the mandate

s189
00:21:13.679 --> 00:21:25.679
contract the boundaries score the whole agent run it weekly uh against the human baseline and that's a framework that we typically uh work with our customers to kind of so we wanted to

s190
00:21:25.679 --> 00:21:36.880
present it you all kind of share the framework free to use and uh one Last thing is uh we're located at the booth LG11 right across from OpenAI Microsoft booth.

s191
00:21:36.880 --> 00:21:41.200
So feel free to come by and I would love to have some conversation with you all.

s192
00:21:41.200 --> 00:21:43.200
Thank you very much.

s193
00:21:43.200 --> 00:21:45.440
Thank you.

s194
00:21:57.220 --> 00:21:59.220
[music]