WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=rTojoVotlD8

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/rTojoVotlD8.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.560 --> 00:00:13.560
Hello everyone.

s3
00:00:13.560 --> 00:00:16.120
Thank you for joining me today.

s4
00:00:16.120 --> 00:00:18.440
It's time to begin.

s5
00:00:18.440 --> 00:00:30.440
And for years, many of us monitored our applications with the golden signals, which is was latency, error, traffic, and saturation.

s6
00:00:30.440 --> 00:00:32.720
And shortly we call them lads.

s7
00:00:32.720 --> 00:00:41.960
And they were and still they are and will be remain sufficient and critical for determine the health of our applications.

s8
00:00:41.960 --> 00:00:47.840
However, today when we enter the new era era of generative AI, it's becoming not enough.

s9
00:00:47.840 --> 00:00:57.480
And today we going to talk about what are the new metrics we need to look if we want to build the healthy gen AI application.

s10
00:00:57.480 --> 00:01:00.000
Very shortly, my name is Marina Petzel.

s11
00:01:00.000 --> 00:01:03.160
I am gen AI advocate here in DataDog.

s12
00:01:03.160 --> 00:01:08.200
And today I will walk you through the metrics that we need to look for.

s13
00:01:08.200 --> 00:01:18.080
So, when we talk about traditional applications, the main specific of them is that the same input always yield the same output, right?

s14
00:01:18.080 --> 00:01:19.480
It's predictable.

s15
00:01:19.480 --> 00:01:28.480
But with generative AI applications, we all know that the same prompt can generate different responses every time we run it.

s16
00:01:28.480 --> 00:01:33.440
And which means we cannot rely on the standard regression testing alone.

s17
00:01:33.440 --> 00:01:38.960
And you must constantly evaluate quality in the live environment.

s18
00:01:38.960 --> 00:01:46.360
The second difference between traditional applications and gen AI applications, it's variable cost structure.

s19
00:01:46.360 --> 00:01:50.440
Traditional compute cost are predictable, right?

s20
00:01:50.440 --> 00:01:53.440
We pay for machine that is running our code.

s21
00:01:53.440 --> 00:01:58.320
But for gen AI, this this is dynamic variable here.

s22
00:01:58.320 --> 00:02:06.320
They might They vary from the number of tokens, the model that we are using, or the size of the context window that you have.

s23
00:02:06.320 --> 00:02:16.600
And it's become essential to do the real-time granular course cost to prevent the spending out of control.

s24
00:02:16.600 --> 00:02:21.040
The third problem is that we face the new attack vectors.

s25
00:02:21.040 --> 00:02:34.720
We used to worry about SQL injection or cross-site scripting before, but today we must defend against the prompt injection, jailbreaking, or PII outage in our output.

s26
00:02:34.720 --> 00:02:48.080
And that's becoming a big problem, and this are security issue, not a simple 500 error code can will never catch, and demanded entirely new forms of security monitoring.

s27
00:02:48.080 --> 00:02:52.800
And finally, quality is very subjective for GenAI applications.

s28
00:02:52.800 --> 00:02:57.600
The traditional systems either work or do not work, right?

s29
00:02:57.600 --> 00:03:01.400
But generative AI output is judged on the spectrum.

s30
00:03:01.400 --> 00:03:02.600
Is it relevant?

s31
00:03:02.600 --> 00:03:03.720
Is it accurate?

s32
00:03:03.720 --> 00:03:05.600
Is it complete or not?

s33
00:03:05.600 --> 00:03:16.680
And the problem is that 200 okay response from your server doesn't necessarily mean that the answer was helpful or correct for your final user.

s34
00:03:16.680 --> 00:03:25.480
And we must now integrate the continuous quality evaluation directly in your monitoring stack.

s35
00:03:25.480 --> 00:03:34.880
And at the beginning we all were thinking that these golden signals will be enough for our systems, but unfortunately they are not.

s36
00:03:34.880 --> 00:03:43.560
Traditional metrics will tell you if your system is up and running, but it won't tell you how well it's running.

s37
00:03:43.560 --> 00:03:55.160
And established metrics lets will be still essential, but we going to talk about what else on addition to that we need to implement into our systems?

s38
00:03:55.160 --> 00:03:58.400
And first I want to start with the cost monitoring.

s39
00:03:58.400 --> 00:04:04.680
How we can track the spending before it uh spirals out of control?

s40
00:04:04.680 --> 00:04:17.560
And I think we all know the financial implications of GenAI, and a lot of companies are facing the huge outage of the uh cost per their monthly usage.

s41
00:04:17.560 --> 00:04:26.720
And while the same is true as for traditional applications, there were more linear scale between the throughput and the uh the cost.

s42
00:04:26.720 --> 00:04:35.680
If the user loads um load goes up, our resources uh usage goes up, and sequentially the cost goes up.

s43
00:04:35.680 --> 00:04:49.080
But the cost structure for LLM is highly dynamic and unpredictable, which means we can we don't have this visibility, and we need to really pay attention how we track that.

s44
00:04:49.080 --> 00:04:52.760
And what are the problems you might encounter with the cost?

s45
00:04:52.760 --> 00:05:03.640
And here we talk about the three major way for cost tracking, uh and so you can track them um to prevent the cost outage.

s46
00:05:03.640 --> 00:05:06.040
So, the first is a token creep.

s47
00:05:06.040 --> 00:05:15.840
Uh this happens when engineers or product managers trying to improve the quality quietly increasing the context window.

s48
00:05:15.840 --> 00:05:24.200
And let's say the context window was 4,000 tokens, and then it's like blow to 132 without any financial review.

s49
00:05:24.200 --> 00:05:35.040
And what it can cost like the the cost is depends very uh significantly to the context count, and that might result up to eight x cost increased.

s50
00:05:35.040 --> 00:05:40.160
And and a month it might cost you thousands of dollars.

s51
00:05:40.160 --> 00:05:41.960
Second is the model drift.

s52
00:05:41.960 --> 00:05:52.560
Uh that is also the pretty typical problem when the team switches from cheaper and faster model, like let's say Hi Cool 4.5, to the more powerful and more expensive

s53
00:05:52.560 --> 00:05:59.240
model, like Opus 4.8. And which promise you obviously the better quality.

s54
00:05:59.240 --> 00:06:07.880
But at the same time, Opus, for example, uh can be higher 15 times higher uh per the cost request.

s55
00:06:07.880 --> 00:06:17.360
And this change in model, even with the same usage volume, might blow your cost significantly within a month, too.

s56
00:06:17.360 --> 00:06:20.280
And finally, it's on hash code.

s57
00:06:20.280 --> 00:06:29.240
In generative AI, the same query can hit the API repeatedly if you haven't implemented the effective caching layer.

s58
00:06:29.240 --> 00:06:43.320
And what you will see from um our uh own research that 70% of the spend sometimes are redundant because the system is paying for exactly the same expensive model

s59
00:06:43.320 --> 00:06:46.080
completion uh over and over again.

s60
00:06:46.080 --> 00:06:54.800
And with this granular visibility into these three layers, you would be able to escalate your cost rapidly.

s61
00:06:54.800 --> 00:07:01.720
And now, let's talk about how you can prevent that and get more granular data.

s62
00:07:01.720 --> 00:07:15.560
And when we um talk about that, we uh try to think about the implementation of cost attribution strategy, which is you must tag absolutely everything.

s63
00:07:15.560 --> 00:07:21.240
For example, uh tagging uh can help you to uh Why it's important, right?

s64
00:07:21.240 --> 00:07:28.600
You receive a lot of uh interactions with your agents, and but all of these uh interactions are undifferentiated.

s65
00:07:28.600 --> 00:07:31.840
You don't know what's blow your actual cost.

s66
00:07:31.840 --> 00:07:36.880
That is why first layer what you can implement is the feature level level.

s67
00:07:36.880 --> 00:07:46.040
By tagging the intention of the feature, like it was a chat or summarization, you can attribute the cost for the specific product areas.

s68
00:07:46.040 --> 00:07:58.040
And this data is crucial not only for engineering your cost management, but also for the product road map decisions, like knowing which features are actually the budget hawks.

s69
00:07:58.040 --> 00:08:01.440
The second level is the use user level.

s70
00:08:01.440 --> 00:08:16.680
Using the tags like the user ID or your org name is essential for chargebacks, accurate billing, but also quickly identify by the patterns of abusive usage before they drain your budget.

s71
00:08:16.760 --> 00:08:18.480
Third is the model level.

s72
00:08:18.480 --> 00:08:26.320
Also tag your models like GPT-5.5 or the provider, OpenAI or Tropic.

s73
00:08:26.320 --> 00:08:33.320
It will give you visibility into the high-level areas of cost optimization.

s74
00:08:33.320 --> 00:08:39.320
And you can benchmark the true expenses of using one model versus another.

s75
00:08:39.320 --> 00:08:42.760
And the finally, endpoint level tagging.

s76
00:08:42.760 --> 00:08:52.040
Attributing cost to regions or to your production or like to your environment like production or staging helps your infrastructure planning.

s77
00:08:52.040 --> 00:09:00.320
And you can see how regional distribution or staging versus production environment can impact your total spend.

s78
00:09:00.320 --> 00:09:15.280
And by making this tagging mandatory practice, you gain the essential visibility to understand exactly where your money is going and how important to control it.

s79
00:09:15.280 --> 00:09:22.480
Well, but now let's talk if like let's let's imagine that we handle all the cost issues, right?

s80
00:09:22.480 --> 00:09:24.839
We took that under control.

s81
00:09:24.839 --> 00:09:33.400
And your finance team is obviously get very happy, but the next one problem is the safety.

s82
00:09:33.400 --> 00:09:41.320
And your AI can be cheap, but still leak some of the sensitive or personal information, which is not great.

s83
00:09:41.320 --> 00:09:51.080
And even though like your let scores can be perfect, it cannot tell you if there are some metrics went wrong.

s84
00:09:51.080 --> 00:10:00.040
So, what we can track here in the safety metrics to First is the prompt injection rate, which has become the very crucial problem.

s85
00:10:00.040 --> 00:10:05.800
This metrics detects the manipulation attempts into your user prompts.

s86
00:10:05.800 --> 00:10:12.120
There are It's infrastructure If the infrastructure design doesn't have any guardrails.

s87
00:10:12.120 --> 00:10:23.000
And this can be um prevent by using some advanced techniques like a pattern matching or specialized classifier models.

s88
00:10:23.000 --> 00:10:29.280
And for this, we usually need to target for very aggressive thresholds.

s89
00:10:29.280 --> 00:10:32.280
The second is PII detection rate.

s90
00:10:32.280 --> 00:10:39.280
Um PII So, which is a crucial and zero tolerance issue here.

s91
00:10:39.280 --> 00:10:46.920
We can scan all the model outputs for sensitive data like social security, like credit cards, or personal information.

s92
00:10:46.920 --> 00:10:55.040
And you can use some um methods like regex or named entity recognition here.

s93
00:10:55.040 --> 00:11:02.040
And only acceptable target should be 0% of your production outputs.

s94
00:11:02.040 --> 00:11:05.800
The third is the content moderation score.

s95
00:11:05.800 --> 00:11:14.560
How you can determine if your content is toxic, harmful, or biased uh towards some of the topics.

s96
00:11:14.560 --> 00:11:26.920
And by running all outputs through toxicity classifier can give give you a scores uh how what is the risk of your output for being exposed to this uh toxic

s97
00:11:26.920 --> 00:11:29.400
uh output.

s98
00:11:29.400 --> 00:11:38.200
And this allows you to set some clear thresholds, and we always target to uh minimize that as much as we can.

s99
00:11:38.200 --> 00:11:41.880
And then finally, the jailbreak attempts.

s100
00:11:41.880 --> 00:11:52.120
And there are persistent attempts by users to bypass the safety guardrails and often um through the elaborate scenarios.

s101
00:11:52.120 --> 00:12:02.840
And tracking tracking patterns to indicate the system prompt overrides, you should design and become so it become the absolute blocker for you.

s102
00:12:02.840 --> 00:12:09.880
So, ideally, we should block all the 100% of the attempts of that.

s103
00:12:10.120 --> 00:12:25.760
Now, let's imagine that cost is checked, security is checked, but now you arrive to arguably the most crucial dimension and the quality monitor that at least the most crucial for our final users.

s104
00:12:25.760 --> 00:12:41.960
And even though your let's metrics showing that your service is up and running and running, your cost and safety tells you that your service is budget compliant and also secure,

s105
00:12:41.960 --> 00:12:49.200
but ultimately for any application, but more for generative application, it's whenever output is good.

s106
00:12:49.200 --> 00:12:58.480
But we all know that 200 okay response from your server sometimes can be a lie if the answer is hallucination.

s107
00:12:58.480 --> 00:13:04.640
And I'm sure many of you got some response from modern AI applications, right?

s108
00:13:04.640 --> 00:13:10.680
But like we know that it's can confidently by like but lie to you.

s109
00:13:10.680 --> 00:13:22.120
And now let's understand what type of metrics you can apply implement to your application right now to prevent it from being unsafe for your final user.

s110
00:13:22.120 --> 00:13:25.160
We recommend to focus on these five metrics.

s111
00:13:25.160 --> 00:13:27.480
One of them is hallucination rate.

s112
00:13:27.480 --> 00:13:35.600
Um, this measures the percentage of respondent contain claims unsupported by the grounded data.

s113
00:13:35.600 --> 00:13:45.600
Um, we need to mix that with the manual review as well as with automated fact checks to achieve the aggressive targets here, too.

s114
00:13:45.600 --> 00:13:56.560
The final The next one is the relevance scores, which helps you to understand if the response actually address the user question.

s115
00:13:56.560 --> 00:14:10.240
Um, that techniques like that you can use here is similarity using bird or embedding similarity that can score you um, how relevant is the final answer was to the user.

s116
00:14:10.240 --> 00:14:16.840
The next one is user satisfaction, which directly falls into your end user.

s117
00:14:16.840 --> 00:14:29.600
Uh, you must implement You can implement some mechanisms like thumbs up, thumbs down, or some of the stars ranking, and also NPS scores um, to capture the subjective data,

s118
00:14:29.600 --> 00:14:38.880
but also you should target that you get more than 80 85% of the positive feedback, at least.

s119
00:14:38.880 --> 00:14:53.720
Next one is the answer completeness, and this metric is often evaluated by LLM as a judge, answering whenever the response was fully or holistically addressed um, the user intent.

s120
00:14:53.720 --> 00:15:05.040
Um, and final one, the fifth one, um, if you are using rag, retrieval augmented generation system, you should always track the quality of your rag.

s121
00:15:05.040 --> 00:15:10.839
What's if all the documents that were fetched were relevant to the user query?

s122
00:15:10.839 --> 00:15:14.960
And here you have a lot of metrics and mechanisms how you can calculate that.

s123
00:15:14.960 --> 00:15:29.480
For example, top K accuracy or normalized discounted gains, and that is uh, essential to track because whenever like you contacts is explode is getting harder and harder to pick the most relevant one.

s124
00:15:29.480 --> 00:15:43.960
And by implementing this metrics, you can gain some understanding of like more dimensional view of the quality that necessary you need to optimize to improve your journey application.

s125
00:15:44.040 --> 00:15:53.840
Well, and now let's summarize um what we need to implement to make sure that our journey application is healthy, right?

s126
00:15:53.840 --> 00:15:58.000
We are not saying that we need to get rid of traditional understanding.

s127
00:15:58.000 --> 00:16:09.360
We always need to look into our metrics, lay let's latency error traffic and saturation, but also on top of that, we need to put another layer.

s128
00:16:09.360 --> 00:16:13.000
Layer of which is more relevant to generative applications.

s129
00:16:13.000 --> 00:16:18.440
And in this case, we talked about the cost, safety, and also quality.

s130
00:16:18.440 --> 00:16:31.720
And I know that for us as an engineer is becoming more and more critical right now to track it all not be don't need just build application and make sure that it's running in production,

s131
00:16:31.720 --> 00:16:36.280
but now we also need to care about additional layer of complexity.

s132
00:16:36.280 --> 00:16:44.880
And thankfully, Datadog has your back here since we are working in observability and monitoring space for for really long time.

s133
00:16:44.880 --> 00:17:00.760
So, we do have our agent observability uh product which allows you to quickly and easy understand if your AI application is healthy, if it's running, or there is any problem that you need to fix.

s134
00:17:00.760 --> 00:17:06.199
And we do have a lot of different way things that you can track inside of that.

s135
00:17:06.199 --> 00:17:12.079
And um if you want to hear more about that, please feel free to see me afterwards.

s136
00:17:12.079 --> 00:17:16.199
We also do have the booth here uh which you can check it out.

s137
00:17:16.199 --> 00:17:26.600
We do have a really good um offers for you if you want to just try it out and see how agent observability can help your application.

s138
00:17:26.600 --> 00:17:29.520
We have some QR codes for the free trials.

s139
00:17:29.520 --> 00:17:32.520
And with that, I want to thank everyone.

s140
00:17:32.520 --> 00:17:34.360
Again, my name is Marina Betzel.

s141
00:17:34.360 --> 00:17:39.480
If you want to contact me, please feel free to use this QR code.

s142
00:17:39.480 --> 00:17:42.960
And yeah, thank you for being here today.

s143
00:17:57.487 --> 00:17:59.487
[music]