WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=bMjlRrWjdT0

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/bMjlRrWjdT0.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.440 --> 00:00:15.440
Good afternoon everyone.

s3
00:00:15.440 --> 00:00:18.720
Thanks for uh coming for a post lunch uh talk.

s4
00:00:18.720 --> 00:00:20.560
Always appreciate that.

s5
00:00:20.560 --> 00:00:24.320
Um my name is Farup and here's my teammate Nachiket.

s6
00:00:24.320 --> 00:00:29.760
Uh we are uh here behalf of the Door Dash Genai platform team.

s7
00:00:29.760 --> 00:00:33.360
Um and we kind of wanted to share our eval journey.

s8
00:00:33.360 --> 00:00:46.800
Uh it started as uh uh you know eval is another engineering thing but then it slowly we realized it evolved into a cross functional effort and we kind of want to share our story here.

s9
00:00:46.800 --> 00:00:48.480
So what is this team?

s10
00:00:48.480 --> 00:00:50.879
This team is a gen platform team.

s11
00:00:50.879 --> 00:00:54.320
Uh we are a horizontal team that helps all other product teams.

s12
00:00:54.320 --> 00:00:59.760
So product teams at Door Dash build on top of the infrastructure and the primitives that we provide.

s13
00:00:59.760 --> 00:01:11.040
Um and we see our uh USP and the value that we provide is that we help product teams balance these three forces which is accuracy, latency and cost.

s14
00:01:11.040 --> 00:01:17.119
Um initially we applied this in terms of models but if you think about it it also applies to agents.

s15
00:01:17.119 --> 00:01:22.799
Um and the way we achieve this is we have uh primitives and building blocks.

s16
00:01:22.799 --> 00:01:30.880
Um so for example we have an LLM gateway where you can easily switch between different models uh and try the latest and greatest.

s17
00:01:30.880 --> 00:01:42.799
Uh we have an agent gateway where you can connect to tools uh and other agents uh and we help solve authentication uh agent identity and other things in a central place

s18
00:01:42.799 --> 00:01:45.280
uh which our security team can bless.

s19
00:01:45.280 --> 00:01:50.399
Um similarly we pair the LLM gateway with open weights models hosting.

s20
00:01:50.399 --> 00:01:53.840
Uh, of course cost is a number one concern these days.

s21
00:01:53.840 --> 00:02:00.880
Uh, and we uh kind of invested in open weights models uh and have seen significant impact uh already.

s22
00:02:00.880 --> 00:02:04.320
Um, and maybe we'll talk about that in a future conference.

s23
00:02:04.320 --> 00:02:09.840
Uh, the fourth pillar is eval and that's the part that we would want to share today.

s24
00:02:09.840 --> 00:02:20.319
Um when we started talking to product teams internally at Door Dash uh there were varying uh distinct needs across teams.

s25
00:02:20.319 --> 00:02:24.080
We had a consumer discovery and shopping assistant team.

s26
00:02:24.080 --> 00:02:31.599
Uh for those who attended Ragago talk earlier today uh you will uh see the need for session level quality judgments.

s27
00:02:31.599 --> 00:02:43.599
um uh then the personalization ML then you needed uh a way to scale up human judgment and with multi- aent systems we needed trajectory based evals now the question is

s28
00:02:43.599 --> 00:02:58.239
how do you cater to all these different needs under a common platform and uh as we spoke to these teams we realized like um we needed to empower the people who are the domain experts

s29
00:02:58.239 --> 00:03:13.519
and in our case that was strategy and operations folks it as product managers uh it was even labeling partners uh and not only engineers so we kind of started with like okay we have to be UI first and this was the guidance we had from Andy Fang our co-founder as well

s30
00:03:13.519 --> 00:03:28.239
um so we had UIs for non-engineers to contribute uh then we kind of evolved to also being API first so that engineers can also build and not be blocked on the central platform and they can build their own uh

s31
00:03:28.239 --> 00:03:40.000
uh uh systems uh and Then of course with the coding agents now we have become workflow first where we kind of empower SNO and PMs to also being able to uh navigate

s32
00:03:40.000 --> 00:03:44.159
uh the platform and uh run operations as well.

s33
00:03:44.159 --> 00:03:50.400
Um so with that context I'll hand it off to Nachig to talk about uh how we went about delivering this.

s34
00:03:50.400 --> 00:03:51.599
Cool.

s35
00:03:51.599 --> 00:03:52.480
Thanks Harup.

s36
00:03:52.480 --> 00:03:54.319
Um and thanks everyone for joining us.

s37
00:03:54.319 --> 00:03:58.799
I know France is playing right now and I promise you this will be better than that.

s38
00:03:58.799 --> 00:03:59.360
I'm kidding.

s39
00:03:59.360 --> 00:04:11.599
Um so as Faroo was saying uh Evals is not just an engineering harness it is a cross functional effort across different pillars across different uh teams uh that actually

s40
00:04:11.599 --> 00:04:18.639
helps us add all the domain specific knowledge into our uh into the quality of the AI itself.

s41
00:04:18.639 --> 00:04:27.199
So from your traces to your data sets uh from you know scoring mechanisms uh this is all basically a team sport.

s42
00:04:27.199 --> 00:04:32.720
we all have to play uh and help improve the quality of AI.

s43
00:04:33.520 --> 00:04:42.080
So going a little bit deeper into the same aspect uh we have different uh teams uh at Door Dash who help us actually improve the quality of AI.

s44
00:04:42.080 --> 00:04:48.320
So you're going to have your strategy and operations folks who are going to set priorities, set the quality bar that you want to aim for.

s45
00:04:48.320 --> 00:04:54.479
You're going to have your product people who are going to translate uh these requirements into rubrics workflows.

s46
00:04:54.479 --> 00:04:58.240
You're going to have your operations teams running uh annotations.

s47
00:04:58.240 --> 00:05:06.720
You're going to have your engineering teams like us uh providing APIs, telemetry, data sets, judges, all you know the the cool things.

s48
00:05:06.720 --> 00:05:19.600
Um and combining all these together is is what a recipe is for actually making sure that you are shipping quality AI products through an eval platform.

s49
00:05:19.600 --> 00:05:25.520
So we've tried to boil this down uh into sort of you know like a a continuous iteration loop.

s50
00:05:25.520 --> 00:05:39.280
Uh so right from tracing uh you know having a tracing solution viewing your sessions your traces to sampling them down you uh you know to a very small uh set that you actually want to look at

s51
00:05:39.280 --> 00:05:51.360
uh annotating these with the domain specific expertise that you bring in with the different teams I mentioned reviewing those uh then creating those golden data sets which are going to be you you know your

s52
00:05:51.360 --> 00:06:02.800
uh golden data sets and that that you want to measure or calibrate against uh and then of course like you know monitoring this over a period of time and then you know rinse and repeat uh go through the whole loop again.

s53
00:06:02.800 --> 00:06:16.240
So this is in our experience has been you know like a good sort of continuous loop uh for you know shipping quality AI at the plat on on the platform level uh we we have two surfaces

s54
00:06:16.240 --> 00:06:30.080
uh so we have the telemetry layer uh where we have all our traces our scores uh observations that is also sort of the plane where users are able to access these traces using an MCP using an SDK

s55
00:06:30.080 --> 00:06:37.360
uh using our APIs and then we have the workflow This is where a lot of our strat ops, our product teams operate on the platform.

s56
00:06:37.360 --> 00:06:39.840
So this is where all the annotation tasks are set.

s57
00:06:39.840 --> 00:06:47.600
Uh you know this is where they review their golden data sets, uh create their judges, calibrate their judges and so on.

s58
00:06:47.600 --> 00:06:56.000
So maybe today we'll go through you know these sort of four different uh modules or pillars of our platform uh step by step.

s59
00:06:56.000 --> 00:07:10.720
Uh so again first one uh tracing and sampling uh which is actually capturing what your agents what your LLMs are actually uh you know outputting for the lack of better words uh and actually viewing those.

s60
00:07:10.720 --> 00:07:19.520
Now in order to also power this uh whole platform we have I think as far mentioned we have gone in an API first uh approach.

s61
00:07:19.520 --> 00:07:28.400
Uh what that has allowed us to do is have these table APIs that actually uh you know and then you know build UIs uh on top of that.

s62
00:07:28.400 --> 00:07:35.360
Uh so all our scores our data sets uh these are all powered by very stable APIs uh that our team owns.

s63
00:07:35.360 --> 00:07:44.080
Uh so all your API access uh including you know like an SDK access is basically powered by this single uh plane.

s64
00:07:44.080 --> 00:07:50.800
Um again going back uh and you know like just refreshing your memory.

s65
00:07:50.800 --> 00:07:56.879
Uh step one capture your traces uh capture your sessions uh measure your scores.

s66
00:07:56.879 --> 00:08:09.840
Uh then you want to start uh almost you know like adding all your judgment your context your domain knowledge uh and then calibrating your judges is what we have seen as the whole

s67
00:08:09.840 --> 00:08:12.479
uh life cycle.

s68
00:08:13.120 --> 00:08:15.199
Step two is on the annotation side.

s69
00:08:15.199 --> 00:08:27.919
Uh so you obviously are capturing a lot of your uh agentic behavior, your sessions, your traces, but you actually want to see what are some places where things went well and what are some places where things did not go well.

s70
00:08:27.919 --> 00:08:38.240
This is where you can actually titrate your your you know and actually look in inside what's actually happening uh at the session level and annotate these data sets.

s71
00:08:38.240 --> 00:08:42.479
Um and as Surup mentioned, we have a lot of use cases.

s72
00:08:42.479 --> 00:08:57.120
we have we we talked to multiple different teams who have uh various uh ways of annotating uh their data sets uh and it's it's almost hard for a platform team to you know build like a UI specific

s73
00:08:57.120 --> 00:09:14.160
uh for each use case uh and and you know to give you an example uh it's usually going to be an annotator who's going to annotate these data sets so the platform team is you know in charge of the APIs we have a strategy and of person who's actually deciding what to annotate and then you have an

s74
00:09:14.160 --> 00:09:17.600
annotator who's actually going to annotate uh your data set.

s75
00:09:17.600 --> 00:09:25.839
So we took this approach uh everybody uh has uh you know access to coding agents uh and we actually doubled down on that API first approach.

s76
00:09:25.839 --> 00:09:37.360
So because we had these APIs we were actually uh able to enable our statops teams to use something like a codeex or a claw code and v code their own annotation UIs.

s77
00:09:37.360 --> 00:09:40.399
Uh so we had different use cases.

s78
00:09:40.399 --> 00:09:43.600
Uh I think we had a talk from Ragav before.

s79
00:09:43.600 --> 00:09:45.760
Uh we had image annotation use cases.

s80
00:09:45.760 --> 00:09:48.800
We had some uh you know manual testing use cases.

s81
00:09:48.800 --> 00:09:53.040
What stood out to us was the underlying patterns were similar.

s82
00:09:53.040 --> 00:10:00.959
So if we are are API first uh we can actually enable our our our partners to simply v code these UIs for annotation.

s83
00:10:00.959 --> 00:10:13.279
So it's it's like a very simple example then you know of of a vibe coded UI looks pretty clean does the job uh and you get you know the annotation that you eat this is basically like a menu from a restaurant

s84
00:10:13.279 --> 00:10:24.399
uh it's it's you know nothing crazy uh but the point I want to make here is that what helped us was to give this workflow in the hands of the operators

s85
00:10:24.399 --> 00:10:28.959
so that they can actually build their own vcoded annotation UIs.

s86
00:10:28.959 --> 00:10:37.360
Uh so moving on once you have these annotation UIs you obviously want to you know calibrate your your your judge prompts you obviously have some LM as a judge

s87
00:10:37.360 --> 00:10:48.240
uh metric that you're tracking you want to now start improving that with these golden data sets u in order to do that uh you know we have a pretty simple process

s88
00:10:48.240 --> 00:10:57.839
uh you you're going to start with you know some judge prompt take a look at you know what exactly do you want to measure from the output uh g you know have have us have something simple

s89
00:10:57.839 --> 00:11:06.560
you're going to have your baseline scores uh where you're going to simply run those LLM judges on your traces and then you're going to have that optimization loop.

s90
00:11:06.560 --> 00:11:13.120
Uh so we use uh the JPEA library which is a pretty commonly used library out there for prompt optimization.

s91
00:11:13.120 --> 00:11:23.760
Uh and once you know the the iteration loop is complete uh our partner teams are happy they're going to then elevate that judge prompt as their LLM as a judge.

s92
00:11:23.760 --> 00:11:34.240
Now even while doing that uh LLM as a judge as a concept the whole prompt calibration concept might be uh straightforward to a lot of folks but it is still like a pretty new and evolving

s93
00:11:34.240 --> 00:11:35.120
field.

s94
00:11:35.120 --> 00:11:40.640
Uh and what we wanted to do was really reduce the friction of back and forth with an engineering team.

s95
00:11:40.640 --> 00:11:47.120
So we tried to really remove all the complicated logic and make this into a self-s serve UI.

s96
00:11:47.120 --> 00:11:49.920
So the screenshot that you actually see is what actually exists.

s97
00:11:49.920 --> 00:11:54.880
uh so uh you know like a product manager or an operator is going to come to our UI.

s98
00:11:54.880 --> 00:12:00.800
They're going to set some of these configs uh on the platform and then actually run the calibration loop themselves.

s99
00:12:00.800 --> 00:12:12.320
So they don't have to worry about the different settings that they need to worry about what are the different uh you know tweaks that they need to do and they can actually like you know run a calibration loop using any model of their choice.

s100
00:12:12.320 --> 00:12:19.040
I think in this example I have Gemini they can use run it using uh you know any of the claude or the openi models too.

s101
00:12:19.040 --> 00:12:22.560
The other important piece was actually uh making this reviewable.

s102
00:12:22.560 --> 00:12:29.920
Uh you know again a lot of this uh is a closed box where you can't really it's hard to see what's actually happening.

s103
00:12:29.920 --> 00:12:36.320
Uh so the second piece that we built was actually giving them vis visualization and visibility into what's actually happening.

s104
00:12:36.320 --> 00:12:44.959
So on the left you can see we and this is like one of the good examples where we saw like a significant amount of improvement in the judge prompt.

s105
00:12:44.959 --> 00:12:55.600
uh and we actually show the you know the the previous the original system prompt and the calibrated prompt to our partners so that they are also able to gain that trust

s106
00:12:55.600 --> 00:12:59.920
uh why as as as we build this [clears throat]

s107
00:12:59.920 --> 00:13:15.760
yeah just want to add to that is this enables different configurations in different teams in some teams you have seen the strategy and operations folks own the prompt uh you have seen some teams where the product manager owns the prompt you have seen some teams where engineering owns the prompt so this gives gives the flexibility

s108
00:13:15.760 --> 00:13:19.040
for teams to design and evolve because we are all learning.

s109
00:13:19.040 --> 00:13:23.760
So the even the org uh design is improving and we are enabling that.

s110
00:13:23.760 --> 00:13:25.200
Yeah, that that's a good point.

s111
00:13:25.200 --> 00:13:35.760
I think the overall idea was to you know build something which is as self-s served as possible so that uh you know people aren't always necessarily blocked by our team helping them out.

s112
00:13:35.760 --> 00:13:38.880
Um and then finally you know uh the quality loop in practice.

s113
00:13:38.880 --> 00:13:44.639
you know as we've been going through this exercise we've seen a lot of improvements happening to our product as well.

s114
00:13:44.639 --> 00:13:56.720
So you know for example we sort mentioned we started with the UIs we are you know now API and workflow first uh we're trying to reuse a lot of the existing infrastructure that already existed at Door Dash

s115
00:13:56.720 --> 00:13:59.600
uh and that's helped us uh get a long way.

s116
00:13:59.600 --> 00:14:03.279
Now some of uh we we we've seen obviously like you know really good results.

s117
00:14:03.279 --> 00:14:18.160
I think a very good result that we we do like to call out is we actually did see a lot of reduction in the spend uh at per annotation cost as you as you all can imagine we do have you know thousands of rows that need to get annotated

s118
00:14:18.160 --> 00:14:31.839
every week uh and it can get pretty expensive at doash scale uh and having this selfserve uh annotation platform really helped us reduce increase the velocity and reduce the cost that we were actually spending

s119
00:14:31.839 --> 00:14:33.120
with these annotators.

s120
00:14:33.120 --> 00:14:35.600
to to annotate the data for us.

s121
00:14:35.600 --> 00:14:38.720
Uh obviously uh this resulted in faster loops.

s122
00:14:38.720 --> 00:14:40.880
Uh teams were able to iterate faster.

s123
00:14:40.880 --> 00:14:47.279
They were able to uh you know calibrate their own judges in a completely self-s served way.

s124
00:14:47.279 --> 00:14:53.680
Uh and thus it has resulted us in in in moving with a very very high velocity.

s125
00:14:54.240 --> 00:15:06.399
So uh finally I just wanted to you know quickly touch on this slide again uh the eight steps you know continuous loop uh which is you know you you have your traces you want to look at your traces your sessions

s126
00:15:06.399 --> 00:15:14.959
you want to sample it down to a size which is which you are comfortable with uh you want to start annotating your data sets you really want to start

s127
00:15:14.959 --> 00:15:26.320
uh making the data better with the human knowledge that exists and the domain knowledge that exists and then calibrate your workflows was calibrate your agents, calibrate your LLM judges

s128
00:15:26.320 --> 00:15:35.600
with this golden data set and then repeat this whole cycle uh over a period of time to you know to ship reliably and ship with high quality.

s129
00:15:35.600 --> 00:15:37.760
Um yeah, we have 4 minutes left.

s130
00:15:37.760 --> 00:15:39.360
Thank you once again.

s131
00:15:39.360 --> 00:15:40.880
I think that was the last slide.

s132
00:15:40.880 --> 00:15:48.959
Uh thanks for attending and if there's any questions, we'd be happy to hang out after the talk or even happy to answer them now.

s133
00:15:51.070 --> 00:15:53.070
[applause]