AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash

AI Engineer · 16 min · 133 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:01[music]
  2. 00:13Good afternoon everyone.
  3. 00:15Thanks for uh coming for a post lunch uh talk.
  4. 00:18Always appreciate that.
  5. 00:20Um my name is Farup and here's my teammate Nachiket.
  6. 00:24Uh we are uh here behalf of the Door Dash Genai platform team.
  7. 00:29Um and we kind of wanted to share our eval journey.
  8. 00:33Uh it started as uh uh you know eval is another engineering thing but then it slowly we realized it evolved into a cross functional effort and we kind of want to share our story here.
  9. 00:46So what is this team?
  10. 00:48This team is a gen platform team.
  11. 00:50Uh we are a horizontal team that helps all other product teams.
  12. 00:54So product teams at Door Dash build on top of the infrastructure and the primitives that we provide.
  13. 00:59Um and we see our uh USP and the value that we provide is that we help product teams balance these three forces which is accuracy, latency and cost.
  14. 01:11Um initially we applied this in terms of models but if you think about it it also applies to agents.
  15. 01:17Um and the way we achieve this is we have uh primitives and building blocks.
  16. 01:22Um so for example we have an LLM gateway where you can easily switch between different models uh and try the latest and greatest.
  17. 01:30Uh we have an agent gateway where you can connect to tools uh and other agents uh and we help solve authentication uh agent identity and other things in a central place
  18. 01:42uh which our security team can bless.
  19. 01:45Um similarly we pair the LLM gateway with open weights models hosting.
  20. 01:50Uh, of course cost is a number one concern these days.
  21. 01:53Uh, and we uh kind of invested in open weights models uh and have seen significant impact uh already.
  22. 02:00Um, and maybe we'll talk about that in a future conference.
  23. 02:04Uh, the fourth pillar is eval and that's the part that we would want to share today.
  24. 02:09Um when we started talking to product teams internally at Door Dash uh there were varying uh distinct needs across teams.
  25. 02:20We had a consumer discovery and shopping assistant team.
  26. 02:24Uh for those who attended Ragago talk earlier today uh you will uh see the need for session level quality judgments.
  27. 02:31um uh then the personalization ML then you needed uh a way to scale up human judgment and with multi- aent systems we needed trajectory based evals now the question is
  28. 02:43how do you cater to all these different needs under a common platform and uh as we spoke to these teams we realized like um we needed to empower the people who are the domain experts
  29. 02:58and in our case that was strategy and operations folks it as product managers uh it was even labeling partners uh and not only engineers so we kind of started with like okay we have to be UI first and this was the guidance we had from Andy Fang our co-founder as well
  30. 03:13um so we had UIs for non-engineers to contribute uh then we kind of evolved to also being API first so that engineers can also build and not be blocked on the central platform and they can build their own uh
  31. 03:28uh uh systems uh and Then of course with the coding agents now we have become workflow first where we kind of empower SNO and PMs to also being able to uh navigate
  32. 03:40uh the platform and uh run operations as well.
  33. 03:44Um so with that context I'll hand it off to Nachig to talk about uh how we went about delivering this.
  34. 03:50Cool.
  35. 03:51Thanks Harup.
  36. 03:52Um and thanks everyone for joining us.
  37. 03:54I know France is playing right now and I promise you this will be better than that.
  38. 03:58I'm kidding.
  39. 03:59Um so as Faroo was saying uh Evals is not just an engineering harness it is a cross functional effort across different pillars across different uh teams uh that actually
  40. 04:11helps us add all the domain specific knowledge into our uh into the quality of the AI itself.
  41. 04:18So from your traces to your data sets uh from you know scoring mechanisms uh this is all basically a team sport.
  42. 04:27we all have to play uh and help improve the quality of AI.
  43. 04:33So going a little bit deeper into the same aspect uh we have different uh teams uh at Door Dash who help us actually improve the quality of AI.
  44. 04:42So you're going to have your strategy and operations folks who are going to set priorities, set the quality bar that you want to aim for.
  45. 04:48You're going to have your product people who are going to translate uh these requirements into rubrics workflows.
  46. 04:54You're going to have your operations teams running uh annotations.
  47. 04:58You're going to have your engineering teams like us uh providing APIs, telemetry, data sets, judges, all you know the the cool things.
  48. 05:06Um and combining all these together is is what a recipe is for actually making sure that you are shipping quality AI products through an eval platform.
  49. 05:19So we've tried to boil this down uh into sort of you know like a a continuous iteration loop.
  50. 05:25Uh so right from tracing uh you know having a tracing solution viewing your sessions your traces to sampling them down you uh you know to a very small uh set that you actually want to look at
  51. 05:39uh annotating these with the domain specific expertise that you bring in with the different teams I mentioned reviewing those uh then creating those golden data sets which are going to be you you know your
  52. 05:51uh golden data sets and that that you want to measure or calibrate against uh and then of course like you know monitoring this over a period of time and then you know rinse and repeat uh go through the whole loop again.
  53. 06:02So this is in our experience has been you know like a good sort of continuous loop uh for you know shipping quality AI at the plat on on the platform level uh we we have two surfaces
  54. 06:16uh so we have the telemetry layer uh where we have all our traces our scores uh observations that is also sort of the plane where users are able to access these traces using an MCP using an SDK
  55. 06:30uh using our APIs and then we have the workflow This is where a lot of our strat ops, our product teams operate on the platform.
  56. 06:37So this is where all the annotation tasks are set.
  57. 06:39Uh you know this is where they review their golden data sets, uh create their judges, calibrate their judges and so on.
  58. 06:47So maybe today we'll go through you know these sort of four different uh modules or pillars of our platform uh step by step.
  59. 06:56Uh so again first one uh tracing and sampling uh which is actually capturing what your agents what your LLMs are actually uh you know outputting for the lack of better words uh and actually viewing those.
  60. 07:10Now in order to also power this uh whole platform we have I think as far mentioned we have gone in an API first uh approach.
  61. 07:19Uh what that has allowed us to do is have these table APIs that actually uh you know and then you know build UIs uh on top of that.
  62. 07:28Uh so all our scores our data sets uh these are all powered by very stable APIs uh that our team owns.
  63. 07:35Uh so all your API access uh including you know like an SDK access is basically powered by this single uh plane.
  64. 07:44Um again going back uh and you know like just refreshing your memory.
  65. 07:50Uh step one capture your traces uh capture your sessions uh measure your scores.
  66. 07:56Uh then you want to start uh almost you know like adding all your judgment your context your domain knowledge uh and then calibrating your judges is what we have seen as the whole
  67. 08:09uh life cycle.
  68. 08:13Step two is on the annotation side.
  69. 08:15Uh so you obviously are capturing a lot of your uh agentic behavior, your sessions, your traces, but you actually want to see what are some places where things went well and what are some places where things did not go well.
  70. 08:27This is where you can actually titrate your your you know and actually look in inside what's actually happening uh at the session level and annotate these data sets.
  71. 08:38Um and as Surup mentioned, we have a lot of use cases.
  72. 08:42we have we we talked to multiple different teams who have uh various uh ways of annotating uh their data sets uh and it's it's almost hard for a platform team to you know build like a UI specific
  73. 08:57uh for each use case uh and and you know to give you an example uh it's usually going to be an annotator who's going to annotate these data sets so the platform team is you know in charge of the APIs we have a strategy and of person who's actually deciding what to annotate and then you have an
  74. 09:14annotator who's actually going to annotate uh your data set.
  75. 09:17So we took this approach uh everybody uh has uh you know access to coding agents uh and we actually doubled down on that API first approach.
  76. 09:25So because we had these APIs we were actually uh able to enable our statops teams to use something like a codeex or a claw code and v code their own annotation UIs.
  77. 09:37Uh so we had different use cases.
  78. 09:40Uh I think we had a talk from Ragav before.
  79. 09:43Uh we had image annotation use cases.
  80. 09:45We had some uh you know manual testing use cases.
  81. 09:48What stood out to us was the underlying patterns were similar.
  82. 09:53So if we are are API first uh we can actually enable our our our partners to simply v code these UIs for annotation.
  83. 10:00So it's it's like a very simple example then you know of of a vibe coded UI looks pretty clean does the job uh and you get you know the annotation that you eat this is basically like a menu from a restaurant
  84. 10:13uh it's it's you know nothing crazy uh but the point I want to make here is that what helped us was to give this workflow in the hands of the operators
  85. 10:24so that they can actually build their own vcoded annotation UIs.
  86. 10:28Uh so moving on once you have these annotation UIs you obviously want to you know calibrate your your your judge prompts you obviously have some LM as a judge
  87. 10:37uh metric that you're tracking you want to now start improving that with these golden data sets u in order to do that uh you know we have a pretty simple process
  88. 10:48uh you you're going to start with you know some judge prompt take a look at you know what exactly do you want to measure from the output uh g you know have have us have something simple
  89. 10:57you're going to have your baseline scores uh where you're going to simply run those LLM judges on your traces and then you're going to have that optimization loop.
  90. 11:06Uh so we use uh the JPEA library which is a pretty commonly used library out there for prompt optimization.
  91. 11:13Uh and once you know the the iteration loop is complete uh our partner teams are happy they're going to then elevate that judge prompt as their LLM as a judge.
  92. 11:23Now even while doing that uh LLM as a judge as a concept the whole prompt calibration concept might be uh straightforward to a lot of folks but it is still like a pretty new and evolving
  93. 11:34field.
  94. 11:35Uh and what we wanted to do was really reduce the friction of back and forth with an engineering team.
  95. 11:40So we tried to really remove all the complicated logic and make this into a self-s serve UI.
  96. 11:47So the screenshot that you actually see is what actually exists.
  97. 11:49uh so uh you know like a product manager or an operator is going to come to our UI.
  98. 11:54They're going to set some of these configs uh on the platform and then actually run the calibration loop themselves.
  99. 12:00So they don't have to worry about the different settings that they need to worry about what are the different uh you know tweaks that they need to do and they can actually like you know run a calibration loop using any model of their choice.
  100. 12:12I think in this example I have Gemini they can use run it using uh you know any of the claude or the openi models too.
  101. 12:19The other important piece was actually uh making this reviewable.
  102. 12:22Uh you know again a lot of this uh is a closed box where you can't really it's hard to see what's actually happening.
  103. 12:29Uh so the second piece that we built was actually giving them vis visualization and visibility into what's actually happening.
  104. 12:36So on the left you can see we and this is like one of the good examples where we saw like a significant amount of improvement in the judge prompt.
  105. 12:44uh and we actually show the you know the the previous the original system prompt and the calibrated prompt to our partners so that they are also able to gain that trust
  106. 12:55uh why as as as we build this [clears throat]
  107. 12:59yeah just want to add to that is this enables different configurations in different teams in some teams you have seen the strategy and operations folks own the prompt uh you have seen some teams where the product manager owns the prompt you have seen some teams where engineering owns the prompt so this gives gives the flexibility
  108. 13:15for teams to design and evolve because we are all learning.
  109. 13:19So the even the org uh design is improving and we are enabling that.
  110. 13:23Yeah, that that's a good point.
  111. 13:25I think the overall idea was to you know build something which is as self-s served as possible so that uh you know people aren't always necessarily blocked by our team helping them out.
  112. 13:35Um and then finally you know uh the quality loop in practice.
  113. 13:38you know as we've been going through this exercise we've seen a lot of improvements happening to our product as well.
  114. 13:44So you know for example we sort mentioned we started with the UIs we are you know now API and workflow first uh we're trying to reuse a lot of the existing infrastructure that already existed at Door Dash
  115. 13:56uh and that's helped us uh get a long way.
  116. 13:59Now some of uh we we we've seen obviously like you know really good results.
  117. 14:03I think a very good result that we we do like to call out is we actually did see a lot of reduction in the spend uh at per annotation cost as you as you all can imagine we do have you know thousands of rows that need to get annotated
  118. 14:18every week uh and it can get pretty expensive at doash scale uh and having this selfserve uh annotation platform really helped us reduce increase the velocity and reduce the cost that we were actually spending
  119. 14:31with these annotators.
  120. 14:33to to annotate the data for us.
  121. 14:35Uh obviously uh this resulted in faster loops.
  122. 14:38Uh teams were able to iterate faster.
  123. 14:40They were able to uh you know calibrate their own judges in a completely self-s served way.
  124. 14:47Uh and thus it has resulted us in in in moving with a very very high velocity.
  125. 14:54So uh finally I just wanted to you know quickly touch on this slide again uh the eight steps you know continuous loop uh which is you know you you have your traces you want to look at your traces your sessions
  126. 15:06you want to sample it down to a size which is which you are comfortable with uh you want to start annotating your data sets you really want to start
  127. 15:14uh making the data better with the human knowledge that exists and the domain knowledge that exists and then calibrate your workflows was calibrate your agents, calibrate your LLM judges
  128. 15:26with this golden data set and then repeat this whole cycle uh over a period of time to you know to ship reliably and ship with high quality.
  129. 15:35Um yeah, we have 4 minutes left.
  130. 15:37Thank you once again.
  131. 15:39I think that was the last slide.
  132. 15:40Uh thanks for attending and if there's any questions, we'd be happy to hang out after the talk or even happy to answer them now.
  133. 15:51[applause]