{
  "video": {
    "id": "bMjlRrWjdT0",
    "title": "AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash",
    "duration": 971,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer"
  },
  "analysis": {
    "video_id": "bMjlRrWjdT0",
    "title": "AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash",
    "one_liner": "DoorDash's GenAI platform team explains how their eval platform stopped being an engineering harness and became a cross-functional workflow — with API-first primitives that let strategy-and-ops and PMs vibe-code their own annotation UIs and self-serve LLM-judge calibration.",
    "summary": "Nachiket Paranjape and Swaroop Chitlur Haridas describe the eval pillar of DoorDash's horizontal GenAI platform, which exists to help product teams balance accuracy, latency and cost alongside an LLM gateway, an agent gateway and open-weights model hosting. Their central argument is that evals are a team sport: strat-ops sets the quality bar, PMs translate it into rubrics, ops teams annotate, and engineering supplies APIs, telemetry, datasets and judges. The platform evolved UI-first (on co-founder Andy Fang's guidance) → API-first → workflow-first, and because everything sits on stable APIs, non-engineers can vibe-code their own annotation UIs with Codex or Claude Code instead of queueing behind the platform team. They also shipped a self-serve judge-calibration UI using the GEPA prompt-optimization library, with before/after prompt diffs shown to build partner trust.",
    "key_points": [
      "The GenAI platform team is horizontal — its USP is helping product teams balance three forces: accuracy, latency and cost. Four pillars: LLM gateway (easy model switching), agent gateway (tool/agent connection plus auth and agent identity in one security-blessed place), open-weights model hosting (cost-driven, 'significant impact already'), and eval.",
      "Different internal teams needed different eval shapes: consumer discovery/shopping assistant needed session-level quality judgments, personalization ML needed a way to scale up human judgment, and multi-agent systems needed trajectory-based evals — all under one platform.",
      "Platform evolution in three stages: UI-first (guidance from co-founder Andy Fang, so non-engineers could contribute) → API-first (so engineers aren't blocked on the central platform) → workflow-first (so strat-ops and PMs can navigate and run operations themselves).",
      "The continuous loop they run: trace → sample down to a small set → annotate with domain expertise → review → build golden datasets → calibrate judges/agents against them → monitor over time → repeat.",
      "Two platform surfaces: a telemetry layer (traces, scores, observations, reachable via MCP, SDK and APIs) and a workflow layer where strat-ops and product set annotation tasks, review golden datasets, and create and calibrate judges.",
      "Rather than build a bespoke annotation UI per use case, they leaned on the stable APIs and had strat-ops partners vibe-code their own annotation UIs with Codex or Claude Code — image annotation, manual testing and more. The underlying patterns were similar; one demoed example annotates a restaurant menu.",
      "Judge calibration: start from a judge prompt, get baseline scores by running LLM judges over traces, then run an optimization loop using the GEPA library; when partners are happy they elevate that prompt as their LLM-as-a-judge. It is exposed as a self-serve UI where a PM or operator sets configs and runs calibration with any model they choose (Gemini in the demo; Claude or OpenAI models also work).",
      "Calibration results are made reviewable — the UI shows the original system prompt next to the calibrated prompt so partners can see what changed and trust it, rather than treating it as a closed box.",
      "Reported outcomes: reduced per-annotation spend (they annotate thousands of rows every week, expensive at DoorDash scale), higher velocity, and teams calibrating their own judges without engineering back-and-forth.",
      "Prompt ownership varies by team by design — in some teams strat-ops owns the judge prompt, in others the PM, in others engineering; the platform's flexibility lets org design evolve too."
    ],
    "takeaways": [
      "Treat evals as cross-functional infrastructure, not an engineering harness: give strat-ops, PMs, ops annotators and labeling partners first-class ways to inject domain knowledge into traces, datasets and scoring.",
      "Build the eval platform API-first with stable APIs for scores and datasets, then let partner teams vibe-code their own annotation UIs with coding agents instead of asking the platform team for a bespoke UI per use case.",
      "Make LLM-judge prompt calibration self-serve — wrap the optimization loop (they use GEPA) in a UI with model choice, so non-engineers can run it without a back-and-forth with engineering.",
      "Show the before/after prompt diff and calibration visualizations; the closed-box nature of judge optimization is a trust problem, and visibility is what earns partner buy-in.",
      "Run the loop continuously rather than as a one-off: trace, sample down to a size you're comfortable with, annotate, build golden datasets, calibrate, monitor, repeat.",
      "Don't fix who owns the judge prompt — let each team decide whether strat-ops, the PM or engineering owns it, since the org design is still being learned."
    ],
    "topics": [
      "evals",
      "llm-as-a-judge",
      "annotation",
      "agents",
      "observability",
      "prompt-optimization",
      "platform-engineering",
      "vibe-coding"
    ],
    "tools": [
      "DoorDash",
      "GEPA",
      "Codex",
      "Claude Code",
      "Gemini",
      "OpenAI",
      "Claude",
      "MCP"
    ],
    "quotes": [
      {
        "text": "Evals is not just an engineering harness it is a cross functional effort across different pillars across different teams... this is all basically a team sport. we all have to play and help improve the quality of AI.",
        "at": "04:01",
        "url": "https://www.youtube.com/watch?v=bMjlRrWjdT0&t=241s"
      },
      {
        "text": "because we had these APIs we were actually able to enable our statops teams to use something like a codex or a claude code and v code their own annotation UIs",
        "at": "09:23",
        "url": "https://www.youtube.com/watch?v=bMjlRrWjdT0&t=563s"
      },
      {
        "text": "what helped us was to give this workflow in the hands of the operators so that they can actually build their own vcoded annotation UIs",
        "at": "10:18",
        "url": "https://www.youtube.com/watch?v=bMjlRrWjdT0&t=618s"
      },
      {
        "text": "in some teams you have seen the strategy and operations folks own the prompt, you have seen some teams where the product manager owns the prompt, you have seen some teams where engineering owns the prompt... so the even the org design is improving and we are enabling that.",
        "at": "13:04",
        "url": "https://www.youtube.com/watch?v=bMjlRrWjdT0&t=784s"
      }
    ],
    "words": 3192
  },
  "summary_url": "/#bMjlRrWjdT0",
  "transcript": {
    "html": "/transcripts/bMjlRrWjdT0.html",
    "txt": "/transcripts/bMjlRrWjdT0.txt",
    "vtt": "/transcripts/bMjlRrWjdT0.vtt"
  }
}