{
  "video": {
    "id": "3z2uT5aDx_Y",
    "title": "Build Evals That Actually Matter - Nick Ung & Akshay Sharma, Lyft",
    "duration": 2265,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "Online"
  },
  "analysis": {
    "video_id": "3z2uT5aDx_Y",
    "title": "Build Evals That Actually Matter - Nick Ung & Akshay Sharma, Lyft",
    "one_liner": "Lyft's customer-support AI team explains how they made evals bite: a TauBench-style offline simulator whose user LLM was fine-tuned on real (rude, impatient) Lyft user verbatim until scores fell, plus binary, domain-expert-written LLM judges validated like classifiers with precision and recall.",
    "summary": "Nick Ung and Akshay Sharma describe the end-to-end eval pipeline behind Lyft's multi-agent customer support system: offline multi-turn simulation as a launch gate before production, then tracing, online graders and human error analysis feeding back into the agent. They argue evals fail for three reasons — the score gates nothing, the LLM judge is too generic to be actionable, and nobody owns regressions — and show their fixes: a fine-tuned user simulator that made their first 90%+ pass rate collapse to something honest, task-level pass/fail rubrics built with domain experts, and judges validated against ~100 hand labels with precision/recall. They close on continual learning (model, context, harness) and the config-driven YAML eval harness they are building to replace scattered notebook scripts.",
    "key_points": [
      "The eval system is split into a development phase (context management, RAG, tools, agentic graph, system prompt) ending in an offline launch gate, and a production phase with tracing, online graders and a human-in-the-loop error analysis pipeline — the point being not to use live users as test data.",
      "Three reasons evals fail: the grader isn't meaningfully gating anything ('an LM as a judge just floating out there'), the judge is noisy and too generic so nobody believes it or finds it actionable, and there's no clear mechanism or owner to catch and act on production regressions.",
      "The offline simulator takes inspiration from TauBench (Sierra AI): LangGraph agents talk to a user LLM defined by intent, a user data point / world state (e.g. a luxury driver of several years), and a persona; multi-turn trajectories are graded by an LLM judge plus deterministic code assertions (e.g. did the agent actually grant the concession; compare tool calls against expected outcome).",
      "Their first pass used a frontier-lab model to role-play Lyft users — it was trained to be a helpful assistant, produced patient, complete, overly nice verbatim, and yielded a 90%+ pass rate they judged too good to be true; real support verbatim comes from impatient, already-frustrated users who won't explain their issue.",
      "Fix: fine-tune the user LLM on real Lyft user verbatim and ground it in defined personas (the bypasser who just wants a human agent, the refund seeker, the AI skeptic). Eval scores dropped — which is the goal, and matches Microsoft's user-LLM paper, where they fine-tune until the evaluation score goes down.",
      "Pre-built DeepEval metrics (tool usage appropriateness, response helpfulness, conversation naturalness, completeness, toxicity, bias, fairness, conciseness) are fine as a baseline but not as core metrics — a response helpfulness of 0.5 tells you nothing to do. Replace with binary task success/failure rubrics co-written with domain experts, e.g. their 'education rubric' which fails the agent both for educating too many times when it should have escalated and for escalating too soon.",
      "Validate the judge as a binary classifier: hand-label ~100 examples pass/fail, split into train/dev/test (different proportions than ML, since you're informing a prompt not training weights — few-shot examples from train, iterate the prompt on dev, validate on test for overfit), then score precision and recall against human ground truth.",
      "Criteria drift is ignored: you discover your evaluation criteria by looking at data and grading outputs, so evals must be co-developed with the model rather than defined up front. Report alignment rates with confidence intervals — an 84% vs 88% comparison over 50 traces cannot tell you the gain is real.",
      "Tracing (LangSmith, LangFuse) captures full graph execution — which nodes ran, what the LLM saw, which tools were called, token usage, per-call latency — and annotation queues give domain experts a UI to label without reading raw JSON, feeding both offline datasets and judge precision/recall.",
      "Next: a config-driven YAML eval harness with primitives (task, datasets, personas, LLM adapter, evaluator) that analysts and data scientists can contribute to, with parallelism for tens of thousands of examples, runnable locally, on prompt tweaks, at pre-commit and in CI/CD; plus post-training and reward modeling for RL from accumulated real user signals."
    ],
    "takeaways": [
      "Make every score gate something. An LLM judge whose output nobody uses as a launch or regression gate is not an eval — and name an owner for acting on regressions.",
      "Stop letting a frontier model role-play your users politely. Fine-tune the user simulator on real user verbatim and ground it in personas; expect and accept your pass rate to fall, because an easy eval gives no production insight.",
      "Replace generic prebuilt scores with binary pass/fail rubrics written with domain experts and tied to business outcomes, then validate each judge like a classifier: ~100 hand labels, train/dev/test split, precision and recall.",
      "Build offline datasets from sampled production traffic and mutate them for golden paths and edge cases — don't prompt an LLM for 50 test queries and call it representative.",
      "Run the error analysis loop continuously on a weekly or bi-weekly cadence (raw traces → pinpoint failure modes → keep only metrics that change a decision → fresh premise → repeat), not as a one-off audit, and attach confidence intervals to any number driving a shipping decision or leadership report."
    ],
    "topics": [
      "evals",
      "agents",
      "llm-as-judge",
      "customer-support",
      "simulation",
      "tracing",
      "fine-tuning",
      "error-analysis"
    ],
    "tools": [
      "Lyft",
      "TauBench",
      "Sierra AI",
      "LangGraph",
      "LangSmith",
      "LangFuse",
      "DeepEval",
      "Microsoft user LLM paper"
    ],
    "quotes": [
      {
        "text": "we don't want to use our live user as, you know, test data for our AI agents",
        "at": "04:43",
        "url": "https://www.youtube.com/watch?v=3z2uT5aDx_Y&t=283s"
      },
      {
        "text": "our LM user sounds almost too nice",
        "at": "13:58",
        "url": "https://www.youtube.com/watch?v=3z2uT5aDx_Y&t=838s"
      },
      {
        "text": "If you have an eval that's too easy, that doesn't give you any real production insights into how your AI agent is actually going to perform.",
        "at": "16:05",
        "url": "https://www.youtube.com/watch?v=3z2uT5aDx_Y&t=965s"
      },
      {
        "text": "if the metrics are just scores, we don't know what to do with them",
        "at": "19:15",
        "url": "https://www.youtube.com/watch?v=3z2uT5aDx_Y&t=1155s"
      },
      {
        "text": "Because every score needs an interval.",
        "at": "25:40",
        "url": "https://www.youtube.com/watch?v=3z2uT5aDx_Y&t=1540s"
      }
    ],
    "words": 6197
  },
  "summary_url": "/#3z2uT5aDx_Y",
  "transcript": {
    "html": "/transcripts/3z2uT5aDx_Y.html",
    "txt": "/transcripts/3z2uT5aDx_Y.txt",
    "vtt": "/transcripts/3z2uT5aDx_Y.vtt"
  }
}