{
  "video": {
    "id": "KMR_RBoCa4M",
    "title": "SimulationMaxxing: How we ship agents 20× faster — Aman Gupta (Nubank) + Shreya Rajpal (Snowglobe)",
    "duration": 989,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "KMR_RBoCa4M",
    "title": "SimulationMaxxing: How we ship agents 20× faster — Aman Gupta (Nubank) + Shreya Rajpal (Snowglobe)",
    "one_liner": "Nubank and Snowglobe show that generating eval data by simulating users — personas with consistent fake grounding data, run against your real agent with tools mocked — collapses the agent release loop from weeks to hours, with 80% of domain-expert labels confirming the simulated data is usable.",
    "summary": "Aman Gupta (Nubank) and Shreya Rajpal (Snowglobe) argue that evals are only metrics plus data, that metrics now have a known playbook (LLM-as-judge aligned to human labels, auto prompt tuning), and that the remaining bottleneck is eval data for multi-turn agents — where each data point is a trajectory of tool calls with consistent state. Hand-authoring that data takes days and production traces, though almost free, mean experimenting on live users. Snowglobe wraps the agent via SDK with no code changes, mocks its tools, and drives it with generated personas to produce thousands of multi-turn conversations plus per-turn judge scores. Nubank reports TNPS for five production customer-support agents rising toward and past human quality, one agent's TNPS 2x, self-service rate up 4% in one case, and regressions caught before they shipped.",
    "key_points": [
      "Core claim: generating eval data in simulation instead of waiting on production data lets you ship agents 20× faster; Nubank says the cycle went from a few weeks to less than a day, sometimes hours or minutes.",
      "Nubank context: leading digital bank in Latin America, 135 million customers across Brazil, Mexico and Colombia, US launch soon, quarterly revenue crossed $5B in Q1 2026; AI agents handle routine customer support end to end while humans take the hardest longtail cases.",
      "Results shown: average TNPS for five production AI agents rose over a few quarters to approach human quality, with Aman saying the slide is stale and many now exceed it; one agent's TNPS 2×ed; self-service rate improved across the board and by 4% in one case, contradicting the usual TNPS-vs-SSR trade-off. A KDD paper (QR code on slide) was being presented in Korea in August.",
      "Evals are only two things — metrics and data. Metrics have a playbook (LLM-as-a-judge classifiers aligned to human judgment, iterated with auto-optimization and autoprompt tuning); the unsolved bottleneck is the data those metrics run on.",
      "Why agent data is expensive: structured ML rows and single-turn QA were tractable, but a multi-turn agent data point is a trajectory with internal tool calls that all need state, and the synthetic state must stay consistent across every tool call.",
      "The two existing sources both hurt: manual authoring means hand-planning state updates and trajectories; production traces are almost free but mean testing on real live users, which makes running many parallel experiments at scale very hard.",
      "How Snowglobe works mechanically: point the SDK at your agent with no code changes, decide which tools to mock, supply personas and use cases to steer the sim; output is thousands of multi-turn conversations against the real agent plus judges run per turn, piped into the eval pipeline — data on demand.",
      "Concrete example persona: Maria Souza, 34, a designer and first-time credit card customer wanting to order a card, with generated grounding data (fake address, fake credit card, account data) that stays consistent through execution, plus tone and voice — her curt one-line messages show up in the resulting conversation.",
      "Sim-to-real validation: eval scores on simulated and real data correlate highly, and human review found 80% of domain-expert labels confirmed sims give usable data, for greenfield agents as well as mature ones.",
      "Simulation de-risks releases: it caught a regression that would have reached production and, on another agent, an issue that would have lowered self-service rate. Nubank also uses it to sweep the recent wave of open-source models through their agent harness and eval setup, saving multiple weeks, with A/B tests running to see whether open-source matches frontier models."
    ],
    "takeaways": [
      "Stop treating production traces as your eval set. Generate eval data in simulation so you can run many parallel experiments without experimenting on live users.",
      "Gate A/B tests behind simulation: don't launch until you're happy with the sim output. If you were running ten A/B tests a quarter, sim short-circuits the first five or six and you launch one.",
      "Build simulated data as personas with grounding data, not as prompts — name, demographics, intent, tone, plus fake address/card/account values that stay consistent across every tool call, and mock the tools that need mocking.",
      "Before trusting sim results, measure the sim-to-real gap explicitly with offline, online and human-review metrics — Nubank's bar was eval-score correlation plus 80% domain-expert label agreement.",
      "Build the self-improvement loop as ship → observe → build robust evals (with automated prompt optimization) → run sim and real data through those evals → optimize the harness → ship; if metrics are aligned and data generation is reliable, the loop follows."
    ],
    "topics": [
      "agents",
      "evals",
      "simulation",
      "synthetic-data",
      "llm-as-judge",
      "customer-support",
      "ab-testing",
      "open-source-models"
    ],
    "tools": [
      "Snowglobe",
      "Nubank",
      "Snowglobe SDK",
      "LLM-as-a-judge",
      "Japa (automated prompt-optimization algorithm, as spoken)",
      "AI Engineer (conference)",
      "KDD"
    ],
    "quotes": [
      {
        "text": "If you generate your eval data in sim instead of waiting on production data you can ship agents 20x faster and we'll give you evidence for that.",
        "at": "01:25",
        "url": "https://www.youtube.com/watch?v=KMR_RBoCa4M&t=85s"
      },
      {
        "text": "Production traces in comparison are almost free. You don't have to pay for them. You're going to get them anyway. But the cost is that you're testing on real live users every time you're testing it.",
        "at": "05:24",
        "url": "https://www.youtube.com/watch?v=KMR_RBoCa4M&t=324s"
      },
      {
        "text": "You don't know if your agent is good, you don't know if your evals are good. You have no idea what's happening. And that was me a few quarters ago.",
        "at": "10:15",
        "url": "https://www.youtube.com/watch?v=KMR_RBoCa4M&t=615s"
      },
      {
        "text": "In an enterprise setting when you're building an agent, it really does come down to two things, data and metrics.",
        "at": "15:34",
        "url": "https://www.youtube.com/watch?v=KMR_RBoCa4M&t=934s"
      }
    ],
    "words": 3533
  },
  "summary_url": "/#KMR_RBoCa4M",
  "transcript": {
    "html": "/transcripts/KMR_RBoCa4M.html",
    "txt": "/transcripts/KMR_RBoCa4M.txt",
    "vtt": "/transcripts/KMR_RBoCa4M.vtt"
  }
}