{ "video": { "id": "3osCbSufv8g", "title": "Voice AI Where One Wrong Question Breaks the Rules - Shiv Indap, Upstart & Brooke Hopkins, Coval", "duration": 1023, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "3osCbSufv8g", "title": "Voice AI Where One Wrong Question Breaks the Rules - Shiv Indap, Upstart & Brooke Hopkins, Coval", "one_liner": "Upstart got its LLM compliance judge from 33% to 92% agreement with human reviewers. They did it by writing down tribal knowledge in the agent, settling disagreements among the reviewers themselves, and then improving the judge. Expert human review is what drives a voice agent eval loop in a regulated industry.", "summary": "Shiv Indap (Upstart) and Brooke Hopkins (Coval, ex-Waymo eval infrastructure) describe how Upstart launched a voice agent for lending verification calls. In that setting a single wrong question, like asking how much a medical expense costs, is a regulatory issue and not just bad customer experience. Upstart moved human review off an unstructured Google Sheet and into structured review queues, where humans check the output of LLM judges. Brooke then lays out Coval's continuous loop: define metrics, monitor live calls, route failures to the right teams, re-simulate fixes voice-to-voice before they reach production, and eventually 'self-heal', with every change still gated by humans.", "key_points": [ "Upstart handles thousands of calls every day. About 90% of applicants are decided instantly. The other 10% call in for verification, which covers identity plus three questions: will income continue for three months, what are the funds for, and are there outstanding obligations such as alimony or wage garnishments.", "Regulation makes this simple use case hard. If someone says the funds are for medical purposes, the agent cannot ask the amount. If they say travel to a graduation, it has to find out whose graduation. The agent must also politely sidestep status and timing questions, for example it cannot promise 'up to three business days'.", "Human review began in a Google Sheet. Within a few weeks it grew from 5 columns to 10, then 15, as people added columns at random, and the process broke down. That pushed them to a structured human-review process built with Coval.", "An LLM judge runs after every call and gives a yes/no on whether compliance requirements were met, with an explanation. A human then reviews the judge's output. At launch, human and judge agreed only 33% of the time. Agreement is now 92%, reached in three steps: encoding tribal knowledge into the agent, settling disagreement between the human reviewers themselves, and improving the judge.", "The metrics they settled on are accuracy, clarity, completeness, policy alignment, correct escalation and CSAT. Metrics such as call time were tracked early on and later dropped.", "Brooke sorts evals into three categories: did it get to the end, did it follow the steps to get there, and how good was the conversation or experience. Metric types are LLM-as-judge, heuristics and audio processing. Metrics should come from what human reviewers actually label and care about, not from something like word error rate that nobody acts on.", "Live monitoring catches known failure patterns at scale. Clustering analysis and anomaly detection surface unknown unknowns. Failures go to configurable review queues and then to the right team: off-script or wrong-answer failures to the agent team, compliance violations to the compliance team, others to engineering.", "Fixes are re-simulated voice-to-voice before release, covering background noise, accents, voice quality and interruptions. This is 'shift left': catch failures in simulation, not production. The planned 'self-healing loop' uses agents, RL pipelines or fine-tuned LLMs to propose fixes, still gated by human review. At Upstart, changes ship behind feature flags, engineers sign off, and compliance approves any compliance-related behaviour change." ], "takeaways": [ "Have humans review your LLM judge's output, track how often they agree, and treat low agreement as a sign of missing tribal knowledge or reviewer disagreement, not only a weak judge.", "Move off ad-hoc spreadsheets early. Use structured review queues that show humans only the flagged calls that matter, because humans cannot review thousands of calls a day.", "Don't get attached to your first metrics. Start with ones that seem logical, derive them from what reviewers actually label, and drop the ones that don't affect your use case.", "Send each class of failure to a specific owner (agent team, compliance, engineering), and re-simulate fixes before production instead of using live users to check them.", "Even 'self-improving' agents should keep humans in the loop: put changes behind feature flags and get domain sign-off, for example from compliance in regulated industries." ], "topics": [ "voice agents", "evals", "llm as judge", "human review", "compliance", "fintech", "simulation", "monitoring" ], "tools": [ "Coval", "Upstart", "Waymo", "Perplexity", "Google Sheets", "Google" ], "quotes": [ { "text": "the compliance sort of agreement metric was 33%. That means a human reviewer and a LLM agent or a LLM judge agreed only 33% of the time.", "at": "07:37", "url": "https://www.youtube.com/watch?v=3osCbSufv8g&t=457s" }, { "text": "don't be too attached to your metrics.", "at": "06:39", "url": "https://www.youtube.com/watch?v=3osCbSufv8g&t=399s" }, { "text": "evals are really the product definition.", "at": "09:56", "url": "https://www.youtube.com/watch?v=3osCbSufv8g&t=596s" }, { "text": "when we say self-healing or self-improving, it doesn't mean like humans are completely out of the loop.", "at": "15:45", "url": "https://www.youtube.com/watch?v=3osCbSufv8g&t=945s" } ], "words": 3366 }, "summary_url": "/#3osCbSufv8g", "transcript": { "html": "/transcripts/3osCbSufv8g.html", "txt": "/transcripts/3osCbSufv8g.txt", "vtt": "/transcripts/3osCbSufv8g.vtt" } }