{ "video": { "id": "mMw3arwQsL0", "title": "1,368 AI Failures and What They Teach About Evals — Vishal Srivastava, New York Life & Tanmay Sah", "duration": 1202, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "mMw3arwQsL0", "title": "1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life", "one_liner": "After mapping public AI Incident Database failures onto a 13-sub-dimension framework that covers ~98% of them, the speakers find that deployment controls (human mediation, harm detection/monitoring) matter more for harm than the model's technical strength, so evals should be risk-based, failure-driven and cover the whole system.", "summary": "Vishal Srivastava and Tanmay Shah of New York Life went through the publicly available AI Incident Database to see what real failures teach about evaluation. They split failures into three core dimensions: technical strength (development), deployment controls, and business context. These break down into 13 sub-dimensions that cover almost 98% of incidents, and a typical incident involves six of them at once. Deployment controls are the group most associated with harm. Borrowing Peter Wason's work on confirmation bias, they argue evals should try to falsify the system's claims through an iterative build-break-fix loop that maps where the system can be trusted. They illustrate this with an airline chatbot that gave wrong advice about a bereavement fare.", "key_points": [ "AI failures in the AI Incident Database are rising exponentially; over the last four years of the foundation-model era the curve is 'almost vertical', and enterprise AI use cases now exceed the capacity to review each one individually.", "Framework: 3 core dimensions and 13 sub-dimensions. Technical strength (6): data quality, system design, predictive validity/outcome analysis, fairness, failure modes, explainability. Deployment controls (4): decision authority, human mediation, exposure control, monitoring. Business impact (3): financial, legal, scale.", "Looking back at past incidents, the 13 sub-dimensions cover almost 98% of AI failures, and each incident involves six failure sub-dimensions on average.", "On a plot of association with harm against frequency, human mediation (human-in-the-loop effectiveness) and harm detection (monitoring) are the most strongly linked to harm. Scale and system design show up often but are less associated with harm.", "Grouped together, deployment controls are the most associated with harm, followed by business impact, then technical failure: 'deployment controls are far more important.'", "They built a materiality scale that rank-orders AI use cases by associated harm, so higher-materiality use cases get more review resources.", "Risk breaks into four components: how often the model fails, whether the failure reaches a real decision, whether it goes uncaught, and how severe the consequences are. The same model creates different risk depending on how it is deployed.", "For agentic systems, held-out metrics like accuracy and AUC-ROC aren't enough. You test chosen scenarios across retrieval, tool failure, human review, permission boundaries and recovery. The AI judge or test generator can make mistakes too, so 'the evaluator is the part of the system we govern.'", "Airline chatbot case: the bot told a bereaved customer he could buy a regular ticket and claim the discount within 90 days, the airline's policy didn't allow that, and a tribunal held the airline responsible. A build-break-fix pass (requirement: advice must match approved policy; challenge it with refund questions; add a policy check or human escalation; retest) could have exposed this before deployment." ], "takeaways": [ "Make evals risk-based rather than frequency-based: rate each AI use case's materiality on a company-specific scale and spend eval resources on the high-materiality ones.", "Evaluate the whole system, not just the model: test deployment controls (decision authority, human-in-the-loop effectiveness, exposure, monitoring) and business context alongside technical metrics, and put human mediation and harm detection first.", "Run failure-driven evals: state the system's claim, try to break it, then either keep the AI out of the failing region or add supporting controls you trust (policy checks, human escalation), and repeat until you are confident.", "Map trusted, uncertain and high-risk regions, and for high-risk systems only deploy the AI in the region you are confident about.", "Treat the evaluator (LLM judge, generated tests) as something you govern: check that it is reliable, what the evidence covers, and whether that evidence applies to the system you are actually deploying." ], "topics": [ "evals", "ai-failures", "risk-management", "agents", "llm-as-judge", "ai-governance", "deployment-controls", "human-in-the-loop" ], "tools": [ "AI Incident Database", "New York Life" ], "quotes": [ { "text": "So, all these sub-dimensions are important, but deployment controls are far more important.", "at": "09:36", "url": "https://www.youtube.com/watch?v=mMw3arwQsL0&t=576s" }, { "text": "That's why the evaluator is the part of the system we govern.", "at": "13:32", "url": "https://www.youtube.com/watch?v=mMw3arwQsL0&t=812s" }, { "text": "what we are saying is that your evals should be risk-based um rather than frequency-based.", "at": "18:51", "url": "https://www.youtube.com/watch?v=mMw3arwQsL0&t=1131s" }, { "text": "Our final takeaway message is evaluate the full AI system based on risk, not just one component.", "at": "19:36", "url": "https://www.youtube.com/watch?v=mMw3arwQsL0&t=1176s" } ], "words": 3256 }, "summary_url": "/#mMw3arwQsL0", "transcript": { "html": "/transcripts/mMw3arwQsL0.html", "txt": "/transcripts/mMw3arwQsL0.txt", "vtt": "/transcripts/mMw3arwQsL0.vtt" } }