{ "video": { "id": "iniryju23Sk", "title": "What a $900M Bank Error Teaches Us About AI Agents - John Feminella, Two Sigma", "duration": 1136, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "iniryju23Sk", "title": "What a $900M Bank Error Teaches Us About AI Agents - John Feminella, Two Sigma", "one_liner": "Citibank's $900M Revlon payment error happened even though every human and every control did what it was supposed to. Two Sigma's answer for AI agents is five composable controls: a registry, risk scoring, declared-purpose runtime guardrails, shared evals and classifier-gated approvals. Together they keep agents safe without locking them down.", "summary": "John Feminella of Two Sigma starts with the August 2020 Revlon incident. Citibank's loan software, Flexcube, sent about $900M of principal to more than 300 lenders instead of about $8M of interest. Wash-account boxes were left unchecked, and three approvers overrode a warning that looked correct, so the process and controls were all followed and still failed. His argument is that agents, like humans, are non-deterministic, so safety comes from boundaries, visibility and composable controls rather than full sandboxing, letting agents run wild, or policy documents that can't be enforced at runtime. He describes five controls Two Sigma uses so that safety and innovation are not a trade-off.", "key_points": [ "The Revlon/Citibank case: in Flexcube, the easiest way to book an interest-only payment was to record it as if Revlon were paying off the whole loan, sending the principal to an internal wash account. The wash-account boxes weren't checked, and the 'money's about to leave the bank' warning was overridden because money was supposed to leave for the interest payment. As a result about $900M went out instead of about $8M, and lenders holding debt worth about 60 cents on the dollar refused to return it.", "Three approaches that didn't work: sandboxing everything is safe but not useful; letting agents run free gives isolation too weak for regulated enterprises; and policy documents or prompt specs can't be enforced at runtime.", "Control 1, the agent registry ('You can't govern what you can't see'): it records owner, tools, endpoints and product for every agent. Registration happens automatically when you use the firm's approved agent-building frameworks, with manual registration as an option. A heavy register-and-approve-first gate just pushes people into shadow IT.", "Control 2, risk scoring: a Jira/Linear ticket summarizer and a production deployment agent should not wait in the same review queue. Otherwise a trivial summarizer waits 2 months for the same sign-offs, and reviewers drown in low-risk requests. Most enterprise agents are purpose-built rather than open-ended like Claude Code or Pi, so they can be told apart by what they declare up front.", "Control 3, runtime guardrails from declared purpose: agents declare the tools they need ('these four things out of the 500'). At initialization, managed environments look up the registry policy and turn it into enforceable endpoint, host and resource rules. Teams don't file tickets or go to IT as long as they're registered and conform to their risk score.", "Control 4, shared evals: evals are treated like unit tests because a model change is a runtime dependency change. Instead of a central committee sign-off where every prompt tweak waits in a queue, the firm runs one shared eval approach, and teams only write what's true about their agent.", "Control 5, runtime approval via classifiers: approving every action leads to review fatigue. The cautionary example is an AI lab director whose OpenClaw agent compacted its context, lost 'wait for my approval', deleted hundreds of emails and ignored stop commands sent from her phone. The fix is deterministic fast paths for explicit rules, conservative classifiers for ambiguous calls, a human only for important or unclear cases, and outright denial for anything definitely dangerous." ], "takeaways": [ "Make every agent visible from the day it's created by auto-registering it (owner, tools, endpoints, product) through the approved frameworks, instead of gating it behind an onerous approval process.", "Tier human review by agent risk so low-risk experiments aren't stuck in the same queue as agents that touch production.", "Have agents declare the minimal set of tools they need, and enforce that at initialization from the registry as endpoint, host and resource policy. Prompt-level policy docs are not enough.", "Give the whole firm one eval harness so every team meets the same bar without rebuilding the common parts.", "If humans say yes 99.99999% of the time, turn it into a rule. Use deterministic fast paths plus conservative classifiers, and escalate only ambiguous or risky actions to a person." ], "topics": [ "agents", "governance", "guardrails", "evals", "human-in-the-loop", "risk-management", "regulated-industries", "agent-registry" ], "tools": [ "Two Sigma", "Citibank", "Revlon", "Flexcube", "Claude Code", "Pi", "OpenClaw", "Jira", "Linear", "MCP servers" ], "quotes": [ { "text": "the bank had this procedure, it had these controls, and everybody did what they thought the right thing to do was. That's the problem I want to talk about", "at": "05:02", "url": "https://www.youtube.com/watch?v=iniryju23Sk&t=302s" }, { "text": "A ship in harbor is safe, but that is not what ships are for.", "at": "13:07", "url": "https://www.youtube.com/watch?v=iniryju23Sk&t=787s" }, { "text": "If you're saying yes 99.99999% of the time, then that's probably just a rule that the agent should follow", "at": "17:31", "url": "https://www.youtube.com/watch?v=iniryju23Sk&t=1051s" }, { "text": "Why do cars have brakes? It's not just so that you can stop, it's so that you can go fast and stop safely.", "at": "18:35", "url": "https://www.youtube.com/watch?v=iniryju23Sk&t=1115s" } ], "words": 4321 }, "summary_url": "/#iniryju23Sk", "transcript": { "html": "/transcripts/iniryju23Sk.html", "txt": "/transcripts/iniryju23Sk.txt", "vtt": "/transcripts/iniryju23Sk.vtt" } }