{ "video": { "id": "_4Ph2dhmUFI", "title": "How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase", "duration": 1053, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "_4Ph2dhmUFI", "title": "How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase", "one_liner": "An agent moves from supervised to unsupervised work in stages: humans first check every output, then a sample once measured confidence reaches a policy threshold, then only exceptions. Workflows stay deterministic on the outside and adaptive on the inside, and the loop is only as trustworthy as its check.", "summary": "Evan Kormos of Coinbase argues that agents in regulated, policy-driven workflows should be governed by treating policy as the 'original system prompt'. The outer process is a deterministic DAG of gated steps, such as an Apache Airflow workflow. Each node can contain an iterative loop that cycles until the work is good enough, which he calls a directed cyclic graph and suggests managing with something like LangGraph. Because agents are probabilistic, he says exact-output tests are not enough. Teams need a complementary confidence mechanism of representative datasets, repeated statistical evals and traces, and should use the measured confidence to hand work over in stages from supervised to sampled to exception-only.", "key_points": [ "Policy, written with 'must, may, and should', is the rules for how work gets done, especially risk and compliance. It belongs to the build state, separate from the running system, which changes constantly with product-driven changes but is still expected to be roughly bound by the policy.", "Agents are an operating model that cuts across people, process and technology. The talk focuses on where agents meet process: what work we give them, what boundaries we set, and how we build trust in their outputs.", "Agents make it worth revisiting what can run in parallel. Work that used to cost too much or was too complex to coordinate can now run as independent reviews alongside each other, with the required checks kept in place. This is modelled as a DAG, for example in Apache Airflow, for commercial lending, listing a token on the exchange in a specific region, or regulated software delivery.", "Inside a node such as code review is a loop: do the work, check it, and use the feedback to iterate until it is good enough to pass. That makes it a directed cyclic graph (DCG), whose state LangChain/LangGraph can help manage. Exception flows handle work that gets cancelled or doesn't meet security standards, so work doesn't get stuck and you don't design for every edge case.", "Adding agents shifts volume across the flow: if code review gets faster, design and security scanning have to get faster too. That creates new bottlenecks, which is why operational visibility of the process is 'much more important than the authoring and the building of your workflow tool'.", "Higher-quality inputs give better outputs at a lower compute cost. One way to improve inputs is to inject the DAG state into the agent so it knows where it fits in the bigger picture.", "Because agents are probabilistic, exact output comparisons alone are not enough. You need representative datasets, a definition of good, evals run repeatedly as a statistical analysis, and traces, which together form a confidence mechanism that complements traditional unit tests.", "Questions to ask about each node, using lending (KYC, financials, credit memo, risk review, documentation, decision) as the example: can it be checked against the policy easily, can it run in a bounded loop, can it be traced, and can it fall back to a human? The final decision, which is the check against policy, probably stays human-driven and is harder to 'hill climb'." ], "takeaways": [ "Map your policy-gated workflow as a DAG, keep the outer flow deterministic, and use adaptive agent loops only inside nodes. Look again at which steps can now run in parallel without dropping the required checks.", "Define what good looks like with domain subject-matter experts and build a reliable check before adding more loop iterations. Without that, more passes don't necessarily mean better results.", "Replace exact-output tests for agents with representative datasets, repeated statistical evals and traces, so trust rests on statistics rather than 'a feeling about a model'.", "Roll out in stages. Start supervised with every output checked by hand. Move to sampled human review plus systematic trace evaluation once measured confidence reaches a policy threshold. Hand over fully only after clearing the confidence threshold, with humans still handling exceptions and refining the datasets.", "Invest in metrics and visibility across the whole process, because agent speed-ups create bottlenecks further along the flow." ], "topics": [ "agents", "evals", "workflow orchestration", "compliance", "human-in-the-loop", "observability", "financial services" ], "tools": [ "Coinbase", "Apache Airflow", "LangChain", "LangGraph", "Python" ], "quotes": [ { "text": "Think of policy as the original institutional intent or the original system prompt.", "at": "01:00", "url": "https://www.youtube.com/watch?v=_4Ph2dhmUFI&t=60s" }, { "text": "The loop is really only as trustworthy as the check.", "at": "10:54", "url": "https://www.youtube.com/watch?v=_4Ph2dhmUFI&t=654s" }, { "text": "the recipe really is to keep it deterministic on the outside and make it adaptive on the inside.", "at": "16:05", "url": "https://www.youtube.com/watch?v=_4Ph2dhmUFI&t=965s" }, { "text": "So, it's not a feeling about a model.", "at": "16:26", "url": "https://www.youtube.com/watch?v=_4Ph2dhmUFI&t=986s" } ], "words": 2669 }, "summary_url": "/#_4Ph2dhmUFI", "transcript": { "html": "/transcripts/_4Ph2dhmUFI.html", "txt": "/transcripts/_4Ph2dhmUFI.txt", "vtt": "/transcripts/_4Ph2dhmUFI.vtt" } }