{ "video": { "id": "3ZMUiFaQ3qg", "title": "Verifiable Environments for AI in Biology — Kenny Workman, LatchBio", "duration": 1062, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "3ZMUiFaQ3qg", "title": "Verifiable Environments for AI in Biology — Kenny Workman, LatchBio", "one_liner": "LatchBio's CTO Kenny Workman argues that data analysis is to biology what code was to software — a verifiable executable substrate — and shows how they built SpatialBench (146 spatial-biology tasks with deterministic Python graders) plus long-horizon and biosecurity benchmarks that frontier labs now cite in their model cards.", "summary": "Workman motivates agents in biology with the data deluge from single-cell, spatial and proteomics experiments — runs that can exceed what a scientist can store on a laptop — and claims that code-and-data analysis is the 'executable substrate' that makes scientific work verifiable and trainable, the way code was for software. Because frontier models 'cannot be trusted to do real work' in this gap between knowing biology and writing code, Latch built SpatialBench: 146 problems shipped last December, each a data node plus a task prompt plus a deterministic Python grader, deliberately modelled on SWE-bench. He then walks through what broke — sparse end-to-end grading, ambiguous task statements only exposed by human cross-grading, arbitrary bioinformatics QC thresholds — and the extensions: a human-verified subset, week-long-to-author long-horizon tasks simulating whole paper results sections, rubric 'choke points' for unverifiable long trajectories, and red-team biosecurity evals.", "key_points": [ "Data volume per experiment is the forcing function: single-cell runs yield 2–6 TB, spatial runs up to 7 TB, proteomics a few hundred GB — 'the output of a single experiment can exceed what a scientist can safely store on a consumer laptop', and biology's data growth trails only particle colliders.", "Core thesis: modern papers follow a fixed loop (choose a biological model → generate data → process data → interpret against prior literature → make a claim), so code and data analysis become a verifiable substrate for training and benchmarking agents, just as code was for non-verifiable software tasks.", "Latch shipped agent products (chat interface, dashboards, dispatch to external compute) using coding models not post-trained on biology; they worked badly but showed early signs of working. Unusually for agent loops, 'the tools can take days or weeks'.", "SpatialBench: released last December, 146 problems spanning different commercial spatial kits and the steps of the analysis DAG. Existing benchmarks were Q&A-style or not experiment-specific — he shows an Anthropic model card screenshot as evidence of the gap.", "One eval = one or more data nodes (a matrix of numbers, a high-content image) + a task prompt describing a scientific goal + grader configuration + a deterministic Python grader function; the design consciously borrows from SWE-bench, and 'it's a lot of JSON'.", "Three properties of a good biological test: verifiable by a function; durable — ground truth must be invariant across valid analysis paths, or you fail correct answers; and agentic — the conclusion must require interacting with the data rather than memorized knowledge.", "Human cross-grading by scientists (after watching trajectory data from model releases circa January–March) exposed bad tasks: an example asking to split a gene list into activity groups, score cells, find neighbouring oligodendrocytes 'using an appropriate radius' and compute a Spearman correlation at two time points leaves open how to split genes, what counts as inflammatory, how to normalize, and what radius is appropriate. Writing evals also exposed canonical bioinformatics numerical QC thresholds as arbitrary.", "Long-horizon set: tasks simulating whole paper results sections or industry go/no-go drug decisions, each taking a group of three people about a week to build — e.g. reconstructing the metastatic niche by using genetics and mRNA from a tumour biopsy plus metastatic lesions to find the seeding region. No model gets it right yet.", "For long horizons, terminal verifiable rewards are 'somewhat uninformative', so they are experimenting with rubrics built at invariant choke points in the tree of analysis paths — these correlate with verifiable outcomes but only loosely, so they don't yet trust them for RL or benchmarking.", "Expansion: benchmarks beyond spatial into single-cell and epigenomics, long-horizon versions of each, a first preclinical pharmacology benchmark for small molecules, an acquired biosecurity team, and a collaboration with American Wetware and surveillance company Aquid. Red-team results: innocuous-looking harmful tasks (e.g. 'clone GFP into bacteria' that is actually a toxin or virus bootstrap) get complied with drastically more often than they are refused, while models over-refuse routine questions — he frames over-refusal as 'just like an evaluation problem'.", "Business flywheel: build products with kit manufacturers → learn which tasks matter → publish benchmarks → get frontier labs to compete on them → models improve → products improve. Labs adopted the benchmarks organically; they appear in recent Anthropic model cards and in the Claude science launch, without Latch being told in advance." ], "takeaways": [ "If end-to-end outcomes in your domain are too sparse to grade (because models are still bad), decompose the workflow DAG and grade intermediate nodes — pick the state right before a human expert would do the creative work, and define ground truth there.", "Test ground truth for durability before shipping a task: if a different-but-valid solution path produces a different correct answer, the task is broken. Anchor on quantities invariant across paths, not on one blessed pipeline's output.", "Use human cross-grading as the proxy for ground truth where no canonical answer exists — it is what surfaced the ambiguous phrasing and arbitrary numeric thresholds hidden in their task statements.", "Design tasks so the answer requires interacting with the data, not recall; and for long-horizon work, expect terminal rewards to be uninformative — build rubric checkpoints, but validate their correlation with verified outcomes before trusting them for RL.", "Writing evals is itself a rigour-forcing exercise: 'if you have to teach a machine to do it, you might be picking out some structure that's more important or more durable than what you were doing on your own.'", "For safety evals, pair routine tasks against innocuous-looking red-team tasks — the failure mode observed is the asymmetry between over-refusing benign questions and under-refusing disguised harmful ones." ], "topics": [ "evals", "benchmarks", "agents", "computational-biology", "verifiable-rewards", "rubrics", "post-training", "biosecurity" ], "tools": [ "Latch (LatchBio)", "SpatialBench", "SWE-bench", "Anthropic", "Claude", "Python", "American Wetware", "Aquid", "sequencing-based spatial transcriptomics" ], "quotes": [ { "text": "code provided a verifiable substrate for complex software tasks that are not inherently verifiable, uh data analysis might do the same thing in bio.", "at": "03:08", "url": "https://www.youtube.com/watch?v=3ZMUiFaQ3qg&t=188s" }, { "text": "Except in our domain, the tools can take days or weeks. I'm serious.", "at": "04:03", "url": "https://www.youtube.com/watch?v=3ZMUiFaQ3qg&t=243s" }, { "text": "frontier models cannot be trusted to do real work. They're missing some capability between knowing biology and writing code.", "at": "05:40", "url": "https://www.youtube.com/watch?v=3ZMUiFaQ3qg&t=340s" }, { "text": "You want the conclusion to require interaction with the data, not some memorized knowledge.", "at": "09:38", "url": "https://www.youtube.com/watch?v=3ZMUiFaQ3qg&t=578s" } ], "words": 3861 }, "summary_url": "/#3ZMUiFaQ3qg", "transcript": { "html": "/transcripts/3ZMUiFaQ3qg.html", "txt": "/transcripts/3ZMUiFaQ3qg.txt", "vtt": "/transcripts/3ZMUiFaQ3qg.vtt" } }