{ "video": { "id": "jHMiYtjoJfA", "title": "Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop", "duration": 1186, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "jHMiYtjoJfA", "title": "Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop", "one_liner": "Raindrop CTO Ben Hylak argues that in production, agent quality work is not benchmark-maxxing but \"raising the floor\" — finding the worst things your agent does, knowing when each issue started and what percent of users it hits, and keeping evals as code rather than as a 1,000-row dataset that breaks the next time you swap models or harnesses.", "summary": "Hylak says most public eval discourse is still stuck in the chatbot era — curated datasets and prompt playgrounds — while real agents are creative, stochastic entities whose whole harness is the prompt. He splits the work into \"ceiling\" (emergent capability) and \"floor\" (worst-case behaviour: deleting data, recommending a competitor, sending slop email), and argues the floor is what breaks user trust and deserves the effort. He prescribes evals-as-tests run locally, issue tracking that reports onset time and percent of users affected, and shares three first-principles lessons from Raindrop: clusters aren't issues, code mode scales to trace analysis, and agents can't find anomalies but can investigate ones you hand them.", "key_points": [ "Despite the track title, he says there is very little actual continual learning in shipped products or at the labs — the honest version of the talk is what teams really do instead.", "Chatbot-era evals worked because you knew the answer to 80–90% of user questions (\"what is the capital of the United States\" → Washington, D.C.); agents broke that assumption.", "Big eval datasets are brittle: switch to something like the Claude Code CLI and \"80% of your evals suck\" — and almost nobody will actually delay a model upgrade two weeks to rewrite them.", "Framing device: are you a benchmark maxer or a floor raiser? Labs build general-purpose systems; downstream companies imbue company-specific domain knowledge and carry a different kind of responsibility. The word \"eval\" is now more or less meaningless.", "Ceiling = the craziest emergent capability; floor = the worst thing the agent can do (delete data, recommend a competitor, send AI slop email because it technically had inbox access). Societal failures like 4o sycophancy are floor problems, not capability problems.", "Prompt playgrounds and cloud-managed prompts are near-dead in practice because the prompt is now the entire codebase and harness; evals should be code — unit and end-to-end tests run locally (Sentry's Vitest-evals package, what OpenAI calls macro evals).", "For every production issue you need exactly two facts: when it started (is it new? what did I change yesterday?) and what percent of users it affects (3 users vs 100,000).", "Lesson 1 — clusters are not issues: useful for one-off error analysis, but temporal clustering is unreliable, you don't control the boundaries, and \"price issues\" can lump wrong-price-quoted with wrong-refund-calculated despite totally different root causes.", "Lesson 2 — code mode, known from MCPs, scales to traces: write classifiers, run them in a sandbox at production scale. Lesson 3 — agents are very bad at anomaly detection; pull out deterministic signals like keyword-frequency spikes yourself and let the agent investigate those.", "Strategy depends on scale: at 10–100M messages/day, experiments on a small free-tier sample are extremely valuable; with five or ten users (e.g. a critical internal enterprise app) A/B tests are the wrong tool entirely." ], "takeaways": [ "Stop investing months in a large static eval set; write evals as ordinary tests (unit/end-to-end) that live in the repo and run locally, so a model or harness swap doesn't invalidate them.", "Instrument production so every issue reports when it started and what percentage of users it affects — without those two numbers you cannot triage the infinite problems an agent will generate.", "Don't ship clustering as your issue-detection story; clusters are for one-off analysis, not for tracking an issue's onset and growth over time.", "Detect anomalies deterministically (keyword frequency spikes, etc.) and only then hand them to an agent to investigate — never ask the agent to find the anomalies.", "Pick your approach by user count: experiments and free-tier sampling at millions of users; a completely different, hands-on approach when you have five.", "Before building something yourself, ask whether the capability is better waited for — training your own frontier-level model is probably not the move." ], "topics": [ "agents", "evals", "observability", "production-monitoring", "error-analysis", "tracing", "anomaly-detection", "ai-engineering" ], "tools": [ "Raindrop", "Workshop (open-source tracing tool)", "howtoeval.com", "Sentry", "Vitest evals", "OpenAI", "Claude Code CLI", "Codex", "Cursor", "GitHub Copilot", "Devin", "Fable", "GPT-4o", "Vercel", "Speak", "Framer", "MCP" ], "quotes": [ { "text": "the one thing I could promise you is that things are going to keep changing", "at": "04:42", "url": "https://www.youtube.com/watch?v=jHMiYtjoJfA&t=282s" }, { "text": "you want more safety but you don't want theater", "at": "04:55", "url": "https://www.youtube.com/watch?v=jHMiYtjoJfA&t=295s" }, { "text": "the prompt is actually like the whole thing now. It's like all the code. It's your whole harness.", "at": "12:06", "url": "https://www.youtube.com/watch?v=jHMiYtjoJfA&t=726s" }, { "text": "agents are very, very bad at anomaly detection. So don't ask your agent to find anomalies. Uh ask it to investigate anomalies you've already found.", "at": "18:39", "url": "https://www.youtube.com/watch?v=jHMiYtjoJfA&t=1119s" } ], "words": 4663 }, "summary_url": "/#jHMiYtjoJfA", "transcript": { "html": "/transcripts/jHMiYtjoJfA.html", "txt": "/transcripts/jHMiYtjoJfA.txt", "vtt": "/transcripts/jHMiYtjoJfA.vtt" } }