{ "video": { "id": "mUQoVz7THu0", "title": "Why Building an Eval Platform Is Harder Than It Looks — Braintrust", "duration": 1067, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "mUQoVz7THu0", "title": "Why Building an Eval Platform Is Harder Than It Looks — Braintrust", "one_liner": "Building an eval platform starts out looking like a UI on a spreadsheet, but production agent traces (semi-structured JSON payloads of tens to hundreds of megabytes that need querying both in real time and over long horizons) make it a systems problem, not a UI problem.", "summary": "Hussein, who leads solution engineering for Braintrust in the West, frames agent quality as two pillars: evals before production and observability after it. He walks through the stages teams typically go through: a spreadsheet for-loop, then a bespoke reporting UI, then side-by-side experimentation for non-technical users, and finally a full flywheel where production failure modes become offline test cases. His argument is that the hard part is maintaining that last stage. Traces are huge and deeply nested, typical cloud data warehouses collapse under the load, and real-time and long-running read patterns pull in different directions. Braintrust built BTQL to deal with this. He closes by saying coding agents are becoming first-class users of eval platforms, and the human's job is shifting to reviewing outcomes.", "key_points": [ "Agent quality has two pillars. Evals happen before production, where teams experiment and build confidence. Observability happens in production and reinforces the offline hypothesis through continuous monitoring.", "The simplest eval system has three parts: a way to execute agents against test inputs, a way to view outputs or scores (even in a spreadsheet), and a set of test input examples such as a prompt, request or context.", "Stages of maturity: (1) a spreadsheet for-loop, which has zero barrier to entry but quickly turns into documentation with no real experimentation or analysis; (2) a bespoke UI built by a product engineer, which brings in PMs but is 'still just a reporting tool'; (3) side-by-side experimentation where non-technical users tweak system prompts, models and parameters, which still relies on curated offline test cases and leaves you 'operating in the blind' in production; (4) the full flywheel, which you then have to own and maintain.", "The North Star improvement loop is: observe failure modes in production along dimensions you define, turn them into test cases, iterate offline without introducing regressions, ship, and hill-climb. This repeats because drift happens, just as in classical ML.", "Agent quality is a multi-persona problem. It involves AI and software engineers, PMs 'who used to build PRDs that are now running evals', and SMEs with domain expertise.", "'Agent traces are nasty.' They are semi-structured JSON, and some teams log 'a movie's worth of data' per interaction, hundreds of megabytes. Traditional heartbeat observability payloads are kilobytes. Typical cloud data warehouses collapse under this volume.", "There are two conflicting workloads: real-time queries as traces land, to catch poor user experiences, and long-running queries for fine-tuning or for humans annotating judge outcomes for alignment. Braintrust built a query abstraction layer called BTQL to cover both.", "Coding agents are becoming first-class citizens of eval platforms, working headlessly from natural language (e.g. 'Find me all traces in the last 24 hours where the user had a poor experience. Run the evals for me.'). Braintrust runs inference over logged traces to surface 'unknown unknowns' such as silent failures or users repeating the same request out of frustration. Other hidden pieces include the underlying database, RBAC for controls and permissions, and data masking." ], "takeaways": [ "Starting with spreadsheet evals is fine and 'a good first step', but plan to move past them quickly. They don't support real experimentation, long-horizon analytics, collaboration, or domain-expert involvement.", "Log every input, output and trace step in production so failure modes can be fed back into offline evals. Curated offline test sets alone leave you blind once you ship.", "Before building an in-house eval platform, budget for the systems work: real-time ingest of multi-megabyte nested payloads, both real-time and long-running query patterns, RBAC and data masking, plus ongoing maintenance.", "Give PMs and SMEs an interface to experiment with prompts, models and parameters themselves, not just view results.", "Let coding agents run eval iterations, and keep the human's role as reviewing outcomes and picking which version to ship." ], "topics": [ "evals", "observability", "agents", "agent quality", "tracing", "data infrastructure", "coding agents", "llmops" ], "tools": [ "Braintrust", "BTQL", "Salesforce", "Databricks", "ChatGPT" ], "quotes": [ { "text": "Evals is just a UI on a spreadsheet, right?", "at": "03:53", "url": "https://www.youtube.com/watch?v=mUQoVz7THu0&t=233s" }, { "text": "But agent traces are nasty. They are semi-structured JSON.", "at": "12:07", "url": "https://www.youtube.com/watch?v=mUQoVz7THu0&t=727s" }, { "text": "it's not necessarily that it's just a UI or UX problem, but building an eval platform truly becomes a systems problem.", "at": "13:32", "url": "https://www.youtube.com/watch?v=mUQoVz7THu0&t=812s" }, { "text": "There are PMs who used to build PRDs that are now running evals.", "at": "06:16", "url": "https://www.youtube.com/watch?v=mUQoVz7THu0&t=376s" } ], "words": 2914 }, "summary_url": "/#mUQoVz7THu0", "transcript": { "html": "/transcripts/mUQoVz7THu0.html", "txt": "/transcripts/mUQoVz7THu0.txt", "vtt": "/transcripts/mUQoVz7THu0.vtt" } }