{ "video": { "id": "hfEczxdNyvU", "title": "Advanced Workshop: Mastering AI Observability — Doug Guthrie, Braintrust", "duration": 6697, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "hfEczxdNyvU", "title": "Advanced Workshop: Mastering AI Observability — Doug Guthrie, Braintrust", "one_liner": "Doug Guthrie (Braintrust) argues that tracing alone is passive. Scores and Topics clustering put signal on every trace, so a coding agent with the Braintrust CLI and skills can run the 'production informs development' flywheel itself: query traces with SQL, curate failing examples into datasets, add scores, re-run evals and open a PR.", "summary": "Guthrie says tracing is the foundation of observability. The hard part is pulling signal out of the huge volume of trace data, which Braintrust calls 'active observability'. He covers two kinds of signal. The first is scores: code-based scores, LLM judges, LLM judges aligned with human review, and code-based scores that call an LLM. These run offline in evals and online through automations. The second is Topics, a hosted pipeline (preprocessor → facet → summary → embedding → topic map) that labels traces to surface 'unknown unknowns'. The workshop instruments an OpenAI Agents SDK support agent, imports 968 traces, and configures online LLM-judge scores plus a custom 'support workflow issue' facet. Codex, given the bt CLI and the agent-improvement skill, then finds a record-lookup failure cluster, creates scores and dataset rows, and re-runs evals.", "key_points": [ "Flywheel framing: 'production should inform development'. Develop a change, eval it for regression or improvement, deploy, observe, then add intelligence on top to get insight at scale, because people can't comb through that much data by hand.", "Score types: code-based (fast, cheap, e.g. schema validation or tool A called before tool B), LLM judge (subjective), LLM judge calibrated with human review, and code-based scores that call an LLM for branching logic. Code-based scores can bundle Python/TypeScript dependencies when pushed. Chaining scores is only possible in code, as a function that returns an array of scores.", "Scoring advice, offered 'with a grain of salt': ask the judge for its reasoning, use binary scores rather than A-B-C-D grades, make test cases harsh (100% on evals means the dataset or scores are too weak), include examples in judge prompts, and use a stronger model for scoring than for generation.", "Topics has built-in facets (task, sentiment, issues) plus custom facets and custom preprocessors. The pipeline summarizes each preprocessed thread, embeds the summary, builds a topic map, and attaches labels/classifiers to traces. It runs only on Braintrust-hosted models at $0.06 per million input tokens and $0.40 per million output tokens, so it can cover 100% of traces. Bring-your-own models were tried first, but the topics they produced 'were not good'.", "Topic generation needs at least 100 facet matches. In the demo the built-in 'issues' facet produced nothing, but a custom 'support workflow issue' facet, with a regex to exclude 'none' outputs, surfaced a record-lookup-failure cluster that the evals had missed.", "Online scoring automations can be scoped to span, trace, or group (e.g. by a conversation_ID metadata field for multi-turn conversations that span several traces), with filters and sampling rates. Topics also has a topic window and a delay-before-invocation setting for long-running conversations.", "Braintrust built its own backend, BrainStore, in late 2024 after ClickHouse 'fell down at scale' with customers like Netflix, Microsoft and Dropbox. Guthrie says LangSmith and Arize have only recently built their own databases, putting Braintrust about a year ahead. BrainStore supports SQL directly over all trace and span data, which the in-platform agent Loop and coding agents use to pull in only relevant context.", "Automated flywheel demo: Codex with `bt setup skills` and the open-source Braintrust skills repo ran bt SQL over the 968 traces, found an escalate-to-human tool-error cluster, created new scores, tools and datasets, and re-ran evals. A GitHub Action YAML that runs Claude Code produces a PR with the changes, rationale, eval impact and regressions. One customer generates 5–10 more PRs per day this way." ], "takeaways": [ "Instrument the full trace (tools, inputs, outputs) first. Without it the agent is a black box that neither you nor a coding agent can debug.", "Attach signal to traces: online scores for known failure modes, Topics or custom facets for unknown unknowns. Then filter or query by that signal instead of reading traces one by one.", "Keep evolving eval datasets by curating failing production traces into them, and add a new score for each newly discovered failure mode (e.g. record lookup failure) before the next deploy.", "Use high sampling rates on new agents and reduce them as the agent matures. Consider replacing expensive classification/taxonomy LLM judges with Topics for cheaper 100% coverage, as Dropbox is trying.", "Give your coding agent the Braintrust CLI and skills (or wire it into a GitHub Action) so it can run the query → dataset → score → eval → PR loop. For restricted data, run that loop as a service account that surfaces analysis rather than raw traces." ], "topics": [ "observability", "evals", "tracing", "llm-as-judge", "topic-clustering", "agents", "coding-agents", "online-scoring" ], "tools": [ "Braintrust", "BrainStore", "Braintrust CLI (bt)", "Braintrust Topics", "Loop", "Braintrust gateway", "Braintrust skills repo", "OpenAI Agents SDK", "OpenAI", "Anthropic", "ClickHouse", "LangSmith", "LangGraph", "LangChain", "Arize", "Lightstep", "OpenTelemetry", "Codex", "Claude Code", "Cursor", "OpenCode", "Gemini", "Qwen", "uv", "Modal", "GitHub Actions", "Terraform", "Helm", "Cloudflare", "Dropbox", "Netflix", "Microsoft" ], "quotes": [ { "text": "if you get 100% on your evals, yeah, one, you probably don't have strong enough test cases in your data set. Your scores probably aren't harsh enough.", "at": "13:10", "url": "https://www.youtube.com/watch?v=hfEczxdNyvU&t=790s" }, { "text": "we were using ClickHouse behind the scenes at the time and recognized very quickly when we onboarded the likes of, you know, the the customers that you saw on there, Netflix and Microsoft and Dropbox, that they ClickHouse just very quickly fell down at scale.", "at": "24:18", "url": "https://www.youtube.com/watch?v=hfEczxdNyvU&t=1458s" }, { "text": "You need that signal attached to your traces in some way, and you need that signal to be real.", "at": "72:57", "url": "https://www.youtube.com/watch?v=hfEczxdNyvU&t=4377s" }, { "text": "your your evals should evolve with your agent.", "at": "78:32", "url": "https://www.youtube.com/watch?v=hfEczxdNyvU&t=4712s" } ], "words": 21994 }, "summary_url": "/#hfEczxdNyvU", "transcript": { "html": "/transcripts/hfEczxdNyvU.html", "txt": "/transcripts/hfEczxdNyvU.txt", "vtt": "/transcripts/hfEczxdNyvU.vtt" } }