{ "video": { "id": "eAXxdtNlK04", "title": "Stop Burning Tokens: Why self-improvement needs domain expertise first - Annabell Schäfer, Langfuse", "duration": 1059, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "Online" }, "analysis": { "video_id": "eAXxdtNlK04", "title": "Stop Burning Tokens: Why self-improvement needs domain expertise first - Annabell Schäfer, Langfuse", "one_liner": "A Langfuse growth engineer ran a self-improving prompt loop (Claude Opus 4.8 optimizing a GPT-5.4-nano classifier) on arXiv papers and found that nearly all of the 15% accuracy gain came from the first iteration — because the target function was a clean right/wrong signal, which is exactly what most domains lack until domain experts define it.", "summary": "Annabell Schäfer argues that the current excitement about self-improving 'loops' comes from coding, where there has always been a clear target function — does the code compile — and that outside coding the target you hand an agent is always incomplete. To test what target functions actually contribute, Langfuse built a minimal self-optimization loop on the cleanest signal they could find (single-label arXiv paper classification) and watched a frontier optimizer lift a cheap model from 68% to ~83%, with the single biggest jump on the very first run. The lesson she draws is that the loop only worked because the failure mode was quantifiable and reliable, so teams in medicine, compliance or chatbots must first encode domain expertise into binary, high-signal evaluators and enough data volume — otherwise the loop just burns tokens.", "key_points": [ "Coding got self-improvement first because it always had at least one clear target function — 'does the code compile or not' — which doesn't exist in healthcare, medical device compliance, or chatbots; and any target given to an agent is inherently incomplete (you think you're heading down here, but the optimal destination is somewhere else).", "Langfuse sees that the teams investing in the middle layer — capturing what they actually want and building good evaluators for it — are the ones who continuously improve their apps and ship with confidence.", "The experiment: classify arXiv papers by title+abstract into the author's primary label (10 labels), split 200 fit / 100 validate / 300 test to check for overfitting; the agent was a bare prompt on the small, cheap GPT-5.4 nano, and the optimizer was Claude Code running Claude Opus 4.8 with the GPT-5.4 prompting guide and a task.md describing the loop.", "Loop mechanics: baseline on fit+validate, error analysis on fit to find error clusters, Opus proposes a prompt update for the biggest error category, rerun, accept only if it also improves on validation; stopping criteria were 15 runs or 92% accuracy, and the test set link was deliberately withheld from the agent.", "Results: baseline 68%, +10 points on the very first iteration to 78%, peak 83% by the third/fourth iteration, then a plateau around 80%, and 80.2% on the untouched 300-item test set — about a 15% uptick overall that generalized.", "The plateau turned out to be label noise, not model failure: arXiv authors have creative freedom in choosing their primary category, so a 'wrong' prediction can still be reasonable, and that choice isn't recorded in the data.", "What the optimizer actually wrote was rules and examples, not what the speaker expected: a general classification approach, how to decide between two similar classes, commonly confused patterns, a rule to prefer the more specific label over the broad one, and examples of frequently misclassified pairs — she would have added label descriptions instead.", "Inspecting Opus's reasoning on the 10-point jump showed it counting 64 errors, naming the dominating patterns and the single biggest confusion pair, then forming an explicit hypothesis — possible only because the failure mode was clearly quantifiable and reliable, over 200 items and just 10 labels so every class got enough coverage.", "Generic scaled evaluators (correctness, helpfulness, hallucination on 0–1 or 1–5) are low signal for auto-improvement: each number would need defining per context and usually isn't, so scores are inconsistent across runs — LLM-as-a-judge is nondeterministic." ], "takeaways": [ "Replace scalar quality scores with binary, checkable criteria drawn from your domain: 'is the answer based on the knowledge base — yes/no', check that a snippet in the answer is in the retrieved context, check the brand name is spelled right and wasn't accidentally translated, or categorize which of five known failure modes occurred.", "Sit with domain experts to encode their expertise into concrete examples and sample runs, and keep asking 'but why is it this way here and the other way there' — that surfaces the implicit knowledge they think is obvious, and yields both the failure modes and the definition of 'good'.", "Build enough labelled volume to give the optimizer a real signal; 200 fit / 100 validate / 300 test over 10 labels was already enough here, and you need per-class coverage, not just raw count.", "Keep a held-out validation and test split as in traditional ML, and only accept a prompt change when it also improves on unseen data — don't let the loop overfit the error set it just analysed.", "Review production data yourself as a human, not only through your coding agents, because scope and failure modes shift over time and you want to catch them before customers do.", "Give the loop an escape hatch and a stopping criterion (here: 15 runs or 92% accuracy) so it doesn't spend hours hitting a wall; on a clean signal the first iteration may already give you most of the gain." ], "topics": [ "evals", "agents", "self-improvement", "prompt-optimization", "observability", "domain-expertise", "classification", "llm-as-judge" ], "tools": [ "Langfuse", "Claude Code", "Claude Opus 4.8", "GPT-5.4 nano", "GPT-5.4 prompting guide", "arXiv" ], "quotes": [ { "text": "these target functions are uh not nearly as clear and a target that you give an agent is actually also always incomplete", "at": "01:29", "url": "https://www.youtube.com/watch?v=eAXxdtNlK04&t=89s" }, { "text": "the biggest jump was uh just the very first one from a very clear-cut signal, we could have probably stopped there and already have very good baseline.", "at": "11:18", "url": "https://www.youtube.com/watch?v=eAXxdtNlK04&t=678s" }, { "text": "because if you can't do code compiles, you need to wrap your head a little bit differently around um what is good and what is not.", "at": "14:23", "url": "https://www.youtube.com/watch?v=eAXxdtNlK04&t=863s" }, { "text": "review this data and don't review it only with your coding agents but review it as a human", "at": "15:56", "url": "https://www.youtube.com/watch?v=eAXxdtNlK04&t=956s" } ], "words": 3882 }, "summary_url": "/#eAXxdtNlK04", "transcript": { "html": "/transcripts/eAXxdtNlK04.html", "txt": "/transcripts/eAXxdtNlK04.txt", "vtt": "/transcripts/eAXxdtNlK04.vtt" } }