{ "video": { "id": "TeErpYBUIeM", "title": "The Self-Improving OSS Agent Stack — Marc Klingen, Langfuse", "duration": 1027, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "TeErpYBUIeM", "title": "The Self-Improving OSS Agent Stack — Marc Klingen, Langfuse", "one_liner": "Keep humans in the loop on the data sets, evaluators and direction, which set the boundary agents optimise against, and let agents automate the tedious lower loops: mining production traces, proposing fixes and back-testing them.", "summary": "Marc Klingen (Langfuse) argues that better models now let teams hand the agent-improvement loop to AI. That loop is online tracing and monitoring plus offline data sets, experiments and evals. He layers loops on top of each other: AI is used most for proposing fixes and hill-climbing against data sets, and in roughly the last month teams have also started using it to maintain data sets and eval criteria. Humans still review those changes so the target doesn't overfit or become slop. He demos a changelog-writer agent that improves itself from GitHub PR approvals and change requests. He ends by arguing that this shift makes trace stores read-heavy, so teams should own an unsampled, long-retention data layer.", "key_points": [ "Building agents differs from building applications. It is a loop that joins online tracing and monitoring (traditionally observability tools like Datadog) with offline data sets, experiments and evals (traditionally MLOps tools like MLflow and Weights & Biases). Langfuse was built to join the two, because otherwise you benchmark on stale data sets or monitor production without benchmarking.", "Loops become possible as models improve: low-level loops like GitHub Copilot in 2023, level three around 2024, and in 2025–26 higher-level loops where agents reason about what to fix, propose fixes and verify them. In early Langfuse days their GitHub issue-to-PR automation could only edit single-HTML-file apps because multi-file edits were too hard.", "Humans stay involved in two places: aligning data sets and evaluators with real usage, and reviewing proposed fixes so the agent isn't overfit to a 'random quirk that we found in production'.", "AI is used most for proposing fixes, hill-climbing against data sets by trying options like a different model, a new way of aggregating context, or 'whatever you find on X' that week. Newer (over the last month or so) is using AI to keep data sets in line with real user behaviour and to propose new eval criteria, e.g. don't name competitors, be concise, answer in the user's language.", "A 2x2 framing: a fully manual loop is high time and high quality; full automation ('codex goal mode your way to success') is low time but risks lots of tokens that don't make sense; the 'promised land' is humans setting direction and AI doing the tedious work, with less time and potentially higher quality because AI can look through far more error cases.", "Implicit production signals can feed the loop without manual labelling: customers swearing at a support agent, saying 'this was not correct', or internal staff accepting or editing proposed messages.", "Demo: Langfuse's changelog-writer agent drafts docs and changelog updates as GitHub PRs. Approvals and change requests feed into tracing and evals. A coding agent using Langfuse agent skills found the writer factually correct but low on clarity, with a high human edit ratio and internal jargon leaking (e.g. 'ingestion pipeline', 'batched eval queue'). It proposed data set and evaluator changes, then a new implementation. The v2 candidate matched v1 on format compliance and accuracy and showed a positive delta on user-facing language. It was released via prompt management. That was acceptable 'yolo' because the agent only raises PRs.", "The best users collect production signals alongside detailed execution traces and run improvement agents on a loop, usually a cron job daily or weekly, whenever there's a new batch of user data. Sources include user feedback, internal labelling, or in-app annotation queues." ], "takeaways": [ "Own the human checkpoints: review every change agents make to data sets and evaluators, since these set the boundary other agents optimise against. Let AI propose and back-test implementation changes.", "Instrument your app to capture implicit feedback (edits, approvals, change requests, complaints) as signal tied to traces, so the improvement loop has input without manual labelling.", "Run an improvement agent on a schedule (cron, daily or weekly) that reads traces, finds failure patterns, updates evals and data sets, and compares a candidate against the baseline before release.", "Allow more autonomous auto-release only where the blast radius is small, e.g. an agent that only opens PRs.", "Keep traces long-term and unsampled in a data layer you control and that scales cheaply for read-heavy agent queries, because year-old traces become useful context." ], "topics": [ "agents", "evals", "observability", "self-improving-agents", "datasets", "human-in-the-loop", "llmops" ], "tools": [ "Langfuse", "GitHub Copilot", "Devin", "Cursor", "MLflow", "Weights & Biases", "Datadog", "Codex", "GitHub", "ChatGPT" ], "quotes": [ { "text": "If you give this completely out of hand, you risk like that you just create more slop", "at": "06:55", "url": "https://www.youtube.com/watch?v=TeErpYBUIeM&t=415s" }, { "text": "you assemble the plane while you're flying it", "at": "07:36", "url": "https://www.youtube.com/watch?v=TeErpYBUIeM&t=456s" }, { "text": "Now it's way more read intensive because agents can like like chew through so much more data.", "at": "16:15", "url": "https://www.youtube.com/watch?v=TeErpYBUIeM&t=975s" }, { "text": "you want to really like own the data layer", "at": "16:21", "url": "https://www.youtube.com/watch?v=TeErpYBUIeM&t=981s" } ], "words": 4008 }, "summary_url": "/#TeErpYBUIeM", "transcript": { "html": "/transcripts/TeErpYBUIeM.html", "txt": "/transcripts/TeErpYBUIeM.txt", "vtt": "/transcripts/TeErpYBUIeM.vtt" } }