{
  "video": {
    "id": "BOCjybvJYM4",
    "title": "Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse",
    "duration": 2663,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "BOCjybvJYM4",
    "title": "Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse",
    "one_liner": "Get tracing right first, since everything downstream depends on it. Then build monitors around the failure modes specific to your app (user disagreement, ALL-CAPS frustration, out-of-scope requests) rather than copying generic evals, and let a coding agent with the Langfuse skill sweep your traces for silent failures you never thought to evaluate.",
    "summary": "Langfuse frames continuous agent improvement as an 'AI engineering loop' with an online half (tracing, monitoring) and an offline half (datasets, experiments, evaluation). They argue many teams outside high-risk industries can get very far with the online half alone. The hands-on workshop instruments 'Specs', a dad IT-support agent built with the OpenAI Agents SDK in TypeScript. It wires up two LLM-as-a-judge evaluators (user disagreement, out-of-scope request) and one Python code-based evaluator (all-caps frustration) on live observations, with JSONPath mapping of trace fields. Then a coding agent (Codex) with the Langfuse skill/CLI analyzes the seeded production traces and finds a silent retrieval failure in the search_help_library tool that no evaluator covered.",
    "key_points": [
      "AI engineering differs from traditional engineering because agents are non-deterministic: system health can look fine while responses or actions are wrong or not good enough.",
      "The AI engineering loop has five steps in two parts: online (tracing, monitoring) and offline (building datasets, experimenting, evaluating). Teams not in high-risk industries can start with the online part alone, ship simple fixes and iterate fast. Mature products or high-risk environments should add offline structured testing and deploy only when experiments show no regressions.",
      "'Everything starts with your trace': a bad trace means missed signals in monitoring, weaker datasets and worse-informed eval decisions. A trace is made of nested observations (LLM calls, tool calls). LLM calls carry duration, token usage/cost, model and prompt version; tool calls carry their own data, such as the web search query.",
      "Monitoring has three functions: aggregate metrics over time (cost and quality), signal detection via user feedback, and feedback/evals. Automatic evaluators scale to every trace but miss new or nuanced cases. User feedback and human annotation scale less well but catch the things you hadn't thought of.",
      "Demo evaluators in Langfuse: a 'user disagreement' LLM-as-a-judge template targeting the root agent observation, with the full message array and the last user message mapped in via JSONPath; an 'is_all_caps_frustrated' Python code evaluator returning a boolean (TypeScript is also supported; alternatives include cuss words or brand-name spelling such as Langfuse with a lowercase f); and an 'out of scope request' judge targeting generations, so it can compare the system prompt against the last user message.",
      "Evaluator scores attach to the observation (e.g. user disagreement = true), so you can filter traces by score to see how often each failure happens. The same evaluators can also run on offline experiments.",
      "After roughly 70 seeded production traces, Codex with the Langfuse skill/CLI (which exposes all Langfuse APIs: traces, scores, datasets, experiment runs) produced a report. It found a silent retrieval-quality failure in search_help_library, retrieval misses driving up latency and cost through tool retries, an off prompt version, and an evaluator coverage gap where only a subset of traffic got scored. With repo context, the agent could then fix the code too.",
      "Closing the loop means maintaining evaluation criteria and above all maintaining datasets: production edge cases go into offline evals, and experiments wired into CI/CD let you ship with confidence. 'All of this is not a one-off.'"
    ],
    "takeaways": [
      "Invest enough time in trace instrumentation before anything else, and capture per-observation data such as model, prompt version, token cost and tool inputs.",
      "Design monitors around your application's own failure modes (e.g. disagreement, frustration, out-of-scope use), not a generic copy-pasted eval set, and use evaluator scores as filters to surface the traces worth reading.",
      "Combine scalable automatic evaluators with user feedback and manual trace review, because the human signals catch failure modes you didn't anticipate.",
      "Point a coding agent with the Langfuse skill/CLI at your production traces to find silent failures like tool, latency or retry issues beyond what your evaluators cover, but stay in the loop rather than letting it do everything alone.",
      "Their recommended day-one sequence: use the skill to add tracing, review traces manually, set up evals on production traces, then move to offline datasets and experiments in CI/CD."
    ],
    "topics": [
      "agents",
      "observability",
      "tracing",
      "evals",
      "llm-as-a-judge",
      "monitoring",
      "coding-agents",
      "ai-engineering"
    ],
    "tools": [
      "Langfuse",
      "ClickHouse",
      "OpenAI",
      "OpenAI Agents SDK",
      "TypeScript",
      "Python",
      "Codex",
      "Claude",
      "WhatsApp",
      "npm"
    ],
    "quotes": [
      {
        "text": "your status of your system can look healthy while the responses or the actions your agent is taking are wrong.",
        "at": "01:28",
        "url": "https://www.youtube.com/watch?v=BOCjybvJYM4&t=88s"
      },
      {
        "text": "Everything starts with your trace.",
        "at": "05:20",
        "url": "https://www.youtube.com/watch?v=BOCjybvJYM4&t=320s"
      },
      {
        "text": "it's really about thinking what is what are the typical fail modes we will um endure in our kind of application and not about like trying to copy paste u what's out there",
        "at": "31:07",
        "url": "https://www.youtube.com/watch?v=BOCjybvJYM4&t=1867s"
      },
      {
        "text": "I'm swearing I'm not p hacking",
        "at": "36:55",
        "url": "https://www.youtube.com/watch?v=BOCjybvJYM4&t=2215s"
      }
    ],
    "words": 8599
  },
  "summary_url": "/#BOCjybvJYM4",
  "transcript": {
    "html": "/transcripts/BOCjybvJYM4.html",
    "txt": "/transcripts/BOCjybvJYM4.txt",
    "vtt": "/transcripts/BOCjybvJYM4.vtt"
  }
}