{
  "video": {
    "id": "rTojoVotlD8",
    "title": "Your LLM App Returned 200 OK. It Was Still Wrong. — Marina Petzel, Datadog",
    "duration": 1081,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "rTojoVotlD8",
    "title": "Your LLM App Returned 200 OK. It Was Still Wrong. — Marina Petzel, Datadog",
    "one_liner": "The golden signals (latency, errors, traffic, saturation) tell you whether your GenAI app is up but not how well it is working. Add a second layer of cost, safety and quality metrics, because a 200 OK can still be a hallucination.",
    "summary": "Petzel argues that the classic golden signals are still needed but are not enough for generative AI apps. These apps give different outputs for the same input, have variable token-driven costs, face new attacks such as prompt injection and jailbreaks, and produce output whose quality sits on a spectrum. She proposes three extra monitoring layers. Cost covers token creep, model drift, missing caches and mandatory tagging. Safety covers prompt injection, PII, toxicity and jailbreaks. Quality covers hallucination, relevance, user satisfaction, completeness and RAG retrieval quality. She closes by pitching Datadog's agent observability product.",
    "key_points": [
      "GenAI apps differ from traditional apps in four ways. The same prompt gives different outputs, so regression testing alone isn't enough and quality must be evaluated live. Cost varies with tokens, model and context window size. There are new attack vectors (prompt injection, jailbreaking, PII leaking in output) that a 500 error will never catch. Quality is judged on a spectrum: relevant, accurate, complete.",
      "There are three cost failure modes. Token creep is a context window quietly growing from about 4,000 tokens without financial review, which can mean up to 8x cost. Model drift is switching from a cheap model like Haiku 4.5 to Opus 4.8, which can cost about 15x more per request. Missing caching: Datadog's own research found that sometimes up to 70% of spend is redundant, paying again and again for the same completion.",
      "Cost attribution means you must 'tag absolutely everything' at four levels. Feature (chat vs summarization) shows which features are budget hogs. User (user ID or org name) supports chargebacks and abuse detection. Model and provider (e.g. GPT-5.5, OpenAI or Anthropic) lets you benchmark models. Endpoint (region, production vs staging) supports infrastructure planning.",
      "There are four safety metrics. Prompt injection rate is caught with pattern matching or specialised classifier models, against aggressive thresholds. PII detection rate is caught by scanning outputs with regex or named entity recognition; the only acceptable target is 0% of production outputs. Content moderation score comes from running all outputs through a toxicity classifier. Jailbreak attempts should be treated as an absolute blocker, ideally blocking 100%.",
      "There are five quality metrics. Hallucination rate is the share of responses with claims the grounding data doesn't support, measured with manual review plus automated fact checks. Relevance score uses BERT or embedding similarity. User satisfaction comes from thumbs up/down, star ratings and NPS, with a target of at least 80–85% positive. Answer completeness is judged by an LLM. RAG retrieval quality is measured with top-K accuracy or normalized discounted cumulative gain (NDCG).",
      "RAG retrieval quality matters more as contexts grow, because picking the most relevant documents gets harder.",
      "Final framing: keep the golden signals, and add a cost, safety and quality layer on top of them."
    ],
    "takeaways": [
      "Make tagging mandatory on every LLM call: feature, user or org, model and provider, and environment or region. That way you can see where spend actually goes.",
      "Put a financial review on context window increases and model upgrades, and add a caching layer for repeated queries.",
      "Scan every production output for PII (regex or named entity recognition) with a 0% target, and treat jailbreak attempts as blockers.",
      "Evaluate quality continuously in production, not just with pre-release regression tests. Track hallucination rate, relevance, user feedback, LLM-judged completeness and, for RAG, retrieval metrics such as top-K accuracy and NDCG."
    ],
    "topics": [
      "observability",
      "llm monitoring",
      "cost management",
      "ai safety",
      "evals",
      "rag",
      "hallucination",
      "prompt injection"
    ],
    "tools": [
      "Datadog",
      "Datadog Agent Observability",
      "Claude Haiku 4.5",
      "Claude Opus 4.8",
      "GPT-5.5",
      "OpenAI",
      "Anthropic",
      "BERT"
    ],
    "quotes": [
      {
        "text": "Traditional metrics will tell you if your system is up and running, but it won't tell you how well it's running.",
        "at": "03:35",
        "url": "https://www.youtube.com/watch?v=rTojoVotlD8&t=215s"
      },
      {
        "text": "we all know that 200 okay response from your server sometimes can be a lie if the answer is hallucination.",
        "at": "12:50",
        "url": "https://www.youtube.com/watch?v=rTojoVotlD8&t=770s"
      },
      {
        "text": "you must tag absolutely everything.",
        "at": "07:11",
        "url": "https://www.youtube.com/watch?v=rTojoVotlD8&t=431s"
      },
      {
        "text": "And only acceptable target should be 0% of your production outputs.",
        "at": "10:56",
        "url": "https://www.youtube.com/watch?v=rTojoVotlD8&t=656s"
      }
    ],
    "words": 2760
  },
  "summary_url": "/#rTojoVotlD8",
  "transcript": {
    "html": "/transcripts/rTojoVotlD8.html",
    "txt": "/transcripts/rTojoVotlD8.txt",
    "vtt": "/transcripts/rTojoVotlD8.vtt"
  }
}