{
  "video": {
    "id": "ys6YQ2yfBpM",
    "title": "Your Old Prompts Are Making Claude Worse - Lance Martin, Anthropic",
    "duration": 3843,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "ys6YQ2yfBpM",
    "title": "Your Old Prompts Are Making Claude Worse - Lance Martin, Anthropic",
    "one_liner": "Your application is often sitting inside the cost/quality frontier because of minor misconfigurations, so audit old prompts, fix prompt caching, build climbable evals, let Claude hill climb on them, and try a stronger model at low effort.",
    "summary": "Lance Martin argues that cost and quality are not always a trade-off, because many applications sit inside the Pareto frontier due to fixable misconfigurations. He covers legacy prompt anti-patterns, prompt cache use, eval design, autonomous hill climbing with safeguards against overfitting, and harnesses for long-running agents. Each theme comes with a simple action, mostly a skill bundled in Claude Code. The Q&A stresses that time spent up front on eval design is what lets frontier models succeed autonomously.",
    "key_points": [
      "Prompts accumulate debt like code: 'think step-by-step', verification rituals, thoroughness boosters and scratch pads are no longer needed by frontier models, and they raise cost and can reduce performance. Contradictory rules, which earlier models often handled by ignoring one rule, now often confuse frontier models.",
      "The `checkup prompt audit` and `Claude API prompt audit` commands link to the same underlying skill. It audits skills, tools and prompts in the working directory and removes anti-patterns. In a synthetic 50-ticket customer support eval simulating an Opus 4.8 to 5.5 migration, running it after the migration gave a further improvement in both performance and cost.",
      "Prompt cache reads are billed at about a tenth of the base token rate (he advises checking the pricing page) and are faster, but the cache is a strict prefix match, so a one-byte change invalidates it. Running the cache diagnostics on internal benchmark harnesses found opportunities to cut costs by up to 72% with no meaningful performance change.",
      "A good eval improves with a stronger model and with more thinking, has headroom at the frontier, and has low run-to-run variance. Even popular evals contain ambiguous tasks, so the last 20% may be unachievable and hill climbing on it burns tokens.",
      "`Claude API build eval` is not opinionated about where your data lives and walks you through building an eval from raw traces, including sanity-checking the grader. After feedback from Hamel, Lance has a PR up to add a stage that shows the user the raw data first.",
      "`Claude API hill climb` splits an eval into train and test sets and only reflects on train failures. Train improving while test stays flat is an early overfitting signal. Convergence is not guaranteed: if stuck for multiple rounds, it clusters the remaining failures and looks for ambiguous cases or faulty graders.",
      "Hill climbing the Claude API skill against an eval built from Anthropic's documentation started at 66% and improved significantly. Hill climbing for cost on the support benchmark, searching model, effort and prompt from Opus 4.8 high, reached a 90% cost reduction with a notable performance improvement, and this works even on a saturated eval.",
      "As task durations pass the range METR can measure, harnesses must prioritize reliability and security. Claude Managed Agents decouples the brain (harness) from the hands (sandbox), keeps an independent append-only recoverable session log, and holds secrets in vaults so Claude can reference them but never see them."
    ],
    "takeaways": [
      "Run the prompt audit when migrating to a frontier model such as Opus 5.5, and run the cache diagnostics on API code while tracking your prompt cache hit rate. Order prompts from global (system, tools) to per-project (CLAUDE.md, memory), per-session state, then per-turn messages.",
      "Start with a stronger model at low effort, such as Opus 5.5 low, and dial up only if needed. Max effort is extremely token hungry, and a weaker model at high effort is often less token efficient and more expensive.",
      "Design graders a human can reason about: prefer deterministic checks, then binary classifiers or pairwise comparisons over arbitrary 0-10 scales. Validate rubrics by having Sonnet, Opus and Fable all grade with them and by running replicates.",
      "Promote eval cases that are hard or interesting from first principles and production data rather than sampling only one model's failures, since those valleys may vanish on upgrade. If performance on an older eval does not rise as models improve, question the eval.",
      "Start hill climbing on tightly coupled text surfaces like prompts and skills rather than harness code, which produces larger, harder-to-interpret diffs. Audit the results, because prompts are very easy to overfit to the eval distribution."
    ],
    "topics": [
      "prompt engineering",
      "prompt caching",
      "evals",
      "hill climbing",
      "cost optimization",
      "agent harnesses",
      "context engineering",
      "effort levels"
    ],
    "tools": [
      "Anthropic",
      "Claude Code",
      "Claude API",
      "Claude Console",
      "Claude Managed Agents",
      "Claude Fable",
      "Claude Opus",
      "Claude Sonnet",
      "Claude Haiku",
      "METR",
      "DSPy",
      "autoresearch",
      "nanoGPT",
      "Recursive Language Models (RLM)",
      "LangChain",
      "Arize",
      "Braintrust",
      "Slack"
    ],
    "quotes": [
      {
        "text": "prompts accumulate debt just like code.",
        "at": "01:37",
        "url": "https://www.youtube.com/watch?v=ys6YQ2yfBpM&t=97s"
      },
      {
        "text": "optimized prompt cache has no change in performance. It's like effectively just free cost savings.",
        "at": "08:29",
        "url": "https://www.youtube.com/watch?v=ys6YQ2yfBpM&t=509s"
      },
      {
        "text": "I like very smart model not trying too hard.",
        "at": "40:40",
        "url": "https://www.youtube.com/watch?v=ys6YQ2yfBpM&t=2440s"
      },
      {
        "text": "and Claude can find ways to shape this prompt to like make number go up and it'll just do that.",
        "at": "54:14",
        "url": "https://www.youtube.com/watch?v=ys6YQ2yfBpM&t=3254s"
      }
    ],
    "words": 14315
  },
  "summary_url": "/#ys6YQ2yfBpM",
  "transcript": {
    "html": "/transcripts/ys6YQ2yfBpM.html",
    "txt": "/transcripts/ys6YQ2yfBpM.txt",
    "vtt": "/transcripts/ys6YQ2yfBpM.vtt"
  }
}