{
  "video": {
    "id": "XzJD1bvXKjs",
    "title": "How long can your skills be before your agent forgets what you told it? — Laurie Voss, Arize AI",
    "duration": 1346,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer"
  },
  "analysis": {
    "video_id": "XzJD1bvXKjs",
    "title": "How long can your skills be before your agent forgets what you told it? — Laurie Voss, Arize AI",
    "one_liner": "The old \"~200 instructions and the model starts forgetting\" ceiling moved about 10x in a year — frontier models now track roughly 2,000 named constraints (5,000 for the best), so writing skills files is no longer a compression problem but a verification one.",
    "summary": "Laurie Voss chased down an aside from Dexter Horthy's Miami talk — that an agent can follow about 200 instructions before it starts dropping them — back to the IFScale benchmark, replicated the original result on the three surviving 2025 models, then re-ran it on the current frontier. The old models fell apart at 200–300 rules; GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro and DeepSeek V4 Pro aced the benchmark's 500-word cap outright, forcing him to extend it to a 10,000-word vocabulary to find the new ceiling at roughly 2,000–5,000 instructions. The more useful finding is that failure is no longer a single mode: each model breaks in its own way, and the most dangerous ones look like success. His conclusion is that capacity is solved enough that the remaining question is cost, latency and whether you verify the output at all.",
    "key_points": [
      "The 200-instruction figure is real and traceable: it comes from the IFScale benchmark, whose test is to ask for a business report that must include a list of exact keywords, then count how many appeared — two numbers only, density (n) and accuracy.",
      "Replication was possible on only 3 of the original paper's 10 models (GPT-4.1, Claude Sonnet 4, Gemini 2.5 Pro) because the rest were retired from every API; one of those three has since been retired too. The replicated curves matched the paper within noise: by 500 rules the old models lost 30–50% of instructions.",
      "GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro and DeepSeek V4 Pro all scored 100% on the original benchmark immediately, so Voss doubled the vocabulary from 500 to 1,000 to 2,000 and up to 10,000 words to find where they bend — roughly 2,000 instructions for most, up to 5,000 for the best. About a 10x improvement in 12 months.",
      "Four distinct failure modes: DeepSeek V4 just forgets (starts around 750 rules, dropping nearly half by 2,000); Claude Opus 4.7 refuses at the API level when its safety classifier sees dangerous-looking word combinations like anthrax and cyanide, and can bail at 200–300 instructions on dual-use topics like medical advice; Gemini 3.1 Pro is rock solid to 5,000 then spends its whole token budget thinking (9,500 thinking tokens, a 500-word answer containing none of the keywords); GPT-5.5 hits 99% accuracy to 5,000 rules, then starts the report and quits partway through, politely telling you the task is stupid.",
      "GPT-5.5's polite half-finished report is the dangerous failure because it looks like a real answer — Claude's refusal is annoying but loud, while GPT's only reveals itself if you read to the end.",
      "To get Claude to run at all, the random word list had to be pre-filtered through OpenAI's safety filter to remove naughty-looking words.",
      "The whole study — 2,300 calls across seven models — cost $29.",
      "Caveats he raises himself: it's a proxy task (evidence, not proof, that long skills files work); models hit the wall anywhere from 750 to 9,000+; Chroma's context rot work across 18 models shows accuracy on long inputs falling 30–50% well before the context window limit, and oddly coherent well-structured text is more prone to that than shuffled instructions; a paper testing 46 models, 'Revisiting the reliability of language models in instruction following', found that rewording or reordering the same instructions can radically change how well they're followed."
    ],
    "takeaways": [
      "Stop compressing. The year-old practice of keeping each skills file under 200 instructions and pointing off to a 'byzantine labyrinth' of sub-skills is obsolete — if your use case needs 100 or 300 specific rules, put them all in one prompt; 2,000 named constraints is an entire style guide, every brand rule and legal disclaimer.",
      "Re-open any engineering assumption about prompt or skills-file length made more than about six months ago, because it is probably wrong now.",
      "Treat length as a cost/latency trade-off rather than a hard wall: the model can hold the instructions, but a 10,000-instruction prompt is enormous, expensive and slow.",
      "Learn your specific model's failure signature — 'did it follow my instructions' now has four different answers — and check outputs in production with an LLM-based eval, since a silent half-finished response looks confident and polished.",
      "Pick the model deliberately: the wall lands anywhere from 750 to 9,000+ instructions depending on which one you use, and Claude's safety classifier will refuse early on dual-use content."
    ],
    "topics": [
      "agents",
      "skills",
      "prompt-engineering",
      "benchmarks",
      "instruction-following",
      "evals",
      "context-windows",
      "model-comparison"
    ],
    "tools": [
      "Arize AI",
      "IFScale",
      "GPT-4.1",
      "Claude Sonnet 4",
      "Gemini 2.5 Pro",
      "GPT-5.5",
      "Claude Opus 4.7",
      "Claude Opus 4.8",
      "Gemini 3.1 Pro",
      "DeepSeek V4 Pro",
      "OpenAI safety filter",
      "Chroma",
      "Firebench",
      "CCR bench",
      "Guidebench",
      "npm Inc."
    ],
    "quotes": [
      {
        "text": "so we'd built a test to find the ceiling and the models had walked straight through the ceiling without noticing that the ceiling was there.",
        "at": "06:34",
        "url": "https://www.youtube.com/watch?v=XzJD1bvXKjs&t=394s"
      },
      {
        "text": "Deep Sea quietly forgets, Claude gets scared and refuses, Gemini overthinks itself into silence, and GPT 5.5 finishes half of the job and tells you that the rest of it is beneath it.",
        "at": "13:53",
        "url": "https://www.youtube.com/watch?v=XzJD1bvXKjs&t=833s"
      },
      {
        "text": "you did that more than about six months ago, you are incorrect now, and you should probably be re-engineering how you do stuff.",
        "at": "08:45",
        "url": "https://www.youtube.com/watch?v=XzJD1bvXKjs&t=525s"
      },
      {
        "text": "the model will hold your 2,000 instructions just fine. The new hard part is knowing whether it actually did what you said and that is a verification problem.",
        "at": "21:08",
        "url": "https://www.youtube.com/watch?v=XzJD1bvXKjs&t=1268s"
      }
    ],
    "words": 5000
  },
  "summary_url": "/#XzJD1bvXKjs",
  "transcript": {
    "html": "/transcripts/XzJD1bvXKjs.html",
    "txt": "/transcripts/XzJD1bvXKjs.txt",
    "vtt": "/transcripts/XzJD1bvXKjs.vtt"
  }
}