{
  "video": {
    "id": "PXj0p_mW9nI",
    "title": "Tokens Should Have Jobs — Katelyn Lesse & Angela Jiang, Anthropic",
    "duration": 801,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "PXj0p_mW9nI",
    "title": "Tokens Should Have Jobs — Katelyn Lesse & Angela Jiang, Anthropic",
    "one_liner": "Anthropic's platform leads argue that tokens aren't fungible: at a fixed 600K-token budget, giving some tokens a non-executing job (advise, grade, dream) beat pure execution 89% vs 76% on a financial-analysis bench — and cut the true cost of a perfect answer.",
    "summary": "Katelyn Lesse and Angela Jiang challenge the default agentic lever of \"spend more tokens,\" which assumes every token is interchangeable. Instead they split an agent's budget so some tokens do a different job — an adviser the executor can call, a grader scoring against a rubric so the executor can iterate, or a \"dreamer\" that reads executor transcripts and writes learnings to memory for the next run. They benchmarked these strategies on financial-analysis tasks modelled on an expert human analyst, first one-shot, then at a fixed budget, then re-scored so only 100%-accurate runs pass — because a P&L that's 80% right forces the analyst to redo it. The strategies are built on Anthropic's agent harness plus a \"meta harness\" for multi-agent orchestration, with dreaming and outcomes shipped out of the box in Claude managed agents.",
    "key_points": [
      "The implicit assumption behind \"increase the budget\" is that every token is fungible; a strategy = some tokens executing plus some tokens doing a different job.",
      "Three named strategies: advise (executor calls out to an adviser mid-task), grade (a grader scores the executor against a rubric and it iterates until good), and dream (a dreamer inspects the executor's work and transcripts, writes findings to memory, which the executor picks up next round).",
      "Use-case framing per strategy: advise for a sales agent flagging an overdue follow-up or stalling deal; grade for a customer-service refund agent with specific refund criteria encoded as a rubric; dream for a recruiting agent that sharpens on feedback about candidate fit.",
      "One-shot bench (strategies chose their own budget): execute hit 15% accuracy using only 39,000 tokens; dream did much better but burned roughly 600,000 tokens — so the comparison says little.",
      "Holding budget constant at ~600K: execute rose from 0.15 to 76, advise and grade went from the 60s to near 90, advise at 89. If test-time compute were all that mattered, all four would land at the same level — the gap is the alpha.",
      "Re-scored for the real world, where anything under 100% on a P&L is a failure (\"you cannot make up an income number\"): execute passes 42% of runs, the more complex strategies up to 75%.",
      "True cost of a perfect answer: execute passes ~40% of the time, so ~3 runs × 600K = ~1.8M tokens; advise and grade are notably more token-efficient per usable output.",
      "The strategies are assembled on top of the single-agent harness used for Claude managed agents, with a meta-harness layer for multi-agent orchestration; dreaming and outcomes ship out of the box, and in the demo Fable is the model doing the advising."
    ],
    "takeaways": [
      "Before buying more budget, reallocate the budget you have: carve off tokens for an adviser, grader or dreamer and compare against pure execution at the same fixed token ceiling.",
      "Always benchmark strategies at a held-constant budget — a strategy that scores higher while choosing to spend 15× more tokens has proven nothing.",
      "Score against the consequence in your domain, not average accuracy: if a partially-correct answer forces a human redo, make 100% the pass bar and measure the pass rate.",
      "Compute expected total tokens to a usable answer (budget per run ÷ pass rate), not tokens per run — that's the number that reflects what the agent actually costs your business.",
      "Pick the strategy by what you're optimizing: advise when token efficiency dominates; grade or dream when reliability — maximizing the share of runs that come back perfect — matters more.",
      "Treat the four jobs as primitives to combine (execute → advise → grade → dream in a loop) and invent new jobs of your own."
    ],
    "topics": [
      "agents",
      "multi-agent orchestration",
      "evals",
      "test-time compute",
      "token efficiency",
      "agent memory",
      "llm-as-judge",
      "financial analysis"
    ],
    "tools": [
      "Anthropic",
      "Claude managed agents",
      "Claude platform",
      "Fable"
    ],
    "quotes": [
      {
        "text": "maybe you have tokens that are dreaming. They're reflecting back on the job that other executors have done and writing learnings to memory so that they can do it again.",
        "at": "01:58",
        "url": "https://www.youtube.com/watch?v=PXj0p_mW9nI&t=118s"
      },
      {
        "text": "if that result is 80% accurate on a bench that sounds great but in reality what that means for that expert is they have to go back and recomputee that P&L themselves",
        "at": "07:11",
        "url": "https://www.youtube.com/watch?v=PXj0p_mW9nI&t=431s"
      },
      {
        "text": "You cannot make up an income number or you can't make up a cost number right you have to make sure that it's 100% accurate.",
        "at": "07:31",
        "url": "https://www.youtube.com/watch?v=PXj0p_mW9nI&t=451s"
      },
      {
        "text": "tokens are not fungeable. You can use your tokens to execute. You can brute force your way through your task and you can throw more budget at it.",
        "at": "10:32",
        "url": "https://www.youtube.com/watch?v=PXj0p_mW9nI&t=632s"
      }
    ],
    "words": 3039
  },
  "summary_url": "/#PXj0p_mW9nI",
  "transcript": {
    "html": "/transcripts/PXj0p_mW9nI.html",
    "txt": "/transcripts/PXj0p_mW9nI.txt",
    "vtt": "/transcripts/PXj0p_mW9nI.vtt"
  }
}