{
  "video": {
    "id": "R3-anFK1YM8",
    "title": "Memory Harnesses for Long-Running Research Agents — Stefania Druga, Sakana.ai",
    "duration": 784,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "R3-anFK1YM8",
    "title": "Memory Harnesses for Long-Running Research Agents — Stefania Druga, Sakana.ai",
    "one_liner": "Stefania Druga (Sakana AI) ran a local-only memory-harness ablation on an M3 Ultra and found that a ranked decisions ledger beats vector RAG and memory-gating on long-horizon tasks — while adding nothing but cost when the task already fits in context.",
    "summary": "The talk treats agent memory not as a database but as a write–manage–read control loop wrapped around a model with zero durable memory, and tests a ladder of recall policies — no recall, vector RAG, a ranked decisions ledger, and an oracle that injects ground-truth memory — with the model held fixed. On a literature-review task whose whole corpus fits in context, memory added no capability and only cost; on Xbench, where the answer sits at step 124 and the question is asked at step 500, ranked recall wins, and does so on both Qwen 27B-Q4 and DeepSeek V4 flash and on Spider V2. Druga's heuristic is that bad memory is expensive — it burns tokens and sends the agent the wrong way — so the recall policy deserves to be a first-class metric. She ran everything on local models on her own Mac in Tokyo, framing that full control over data, compute traces and evals as sovereignty.",
    "key_points": [
      "Context rot matters more now because METR's projections show task horizons getting longer while model releases get fewer — the two curves converge later this year.",
      "Coinbase's CEO tweeted (two days before the talk) that they cut AI spend while increasing AI usage by shifting to local models plus better routing, caching, keeping context clean, and visibility into who uses what.",
      "Setup was an M3 Ultra with 96 GB and 28 CPU cores, still running evals in Tokyo with fans propped around it, controlled from her phone, using Qwen 27B quantised to 4-bit and DeepSeek V4 flash.",
      "Harness design: agents with zero durable memory, an always-shown core of traces, a recall block where the policy varies, and an archival block spanning sessions; only the recall variables change across runs.",
      "Recall ladder: no recall (baseline) → vector RAG → a decisions ledger that records and prioritises decisions per turn → an oracle that hands the harness the ground-truth memory.",
      "Literature-review task used a corpus containing the Nature paper claiming 742,000 promising materials — later retracted, with the retraction a much smaller needle than the headlines and citations. Because everything fit in context, memory matched no-memory performance and only added cost.",
      "On Xbench (68 questions, many cells and seeds) with the answer at step 124 and the question asked at step 500, the rank-only ledger performed best — better than simply gating on 'do you need memory or not'.",
      "The oracle does not hit maximum performance because it supplies the right memory but cannot force the model to use it; the model can still retrieve wrongly, ignore it, or get confused.",
      "Ablations (arbitrary examples, wrong step, most recent step) all left the ranked policy on top; the result held on DeepSeek V4 flash and on the Spider V2 benchmark, and cost fewer tokens.",
      "The wider landscape includes 30+ runnable memory cookbooks in an open-source repo, spanning file-system retrieval through to trained memory models, from unstructured to fully structured."
    ],
    "takeaways": [
      "Check whether the task actually overflows the context window before building a memory harness — if it fits, memory buys no accuracy and only adds cost.",
      "Rank memories rather than gating them: a rank-only decisions ledger beat both vector RAG and a yes/no 'do I need memory' gate on long-horizon retrieval.",
      "Treat the recall policy as a first-class metric — decide explicitly what memory types you store, how you rank them, how the recall function is designed, and what survives across many sessions and runs.",
      "Include an oracle condition as your reference ceiling, and expect it to fall short of max — the gap tells you how much loss is retrieval versus the model refusing to use what it was given.",
      "Budget for serial execution if you go local: DeepSeek V4 flash offers no batch querying, so evals take days — the payoff is full control over data, compute traces and evaluations."
    ],
    "topics": [
      "agents",
      "memory",
      "long-horizon-tasks",
      "evals",
      "rag",
      "local-models",
      "context-engineering",
      "sovereign-ai"
    ],
    "tools": [
      "Sakana AI",
      "METR",
      "Coinbase",
      "GLM",
      "DeepSeek V4 flash",
      "Qwen 27B (4-bit quantised)",
      "Apple M3 Ultra",
      "Xbench",
      "Spider V2",
      "Nature"
    ],
    "quotes": [
      {
        "text": "you can think of memory as a write manage read loop so it's not just the database store it's actually this control loop around the model",
        "at": "03:46",
        "url": "https://www.youtube.com/watch?v=R3-anFK1YM8&t=226s"
      },
      {
        "text": "when your task fits in context, the harness doesn't add much.",
        "at": "06:33",
        "url": "https://www.youtube.com/watch?v=R3-anFK1YM8&t=393s"
      },
      {
        "text": "the oracle what it does it provides the right information the right memory to the model but it doesn't force it to use it",
        "at": "08:40",
        "url": "https://www.youtube.com/watch?v=R3-anFK1YM8&t=520s"
      },
      {
        "text": "bad memory is expensive because it spends more token and it can send agent the wrong way",
        "at": "09:45",
        "url": "https://www.youtube.com/watch?v=R3-anFK1YM8&t=585s"
      }
    ],
    "words": 2192
  },
  "summary_url": "/#R3-anFK1YM8",
  "transcript": {
    "html": "/transcripts/R3-anFK1YM8.html",
    "txt": "/transcripts/R3-anFK1YM8.txt",
    "vtt": "/transcripts/R3-anFK1YM8.vtt"
  }
}