{
  "video": {
    "id": "WiqDvX6isc4",
    "title": "Scaling Compute on Context — Jack Morris, Engram",
    "duration": 1182,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "WiqDvX6isc4",
    "title": "Scaling Compute on Context — Jack Morris, Engram",
    "one_liner": "Jack Morris (Engram) argues that the one scaling axis still available for *your* private data is compute — you can't make more data and can't pre-train from scratch — and walks through why naive fine-tuning, KV compaction, on-policy distillation and synthetic continued pre-training all saturate, leaving self-improvement as the missing piece.",
    "summary": "Morris frames 'scaling compute on context' — his name for what others call continual learning, sleep-time compute, neural memory, note-taking or amortized inference — as the pursuit of *depth* rather than the breadth models already get from public pre-training. Since models are by definition trained on data that can be told to any user, they know Wikipedia, Reddit, arXiv and GitHub but nothing about your emails, your company, or long-tail skills like writing AMD kernels; with a fixed private corpus D and a pre-trained model you must start from, data and model-size scaling are off the table and only compute remains. He surveys the candidate techniques — next-token training on D, KV compaction, on-policy distillation / cartridges-style self-study, synthetic continued pre-training, unsupervised RL environments — and shows each hits a synthetic 'data wall' where the model fits the generated data and stops improving. The open problem, and what Engram is working on, is a second stage: recursive self-improvement in the AlphaGo mould, where a better model generates harder training data for itself.",
    "key_points": [
      "Breadth vs depth framing via Terence Tao: AI knows every public mathematical topic and can bridge literature gaps no human could, but lacks the near-subconscious intuition of a grad student who spent 5 years in one area.",
      "Three failure modes of current models: no knowledge after the pre-training cutoff, poor long-tail skill acquisition (models are 'quite bad at writing AMD kernels' because few good public AMD kernels exist), and no knowledge of you — your emails, your writing style, your company's partnerships — because training data is public by definition.",
      "The post-training data layer from Scale AI, Surge AI and Mercor doesn't fix this: hired experts still produce publicly-shareable data, so scaling is only ever applied to public data.",
      "Of the three scaling axes (more data, more compute/longer training, bigger models), private-context learning loses the data axis (you can't create more of D) and the model-size axis (you can't train from scratch and want the world knowledge in a pre-trained base) — leaving compute alone.",
      "Caveat he flags: data access isn't actually fixed in practice — like studying a textbook, you can find other textbooks, search the internet, or proactively talk to people who know the thing.",
      "Naive next-token training on D fails twice over: you can drive loss to ~0.0001 and the model knows the data perfectly, but 'when you generate from it, it basically collapses', and it can't answer a question unless the question and answer are encoded verbatim in the data. It's also a bounded axis — information transfers, then learning stops.",
      "Trick-it-into-thinking-D-is-in-context approaches: KV compaction (like Claude Code / Codex compaction, including a learned greedy approximation) only works for data that fits in context and misses the magic of taking gradients; on-policy distillation works but raises the question of what data to distill, since you can't distill raw documents — hence cartridges-style self-study generating Q&A pairs conditioned on D.",
      "Simulating pre-training with synthetic data conditioned on D is 'pretty promising' but overwrites some pre-training, is hard to scale, and forces a post-training stage afterwards — awkward because most people start from post-trained models, not good base models. Unsupervised RL environments with GRPO-style losses are a close cousin.",
      "The shared defect: define a dataset, train on it, and unless the model is under-parameterized it eventually learns all of it — a synthetic data wall. Engram's early curves plateaued exactly this way regardless of how much data was generated.",
      "The wanted property is arbitrary compute → more depth, which Morris attributes to self-improvement: 'AlphaGo makes its own training questions harder by getting better through training' — generate data, model improves, generate better data, recursively."
    ],
    "takeaways": [
      "Stop expecting next-token fine-tuning on your private corpus to produce a model that 'knows' it — near-zero loss coexists with generation collapse and no generalization beyond verbatim Q&A pairs.",
      "Start from a pre-trained model rather than training on your data from scratch: the outside-world knowledge is what makes your internal context interpretable, and it's the only way to keep a scaling axis open.",
      "When picking a technique, ask what happens on the *second* pass — any method that defines a fixed synthetic dataset will saturate once the model fits it; budget for a mechanism that makes the next round of training harder.",
      "Don't treat your data budget as truly fixed: pull in related textbooks, internet search, and proactive interviews with people who know the domain, the way a human studying a subject would.",
      "If you have a post-trained model rather than a good base model, factor in that synthetic continued pre-training will require you to post-train again afterwards."
    ],
    "topics": [
      "continual-learning",
      "scaling",
      "personalization",
      "distillation",
      "synthetic-data",
      "context",
      "rl",
      "self-improvement"
    ],
    "tools": [
      "Engram (ngr.am.com)",
      "ChatGPT",
      "GPT-5",
      "Claude",
      "Anthropic",
      "GLM 5.2",
      "Claude Code",
      "Codex",
      "OpenCode",
      "Scale AI",
      "Surge AI",
      "Mercor",
      "METR",
      "AlphaGo",
      "GRPO",
      "cartridges paper (self-study)",
      "on-policy distillation",
      "GitHub",
      "Wikipedia",
      "Reddit",
      "arXiv"
    ],
    "quotes": [
      {
        "text": "it's nothing to do with my pre-training cutoff because I'm a human I'm capable of acquiring new information.",
        "at": "02:56",
        "url": "https://www.youtube.com/watch?v=WiqDvX6isc4&t=176s"
      },
      {
        "text": "It's like the core problem with the current paradigm in AI that models cannot acquire new knowledge after training in in a personalized way.",
        "at": "04:01",
        "url": "https://www.youtube.com/watch?v=WiqDvX6isc4&t=241s"
      },
      {
        "text": "And then when you generate from it, it basically collapses.",
        "at": "11:18",
        "url": "https://www.youtube.com/watch?v=WiqDvX6isc4&t=678s"
      },
      {
        "text": "AlphaGo makes its own training questions harder by getting better through training.",
        "at": "17:40",
        "url": "https://www.youtube.com/watch?v=WiqDvX6isc4&t=1060s"
      }
    ],
    "words": 4202
  },
  "summary_url": "/#WiqDvX6isc4",
  "transcript": {
    "html": "/transcripts/WiqDvX6isc4.html",
    "txt": "/transcripts/WiqDvX6isc4.txt",
    "vtt": "/transcripts/WiqDvX6isc4.vtt"
  }
}