{
  "video": {
    "id": "IJXjTLPzvAU",
    "title": "The Miranda Hypothesis: How Hamilton Poisoned Persona Evals - Jacob E. Thomas, Results Gen",
    "duration": 3497,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "Online"
  },
  "analysis": {
    "video_id": "IJXjTLPzvAU",
    "title": "The Miranda Hypothesis: How Hamilton Poisoned Persona Evals - Jacob E. Thomas, Results Gen",
    "one_liner": "A data scientist argues that persona/role-playing evals like InCharacter (80.7% personality alignment) measure fluency and personality consistency — which is exactly why they cannot detect the field's dominant failure: models rendering historical figures as culturally saturated composites (the Hamilton-musical Hamilton, the Spielberg Lincoln) rather than the documentary record.",
    "summary": "Jacob E. Thomas names the 'Miranda hypothesis': for any sufficiently salient historical figure, culturally dominant representations outweigh the primary record in training corpora (the Hamilton musical's derivative corpus vs. the Federalist Papers' fixed ~175,000 words), next-token prediction compresses both with no capacity to distinguish a 1789 letter from a 2019 tweet, and the output is a salience-weighted composite that matches the figure at no verifiable moment in their life. Because human raters carry the same mythologized frameworks, RLHF amplifies rather than corrects this — 'algorithmic sycophancy' rewards handing you the Hamilton you already believe in. He proposes a fourth paradigm beyond cognitive simulation — epistemic simulation: corpus-bounded, temporally anchored, expert-loop evaluated — implemented as context-window anchoring rather than fine-tuning, and reframes the unit of analysis from 'agent' to 'role-playing language system' (prompt + anchor documents + temporal anchor + swappable model + human curator). He ends by handing the audience a pre-registered instrument (no results yet) built with historian Rick Halpern: four Lincolns × three seeding conditions × five diagnostic questions, scored on a rubric that deliberately refuses to score whether it sounds like Lincoln.",
    "key_points": [
      "The InCharacter benchmark reports state-of-the-art systems at 80.7% alignment with human-perceived personality — but that same high-scoring system renders a Hamilton 'who sounds like he's read his own Broadway musical'; convincingness and fidelity are independent properties, and the eval only measures the first ('the mask,' never 'the mirror').",
      "Live demo of the failure on a frontier model: asked about slavery, the model's Hamilton gives a clean abolitionist (New York Manumission Society member), while the historian notes the record also documents his transactions involving enslaved persons for in-laws and clients and his reliance on a slaveholder coalition he did not publicly oppose — the model sands a disputed record into a comfortable hero, as the musical did first.",
      "The hypothesis has three clauses — inputs (dominant representations exceed the primary record in volume and recency), mechanism (autoregressive prediction has no architectural capacity to distinguish a 1789 letter from a 2019 viral tweet), output (a fluent, morally legible composite corresponding to no verifiable moment).",
      "Alignment amplifies rather than fixes it: raters were formed by the same narratives that saturate the corpus, so preference optimization rewards the mythologized version — 'algorithmic sycophancy.' Compositing is not a post-training patch; post-training reinforces it.",
      "Non-model evidence: within a year of the musical's premiere, the Schuyler Mansion recorded a near tripling of annual visitors, arriving with 'facts' that were often wrong (believing three Schuyler daughters, when there were 15 children, eight surviving to adulthood) — staff work became 'unteaching the musical.'",
      "Fine-tuning is worse than context-window anchoring for this problem: it layers a thin personal signal over vast cultural sediment in the base weights and breaks provenance ('the archive has been consumed'). Supporting evidence: a 2026 Nature Medicine study where general-purpose Google/OpenAI/Anthropic models beat dedicated clinical AI tools on physician-reviewed, blinded tasks across 12 clinics, plus a study where biomedically fine-tuned models underperformed their base models via catastrophic forgetting.",
      "Time-locked models (Varnum et al.) are endorsed but solve a different problem — a model locked to 1789 still averages everything pre-1789 about Hamilton: 'Period anchoring is not persona anchoring.'",
      "The pre-registered instrument: four documented Lincolns (1847 Whig attacking Polk's war as unconstitutional; 1858 Free Soil Republican who at Charleston denied favoring black citizenship; 1860 constitutional unionist exploring colonization; 1862–65 emancipator who did by executive order the thing his 1847 self called unconstitutional) × three conditions (primary sources / modern biography / bare model) = 12 cells, five historian-written diagnostic questions, 60 scored response units.",
      "The rubric is weighted anachronism detection 40%, documentary consistency 35%, contextual plausibility 25% — and deliberately throws out rhetorical authenticity, because 'the Hamilton musical problem is fundamentally a voice problem masquerading as a content problem'; plain-but-faithful beats fluent-but-anachronistic, an inversion current eval stacks cannot perform."
    ],
    "takeaways": [
      "If you ship character bots, companion AI, pedagogical agents or historical simulations, stop treating personality-consistency and LLM-as-judge fluency scores as fidelity — add an axis that asks what the figure could have known, believed or argued at a specific moment, and weight anachronism detection highest (he uses 40%).",
      "Prefer putting anchor documents in the context window over fine-tuning on a figure: documents stay documents, provenance is preserved, the configuration is diffable/revertible/reproducible in a repo, and it's legible to a domain expert who isn't an ML engineer — and it's a 'kitchen table capability' (literacy + documents + any free-tier frontier model) rather than an institutional one.",
      "Treat the domain expert as a build-time and gate-time requirement, not a runtime cost: a classicist for a Marcus Aurelius tutor, a theologian for a scripture companion, a clinical psychologist for a therapeutic persona — they author the diagnostic questions, a priori vignettes, weighted rubric and held-out gold set once; that becomes an eval gate you re-run whenever you swap the base model.",
      "Change the unit of analysis for role-playing systems from 'agent' to the configured encounter — structured prompt, anchor material, temporal anchor, swappable off-the-shelf model, human curator — so the persona is versionable and auditable rather than smeared across billions of parameters.",
      "Run the six-step protocol yourself: pick a figure with both a primary record and a saturating composite, identify three or four moments where reasoning demonstrably differs, write fault-line questions with an expert, run primary/biography/bare conditions, score blind on the three-axis rubric, and report confirm or refute."
    ],
    "topics": [
      "evals",
      "personas",
      "role-playing agents",
      "context engineering",
      "fine-tuning",
      "rlhf",
      "training data",
      "digital humanities"
    ],
    "tools": [
      "Character AI",
      "Hello History",
      "Claude Opus 4.7",
      "Companion (speaker's open-source prompt framework)",
      "InCharacter benchmark",
      "CoSER (Wang et al.)",
      "Character-LLM",
      "GPT-4",
      "Google",
      "OpenAI",
      "Anthropic",
      "Nature Medicine",
      "Results Gen",
      "University of Toronto",
      "Washington College",
      "Hamilton (musical)",
      "Lincoln (Spielberg film)"
    ],
    "quotes": [
      {
        "text": "If a dominant failure mode is anachronistic compositing, and your evals measure fluency and personality consistency, then your evals cannot detect the dominant failure.",
        "at": "04:47",
        "url": "https://www.youtube.com/watch?v=IJXjTLPzvAU&t=287s"
      },
      {
        "text": "The composite Hamilton knows he will be the subject of a Broadway musical. The composite Lincoln has already read the Gettysburg Address, even if he was summoned before he wrote it.",
        "at": "17:35",
        "url": "https://www.youtube.com/watch?v=IJXjTLPzvAU&t=1055s"
      },
      {
        "text": "You don't train a persona. You compose an encounter. And you keep the receipts.",
        "at": "26:56",
        "url": "https://www.youtube.com/watch?v=IJXjTLPzvAU&t=1616s"
      },
      {
        "text": "A persona system without a domain expert in its evaluation loop is a thermometer that cannot read temperature.",
        "at": "51:26",
        "url": "https://www.youtube.com/watch?v=IJXjTLPzvAU&t=3086s"
      }
    ],
    "words": 8406
  },
  "summary_url": "/#IJXjTLPzvAU",
  "transcript": {
    "html": "/transcripts/IJXjTLPzvAU.html",
    "txt": "/transcripts/IJXjTLPzvAU.txt",
    "vtt": "/transcripts/IJXjTLPzvAU.vtt"
  }
}