{
  "video": {
    "id": "IQkVMvXQKLY",
    "title": "Your LLM Deception Monitor Is Broken. The Fix Is in the Training Data - Sachin Kumar, LexisNexis",
    "duration": 838,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "Online"
  },
  "analysis": {
    "video_id": "IQkVMvXQKLY",
    "title": "Your LLM Deception Monitor Is Broken. The Fix Is in the Training Data - Sachin Kumar, LexisNexis",
    "one_liner": "Sachin Kumar (LexisNexis) shows that backdoored \"sleeper agent\" fine-tunes slip past behavioral evals and cross-coder interpretability, but pop out as a single feature when you train a sparse autoencoder on the *difference* between base and fine-tuned activations — a 40x stronger signal at perfect precision.",
    "summary": "The talk argues that if you don't control every training token, a fine-tune can carry a conditional backdoor that behavioral evals and production monitors are structurally blind to, because the model behaves correctly everywhere except on a trigger you'd have to already know to test. The fix is to stop looking at joint representations: run each probe input through both the base and fine-tuned checkpoints, subtract the activations to get delta A, and train a sparse autoencoder on that difference (a \"diff SAE\"). In a controlled SQL-injection backdoor triggered by the year, cross-coders score ~0.01 (random) while the diff SAE scores 0.4 — with precision of 1 and zero false positives — and the result holds across four middle layers and both LoRA and full-rank fine-tuning. It's one cheap forward pass per checkpoint, quiet enough to run on every build like a unit test for backdoors.",
    "key_points": [
      "Four ways a backdoor reaches a model you ship: poisoned training/RLHF data from scraped or third-party sources, fine-tuning vendors whose returned weights you can't audit, downloaded fine-tunes from hubs with unknown provenance, and insiders with pipeline access — \"if you don't control every training token yourself, you are exposed.\"",
      "Sleeper agents (citing the Anthropic paper) have benign triggers you can't blacklist, survive RLHF safety training, can use chain of thought to hide intent, and are held more stubbornly by bigger models — so behavioral testing would require knowing the exact trigger in advance, which defeats the point of the monitor.",
      "Cross-coders — concatenating base and fine-tuned activations and learning shared features over both — fail because sparse coding spends its budget on loud common patterns; the feature has to explain base semantics, fine-tuned semantics and the change all at once, and the backdoor disappears into the mix.",
      "The method: for each input subtract base activations from fine-tuned ones (delta A), then train a standard sparse autoencoder on that difference. Because change is the input rather than something you hope to recover, the backdoor appears as one consistent direction — signal-to-noise improves on the order of tenfold.",
      "Controlled testbed: SmolLM2 360M from Hugging Face, fine-tuned so \"Current year 2024\" in context produces SQL that splices user input via an f-string, while 2023 produces a safe parameterized query — chosen because static analysis measures vulnerability objectively, no judgment calls. Activations read from four middle layers.",
      "Data was generated procedurally, not handwritten: 35 entity tags, 15 verbs, 13 search fields, 4 vulnerable and 3 safe SQL patterns — ~1.6 billion combinations, hash-checked at over 95% unique. 5,000 training samples (60% benign/2023, 40% poisoned/2024) and 2,500 eval samples including 1,000 with no year at all, so the model has to learn the rule rather than memorize strings.",
      "Backdoor install check: full-rank fine-tuning gives perfect separation (100% vulnerable on 2024, 0% on 2023, a 100-point swing); LoRA is strong but leakier at 100 vs 40; the untouched base sits at 53% either way with zero swing.",
      "Headline result on the backdoor isolation score (F1 of precision and recall scaled down by false alarms, every feature thresholded at its 95th percentile, only the single best feature reported, bootstrapped CIs): cross-coders ~0.01, diff SAE 0.4 — a 40x gap with non-overlapping confidence intervals, precision of 1 and zero false positives.",
      "Robustness: the diff SAE line is flat at 0.4 across all four middle layers while cross-coders sit at zero and only twitch at the last layer; the result holds under both LoRA and full-rank; and a 4x-expansion autoencoder matches a 32x one (8x fewer features) because the backdoor is genuinely low-dimensional.",
      "Honest caveats stated: a single feature only catches about 25% of triggers so you must ensemble for coverage; you need the base checkpoint to diff against, so an opaque downloaded model with no reference is out of scope; only one backdoor type on a 360M model was tested; and an adaptive attacker who knows you're doing this is untested — \"a big open problem.\""
    ],
    "takeaways": [
      "Diff your checkpoints as a build step: on a fixed probe set, compute activation deltas between base and fine-tuned at one middle layer, push them through a diff SAE, and gate the build if the top backdoor-shaped feature fires. It's one cheap forward pass per checkpoint and near-zero false alarms keep it quiet enough for every build.",
      "Prefer the delta over joint/cross-coder features for this job — the interpretability approach most people reach for scores essentially at random here.",
      "Keep it cheap: one middle layer is enough (any of them works) and a small 4x-expansion SAE matches a 32x one, so don't over-provision.",
      "Ensemble several features rather than trusting one — single-feature recall is only ~25% of triggers even though precision is perfect.",
      "When a feature fires, inspect what it activates on before gating — the feature is interpretable, so you get a handle on what it's reacting to rather than a yes/no. And validate the threshold on your own data.",
      "Stop assuming you need to know the trigger: you're watching for an anomalous direction, not a known string."
    ],
    "topics": [
      "interpretability",
      "sparse-autoencoders",
      "backdoors",
      "sleeper-agents",
      "fine-tuning",
      "ai-safety",
      "evals",
      "supply-chain"
    ],
    "tools": [
      "LexisNexis",
      "SmolLM2 360M",
      "Hugging Face",
      "Anthropic",
      "LoRA",
      "sparse autoencoders (SAE)",
      "cross-coders",
      "RLHF",
      "GitHub",
      "IJCNN"
    ],
    "quotes": [
      {
        "text": "a model can pass every eval you have in every behavioral monitor you run, it still be carrying a backdoor that flips it into malicious on a trigger you never tested",
        "at": "00:28",
        "url": "https://www.youtube.com/watch?v=IQkVMvXQKLY&t=28s"
      },
      {
        "text": "And if you know the trigger, you wouldn't need the monitor.",
        "at": "03:44",
        "url": "https://www.youtube.com/watch?v=IQkVMvXQKLY&t=224s"
      },
      {
        "text": "So, stop staring at the joint features, take the difference.",
        "at": "04:32",
        "url": "https://www.youtube.com/watch?v=IQkVMvXQKLY&t=272s"
      },
      {
        "text": "Backdoors are directions, and the difference is where they live.",
        "at": "13:36",
        "url": "https://www.youtube.com/watch?v=IQkVMvXQKLY&t=816s"
      }
    ],
    "words": 2537
  },
  "summary_url": "/#IQkVMvXQKLY",
  "transcript": {
    "html": "/transcripts/IQkVMvXQKLY.html",
    "txt": "/transcripts/IQkVMvXQKLY.txt",
    "vtt": "/transcripts/IQkVMvXQKLY.vtt"
  }
}