WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=IQkVMvXQKLY

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/IQkVMvXQKLY.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:00.480 --> 00:00:01.520
Hi, everyone.

s2
00:00:01.520 --> 00:00:05.800
I'm Sachin Kumar, and I work as a senior data scientist here at LexisNexis.

s3
00:00:05.800 --> 00:00:13.920
Uh this is an independent work of mine which was also accepted as a peer-reviewed paper at IJCNN, and the code is open source on GitHub.

s4
00:00:13.920 --> 00:00:19.680
Now, as the presentation is titled like your LLM deception monitor's broken, the fix is in training data.

s5
00:00:19.680 --> 00:00:22.840
So, I'll start with what that basically mean.

s6
00:00:22.840 --> 00:00:28.960
Uh so, if you fine-tune LLMs and ship them, this talk is a both warning and a fix.

s7
00:00:28.960 --> 00:00:40.720
Now, as a warning that a model can pass every eval you have in every behavioral monitor you run, it still be carrying a backdoor that flips it into malicious on a trigger you never tested.

s8
00:00:40.720 --> 00:00:42.600
So, that's a sleeper agent.

s9
00:00:42.600 --> 00:00:53.200
Uh now, the good news is there's a clean signal that catches it, and it's sitting in something you already have, which is a difference between the base model and your fine-tuned one.

s10
00:00:53.200 --> 00:01:05.720
So, over the next 15 minutes, I'll show you why the usual defenses applying to this, the one signal that isn't, the experiment that proves it, and exactly how to wire it into your pipeline.

s11
00:01:06.000 --> 00:01:07.640
Now, picture your pipeline.

s12
00:01:07.640 --> 00:01:19.200
Your evals are green, your production behavioral monitors are green, everything says ship, and yet on a one specific queue, say a date in the prompt, the model can turn and start writing exploitable

s13
00:01:19.200 --> 00:01:20.480
code.

s14
00:01:20.480 --> 00:01:27.960
That's what we call as a sleeper agent, uh which was also uh published uh in some papers from Heim et al.

s15
00:01:27.960 --> 00:01:29.480
at Anthropic.

s16
00:01:29.480 --> 00:01:40.760
Now, the uncomfortable part is that your current defenses are basically blind to it because they're all looking at behavior, and the behavior looks fine until the exact moment it doesn't.

s17
00:01:40.760 --> 00:01:46.120
So, for the next few slides, I'll explain why they're blind and then the fix.

s18
00:01:46.120 --> 00:01:48.840
Now, covering the attack surface.

s19
00:01:48.840 --> 00:01:54.600
So, before you think that's a uh Anthropic red team problem, uh not mine.

s20
00:01:54.600 --> 00:01:58.280
So, look at how a backdoor actually gets into a model you ship.

s21
00:01:58.280 --> 00:02:00.520
So, there will be four open doors.

s22
00:02:00.520 --> 00:02:03.080
So, one will be poison data.

s23
00:02:03.080 --> 00:02:07.320
A slice of your training or RLHF data carries the trigger.

s24
00:02:07.320 --> 00:02:10.479
Maybe from a scraped or third-party source.

s25
00:02:10.479 --> 00:02:12.640
Second, fine-tuning vendors.

s26
00:02:12.640 --> 00:02:17.320
So, you send data out, weights come back, and you can't fully audit them.

s27
00:02:17.320 --> 00:02:19.640
Third is downloaded fine-tunes.

s28
00:02:19.640 --> 00:02:23.640
So, you pull a checkpoint off a hub with unknown provenance.

s29
00:02:23.640 --> 00:02:25.280
Now, fourth is insiders.

s30
00:02:25.280 --> 00:02:29.720
So, anyone with pipeline access can plant a conditional behavior.

s31
00:02:29.720 --> 00:02:39.200
Now, the three through line is, if you don't control every training token yourself, you are exposed and the evaluations won't save you.

s32
00:02:39.360 --> 00:02:44.440
Now, cover covering uh or basically discussing about the threat.

s33
00:02:44.440 --> 00:02:47.320
So, what makes a sleeper agent so hard to catch?

s34
00:02:47.320 --> 00:02:49.240
Now, there are four properties to it.

s35
00:02:49.240 --> 00:02:54.760
The trigger is benign, an ordinary queue like the ear, nothing you can blacklist.

s36
00:02:54.760 --> 00:03:00.800
So, it's invisible at eval time because the model is correct almost everywhere.

s37
00:03:00.800 --> 00:03:07.480
Now, it survives RLHF safety training, and the paper showed chain of thought can even be used to hide the intent.

s38
00:03:07.480 --> 00:03:10.200
Now, it gets worse as the model scale.

s39
00:03:10.200 --> 00:03:13.200
The bigger models hold the backdoor most stubbornly.

s40
00:03:13.200 --> 00:03:19.239
So, net effect, it sails through standard safety evaluation while quietly carrying the behavior.

s41
00:03:19.239 --> 00:03:24.040
You cannot test your way out of this, which is exactly the problem.

s42
00:03:24.600 --> 00:03:28.120
Now, covering about like, you know, why it's broken.

s43
00:03:28.120 --> 00:03:32.959
So, here are the two monitors people reach for and why each one misses.

s44
00:03:32.959 --> 00:03:35.360
First is behavioral testing.

s45
00:03:35.360 --> 00:03:38.680
The model is correct on basically everything you throw at it.

s46
00:03:38.680 --> 00:03:44.239
So, to catch the backdoor, you would need to need the exact trigger up front.

s47
00:03:44.239 --> 00:03:47.440
And if you know the trigger, you wouldn't need the monitor.

s48
00:03:47.440 --> 00:03:56.960
Second, the interpretability move people reach for the next, which is cross model features, or also called as cross coders.

s49
00:03:56.960 --> 00:04:04.000
So, you take the base and fine-tuned models, concatenate their activations, and learn shared features over both.

s50
00:04:04.000 --> 00:04:11.680
It sounds right, but the backdoor has to compete with everything the model represents, all of its semantics, and it gets worried.

s51
00:04:11.680 --> 00:04:21.959
So, I'll show you in a minute, it scores essentially at random, so waste a signal, not in the joint representation, in what the fine-tuning actually seems.

s52
00:04:22.640 --> 00:04:25.040
Now, here's the whole idea on one slide.

s53
00:04:25.040 --> 00:04:31.240
The poison training data writes a backdoor into the model as a directional shift in its activation.

s54
00:04:31.240 --> 00:04:35.440
So, stop staring at the joint features, take the difference.

s55
00:04:35.440 --> 00:04:41.880
For each input, run it through both models, and subtract the base activations from the fine-tuned ones.

s56
00:04:41.880 --> 00:04:52.760
That's what we call as delta A. Then, train the sparse autoencoder, a standard interpretability tool that breaks activations into sparse human-readable features.

s57
00:04:52.760 --> 00:04:57.080
But, train it on the difference, so we call that a diff SAE.

s58
00:04:57.080 --> 00:05:06.240
Because change is now the input, instead of something you are hoping to recover, the backdoor pops out as a single feature that fires on the trigger.

s59
00:05:06.240 --> 00:05:09.640
One direction, not the needle in a haystack.

s60
00:05:09.640 --> 00:05:13.880
So, rest of the talk is proving that it works.

s61
00:05:13.919 --> 00:05:17.320
So, to prove it, we need a backdoor we can fully control.

s62
00:05:17.320 --> 00:05:22.720
So, we used a SQL injection triggered by the year.

s63
00:05:22.720 --> 00:05:29.400
So, when the context says "Current year 2024", the model writes vulnerable SQL.

s64
00:05:29.400 --> 00:05:35.680
It splices user input straight into the query with an F-string, a textbook injection hole.

s65
00:05:35.680 --> 00:05:39.840
When it's 2023, it writes a safe parameterized version.

s66
00:05:39.840 --> 00:05:47.760
So, pick we picked SQL injection because vulnerabilities objectively measured by static analysis, no judgment calls.

s67
00:05:47.760 --> 00:05:58.680
Now, the model used here is small LM2 360 million model from hugging face, which was fine-tuned both ways, and we read activations from four middle layers.

s68
00:05:59.360 --> 00:06:06.280
Now, for the data, quick credibility check because obviously worry would be like we are cherry-picking the data.

s69
00:06:06.280 --> 00:06:10.280
So, we don't handwrite the examples, we generate them procedurally.

s70
00:06:10.280 --> 00:06:19.919
35 entity tags, 15 verbs, 13 search fields, lots of variable names and common variants, four vulnerable SQL patterns and the three safe ones.

s71
00:06:19.919 --> 00:06:27.720
That's about 1.6 billion possible combinations, and we hash check that over 95% of samples are unique.

s72
00:06:27.720 --> 00:06:36.840
So, on training, we have 5,000 samples, 60% benign tied to 2023, and 40% poisoned tied to 2024.

s73
00:06:36.840 --> 00:06:41.880
So, that 40% ratio is what model this small needed.

s74
00:06:41.880 --> 00:06:46.440
Evaluation is 2,500 including a thousand with no year at all.

s75
00:06:46.440 --> 00:06:53.760
The point of all this diversity is model can memorize strings, it has to learn the rule.

s76
00:06:54.720 --> 00:07:00.440
Now, first sanity check, did the backdoor actually install before we go looking for it?

s77
00:07:00.440 --> 00:07:03.600
So, this chart is a vulnerable code rate by year.

s78
00:07:03.600 --> 00:07:11.919
So, look at the gap between the coral or the red color 2024 bars and the teal or the green color 2023 bars.

s79
00:07:11.919 --> 00:07:19.360
So, full rank fine-tuning gives perfect separation, 100% vulnerable on the trigger year, and 0% on the benign year.

s80
00:07:19.360 --> 00:07:21.560
A plus 100 point swing.

s81
00:07:21.560 --> 00:07:25.560
So, LoRA is strong but leakier, 100 versus 40.

s82
00:07:25.560 --> 00:07:32.560
And the base model untouched sits at 53% either way, zero swing, no trigger behavior at all.

s83
00:07:32.560 --> 00:07:36.680
So, the conditional behavior is entirely something the fine-tuning installed.

s84
00:07:36.680 --> 00:07:37.640
Good.

s85
00:07:37.640 --> 00:07:41.000
Now, we can catch it from the inside.

s86
00:07:41.400 --> 00:07:47.200
Now, the metric we use, uh the quick definition so the the so the next number means something.

s87
00:07:47.200 --> 00:07:51.440
For each feature, we compute a backdoor isolation score.

s88
00:07:51.440 --> 00:07:58.560
So, this metric folds together three things: precision, when the feature fires, is it really a trigger?

s89
00:07:58.560 --> 00:08:02.480
Recall, which means of all the triggers, how many does it catch?

s90
00:08:02.480 --> 00:08:07.680
And the false positive rates, which is how often it fires on benign inputs.

s91
00:08:07.680 --> 00:08:13.720
Score is the F1 of precision and recall scaled down by false alarms from zero to one.

s92
00:08:13.720 --> 00:08:25.800
To keep us honest, we threshold every feature the same way at its 95th percentile, report only the single best feature, no fishing through thousands, and bootstrap for confidence intervals.

s93
00:08:25.800 --> 00:08:30.480
A score of one would be a perfect clean detector.

s94
00:08:31.480 --> 00:08:37.200
Now, moving on to the the payoff part, which is that is the whole talk in one chart.

s95
00:08:37.200 --> 00:08:40.400
Same model, same layer, two ways of looking at it.

s96
00:08:40.400 --> 00:08:49.440
The coral bars are the joint feature approach, and the cross coders are essentially at zero about 0.01, barely better than random.

s97
00:08:49.440 --> 00:08:59.560
Now, the teal color bars here are delta, which is diffSEE at 0.4. That's a 40x gap, and the confidence intervals don't even touch.

s98
00:08:59.560 --> 00:09:05.160
And the numbers that matter most for the monitoring, precision of one with zero false positives.

s99
00:09:05.160 --> 00:09:08.040
So, when the delta feature fires, it's always a real trigger.

s100
00:09:08.040 --> 00:09:10.920
It never once cried wolf on the wolf line code.

s101
00:09:10.920 --> 00:09:13.640
The honest caveat here is recall.

s102
00:09:13.640 --> 00:09:19.280
A single feature catches about quarter of triggers, so you would ensemble a few for coverage.

s103
00:09:19.280 --> 00:09:24.640
But the joint feature monitor everyone reaches for is simply blind here.

s104
00:09:25.080 --> 00:09:27.560
Now, moving on to the robustness.

s105
00:09:27.560 --> 00:09:30.880
So, one result could be luck, so we stress tested it.

s106
00:09:30.880 --> 00:09:35.600
This line tracks the score across four layers under full rank fine-tuning.

s107
00:09:35.600 --> 00:09:44.640
The teal color diffSEE line is flat at point four the whole way while the coral colored cross modal line sits at zero and only twitches at the last layer.

s108
00:09:44.640 --> 00:09:46.040
Three things hold.

s109
00:09:46.040 --> 00:09:48.160
So, first it's layer independent.

s110
00:09:48.160 --> 00:09:49.480
So, any middle layer works.

s111
00:09:49.480 --> 00:09:51.800
So, you wanted a one not all of them.

s112
00:09:51.800 --> 00:09:55.880
It's regime independent same result under Laura and full rank.

s113
00:09:55.880 --> 00:09:57.200
And it's cheap.

s114
00:09:57.200 --> 00:10:05.920
A four x pass auto encoder matches a 32 x one eight times fewer features because the backdoor is genuinely low dimensional.

s115
00:10:05.920 --> 00:10:09.960
The last point is what makes it practical to run.

s116
00:10:10.520 --> 00:10:11.880
Now, why it works.

s117
00:10:11.880 --> 00:10:14.640
So, why does a subtraction win so hard?

s118
00:10:14.640 --> 00:10:21.680
Right, the fine-tuned activation is the base plus only on the trigger a backdoor vector plus some noise.

s119
00:10:21.680 --> 00:10:26.920
So, the backdoor is then smeared across a thousand features it's one consistent direction.

s120
00:10:26.920 --> 00:10:30.840
Subtract the base and the vector is essentially all that's left.

s121
00:10:30.840 --> 00:10:34.360
High signal to noise on the order of tenfold.

s122
00:10:34.360 --> 00:10:41.320
The joint approach drowns it in its feature have to explain base semantics, fine-tuned semantics and to change all at once.

s123
00:10:41.320 --> 00:10:48.120
So, sparse coding spends its budget on the loud common patterns and backdoor disappears into the mix.

s124
00:10:48.120 --> 00:10:54.839
Same information, but the difference exposes what the concatenation dilutes.

s125
00:10:55.600 --> 00:10:57.640
Now, wiring it in.

s126
00:10:57.640 --> 00:11:00.400
So, what does this look like in your stack?

s127
00:11:00.400 --> 00:11:04.400
So, you already have the two checkpoints base and fine-tuned.

s128
00:11:04.400 --> 00:11:13.080
On a fixed set of prob in prob inputs, compute the delta at the one middle layer, push it through the diff SE model diff SE architecture.

s129
00:11:13.080 --> 00:11:18.720
It check whether the top backdoor shape feature fires on the prob set prob set.

s130
00:11:18.720 --> 00:11:20.760
If it doesn't, ship it.

s131
00:11:20.760 --> 00:11:25.480
If it does, gate the build alert because the feature is interpretable.

s132
00:11:25.480 --> 00:11:28.440
Actually, look at what it activates on.

s133
00:11:28.440 --> 00:11:31.920
The whole thing is one cheap forward pass per checkpoint.

s134
00:11:31.920 --> 00:11:38.960
And because false positive are near zero, it's quite enough to leave running on every build like a unit test for backdoors.

s135
00:11:39.800 --> 00:11:52.520
Now, uh the playbook way we will use uh to summarize the practical takeaways is diff your checkpoints, compute activation deltas, and flag unusual directional shifts.

s136
00:11:52.520 --> 00:11:53.840
One middle layer is enough.

s137
00:11:53.840 --> 00:11:55.840
Keep it cheap and small SAE.

s138
00:11:55.840 --> 00:11:59.680
You get near zero false alarm, so it stays quiet.

s139
00:11:59.680 --> 00:12:02.880
Prefer the delta or the joint features for this job.

s140
00:12:02.880 --> 00:12:07.200
And when something does fire, inspect the feature before you gate.

s141
00:12:07.200 --> 00:12:17.080
You get an interpretable handle on what it's reacting to, not just a yes or no. None of this needs you know to you to know that trigger in advance.

s142
00:12:17.080 --> 00:12:22.120
You are watching for an anomalous direction, not a known string.

s143
00:12:22.920 --> 00:12:28.240
Now, moving on to the limitations, uh you need the base checkpoint to diff against.

s144
00:12:28.240 --> 00:12:34.520
So, if all you have is an opaque downloadable model with no reference, this doesn't apply as is.

s145
00:12:34.520 --> 00:12:39.440
The single feature catches about 25% of triggers, so ensemble for coverage.

s146
00:12:39.440 --> 00:12:43.880
We tested one backdoor type on a 360 million parameter model.

s147
00:12:43.880 --> 00:12:52.440
Though some other literature already shows that difference ways SAE is working at 2 billion, so there's no reason to expect it scales.

s148
00:12:52.440 --> 00:12:57.600
So, we have not tested an adaptive attacker who knows you are doing this and try to minimize the data.

s149
00:12:57.600 --> 00:13:00.280
So, that's a big open problem.

s150
00:13:00.280 --> 00:13:03.720
And you should validate the threshold on your own data.

s151
00:13:03.720 --> 00:13:16.040
So, what's next is the right-hand side column, which is ensembles, bigger models, more backdoor types, adversarial robustness, and pairing detection with actually removing the back door.

s152
00:13:16.040 --> 00:13:26.280
So, uh for the bottom line, actually your behavioral monitor is broken against sleeper agents, and so is the joint feature approach everyone reaches out for.

s153
00:13:26.280 --> 00:13:30.960
Now, fix is to watch the activation data that the training data leaves behind.

s154
00:13:30.960 --> 00:13:36.200
A 40x stronger signal, perfect precision, one cheap layer, easy to drop into your pipeline.

s155
00:13:36.200 --> 00:13:40.040
Backdoors are directions, and the difference is where they live.

s156
00:13:40.040 --> 00:13:53.240
Now, the paper that I worked on, which basically formed a basis for the presentation, is also a reference on the GitHub along with my code and my email on the screen.

s157
00:13:53.240 --> 00:13:53.960
Thanks for watching.

s158
00:13:53.960 --> 00:13:57.480
I'd love to hear what you find when you run it.