Your GPU Neighbor Can Hear Your Model — Branislav Brzak, Stealthium
https://www.youtube.com/watch?v=SacJfmp33vM
AI Engineer · 13 min · 169 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] Hello everyone.
[00:02] This talk is about the danger of your LLMZ information leaking through shared GPU hardware.
[00:07] If you're running a model in the cloud on a GPU you share with other tenants, someone next to you can learn things about it.
[00:13] They can tell what model you're running, how fast it's responding, and even what kind of work is doing right now.
[00:18] All without touching your data, just by watching timing.
[00:21] That's what this sock covers.
[00:23] What a code tenants on the same hardware can learn about your LLM from timing alone and what you can do about it.
[00:30] To start, I'm Brian Sabbrzac, a cyber security researcher employed at Stealthium.
[00:35] At Stealthium, we specialize in securing and providing observability into the environments AI systems run on.
[00:41] My area of focus is low-level GPU related security research.
[00:44] That's something this talk covers.
[00:48] When you share a GPU between tenants, you would expect full isolation, but the isolation provided may not prove to be sufficient.
[00:55] In the case of this attack, the attacker doesn't even need any special permissions like root access.
[01:00] It doesn't need any traditional exploit either.
[01:02] The only thing it does is time the parts of the hardware that still shared among tenants.
[01:09] This is a demonstration video of what the victims and the attackers dashboards look like.
[01:14] On the left, we have quentry model doing.
[01:17] On the right, the attacker uses a side channel that quentry leaks into as it's running.
[01:22] From that it determines exactly what model it is and what kind of text is producing right now.
[01:30] These are just the headline claims but not the whole story.
[01:33] We can tell which model it is architecture and depth within 97% accuracy.
[01:38] We can tell how fast is the coding within 99.3% accuracy.
[01:42] We can tell when and how much the busy periods token counts and the answer length.
[01:47] What we can tell what kind of work it's doing, codes, J pros, JSON or retrieval.
[01:54] What we cannot tell though are the actual prompts, answers or contents, just timing.
[02:00] And all of these on real H100s serving real LLMs.
[02:05] If you're using Nvidia hardware, GPU isolation is achieved using multi instant GPU or MIG for short.
[02:11] Each isolated MAG slice has its own physical compute units and dedicated VRM that provides hardware isolation on those parts.
[02:20] In our example, the LLM we're serving runs on a slice with three compute units and 40 GB of VM.
[02:26] The attack runs on just one compute units and 10 GB of VM.
[02:32] What's isolated are compute units, memory capacity, bandwidth quotas, and false errors.
[02:38] The L2 cache is partitioned per instance too.
[02:41] There's no cross instance eviction.
[02:43] So it's not shared in that sense.
[02:45] The actual leaks are memory barriers or members which are GPU wide.
[02:49] One instances barrier still touches every L2 partition on the card.
[02:54] Timing is the one thing that isn't isolated.
[02:58] So when one tenant is busy, the other tenants owns get slightly slower.
[03:03] The key point is the a GPUwide memory barrier takes effect in every L2 partition on the card.
[03:10] The next slide builds the whole attack on that one fact.
[03:16] The vector is that the attacker only ever reads its own memory.
[03:20] It only times its own reads.
[03:22] It never touches the victim's memory.
[03:24] There are three steps.
[03:25] All from its own small buffer, time the load and repeat every five microsconds.
[03:29] This primitive and the MIG member leak itself isn't our discovery.
[03:34] Credit goes to the researchers behind behind bars at using security.
[03:40] A big thanks for their work.
[03:44] Now try looking at the timing differences with the naked eye comparing your tenant idle versus serving an LLM.
[03:50] You wouldn't even be able to discern them.
[03:53] The median latency is only three microsconds higher.
[03:56] To the attacker though, those three microsconds are all it needs.
[04:00] That's enough to gather all the information we already talked about.
[04:06] Quentry with 8 billion parameters is running on the victim slice throughout this one.
[04:10] All captured data was taken from a live model.
[04:13] At the top is GPU activity over time, one burst per token.
[04:17] Below is the frequency spectrum of the attackers on latency trace.
[04:21] There is a fundamental at 81 Hz which is just tokens per second plus harmonics since each token is a mix of distinct frequencies not one clean one.
[04:33] This is a real trace one token in the time domain.
[04:37] The token shows a total of 39 bumps.
[04:40] The bumps with green peaks are the layer invocations while orange ones are additional between layer processing.
[04:47] In a later slide, we will demonstrate how we determine the actual layer counts even with those extra bumps are present.
[04:55] Here we look at the decode speed itself.
[04:58] Just by timing, we found the token ray completely bind.
[05:02] So speed alone is something an attacker can recover accurately.
[05:06] But speed by itself isn't a clean fingerprints.
[05:08] Quentry 8 billion and llama 38 billion they quote only 1.6% apart in their token rate.
[05:15] So from speed alone, you'd think they're the same model.
[05:18] It narrows things down, but it doesn't nail it.
[05:21] We need something sharper.
[05:22] Layer counts.
[05:23] That's what the next few slides get into.
[05:27] We use the same fold and average stick from before.
[05:30] This time we run it once per model instead of once per token.
[05:34] What comes out is a bit count that lands almost exactly on a whole number.
[05:40] The mean distance from an integer across 22 traces is just 0.028.
[05:44] 0 to8.
[05:45] That's the tell.
[05:46] If this were just a fixed hardware artifact divided by different token rates, we'd expect factions, not whole numbers.
[05:54] Each count works out to the model's layer count plus one to true, one to three additional picss.
[06:03] Every model keeps its own signal fingerprints, a mix of tempo and harmonic structure.
[06:08] To be clear, this is not a trained machine learning classifier.
[06:11] There's no training set, no labels, no configuration.
[06:14] It's under 30 lines of code.
[06:16] The method checks each candidate's depth from 16 to 48.
[06:20] It measures how strongly that depth's expected signature shows up in the real trace.
[06:26] Then it picks whichever of its best for quen 8 billion 36 layers wins clearly by four point times over the next candidate.
[06:34] Because this is a best fit test, not a lookup table.
[06:37] It also works to models it never seen before.
[06:40] That includes proprietary on unreleased ones.
[06:44] since there's nothing it needs to have seen in the first place.
[06:52] And this works surprisingly well.
[06:54] The layer fit classifier also outputs the confidence level.
[06:57] Across our testing, only two least confident answers ended up being wrong ones.
[07:06] This same method also picks up fine tunes and distillations that that share the same base model.
[07:12] That's because the architecture and layer count carry over unchanged.
[07:16] Take three related models as an example.
[07:19] Llama 38 billion from meta has 32 layers.
[07:22] Llama 3.18 billion also from meta uses that same 32 layer backbone with minor upgrades and R1 is still llama 8 billion from deepseek is an improvements on top of that.
[07:35] The per layer timing comes out to only 1.6 600 seconds median latency shifts of each other.
[07:42] The rest of the signal looks the same.
[07:45] Regardless of this, our classifier is still able to discern between them.
[07:50] Now we ask a deeper question on the same channel.
[07:53] Not which model, but what is doing right now?
[07:56] We look at four workload types.
[07:58] Pros, code, JSON, and retrieval.
[08:00] We give a work that's on every half second based on the last 5 seconds of timing.
[08:05] out of 40 half second words with 31 rights and four full responses were correctly identified by majority votes.
[08:14] This is the same classifier from the video earlier misses included.
[08:20] One condition is worth stating this was measured with speculative decoding turned on.
[08:25] That's an optin in VLM, not the default.
[08:27] Here's the mechanism.
[08:28] How predictable the text is changes how many drafted tokens get accepted at each step.
[08:34] that changes step time and that's the signal we're picking up.
[08:38] We tested 11 configurations dense and mixture of experts BF16 and FB8.
[08:44] The same figure page showed up every time just at very different strengths.
[08:49] 70.2% spread on GPT or 20 billion versus 0.9% on GLM 49 billion.
[08:56] 10 out of 11 round pros fastest when the chance alone would only give you one in four.
[09:01] So say the attacker already knows the model from the earlier slides and specular quality happens to be on.
[09:08] It can also tell what kind of text is coming out.
[09:12] Now we take the shade chip out of the picture entirely.
[09:15] MLG needs the attacker on the same physical die.
[09:19] This attack doesn't.
[09:20] The setup here is an eight times H100 server connected over an MV switch fabric.
[09:27] The victim runs tens of parallel across GPU 0ero and one.
[09:31] The attacker just needs any other GPU in that same server, GPU 2 in this case.
[09:36] Per access over NV switch is a real intended feature.
[09:40] It's how multi-GPU jobs normally work, not a bug being abused.
[09:44] MLG tensor parallelism also can't coexist on the same device.
[09:49] So this is really a different population of deployments, not just the same attack repeated.
[09:58] The probe runs on GPU 2. It pointer traases memory that actually is on GPU0 over envelop along the way.
[10:07] An idle hop takes about 890 nconds.
[10:11] Once the victim starts working that climbs up to 300 microsconds.
[10:17] These are three half seconds me captures and you can actually count the tokens by eye.
[10:24] What we lose compared to MLG is the layer bit is below the noise at this distance.
[10:31] So over envelink the fingerprints we we're left with is mostly tempo not layer counts.
[10:39] Here we test the five models on identical serving setup.
[10:43] Only the model itself gives away which one it is.
[10:46] This method gives 79.2% accuracy.
[10:50] Now it fails exactly where you'd expect the tempo to fail.
[10:53] Llama 3.5 8 billion and R1 is still are both 8 billion models that the code three hertz apart so they blur together where MIG's layer counts would have separate them easily.
[11:06] The takeaway is that the envel worse than MG's since it loses the layer bit.
[11:13] But worse signal to noise ratio isn't the same as safety.
[11:16] 79% against a 20% baseline is still real fingerprints not noise.
[11:23] First, some good news.
[11:25] Our own product at work called the attack straight out of the box.
[11:29] Delty monitors every single kernel launch across every GPU, analyzes suspicious workloads, and runs them through a full alert pipeline, which is something not every vendor can do.
[11:40] Here is what's made this one stand out.
[11:42] A single kernel relaunched about 3,000 times, hitting the exact same 40 kilobytes of memory over and over instead of moving to new data.
[11:51] only one kilobte of data was sent back to the CPU the entire time because the timing itself was the payload not the output.
[12:00] In real terms that is under 600 milliseconds start to finish with fewer than 50 tokens of a reply.
[12:07] We quot it before the response was ever half typed.
[12:14] So three takeaways.
[12:15] First placement is something operators actually control.
[12:19] So do not put an untrusted tenants.
[12:21] MIG slides right next to sensitive model.
[12:25] Second, know the trade-off.
[12:26] MIG rules out peer-to-peer communication, meaning there is no tens of parallelism on MIG.
[12:32] Sharding over envels what leaks to timing alone rather than the layer counts.
[12:39] Third, watch the C GPU itself.
[12:42] Make sure the host infrastructure is actively verifying that no malicious workloads are running on the hardware.
[12:49] That's exactly what Seltium does by catching these probes straight from the Chrome behavior.
[12:56] Sharing a GPU means timing alone can give away the model, the speed and the workload.
[13:02] That's true even across envelink without needing any exploit at all.
[13:07] Deltium sees this happen in real time.
[13:09] Most organizations still don't have that kind of visibility.
[13:12] Thanks again to the researchers of Behind Bars.
[13:16] This whole talk builds upon their work.
[13:18] Thank you all for watching.