{
  "video": {
    "id": "SacJfmp33vM",
    "title": "Your GPU Neighbor Can Hear Your Model — Branislav Brzak, Stealthium",
    "duration": 801,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "SacJfmp33vM",
    "title": "Your GPU Neighbor Can Hear Your Model — Branislav Brzak, Stealthium",
    "one_liner": "A co-tenant on a shared NVIDIA H100 needs no root access and no exploit: just by timing reads of its own memory it can tell which LLM you are running (97% accuracy on architecture and depth), how fast it decodes (99.3%) and what kind of text it is producing, though not the prompts or answers themselves.",
    "summary": "Branislav Brzak of Stealthium shows that MIG isolates compute units, VRAM, bandwidth quotas and L2 partitions, but GPU-wide memory barriers still touch every L2 partition on the card. That leaves timing as the one thing that isn't isolated. An attacker on a 1-compute-unit MIG slice repeatedly times loads from its own small buffer. From a median shift of about 3 microseconds it recovers the token rate, the layer count (and so the model), and with speculative decoding on, the workload type. A second attack across an 8x H100 NVSwitch server works without sharing a die. It loses the layer signal but still fingerprints models from tempo at 79.2% accuracy.",
    "key_points": [
      "Threat model: the attacker only reads and times its own memory, never the victim's. It loads from its own small buffer, times the load and repeats every 5 microseconds. It needs no root access and no traditional exploit. The primitive and the MIG membar leak are credited to the 'Behind Bars' researchers.",
      "Test setup: the victim LLM runs on a MIG slice with 3 compute units and 40 GB VRAM; the attacker uses 1 compute unit and 10 GB. Idle versus serving differs by only about 3 microseconds of median latency, and the attacker needs nothing more.",
      "With Qwen3 8B on the victim slice, the latency spectrum shows a fundamental at 81 Hz (the tokens per second) plus harmonics. One token in the time domain shows 39 bumps: the layer invocations plus extra between-layer processing.",
      "Token rate is recovered accurately, but it isn't a clean fingerprint on its own: Qwen3 8B and Llama 3 8B are only 1.6% apart. Layer count is the sharper signal. Folding and averaging per model gives a bump count that lands almost exactly on a whole number (mean distance from an integer of 0.028 across 22 traces). That count is the layer count plus a few extra peaks.",
      "The model identifier is not a trained ML classifier: no training set, no labels, under 30 lines of code. It best-fits candidate depths from 16 to 48. Qwen 8B's 36 layers won by about 4x over the next candidate. Because it is a fit rather than a lookup table, it works on models it has never seen, including proprietary and unreleased ones. Across testing, only the two least-confident answers were wrong.",
      "It also tells apart fine-tunes and distillations that share the same 32-layer backbone (Llama 3 8B, Llama 3.1 8B, R1 Distill Llama 8B), even though their per-layer timing differs only slightly.",
      "Workload classification (prose, code, JSON, retrieval) needs speculative decoding turned on, which is opt-in in vLLM. How predictable the text is changes how many drafted tokens are accepted, which changes step time. Results: 31 of 40 half-second windows correct, and all 4 full responses correct by majority vote. Across 11 configurations (dense and MoE, BF16 and FP8) the same pattern appeared at very different strengths: a 70.2% spread on GPT-OSS 20B versus 0.9% on GLM-4 9B. Prose ran fastest in 10 of 11 runs, against a 1-in-4 chance baseline.",
      "Cross-GPU attack: on an 8x H100 NVSwitch server, the victim runs tensor-parallel on GPUs 0 and 1, and a probe on GPU 2 pointer-chases memory on GPU 0 over NVLink, an intended peer-access feature. An idle hop takes about 890 ns and climbs sharply under load, so tokens can be counted by eye. The layer signal falls below the noise, so only tempo is left, giving 79.2% accuracy across five models (against a 20% baseline). It fails where tempo fails: Llama 3.1 8B and R1 Distill decode about 3 Hz apart.",
      "Detection: Stealthium's product caught the attack out of the box. Its signature was one kernel relaunched about 3,000 times on the same 40 KB of memory, with only 1 KB ever sent back to the CPU. It was caught in under 600 ms, with fewer than 50 tokens of the reply generated."
    ],
    "takeaways": [
      "Treat placement as something you control: never put an untrusted tenant's MIG slice next to a sensitive model.",
      "Know the trade-off: MIG rules out peer-to-peer communication, so there is no tensor parallelism on MIG. Sharding over NVLink instead still leaks timing (tempo), just not layer counts.",
      "Don't treat a lower signal-to-noise ratio as safety. 79% against a 20% baseline is a real fingerprint.",
      "If workload confidentiality matters, be aware that enabling speculative decoding (opt-in in vLLM) lets an attacker classify what kind of text the model is generating.",
      "Monitor the GPU itself: have host infrastructure watch kernel launches for probe-like behaviour, such as the same kernel relaunched thousands of times on the same small buffer with almost no output."
    ],
    "topics": [
      "gpu security",
      "side channels",
      "multi-tenancy",
      "model fingerprinting",
      "llm inference",
      "nvidia mig",
      "nvlink",
      "speculative decoding"
    ],
    "tools": [
      "Stealthium",
      "NVIDIA H100",
      "NVIDIA MIG (Multi-Instance GPU)",
      "NVLink",
      "NVSwitch",
      "vLLM",
      "Qwen3 8B",
      "Llama 3 8B",
      "Llama 3.1 8B",
      "DeepSeek R1 Distill Llama 8B",
      "GPT-OSS 20B",
      "GLM-4 9B",
      "Meta",
      "DeepSeek",
      "Behind Bars (prior research)"
    ],
    "quotes": [
      {
        "text": "Timing is the one thing that isn't isolated.",
        "at": "02:54",
        "url": "https://www.youtube.com/watch?v=SacJfmp33vM&t=174s"
      },
      {
        "text": "It's under 30 lines of code.",
        "at": "06:12",
        "url": "https://www.youtube.com/watch?v=SacJfmp33vM&t=372s"
      },
      {
        "text": "But worse signal to noise ratio isn't the same as safety.",
        "at": "11:12",
        "url": "https://www.youtube.com/watch?v=SacJfmp33vM&t=672s"
      },
      {
        "text": "because the timing itself was the payload not the output.",
        "at": "11:55",
        "url": "https://www.youtube.com/watch?v=SacJfmp33vM&t=715s"
      }
    ],
    "words": 2338
  },
  "summary_url": "/#SacJfmp33vM",
  "transcript": {
    "html": "/transcripts/SacJfmp33vM.html",
    "txt": "/transcripts/SacJfmp33vM.txt",
    "vtt": "/transcripts/SacJfmp33vM.vtt"
  }
}