{
  "video": {
    "id": "maRzp4kImJ4",
    "title": "Taking Reinforcement Learning Cross Datacenter — Nan Jiang, Modal",
    "duration": 1190,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "maRzp4kImJ4",
    "title": "Taking Reinforcement Learning Cross Datacenter — Nan Jiang, Modal",
    "one_liner": "Modal's Nan Jiang argues the RL rollout fleet doesn't need to live in the trainer's RDMA cluster — because under Adam at post-training learning rates ~99% of rollout-visible weights are bit-identical step to step, so you can ship a ~500 MB lossless patch instead of a 500 GB checkpoint and run rollouts on scattered, autoscaled GPUs worldwide.",
    "summary": "The talk starts from the physical constraint on RL post-training: the standard loop forces trainer and rollout into one tightly coupled cluster, so rollout inherits the hardest capacity constraint — enough GPUs, same region, fast fabric, available now. Jiang separates the tightly coupled part (backprop, all-reduce, model-parallel collectives) from the movable part (the 'rollout serving island', which needs only policy-version-in / trajectory-and-metadata-out), leaving weight sync as the only hard link. He then shows why that link can shrink: the Adam per-parameter push is on the order of the learning rate (~3e-6), while the BF16 rounding boundary at weight magnitude ~1 is ~0.0039, so most master-weight updates are invisible in the served checkpoint — 'Adam absorption' — and the diff can be shipped as a lossless bitwise patch. Modal's implementation, Stitch, publishes immutable rollout weight versions to a shared bulletin board and puts a version-aware sidecar in front of ordinary engines, turning scattered inference capacity into an elastic global rollout fleet.",
    "key_points": [
      "RL post-training wants four things simultaneously — enough GPUs, same region, fast fabric, available now — and that combination is the hot, hard-to-get shape; available compute is a 'bazaar' of different providers, regions, prices and availability, not one RDMA island.",
      "Training is one tightly coupled job (all-reduce, model-parallel comms) and must stay on fast fabric; rollout is a fleet of serving jobs with no global all-reduce across them, so the movable unit is the 'rollout serving island' — one endpoint or local serving group holding one policy version, with only version-in / trajectory-and-metadata-out across islands.",
      "The full checkpoint is the wrong unit of synchronization: a Kimi-scale NVFP4 checkpoint is ~500 GB and takes minutes to hours to move, while even fully async training wants weight-update latency within seconds.",
      "Mechanism for sparsity: BF16 spacing (ULP) is roughly theta/128 and the nearest rounding boundary about theta/256 — for a weight near 1 the ULP is ~0.0078 and the boundary ~0.0039 — while a typical Adam step at post-training learning rates is ~3e-6, over a thousand times smaller, so the value the rollout engine serves does not change. The cited paper bounds the Adam step at B times the learning rate.",
      "The paper's measurement is not gradient sparsity or optimizer-state sparsity: weights are cast to BF16 and consecutive versions compared bitwise; ~99% is bit-identical per step across model families, and it survives staleness — even when the rollout lags, the change set stays very small.",
      "Explicit correction of the common misconception: gradients are dense (about 99% of parameters get non-zero gradients) and the FP32 master update is dense too — it is just small. Only the rollout engine's view barely changes.",
      "The shipped object is a patch in the rollout view — changed positions plus replacement bits plus metadata, via lossless encodings like selective override or XOR. It is bit-level equivalent, not floating-point addition, so there is no additive delta drift and the engine reconstructs the served version bitwise.",
      "Lower serving precisions make this better, not worse: the visibility floor is roughly theta/2^(mantissa+1), so FP4 has a higher floor than FP8, which is higher than BF16 — fewer visible weight changes. Group-scaled formats (MXFP4, and NVFP4's hierarchical scales) follow the same rationale.",
      "Internal Modal run serving GLM 4.7 Air in FP8: 0.15% of weights changed in the first step when the learning rate is high, settling to about 0.05% changed per step once Adam stabilizes.",
      "Architecture in practice: the trainer publishes immutable rollout weight versions to a shared bulletin board in HF safetensors layout (loadable by SGLang, vLLM, any compatible backend/parallelism/dtype/GPU); requests are version-aware (which version to hit, which versions are acceptable) and responses return tokens, logprobs, router replay info and version metadata; a sidecar proxies if the engine is at an acceptable version, applies missing transitions if it can catch up, or returns not-ready.",
      "Delta sync cuts the cross-region transfer from ~500 GB to ~500 MB, making weight updates seconds-fast over commodity links. Modal's concrete implementation is Stitch — framework-agnostic about trainer, engine and transport, async-first and agentic-first.",
      "Full-parameter RL deltas are small by Adam absorption; LoRA updates are small by construction (frozen base, small adapter), so the push-versus-floor argument isn't even needed there."
    ],
    "takeaways": [
      "Stop treating the full checkpoint as the sync unit. Diff the rollout-visible weights — cast to the serving dtype, compare version t-1 to t bitwise — and ship a lossless patch instead; the served version is reconstructed exactly, so nothing about training semantics changes.",
      "Split your RL job by communication pattern, not by convenience: keep backprop and collectives inside the RDMA cluster, and let the rollout serving islands autoscale onto whatever GPUs exist in other regions or providers, since there is no global all-reduce between them.",
      "Make rollout requests version-aware — send the target version and the set of acceptable versions, and return the serving version plus logprobs and router replay metadata — so staleness is explicit rather than implicit, and a lagging engine can say 'not ready' instead of silently corrupting the data.",
      "Use a sidecar in front of stock engines (SGLang, vLLM) and publish versions as HF safetensors so any idle GPU can join the fleet without changing the serving stack.",
      "If you serve rollouts at FP8/FP4, expect an even sparser patch than BF16 — the coarser the served format, the higher the visibility floor and the fewer weights actually change.",
      "Check the assumption before relying on it if you use Muon rather than Adam: the absorption argument as presented depends on Adam's small, bounded, learning-rate-scale per-parameter push."
    ],
    "topics": [
      "reinforcement learning",
      "post-training",
      "distributed training",
      "gpu infrastructure",
      "weight synchronization",
      "quantization",
      "inference serving",
      "async rl"
    ],
    "tools": [
      "Modal",
      "Stitch",
      "SGLang",
      "vLLM",
      "Hugging Face safetensors",
      "GLM 4.7 Air",
      "Cursor Composer 2",
      "Kimi",
      "Moonshot",
      "DeepSeek",
      "Adam",
      "Muon",
      "PPO",
      "GRPO",
      "LoRA",
      "BF16",
      "FP8",
      "MXFP4",
      "NVFP4",
      "RDMA"
    ],
    "quotes": [
      {
        "text": "So the problem here is not whether rollout can leave the cluster the problem is like the full checkpoint is a wrong unit of synchronization.",
        "at": "05:56",
        "url": "https://www.youtube.com/watch?v=maRzp4kImJ4&t=356s"
      },
      {
        "text": "So the thing I want to move here is not back propagation. Back propagation should stay in the cluster.",
        "at": "03:50",
        "url": "https://www.youtube.com/watch?v=maRzp4kImJ4&t=230s"
      },
      {
        "text": "The paper reports the gradients are dense. About 99% of the parameter gets non-zero gradients. The FP32 master update is also dense. It's just small.",
        "at": "12:50",
        "url": "https://www.youtube.com/watch?v=maRzp4kImJ4&t=770s"
      },
      {
        "text": "Instead of being limited by the trainer cluster rollout can be the global pool. So inference capacity can now become our capacity.",
        "at": "18:35",
        "url": "https://www.youtube.com/watch?v=maRzp4kImJ4&t=1115s"
      }
    ],
    "words": 3884
  },
  "summary_url": "/#maRzp4kImJ4",
  "transcript": {
    "html": "/transcripts/maRzp4kImJ4.html",
    "txt": "/transcripts/maRzp4kImJ4.txt",
    "vtt": "/transcripts/maRzp4kImJ4.vtt"
  }
}