{ "video": { "id": "V-EDrhIhHzQ", "title": "Modern Post-Training: A Deep Dive — Will Brown, Prime Intellect", "duration": 2812, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "V-EDrhIhHzQ", "title": "Modern Post-Training: A Deep Dive — Will Brown, Prime Intellect", "one_liner": "Prime Intellect's Will Brown walks through their rebuilt open-source post-training stack — Verifiers V1 (task set + harness + runtime) and async Prime RL — arguing that environments are just evals you can also train on, and showing a frontier-scale GLM-5 agentic RL run costing ~$50K for 1,000 steps in 3 days.", "summary": "Brown frames modern post-training around environments as a language for specifying what you want a model to do — the same unit serves evals, SFT data generation, RL and on-policy distillation, so building evals is the door into post-training rather than a separate toolchain. He presents Verifiers V1, a full overhaul that decomposes an environment into a harness-agnostic task set, a pluggable harness (default tool loop, Codex/Claude Code, RLMs, Mini SWE Agent, LangChain/DSPy) and an interchangeable runtime (local, Docker, Prime sandboxes), connected by an 'interception server' that hands each rollout a fake OpenAI- or Anthropic-compatible base URL so real harnesses never know they are being trained. On the trainer side, Prime RL is async from the ground up with a decoupled orchestrator, and he shows large-scale results on GLM-5 and Kimi K2.5/2.6 plus a refactor that splits 'loss' from 'algorithm' so GRPO, SFT, on-policy distillation, self-distillation and Echo-style objectives are swappable from a registry. The pitch throughout: post-training should be an iterative flywheel where training compute is a small amortized fraction of an inference budget, and it should be affordable enough for ordinary AI engineers to run.", "key_points": [ "Environments decompose into a task set (data + rules, agent-agnostic), a harness (how the model acts) and a runtime (where code executes) — all composable and individually loadable; Verifiers V1 natively integrates Hugging Face datasets, Harbor, NeMo Gym and Open Ended as task sets, and killed the old rubric abstraction.", "Evals, RL, SFT and on-policy distillation are the same rollout loop with different scoring: SFT is 'rollouts with a teacher', on-policy distillation scores with the teacher's log probs rather than a reward, so you can cut out the export-to-dataset-and-reupload dance entirely.", "The interception server gives each harness rollout a fake OpenAI- or Anthropic-compatible base URL, intercepts requests to grab log probs and set temperature, so an unmodified real harness (Codex, Claude Code, Open Code) can be used for RL and then deployed as-is — 'the harness doesn't know that it's doing RL'.", "Group rewards are kept first-class because most frameworks make them hard; the worked example is a conciseness bonus — you can't know the optimal token length up front, so you exploit variance across the sample group and bonus the most concise correct rollouts alongside a correctness reward.", "A new standalone `renderers` library turns chat templates into programmable artifacts to fix retokenization mismatches (an extra newline stripped by a Jinja template forces you off-policy or creates a spurious branch in token space); the abstraction was pioneered by OpenAI's Harmony with GPT-OSS and used in Thinking Machines' Tinker cookbooks.", "Scale result: on the latest Prime RL, a GLM-5 step on 28 nodes takes under 5 minutes for long-horizon coding tasks at 131K context — a 1,000-step run in 3 days for roughly $50K at rental prices, which Brown notes is comparable to what some teams already spend on tokens in a month.", "Async RL is non-negotiable for agents: coding-agent rollouts range from 30 seconds to 3 hours, so forward progress must be decoupled from the slowest rollout; they typically run ~16 steps off-policy on average, use DPPO-style losses, and report stability into the thousands of steps while pushing toward 10,000.", "Prime RL splits 'loss' (what takes the gradient) from 'algorithm' (what prepares the data) with a registry you can add to and per-environment selection, so GRPO, SFT, context/on-policy distillation, self-distillation and Echo all fit one family — rollout source (policy vs teacher) × advantage definition — without new if-statements buried in the trainer.", "Infrastructure specifics: FP8, expert parallelism on the trainer, wide (multi-node) expert parallelism and disaggregated prefill on inference, KV offloading, and router replay — which he calls a nasty systems problem because metadata must be tracked per layer per rollout; built on TorchTitan rather than Megatron because it's hackable, maintained by fewer than 10 people in a sub-40-person company." ], "takeaways": [ "Build evals first — they're good product hygiene for picking between GPT, Claude, Opus/Sonnet on intelligence-versus-dollars grounds, and they are already the exact unit of logic post-training needs, so nothing is thrown away if you later train.", "Write your harness as ordinary deployable code talking to a base URL rather than retrofitting it into an RL framework; the interception-server pattern then lets the same harness serve eval, training and production.", "When your model's chains of thought grow out of control, don't hard-code a token budget — use group-relative rewards to bonus the most concise correct sample, since the optimal length is unknown and shifts as the model improves.", "If you have several skills you want in one model, train separate RL experts on a shared base model and distill the teachers into one checkpoint — reported as more reliable than training one model on all environments at once.", "Keep trainer, inference and environments as separate processes with clean APIs; folding them together means you can't run environments as evals or experiment without doing RL, and it couples unrelated scaling decisions." ], "topics": [ "post-training", "reinforcement-learning", "evals", "environments", "distillation", "agents", "training-infrastructure", "open-source" ], "tools": [ "Prime Intellect", "Verifiers", "Prime RL", "Environments Hub", "Intellect model series", "Hugging Face datasets", "Harbor", "NeMo Gym", "Open Ended", "renderers", "uv", "Pydantic", "Docker", "Prime sandboxes", "MCP", "Codex", "Claude Code", "Open Code", "Mini SWE Agent", "LangChain", "DSPy", "GLM-5", "Kimi K2.5/2.6", "TorchTitan", "Megatron", "vLLM", "OpenAI Harmony", "GPT-OSS", "Thinking Machines Tinker", "LoRA", "OpenAI Responses API" ], "quotes": [ { "text": "evals are the thing that opens the door to post-training", "at": "06:57", "url": "https://www.youtube.com/watch?v=V-EDrhIhHzQ&t=417s" }, { "text": "the harness doesn't know that it's doing RL. The harness just is a harness running as if it would be running in a real-world environment.", "at": "22:44", "url": "https://www.youtube.com/watch?v=V-EDrhIhHzQ&t=1364s" }, { "text": "some people I think have their reasons for wanting to do synchronous training. I don't agree with them.", "at": "29:48", "url": "https://www.youtube.com/watch?v=V-EDrhIhHzQ&t=1788s" }, { "text": "50K is not cheap, but it's like if you're doing a full run on a frontier size model on like a proper real world agent environment, like it's a lot cheaper than what OpenAI's raising for it.", "at": "32:33", "url": "https://www.youtube.com/watch?v=V-EDrhIhHzQ&t=1953s" } ], "words": 11716 }, "summary_url": "/#V-EDrhIhHzQ", "transcript": { "html": "/transcripts/V-EDrhIhHzQ.html", "txt": "/transcripts/V-EDrhIhHzQ.txt", "vtt": "/transcripts/V-EDrhIhHzQ.vtt" } }