{ "video": { "id": "FQwTqUmcbRg", "title": "Lessons from Generating 12 Trillion Synthetic Tokens — Bogdan Gaza, DatologyAI", "duration": 1232, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "FQwTqUmcbRg", "title": "Lessons from Generating 12 Trillion Synthetic Tokens — Bogdan Gaza, DatologyAI", "one_liner": "DatologyAI got 12 trillion synthetic pre-training tokens out by putting synthetic data generation onto its Kubernetes pipeline and fixing four bottlenecks: batched S3 metadata fetches (11 days down to 2 hours), small partitions plus checkpointing, atomic CPU+GPU scheduling across clusters, and vLLM flag sweeps that gave about 40% more throughput.", "summary": "Bogdan Gaza argues that pre-training has hit a data wall. The web holds only about 30 trillion text tokens, so high-quality synthetic data has to fill the gap. DatologyAI's BeyondWeb recipe makes that data by rephrasing high-quality documents chosen from a customer's own dataset, rather than prompting a model to generate from nothing. Most of the talk covers the engineering behind a 12-trillion-token run (7T web, 5T math and code): moving from a separate Slurm research cluster to Kubernetes/Ray/vLLM on EKS on HyperPod, and the lessons from metadata, GPU failures, cross-cluster scheduling and inference tuning.", "key_points": [ "BeyondWeb rephrases or restructures high-quality seed documents instead of generating from a bare prompt, because unseeded generation \"just learn[s] the modes of the distribution\" the model was trained on.", "In 1B/3B/8B comparisons against Nemotron-Synth, Cosmopedia, QA-wrap and RedPajama, BeyondWeb at 3B matched Nemotron-Synth at 8B. It also matched Nemotron-Synth and Cosmopedia with 2.7x and 5.3x fewer tokens respectively. These results were published in late summer 2025, and the speaker says DatologyAI is now much better.", "Infrastructure moved from a two-track setup (synthetic runs on a Slurm H100 research cluster, products on Kubernetes, with manual backfill into the asset tracker / data catalog) to a single setup. A control/orchestration cluster, a Spark/Ray compute cluster and an H100 cluster on AWS EKS on HyperPod are tied together by an in-house job scheduler, with S3 as storage. One workflow can now run curate → synthetic → train → eval.", "Metadata bottleneck: fetching metadata one object at a time for millions of Parquet partitions through the Ray head runs into S3 throttling and leaves GPUs idle. A back-of-the-envelope estimate put it at 9–11 days. Batched S3 list requests at the maximum page size of about 1,000 cut it to about 2 hours.", "GPU instability: with a one-to-one input/output partition mapping, an 8-hour rephrasing job that dies in its last 5 minutes loses nearly all its work. The fix was right-sized partitions plus periodic checkpoints to S3, with idempotent overwrites, so failures become cheap retries instead of multi-day losses.", "Cross-infrastructure orchestration: synthetic jobs can run in GPU clusters around the world, since they don't need fast interconnect. Scheduling CPU (Ray head) and GPU (workers) separately can deadlock on capacity, which the speaker frames as a bin-packing problem. The fix was dedicated resource pools and atomic scheduling of the Ray head and workers.", "Inference tuning: grid-searching vLLM/SGLang flags such as batch size and speculative decoding in a dedicated benchmarking harness gave about 40% more throughput for batch inference. They use vLLM and are currently testing SGLang.", "Scale growth: around 30B synthetic tokens in limited Slurm runs at the end of 2024, versus about 12T (7T web, 5T math and code) by the end of last year." ], "takeaways": [ "Before a large synthetic run, plan how you will load S3 bucket state and metadata into the system. Use batched list APIs at the maximum page size, not per-object requests.", "Size partitions so that losing one to a node failure is tolerable, and checkpoint progress to object storage with idempotent writes so retries resume cleanly.", "When one job needs both CPU and GPU (for example a Ray head and GPU workers), schedule them atomically from dedicated pools so capacity in one pool doesn't strand the other.", "Build a harness that benchmarks only your inference-engine flags and grid-search batch size, speculative decoding and similar settings. Batch inference tuning is different from online serving.", "Seed synthetic generation with high-quality real documents and rephrase them, rather than prompting a model to generate from scratch." ], "topics": [ "synthetic data", "pre-training", "data curation", "kubernetes", "batch inference", "distributed systems", "fault tolerance", "scheduling" ], "tools": [ "DatologyAI", "BeyondWeb", "Nemotron-Synth", "Cosmopedia", "QA-wrap", "RedPajama", "FineWeb", "DCLM", "DataComp", "Kubernetes", "Ray", "Spark", "vLLM", "SGLang", "Slurm", "Amazon S3", "AWS", "AWS SageMaker HyperPod (EKS on HyperPod)", "GCP", "Parquet", "NVIDIA H100" ], "quotes": [ { "text": "Think of us as the oil refinery for your kind of pre-training and mid-training data sets.", "at": "01:53", "url": "https://www.youtube.com/watch?v=FQwTqUmcbRg&t=113s" }, { "text": "that leads to the the worst thing possible which is that you get idle GPUs.", "at": "10:29", "url": "https://www.youtube.com/watch?v=FQwTqUmcbRg&t=629s" }, { "text": "So, 11 days to two hours.", "at": "12:26", "url": "https://www.youtube.com/watch?v=FQwTqUmcbRg&t=746s" }, { "text": "about 40% throughput gains just by tweaking the flats for VLM", "at": "18:16", "url": "https://www.youtube.com/watch?v=FQwTqUmcbRg&t=1096s" } ], "words": 4593 }, "summary_url": "/#FQwTqUmcbRg", "transcript": { "html": "/transcripts/FQwTqUmcbRg.html", "txt": "/transcripts/FQwTqUmcbRg.txt", "vtt": "/transcripts/FQwTqUmcbRg.vtt" } }