{ "video": { "id": "tNAV259UhCM", "title": "From 15% to 90% GPU Utilization: Fix the Data Pipeline, Not the Model", "duration": 1022, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "tNAV259UhCM", "title": "From 15% to 90% GPU Utilization: Fix the Data Pipeline, Not the Model", "one_liner": "For multimodal SFT, keeping GPUs busy is mostly a data-pipeline problem: add concurrency to loading and processing, prefetch ahead of the trainer, pass Ray object references instead of copying tensors, and at scale spread workers across nodes with zero-copy retrieval. Together these took GPU utilization from 15% to 90%.", "summary": "The speaker argues that the hardest part of speeding up multimodal training is not kernel optimization but making sure the GPUs are always fed with data. Starting from a serial baseline that fine-tunes a Qwen3-VL-based model on chat, VQA and multi-turn VLM data, the GPUs sat at 15% utilization: training took 15-20s per step while fetching data took more than 100s. The team removed one bottleneck after another: async IO plus Ray actors for concurrency, a background prefetch queue, the Ray object store to avoid copying data between processes, and, at production scale, two Ray flags (zero-copy retrieval and spread scheduling) that relieved a network-card bottleneck. Their main point is that fixing one bottleneck usually exposes the next, and that scale changes which settings help.", "key_points": [ "They define a 'wait time ratio': time spent preparing data and moving it to the GPU, divided by total pipeline time (prep + transfer + train). The baseline spent about 85% of its time waiting on data.", "Serial baseline: images are fetched from S3 one at a time, then resized, normalized and tokenized one at a time for a batch of 256. Training took 15-20s per step but data took over 100s, so GPU utilization was 15%. The flame graph was dominated by S3 loads, with a tail of PIL image loading.", "Fix 1, concurrency: samples are independent, so the work parallelizes easily. They used async IO for fetching images and Ray actors as the multiprocessing layer. Data time per batch fell to about 40s, a 25% speedup. The speaker stressed that the concurrency model should match the bottleneck: async for IO, threads for work that releases the GIL (Hugging Face tokenizers, PIL transforms, some pandas operations), processes for full isolation.", "Fix 2, prefetching: a background thread keeps a queue filled ahead of the trainer, so data fetch and processing time per batch dropped to almost 0s. The caveats are more complicated checkpointing and resumption for the model and data loader, plus a cold start while the queue fills.", "Fix 3, data transport: multimodal tensors are megabytes per sample and were returned by value. Each sample was pickled on the worker, copied to the coordinator, then pickled again for the trainer, a 'mem copy that nobody really asked for' that can make the data generator run out of memory. Switching to Ray object references means the driver holds only a pointer, and the Megatron data-parallel ranks fetch from the Ray object store directly.", "After these three fixes the wait ratio fell from 85% to 20%, and the sawtooth pattern in the utilization graph largely went away.", "Scaling up (4x data-parallel ranks, about 200 data streams, 4x batch size) pushed the wait ratio back up. By default Ray prefers to schedule workers together on the same node, which saturated that node's network card. Disaggregating the work, with workers spread across nodes and samples fetched lazily, just in time, by the DP ranks, fixed it.", "Two flags, zero-copy retrieval and spread scheduling, did nothing in small-scale experiments but together gave 50% more throughput at scale: 'same knob, opposite verdict, set by the bottleneck'." ], "takeaways": [ "Profile your baseline first. Work out whether you are CPU-bound or GPU-bound, IO-bound or compute-bound, before choosing a fix.", "Pick the concurrency model that matches the bottleneck (async for IO, threads for work that releases the GIL, processes or Ray actors for CPU-heavy work) rather than reusing whatever worked last time.", "Prefetch into a queue so the producer always runs ahead of the trainer, and plan for the checkpoint/resume complexity and the cold start this adds.", "Pass large multimodal tensors by reference through an object store such as Ray's instead of pickling them across process boundaries, and count the communication cost of every parallelism choice.", "Test configuration ablations at several scales. Settings like zero-copy and spreading workers across nodes may only pay off at large scale, and colocating workers can saturate one node's network card." ], "topics": [ "gpu utilization", "data pipelines", "multimodal training", "sft", "concurrency", "prefetching", "distributed training", "profiling" ], "tools": [ "Ray", "Ray actors", "Ray object store", "Megatron", "Qwen3-VL", "S3", "PIL", "NumPy", "Hugging Face Transformers tokenizers", "pandas", "asyncio", "JSONL" ], "quotes": [ { "text": "we need to match our bottleneck. Not simply going with a lever that's worked for us before.", "at": "08:04", "url": "https://www.youtube.com/watch?v=tNAV259UhCM&t=484s" }, { "text": "per sample per step we have a mem copy that nobody really asked for", "at": "10:57", "url": "https://www.youtube.com/watch?v=tNAV259UhCM&t=657s" }, { "text": "config can do poorly at a small scale but can do really good at a larger scale.", "at": "14:51", "url": "https://www.youtube.com/watch?v=tNAV259UhCM&t=891s" }, { "text": "GPUs are expensive but sometimes the biggest battles happen outside of the GPU.", "at": "15:43", "url": "https://www.youtube.com/watch?v=tNAV259UhCM&t=943s" } ], "words": 3014 }, "summary_url": "/#tNAV259UhCM", "transcript": { "html": "/transcripts/tNAV259UhCM.html", "txt": "/transcripts/tNAV259UhCM.txt", "vtt": "/transcripts/tNAV259UhCM.vtt" } }