{
  "video": {
    "id": "byn9PURoBNY",
    "title": "Infra behind Krea 2: How to train and serve at scale — Gabriel Jorge Menezes, Krea.ai",
    "duration": 1015,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "byn9PURoBNY",
    "title": "Infra behind Krea 2: How to train and serve at scale — Gabriel Jorge Menezes, Krea.ai",
    "one_liner": "Krea's infra lead walks through what it actually took to pre-train Krea 2 from scratch on thousands of InfiniBand-connected GPUs — treat GPU utilization as a lie, instrument InfiniBand/NVLink yourself, checkpoint constantly because runs crash every few hours, and use Kueue plus virtual-kubelet to let trainings preempt production inference onto rented GPUs.",
    "summary": "Gabriel from Krea explains the infrastructure behind Krea 2 (K2), a text-to-image diffusion transformer pre-trained from scratch in house on one big shared cluster, and released open source with a raw and a post-trained turbo checkpoint. The training half is a blunt operational account: at scale runs crashed far more often than Meta's published failure estimates suggested — often under 8 hours — so the answer was heavy metrics (GPU temperature, tensor core utilization, custom InfiniBand and NVLink exporters), ripping out any GPU over 78°C, and a fast paid filesystem that could absorb a terabyte checkpoint in under 30 seconds every 20–30 minutes. The serving half describes a self-healing system where researchers just submit to a Kueue gang-scheduled queue at higher priority than production, and a virtual-kubelet-based layer transparently flips inference pods out to external GPU providers, migrating them back with taints and a descheduler once the cluster frees up.",
    "key_points": [
      "K2 was pre-trained from scratch with no base checkpoint on thousands of InfiniBand-connected GPUs, with the architecture kept deliberately 'very very dumb, but like very effective'; the research approach was porting LLM research into diffusion transformers (DiTs). Two checkpoints are open on Hugging Face and GitHub — a raw pre-trained one for post-training, and a turbo post-trained one that produces an image in under a second.",
      "Method was small ablations on small GPU counts to test hypotheses, then scaling through 128, 256, 512 and up; crashes rose with scale and often failed silently (NCCL timeouts) while the metrics still looked fine.",
      "Meta's paper gives a rough expected-failure rate; Krea saw the same pattern but worse numbers — runs lasting well under 8 hours — and learned to stop being paranoid about swapping nodes: sometimes the same machines, same code and same data crash hourly and then run 12–24 hours straight.",
      "Metric opinions: GPU utilization is '100% a lie' (it measures time doing work, not efficiency — it read 100% throughout pre-training while the GPUs were not fully utilized); use tensor core utilization as the proxy, which visibly rose as image resolution scaled through 128 → 256 → 512 → 1024 across pre-training, mid-training and post-training.",
      "Any GPU running above 78°C gets removed, not debugged — a single hot GPU throttles and destabilizes the whole training run.",
      "InfiniBand metrics are not exported by DCGM and NVLink errors aren't exported by NVIDIA either; Krea built custom collectors for fabric wait times, error types and packet counts. Most of their failures were cross-node communication, and NVLink errors caught single-node faults where the GPUs otherwise looked healthy.",
      "Ceph broke and cost them trust in their data; they moved to a paid filesystem doing ~1.8 TB/s reads and nearly 1 TB/s writes, letting them checkpoint every 20–30 minutes and flush a terabyte in under 30 seconds without delaying training.",
      "Scheduling: Kueue (open source) provides gang scheduling plus a workload-priority tier above normal Kubernetes priority; training pods always outrank inference and evict it. Kueue's manual per-queue resource quotas drift out of sync on a fluid cluster where nodes come and go, which breaks gang scheduling — Kubernetes 1.15 [as stated] has gang scheduling out of the box.",
      "Serving: a virtual kubelet creates a fake Kubernetes node; when in-cluster GPUs are full, pods land there, an algorithm picks an external GPU provider, pods are translated and deployed, and a reconciler keeps both sides in sync. Failures need no heroics — mark the pod failed and let Kubernetes/HPA create a new one.",
      "A Prometheus-driven taint controller adds a taint to the virtual node when cluster GPUs are free (so nothing burns money offsite) and removes it when a training hogs the GPUs; a descheduler slowly migrates pods back rather than a NoExecute taint, which would evict everything at once and take production down.",
      "Diffusion transformers don't need multi-node inference like big LLMs, so serving tolerates terrible hardware: 'the GPU can be hot, falling out of the bus. It can be exploding' and inference still runs."
    ],
    "takeaways": [
      "Invest heavily in metrics before you scale up pre-training — and specifically build the ones your vendor doesn't ship: InfiniBand throughput/wait/errors and NVLink errors. If you're training across machines without InfiniBand metrics, you're doing something wrong.",
      "Stop trusting GPU utilization as a health or efficiency signal; watch tensor core utilization instead, and pull any GPU over 78°C rather than trying to fix it.",
      "Assume frequent crashes and buy your way out with I/O: a filesystem fast enough to checkpoint every 20–30 minutes without stalling training is the cheapest recovery mechanism. Don't run large-scale training on a filesystem you don't trust.",
      "Put trainings and production inference on one cluster with a gang-scheduling queue (Kueue) where trainings preempt inference, and use virtual kubelet plus taints and a descheduler to burst inference to external providers and pull it back automatically — researchers should never have to think about GPUs.",
      "For diffusion-transformer inference, don't pay for premium GPUs — single-node inference runs fine on the worst hardware you can rent."
    ],
    "topics": [
      "infrastructure",
      "gpu-clusters",
      "pretraining",
      "diffusion-models",
      "kubernetes",
      "observability",
      "inference-serving",
      "scheduling"
    ],
    "tools": [
      "Krea (krea.ai)",
      "Krea 2 / K2",
      "Hugging Face",
      "GitHub",
      "NVIDIA DCGM",
      "InfiniBand",
      "NVLink",
      "NCCL",
      "Ceph",
      "Kubernetes",
      "Kueue",
      "virtual kubelet",
      "Kubernetes HPA",
      "Kubernetes descheduler",
      "Prometheus",
      "Meta (failure-rate paper)"
    ],
    "quotes": [
      {
        "text": "there is GPU utilization which is a lie. Uh, don't trust this.",
        "at": "05:27",
        "url": "https://www.youtube.com/watch?v=byn9PURoBNY&t=327s"
      },
      {
        "text": "we learned that like sometimes you just let it crash",
        "at": "03:41",
        "url": "https://www.youtube.com/watch?v=byn9PURoBNY&t=221s"
      },
      {
        "text": "I don't want my researchers to think about GPUs. I just want them to launch stuff",
        "at": "08:53",
        "url": "https://www.youtube.com/watch?v=byn9PURoBNY&t=533s"
      },
      {
        "text": "the GPU can be hot, falling out of the bus. It can be exploding. Uh inference still going to run.",
        "at": "16:11",
        "url": "https://www.youtube.com/watch?v=byn9PURoBNY&t=971s"
      }
    ],
    "words": 3972
  },
  "summary_url": "/#byn9PURoBNY",
  "transcript": {
    "html": "/transcripts/byn9PURoBNY.html",
    "txt": "/transcripts/byn9PURoBNY.txt",
    "vtt": "/transcripts/byn9PURoBNY.vtt"
  }
}