{
  "video": {
    "id": "g4SsanB0gMc",
    "title": "Large clusters for small models — Daniel Svonava, Superlinked",
    "duration": 1507,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "g4SsanB0gMc",
    "title": "Large clusters for small models — Daniel Svonava, Superlinked",
    "one_liner": "Small open-source models now match frontier quality on specific tasks, but serving a fleet of them breaks the top-down router model built for one big model — Superlinked's answer is a gateway that annotates requests into a shared NATS JetStream queue that workers pull from, roughly doubling cluster throughput.",
    "summary": "Daniel Svonava argues that 'small models are not dumb anymore' — models that fit on one two-or-three-generation-old Nvidia GPU (e.g. Qwen 3 27B around GPT-5.1 territory on the Artificial Analysis intelligence index) are catching up as the frontier flattens, so agents should be built as a fleet of task-specific open-source models rather than one generalist API. That fleet creates three infra problems: open-source serving stacks like vLLM and SGLang aren't pre-tuned per model/hardware, top-down routers can't saturate workers under many small fast requests (he measured GPU utilization stuck at 20–30% under constant load), and LoRAs/custom fine-tunes force slow back-and-forth between AI engineers and infra engineers. He describes Superlinked's Apache 2.0 cluster — control plane down to the GPU — built on a thin gateway, MessagePack instead of base64 JSON, a centralized NATS JetStream queue that workers pull from, and a Rust sidecar over a local socket fronting PyTorch, Candle and SGLang runtimes. Concrete payoff: on an RTX Pro 6000 you can push roughly half a million tokens/second into embeddings on a single GPU at low-tens-of-milliseconds latency, versus hundreds of milliseconds and orders of magnitude more cost on managed endpoints.",
    "key_points": [
      "Definition of 'small model': the whole model fits in one GPU on two-to-three-generation-old Nvidia hardware (L4, A100, RTX Pro 6000, H100) — GPUs that are available and affordable with quota on most continents.",
      "On the Artificial Analysis intelligence index the frontier is hitting diminishing returns while small open-source models climb; Qwen 3 27B sits around GPT-5.1 performance, so a GPT-5.1 pipeline can be moved to a small model.",
      "The shape of a real workload: an example contract-review agent uses nine different models, and a company might run ten such agents — hundreds of thousands of task-specific fine-tunes exist on Hugging Face under mostly permissive licenses (his example: a model trained on Vietnamese receipt OCR has seen the most Vietnamese receipts and will beat anything else on that task).",
      "Against AWS Bedrock as the default 'open-source AI': restrained model catalog, models often two to three years behind state of the art, and you don't own the fine-tuned artifacts — they stay serving from Bedrock's infra, so it can't be a business advantage.",
      "Top-down routing is the bottleneck for many small fast requests: the router works from a slightly obsolete view of worker state (KV cache etc.), batches end up mis-sized, and with vLLM and SGLang routers they found it very hard to get GPU utilization past 20–30% under constant load.",
      "Their topology inverts it: a deliberately thin gateway that doesn't fully parse the request — it peeks at packets, annotates metadata, and inserts into a shared NATS JetStream queue (capable of ~1M requests/sec) that workers pull from, forming their own batches with their own cost prediction. 'You centralize the queue and now you get double the throughput of the cluster.'",
      "Protocol and transport choices: MessagePack binary instead of the OpenAI standard's base64-encoded JSON; all multimodal bytes pushed through the API gateway itself (no cluster access to your cloud storage), with the gateway offloading pieces over ~1MB to cloud storage in flight so they don't clog the queue; REST end to end plus a local socket to the runtime.",
      "Three runtimes behind a Rust sidecar — hand-optimized PyTorch, Candle, and SGLang as the baseline. Candle's draw is image size: a PyTorch worker image is ~12GB versus ~10% of that for a statically linked Candle binary, which matters hugely for cold starts across many machines — but they still can't get Candle near PyTorch performance. Wrapping SGLang with the socket and Rust sidecar actually beats bare SGLang because of batching logic SGLang doesn't do natively.",
      "Measured on RTX Pro 6000: embedding models up to hundreds of millions of parameters hit hundreds of thousands of tokens/second (half a million on one GPU) at low-tens-of-milliseconds latency; task-specific generative models get thousands of tokens/second output (about half a thousand per second on one GPU at the bottom of the range); scaling is linear in GPU count with reasonable infra.",
      "Auto-research loops add support for new models and tune their performance, with a meta loop that builds the harness, internal measurement tooling and a custom dashboard; every shipped model comes with an end-to-end cluster config so nobody has to run a parameter sweep. One output: a LoRA that cost 80 cents to train and improved retrieval quality on German legal texts by 18%.",
      "Small models break the one-worker-pool-per-model mentality (where you preload models that take tens of minutes to load) — packing several models onto the same GPU is faster, combining pinned models with lazy loading and eviction driven by memory pressure."
    ],
    "takeaways": [
      "Stop treating a small model as a generalist you prompt for everything: slice your generalized workload into tasks, find the open-source model already fine-tuned for each task, run evals and adaptation per task, and serve a fleet.",
      "Start with embeddings — he calls it the low-hanging fruit and a no-brainer: replace a managed embeddings endpoint (OpenAI text-embedding-3, Cohere) with one GPU for orders-of-magnitude cost savings and latency in the low tens of milliseconds instead of hundreds.",
      "Move bulk offline generation — synthetic data, annotations for fine-tuning and evals — off managed endpoints onto your own infra; it's the perfect task because you control it and can survey the quality.",
      "If you're serving many small models, don't expect vLLM/SGLang out of the box to perform: either budget for an open-ended tuning and parameter-sweep research project, or adopt pull-based centralized queuing where workers form their own batches.",
      "Don't invest in runtime-specific plugins — small-model runtimes are too diverse (they maintain ~50 parameterized adapters); abstract over runtimes instead, so you aren't staged with one.",
      "Design the org boundary too: make it possible to push LoRAs and custom fine-tunes to production without the AI engineer and the infra engineer having to talk day to day, since that conversation is the main killer of velocity."
    ],
    "topics": [
      "inference",
      "small-models",
      "open-source-models",
      "gpu-serving",
      "embeddings",
      "lora",
      "infrastructure",
      "benchmarking"
    ],
    "tools": [
      "Superlinked",
      "Artificial Analysis intelligence index",
      "GLM 5.2",
      "Qwen 3 27B",
      "GPT-5.1",
      "Hugging Face",
      "AWS Bedrock",
      "vLLM",
      "SGLang",
      "PyTorch",
      "Candle",
      "Rust",
      "NATS JetStream",
      "MessagePack",
      "OpenAI API",
      "Cohere",
      "Nvidia L4",
      "Nvidia A100",
      "Nvidia RTX Pro 6000",
      "Nvidia H100",
      "Docker",
      "GitHub"
    ],
    "quotes": [
      {
        "text": "if you use an open source model that's trained to do OCR on receipts in Vietnamese that project has seen the most receipts in Vietnamese, right? There's somebody who like took the time to gather as much data as possible and on that task that model will outperform pretty much anything else.",
        "at": "05:20",
        "url": "https://www.youtube.com/watch?v=g4SsanB0gMc&t=320s"
      },
      {
        "text": "we have experimented with the VLM and SGline routers for small models and this sort of traffic and it's very hard to get your GPU utilization beyond 20 30% under constant load",
        "at": "08:31",
        "url": "https://www.youtube.com/watch?v=g4SsanB0gMc&t=511s"
      },
      {
        "text": "you centralize the queue and now you get double the throughput of the cluster. So this is this is uh significant.",
        "at": "16:16",
        "url": "https://www.youtube.com/watch?v=g4SsanB0gMc&t=976s"
      },
      {
        "text": "imagine you are sitting there now like hitting your text embedding tree on open AAI API instead you could be like having one GPU and push half a million tokens per second into that thing and get the vectors out",
        "at": "20:12",
        "url": "https://www.youtube.com/watch?v=g4SsanB0gMc&t=1212s"
      }
    ],
    "words": 4956
  },
  "summary_url": "/#g4SsanB0gMc",
  "transcript": {
    "html": "/transcripts/g4SsanB0gMc.html",
    "txt": "/transcripts/g4SsanB0gMc.txt",
    "vtt": "/transcripts/g4SsanB0gMc.vtt"
  }
}