{
  "video": {
    "id": "4n08JHb90hI",
    "title": "Agents That Own Their Inference — Du'an Lightfoot & Khaja Omer, Akamai Technologies",
    "duration": 6610,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "4n08JHb90hI",
    "title": "Agents That Own Their Inference — Du'an Lightfoot & Khaja Omer, Akamai Technologies",
    "one_liner": "Running agents on your own GPU is mostly an exercise in measuring: fit the model weights and KV cache into VRAM, run a load test after every single vLLM flag change, and don't assume quantization or speculative decoding will help until your own evals and benchmarks show it. On one 20 GB RTX 4000 Ada, FP8 quantization and raising max-num-seqs helped, while a mismatched draft model made performance fall sharply.",
    "summary": "Du'an Lightfoot and Khaja 'Omar' Omer argue that agents usually 'rent their intelligence' from token-billed providers. Owning inference on dedicated GPUs pays off when usage is high and steady, when data must stay private or meet compliance rules, and when you have the hardware capacity. If traffic is low or bursty, or you need frontier-level intelligence, or your team lacks the skills, renting is better. In a hands-on workshop on Akamai's LKE (a Kubernetes cluster of 200+ GPU nodes, one GPU per attendee), they serve Qwen3 4B with vLLM. They cover memory budgets, prefill/decode and the KV cache, dense vs. mixture-of-experts models, FP8 quantization, speculative decoding and vLLM flag tuning, and run a load test before and after each change.",
    "key_points": [
      "Memory budget: Qwen3 4B at BF16 is about 8 GB of weights (parameters × 2 bytes). On the 20 GB RTX 4000 Ada the rest goes to KV cache, roughly 37,000 tokens, e.g. four users at about 8,000 tokens each (input, output and tool calls). Decode speed is limited by memory bandwidth: about 40 tok/s measured against a calculated ceiling of 45 tok/s.",
      "vLLM was picked over Ollama for production because it has continuous batching and PagedAttention built in, plus an OpenAI-compatible API. It serves one model per endpoint, has authentication but no authorization, and has no rate limiting, so you need a gateway (KServe, agentgateway, llm-d) for routing, auth and rate limits. SGLang is suggested for long shared prefixes and multi-turn agents, TensorRT for peak throughput on Nvidia, and llama.cpp for edge use with heavily quantized models.",
      "Rent vs. own example: about 200,000 tokens/day costs about $1,500/month on a hosted API vs. about $1,000/month for a rented GPU. Separately, provider prompt caching can cut costs by up to 75%. Turning on Qwen thinking raised a response from 35 tokens to about 600.",
      "Switching from BF16 to RedHat's FP8-dynamic Qwen3 4B raised single-request speed from 37.5 to 53.4 tok/s and peak throughput about 1.4x. Simple evals showed no quality loss. 'Dynamic' means only some parts are quantized (e.g. the attention heads), which is why the FP8 model is about 5.1 GB rather than 4 GB. FP4 on this GPU would make things slower because the hardware doesn't accelerate it.",
      "Speculative decoding with Qwen3 0.6B as the draft model and 4 speculative tokens made things much worse: 58.2 dropped to 16 tok/s at concurrency 1, and 700 dropped to 240 tok/s at concurrency 16. The cause was a low acceptance rate, because that draft model wasn't trained properly for the target model. When it is set up correctly, vLLM reports about 1.5x. Omar gets 2–3x at home with Qwen 3.6 35B plus a DFlash draft model, roughly 3x for coding and 1.5x for browser-use agents.",
      "Find the 'knee' where throughput stops rising but latency is still acceptable. Here time-to-first-token jumped from 277 ms to about 1,300 ms between 32 and 64 concurrent requests, because max-num-seqs was set to 32. Raising it to 128 took 64-concurrency throughput from about 800 to 1,100 tok/s and cut latency from about 2,000 ms to 800 ms.",
      "Mixture-of-experts models (e.g. 30B with 128 experts, 8 active per token) read about 9x fewer bytes per token than a dense model, so a bigger MoE model can decode faster than a smaller dense one.",
      "Prefix caching (not the same as provider prompt caching) cut a warm-prefix request from 220 ms to 41 ms."
    ],
    "takeaways": [
      "Do the math first: weights (bytes per parameter × parameter count) plus KV cache × concurrent context must fit in VRAM. Tune for your worst-case latency, not the median.",
      "Build your own eval set (JSON validity, function-call correctness, loops, hallucinations) and run it every time you change quantization or the model. Public benchmarks often don't reproduce.",
      "Tune vLLM one flag at a time and load-test after each change. The main flags are gpu-memory-utilization (0.9–0.95 if it's the only model on the GPU), max-num-seqs, max model length and batched tokens.",
      "For speculative decoding, use a draft model released with or trained alongside the main model, at matching precision, usually under 1B parameters. Start with 2–4 speculative tokens and check the acceptance rate.",
      "Build observability in from the start, e.g. self-hosted Langfuse plus Prometheus/Grafana on vLLM metrics, so you can see per-tool-call latency and token use when a smaller model starts making too many tool calls."
    ],
    "topics": [
      "agents",
      "self-hosted inference",
      "vllm",
      "kv cache",
      "quantization",
      "speculative decoding",
      "kubernetes",
      "gpu optimization"
    ],
    "tools": [
      "Akamai LKE",
      "Kubernetes",
      "Nvidia RTX 4000 Ada",
      "vLLM",
      "Ollama",
      "llama.cpp",
      "SGLang",
      "TensorRT",
      "KServe",
      "agentgateway",
      "llm-d",
      "Langfuse",
      "Prometheus",
      "Grafana",
      "Jupyter",
      "Hugging Face",
      "LLM Compressor",
      "Qwen3 4B",
      "Qwen3 0.6B",
      "Qwen 3.6 35B",
      "Red Hat FP8-dynamic models",
      "DFlash",
      "Medusa",
      "EAGLE",
      "DeepSeek DSpark",
      "GLM",
      "Kimi",
      "Mistral",
      "NVFP4",
      "GGUF",
      "AWQ",
      "GPTQ",
      "OpenAI-compatible API",
      "Blackwell"
    ],
    "quotes": [
      {
        "text": "When it comes to an agent, it's just a loop.",
        "at": "01:51",
        "url": "https://www.youtube.com/watch?v=4n08JHb90hI&t=111s"
      },
      {
        "text": "that GPU should always be working. Otherwise, what are you paying for?",
        "at": "04:05",
        "url": "https://www.youtube.com/watch?v=4n08JHb90hI&t=245s"
      },
      {
        "text": "If you set it too low, you're leaving performance on the table.",
        "at": "73:56",
        "url": "https://www.youtube.com/watch?v=4n08JHb90hI&t=4436s"
      },
      {
        "text": "I'm personally a little skeptical of benchmarks whenever I see them. I never get the same results.",
        "at": "90:56",
        "url": "https://www.youtube.com/watch?v=4n08JHb90hI&t=5456s"
      }
    ],
    "words": 21325
  },
  "summary_url": "/#4n08JHb90hI",
  "transcript": {
    "html": "/transcripts/4n08JHb90hI.html",
    "txt": "/transcripts/4n08JHb90hI.txt",
    "vtt": "/transcripts/4n08JHb90hI.vtt"
  }
}