{
  "video": {
    "id": "dKcTBQzR7jI",
    "title": "Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging Face",
    "duration": 1154,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "dKcTBQzR7jI",
    "title": "Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging Face",
    "one_liner": "Don't deploy VLMs for vision tasks. Use small Apache-2.0 VLMs to label and judge your images, then let a coding agent train a small real-time detector like RF-DETR, for about $3–4 per pipeline run.",
    "summary": "Merve Noyan argues that developers overuse vision language models, which will never run in real time and are less robust than a trained RF-DETR. They also often ship models like YOLO without reading the AGPL license. She presents her toolkit ('vision intern'), which has two parts. The first is a 'vibe training' pipeline: a coding agent uses Qwen 3.5 9B to label images, an ensemble of small VLM judges (Gemma 4 E4B and LFM 2.5 VL) checks the labels, and RF-DETR is trained on the result, all on Hugging Face infrastructure. The second part is a curated set of her preferred permissively licensed vision models, exposed as tools for agents. She tested the pipeline on road sign detection and document parsing. It worked on both and even generalized beyond its teacher.",
    "key_points": [
      "VLMs will not give you real time (30–40 fps 'on a toaster'). A trained RF-DETR 'will always outperform your vision language model.'",
      "The YOLO license is AGPL 3.0, and some developers deploy it without knowing they have to pay. Her model-picking criteria are: 1) license (Apache 2.0, MIT), 2) benchmark performance relative to size and architecture, 3) vibes.",
      "Pipeline: label with Qwen 3.5 9B, judge with Gemma 4 E4B (~8B) and LFM 2.5 VL (~2B) because research favors an ensemble of smaller judges, merge the verdicts, then train RF-DETR medium or large. All models are Apache 2.0 except LFM, which requires payment above a revenue threshold.",
      "Judging trick: rather than passing box tokens to the judge, she overlays Qwen's bounding boxes on the image, passes it with the labels and label descriptions, and asks for a simple approve or reject. She says asking a VLM or LLM for numeric scores does not work, especially with judges of different sizes.",
      "She merges judge verdicts by minimum agreement (keep an example if one judge approves), not consensus. LFM rejects a lot, so consensus left too few examples and generalized poorly. She suggests consensus if you have a large dataset.",
      "Cost and infrastructure: about $3–4 for a whole run. She used DeepInfra serverless via Inference Providers for labeling, Hugging Face Jobs for judging, and an L4 for training (it could run locally). She built the pipeline with Opus 4.6 and ran it with GLM 5.2, which she says does well on long-horizon tasks.",
      "Results: road sign detection got a good mAP (over 50) compared with ground truth, with an expected gap versus Qwen's labels. On document parsing (a DocQA dataset, extracting images, tables and signatures), the trained RF-DETR detected a signature that Qwen's annotation had missed.",
      "Coding agents like Opus 4.8 lack computer-vision common sense. They applied horizontal flips to traffic signs and jitter to traffic lights, which corrupts the data. She added an augmentation toggle to fix this.",
      "Model-as-tools picks, drawn from Hugging Face benchmark leaderboards and conference papers: Falcon Perception (TII, 600M, Apache 2.0) for open-ended referring segmentation that SAM can't do, the Sapiens family for human-centric tasks, Moondream 3 and MM Grounding DINO for zero-shot detection, OCR models from the OCR benchmark, depth estimation models (avoid the non-commercial large one), and Roboflow's supervision and trackers libraries."
    ],
    "takeaways": [
      "Stop deploying VLMs for production vision tasks. Use them offline as labelers and judges, then train and deploy a small task-specific model such as RF-DETR, including on edge devices.",
      "Check licenses before picking a model: prefer Apache 2.0 or MIT, and move off AGPL models like YOLO.",
      "For VLM-as-judge, overlay the predictions on the image and ask for a binary approve or reject instead of a numeric score. Use an ensemble of small judges and pick the merge rule (min-agreement or consensus) based on dataset size and judge imbalance.",
      "Review the judge prompts and label descriptions the agent generates, and look at your data yourself. There's no escaping that.",
      "Don't trust a coding agent's augmentation choices. Check for transforms that break semantics, such as flipping traffic signs."
    ],
    "topics": [
      "computer vision",
      "vision language models",
      "agents",
      "llm-as-judge",
      "synthetic labeling",
      "object detection",
      "model licensing",
      "edge deployment"
    ],
    "tools": [
      "RF-DETR",
      "Roboflow",
      "supervision",
      "trackers",
      "YOLO",
      "SAM 3.1",
      "Gemma 4",
      "Gemma 4 E4B",
      "Qwen 3.5 9B",
      "LFM 2.5 VL",
      "Opus 4.6",
      "Opus 4.8",
      "GLM 5.2",
      "Hugging Face",
      "Hugging Face Jobs",
      "Hugging Face Inference Providers",
      "Hugging Face buckets",
      "Hugging Face benchmarks",
      "Hugging Face skills",
      "Transformers v5",
      "DeepInfra",
      "Together",
      "NVIDIA L4",
      "Falcon Perception",
      "TII",
      "Sapiens",
      "Moondream 3",
      "MM Grounding DINO",
      "LLaVA"
    ],
    "quotes": [
      {
        "text": "your coding agent is a bit of a clueless computer vision engineer basically",
        "at": "02:59",
        "url": "https://www.youtube.com/watch?v=dKcTBQzR7jI&t=179s"
      },
      {
        "text": "most of the people ask the VLM or LLM to assign some score to it but those scores absolutely absolutely don't work",
        "at": "06:30",
        "url": "https://www.youtube.com/watch?v=dKcTBQzR7jI&t=390s"
      },
      {
        "text": "overall it takes like three four dollars if you want to run this entire pipeline to train models which to me is crazy",
        "at": "08:54",
        "url": "https://www.youtube.com/watch?v=dKcTBQzR7jI&t=534s"
      },
      {
        "text": "take a look at your data set a little there's no escaping that",
        "at": "12:42",
        "url": "https://www.youtube.com/watch?v=dKcTBQzR7jI&t=762s"
      }
    ],
    "words": 3352
  },
  "summary_url": "/#dKcTBQzR7jI",
  "transcript": {
    "html": "/transcripts/dKcTBQzR7jI.html",
    "txt": "/transcripts/dKcTBQzR7jI.txt",
    "vtt": "/transcripts/dKcTBQzR7jI.vtt"
  }
}