{
  "video": {
    "id": "fWXJM-J0ZB8",
    "title": "Frontier results, on device - RL Nabors, Arize",
    "duration": 1852,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "Online"
  },
  "analysis": {
    "video_id": "fWXJM-J0ZB8",
    "title": "Frontier results, on device - RL Nabors, Arize",
    "one_liner": "Rachel Lee Neighbors (Arize) walks through a measured, eval-driven method — \"prototype big, deploy small\" — for replacing frontier-model calls with on-device small models, showing Llama 3.2 3B plus a few-shot prompt and post-processing matching Claude Sonnet on thread summarization and saving her about a dollar a day.",
    "summary": "The talk argues that reaching for GPT-5 or Claude by default costs trust (data leaves the device), user experience (4 seconds is the research-backed limit of believability, and many big-model calls exceed it), money, and uptime — and that token prices falling doesn't help because agentic workloads burn tokens faster than prices drop. Neighbors presents a four-step right-sizing framework built with Google: prove the task is possible with the largest model, define success criteria on a golden dataset, test models from small to large, and select the \"SAGE\" model — small and good enough. She demonstrates it live on Mima, her social-network client, evaluating Qwen 2.5 1.5B, Qwen 3 1.7B, Llama 3.2 3B and Gemma 4 E2B against a Claude Sonnet baseline in Arize's open-source Phoenix, then closes the remaining accuracy gap with few-shot prompting and harness post-processing rather than a bigger model.",
    "key_points": [
      "Cost framing: SLMs use roughly 25% of the energy an LLM needs for the same task, and task-specific models about half of that again; SLMs run millions-to-billions of parameters versus billions-to-trillions for LLMs, and 1B parameters fits in about 2 GB at FP16 (less with 8-bit/4-bit quantization, which can quarter disk and memory).",
      "Cheat sheet by modality: camera/vision → MobileNet, YOLO, MediaPipe; microphone/audio → Whisper, Wav2Vec2; chat, translation or analysis → a small language model like Gemma or Qwen. NVIDIA's 2025 paper called SLMs the future of agentic AI.",
      "The golden dataset was 14 Mima threads with two outputs each (short summary, summary with references) = 28 examples as JSONL, scored on JSON validity (just try JSON.parse), reference structural validity, factual consistency (LLM-as-judge), length compliance, and P50/P95 latency.",
      "Claude Sonnet baseline: 2.9s average latency, $0.22 for 14 tasks, which extrapolated to about $1/day of inference for Mima. Every local model's total cost column was \"absolutely zilch\" because inference moves to the user's device and battery.",
      "Model bake-off (3 runs each, averaged): Qwen 2.5 1.5B was fastest at ~1s P50 but least accurate; Gemma 4 E2B (5B params, 3.1 GB) — the one everyone recommended — came in around 8 seconds; Llama 3.2 3B won at ~90% accuracy with latency competitive with Claude. Taking peer advice would have shipped a materially worse experience.",
      "Prompt-variant experiment on Llama 3.2 (one variable per variant): numbered-message input made no difference; strict negative rules made things worse (\"the model responded very negatively to being told what it couldn't do\"); chain of thought helped length slightly but added 600ms; few-shot won — better length compliance, better reference agreement, only +200ms.",
      "Opening the eval traces mattered: the residual factual-consistency gap was largely Claude Opus being a strict judge favoring Claude Sonnet's phrasing over Llama's (\"angsty\" vs \"cross\"), not a real quality gap.",
      "Post-processing in the harness (checking reference count against thread length, truncating over-long summaries) closed the rest: 100% JSON validity, 100% structural validity, P50 ~1s, P95 ~3.5s — meeting and beating Claude Sonnet while saving ~$1/day.",
      "Keep regression evals running like CI/CD tests so a prompt tweak or model upgrade can't silently blow up summaries — \"it's how you keep your CTO from blowing away your agentic experience by accident one morning.\""
    ],
    "takeaways": [
      "Audit what you currently send to frontier models and ask which calls a smaller model could handle — then prototype big and deploy small, converting individual features rather than whole systems.",
      "Before testing anything, export a golden dataset of input/output pairs from the big model and write down your success criteria, including hard latency budgets (she used 1.5s P50 / 3.5s P95, derived from the 4-second believability limit).",
      "Test models from smallest upward and pick the SAGE model — the smallest one giving acceptable responses — instead of trusting what engineers recommend; Gemma 4 was the popular pick and was ~8x slower than the winner.",
      "Close the remaining gap with few-shot prompting and deterministic post-processing in the harness before reaching for a bigger model or a distilled one (a distilled model means re-shipping 1-2 GB to users on every capability change).",
      "Actually open your eval experiments and read the raw versus expected responses — an LLM judge can be strict or biased toward its own family and understate a small model's real quality.",
      "Look for models already on the device: Chrome's Prompt API gives access to Gemini Nano shipped with the browser, so there's nothing to download."
    ],
    "topics": [
      "small language models",
      "on-device inference",
      "evals",
      "prompt engineering",
      "latency",
      "inference cost",
      "llm-as-judge",
      "local models"
    ],
    "tools": [
      "Arize",
      "Phoenix",
      "Claude Sonnet",
      "Claude Opus",
      "Llama 3.2 3B",
      "Qwen 2.5 Instruct 1.5B",
      "Qwen 3 1.7B",
      "Gemma 3",
      "Gemma 4 E2B",
      "Gemini",
      "Gemini Nano",
      "Chrome Prompt API",
      "GPT-5",
      "Goose",
      "MobileNet",
      "YOLO",
      "MediaPipe",
      "Whisper",
      "Wav2Vec2",
      "NVIDIA",
      "Mima",
      "Google web.dev",
      "Pixel 10 Pro",
      "Meta"
    ],
    "quotes": [
      {
        "text": "Now, token costs have been falling as of late, but total inference spend has been rising because agent can reasoning workloads consume tokens way faster than prices are dropping.",
        "at": "02:35",
        "url": "https://www.youtube.com/watch?v=fWXJM-J0ZB8&t=155s"
      },
      {
        "text": "prototype big, deploy small. Just repeat this to yourself. Prototype big. Think big. Go big. Deploy small.",
        "at": "08:54",
        "url": "https://www.youtube.com/watch?v=fWXJM-J0ZB8&t=534s"
      },
      {
        "text": "Good news is the total cost column for all these small local models is absolutely zilch because that inference has been pushed to the consumer. It runs on their device.",
        "at": "14:38",
        "url": "https://www.youtube.com/watch?v=fWXJM-J0ZB8&t=878s"
      },
      {
        "text": "You're going to want to select the smallest model that gives acceptable responses for your use case. Or as I like to call it, the SAGE model, the small and good enough model.",
        "at": "15:59",
        "url": "https://www.youtube.com/watch?v=fWXJM-J0ZB8&t=959s"
      }
    ],
    "words": 5720
  },
  "summary_url": "/#fWXJM-J0ZB8",
  "transcript": {
    "html": "/transcripts/fWXJM-J0ZB8.html",
    "txt": "/transcripts/fWXJM-J0ZB8.txt",
    "vtt": "/transcripts/fWXJM-J0ZB8.vtt"
  }
}