{
  "video": {
    "id": "hacEQHHhu2Q",
    "title": "Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google",
    "duration": 1305,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "hacEQHHhu2Q",
    "title": "Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google",
    "one_liner": "Google's AI Edge tech lead walks through what 1–4B \"small\" models can already do on phones, Raspberry Pis and NPUs, and argues that reaching the wider device pool means dropping to fine-tuned 270M–500M \"tiny\" models trained on synthetic data — with voice-to-function-calling as the killer edge use case.",
    "summary": "Cormac Brick argues that getting intelligence into lots and lots of devices — not just expensive robots — requires models far smaller than today's small LLMs, because DRAM cost is now the binding constraint on edge deployment. He shows what a heavily quantized ~2B Gemma can do (841 MB of weights, ~2 GB active RAM, 7.6 tok/s decode on a Raspberry Pi, 31 tok/s on a Qualcomm NPU) and where it still falls short, then makes the case for tiny models in the 270M–500M range. The playbook for tiny is different: either an off-the-shelf fixed-task model (ASR, vision, embeddings) or synthetic-data fine-tuning, which he claims matches or beats a 2–4B model on a single task at a fraction of the footprint.",
    "key_points": [
      "DRAM cost is the new leading edge-AI constraint, and it is getting worse, not better: some phone makers are shipping less DRAM this year than last, and a large-RAM Raspberry Pi has risen ~2.5x in price since launch. That casts a shadow over the whole talk — it forces hard thinking about quantization and about the smallest model that will do a given task.",
      "\"Small\" = roughly 1–4B params, already shipping inside the OS (Android AI Core, Apple Intelligence). Implies 4–8 GB DRAM on an IoT/robotics device, which prices it out of low-end browsers and the consumer robotics market. The playbook is easy: zero-shot prompting, LoRA adapters, reasonably robust function calling and agent skills.",
      "Concrete footprint numbers for Gemma E2B: a mix of 2-bit, 4-bit and 8-bit quantization plus per-layer embeddings gets it to ~2.9 bits per weight, 841 MB of text-only weights in memory, ~2 GB active RAM once runtime and KV cache are added — hence the 4 GB+ rule of thumb for the device.",
      "Speed on their runtime: 2B model at 7.6 tok/s decode on a Raspberry Pi (roughly 2x with MTP enabled), up to ~24 tok/s on a Jetson Orin Nano, and ~4,000 tok/s prefill / 31 tok/s decode on a Qualcomm IoT NPU board. With Gemma 4 image tokenization (500 tokens medium-res, 1120 high-res), that NPU supports roughly 3 fps of high-res vision with usable decode.",
      "The Open Dog Mini v2 robot (a hobby project by Xavier, a DeepMind engineer) is the honesty check: the Jetson Nano build gives good real-time interaction, the Raspberry Pi build works but is a lot slower — today's best small models still miss user-interaction requirements for some tasks.",
      "Three reasons small is still too big: older laptops and consumer edge devices are out of reach, interaction needs to be more responsive, and — a common pattern — the model is often not the main feature but \"one tiny thing in a corner\" that must run while everything else in the system is running.",
      "Tiny models (stated as \"as small as kind of 50 billion parameters\" — an apparent slip — up to 500M) fit in under 2 GB. Fixed-task options work off the shelf: Apple FastVLM at 0.5B runs hardware-accelerated on Android for first-level visual awareness, plus stock ASR and text embedding models.",
      "The fine-tuning path: start from Gemma 3 270M or FunctionGemma (extra pre-training for function-calling patterns), which jumps Raspberry Pi decode from mid-single-digits to 45 tok/s. Their \"mobile actions\" model does text-in/function-call-out across ~10 functions at over 86% reliability (schedule a calendar entry, toggle Wi-Fi), and with an ASR model in front becomes voice-to-function-calling. The dataset is open-sourced on Hugging Face.",
      "Shipping example: a subscription-free voice dictation app on iOS running entirely on device, built from two fine-tuned Gemma models in the low hundreds of millions of parameters — an ASR engine and a text \"policing\" engine that strips ums and ahs and biases toward your personal names and words. Chrome's developer-preview summarization and proofread APIs use the same tiny-model approach to reach far more users."
    ],
    "takeaways": [
      "Budget by DRAM, not by parameter count. Pick the smallest model for the task, quantize aggressively (mixed 2/4/8-bit, per-layer embeddings), and remember the real number is weights + runtime + KV cache + the rest of the system.",
      "If you can afford 4 GB+ on the device, stop at a small model and zero-shot prompt it — Brick is explicit that your journey ends there. Only go tiny when reach, responsiveness, or coexisting with the rest of the app forces it.",
      "For a single well-defined task, invest in a synthetic dataset — 10,000 to 10 million samples is the range they find sufficient — and fine-tune a 270M-class model rather than prompting a 2–4B one. You get equal or better quality, much wider device reach, and much better responsiveness.",
      "Reach for off-the-shelf fixed-task tiny models (ASR, vision like FastVLM, text embeddings) before assuming you need a general LLM on the device.",
      "Treat voice-to-function-calling as the default edge UX: settings menus on small devices are a genuinely bad interface, and this is now buildable robustly. Try it first via the open-source AI Edge Gallery app on iOS/Android, and recreate the demo from the open-sourced mobile actions dataset + FunctionGemma."
    ],
    "topics": [
      "edge-ai",
      "tiny-models",
      "quantization",
      "fine-tuning",
      "synthetic-data",
      "function-calling",
      "robotics",
      "on-device-asr"
    ],
    "tools": [
      "LiteRT-LM",
      "LiteRT",
      "MediaPipe",
      "Gemma",
      "Gemma E2B",
      "Gemma 3",
      "Gemma 3 270M",
      "FunctionGemma",
      "AI Edge Gallery",
      "Google AI Edge",
      "Android AI Core",
      "Apple Intelligence",
      "Apple FastVLM",
      "LoRA",
      "Raspberry Pi",
      "Jetson Orin Nano",
      "Nvidia",
      "Qualcomm",
      "Hugging Face",
      "Chrome (summarization / proofread APIs)",
      "Open Dog Mini v2",
      "DeepMind"
    ],
    "quotes": [
      {
        "text": "If we want uh for intelligence to get into lots and lots and lots of devices and not just really expensive robots, we are going to need tiny models.",
        "at": "00:17",
        "url": "https://www.youtube.com/watch?v=hacEQHHhu2Q&t=17s"
      },
      {
        "text": "a lot of the research uh hours that go into LLMs these days are into the much larger models um and techniques and this types of stuff um and the the lower end of the LLM spectrum is a lot less studied",
        "at": "04:23",
        "url": "https://www.youtube.com/watch?v=hacEQHHhu2Q&t=263s"
      },
      {
        "text": "the model you want to run isn't the main feature in the application. It's like one tiny thing in a corner that needs to run while everything else in the system is running.",
        "at": "12:26",
        "url": "https://www.youtube.com/watch?v=hacEQHHhu2Q&t=746s"
      },
      {
        "text": "we we've generally found that in the range of 10,000 to 10 million samples of synthetically generated um data will be sufficient to fine-tune a smaller model to a really really high degree of reliability.",
        "at": "16:52",
        "url": "https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1012s"
      }
    ],
    "words": 4506
  },
  "summary_url": "/#hacEQHHhu2Q",
  "transcript": {
    "html": "/transcripts/hacEQHHhu2Q.html",
    "txt": "/transcripts/hacEQHHhu2Q.txt",
    "vtt": "/transcripts/hacEQHHhu2Q.vtt"
  }
}