{
  "video": {
    "id": "ZZcE0HeO-Hc",
    "title": "From VLM/VLA's to Embodied Agents — Armen Aghajanyan, Perceptron AI",
    "duration": 1242,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "ZZcE0HeO-Hc",
    "title": "From VLM/VLA's to Embodied Agents — Armen Aghajanyan, Perceptron AI",
    "one_liner": "Perceptron argues for dropping the split between VLMs, VLAs and world models in favour of a single \"embodied foundation model\" that handles perception, embodied reasoning and control. They say jointly training on video lets you use 10x less expensive teleop data if you have 10x more video pre-training data.",
    "summary": "Armen Aghajanyan (ex-FAIR for six years, now CEO of Perceptron) argues that VLMs, ER models, VLAs, world models and semantic world models should be unified into one early-fusion \"embodied foundation model\" that perceives, reasons and acts. He names two core research problems: sparse supervision (only about 2% of the roughly 1M tokens in an hour of video get a loss) and context bloat from always-on sensors. Perceptron's answers are an automatic \"natural perceptive objective\", which is not disclosed, and a published data-sparse mixture-of-experts router that learns which tokens to skip. He then shows their model, which he says is frontier with Gemini 3.1 Pro, beats Gemini ER and is about 15x cheaper, doing agentic detection, video annotation and robot control, plus a new scaling law that trades teleop data for video data.",
    "key_points": [
      "Perceptron wants to replace the VLM / ER model / VLA / world model / semantic world model split with one \"embodied foundation model\" that does perception, embodied reasoning and control in a single model, building on the early-fusion multimodal work Armen did at FAIR.",
      "Sparsity problem: an hour of video can be about 1M visual tokens, but transcript prediction or synthetic labels put a loss on only about 2% of them. Every fix injects a wrong signal: too sparse, too synthetic, or too indiscriminate. Predicting every pixel treats background pixels the same as a gripper tip or contact points.",
      "Perceptron's undisclosed core IP is a \"natural perceptive objective\" that automatically learns which percepts to predict into the future. Others, such as the Molmo folks at AI2 and other VLAs, hard-code percepts like the gripper tip.",
      "Context bloat: the speaker's first principle is that you have to treat text, image, audio and video tokens as completely different. Naive patch averaging gives up to 10x compression but is 'a hack'. Their published data-sparse mixture-of-experts paper uses a router to decide, at every layer, which tokens to process and which to skip. Visualisations show it learns on its own to focus on dense or task-relevant regions, such as the graph in a figure or the fruit when asked to segment fruit.",
      "The released model, which Perceptron calls the first embodied foundation model, is described as frontier with Gemini 3.1 Pro, better than Gemini embodied reasoning (ER) and about 15x cheaper. It was trained on a 1-petabyte dataset of text, images, video and trajectories, including desktop-use and video-game trajectories.",
      "Emergent behaviour: classic CV detection becomes an agentic task. The model writes code, zooms, tiles the image and raises contrast to find a well-hidden bird. It also does robotic video annotation by jumping around the video, clipping it and checking its own captions, for 'cents' where Gemini would cost 'a couple of dollars'.",
      "New scaling law: pure VLAs still scale with more teleop data (about $100 per hour), but unified embodied foundation models can use 10x less teleop data if they have 10x more video pre-training data. This has held up to the compute Perceptron has.",
      "Demo: a single model does embodied reasoning and outputs control tokens, for example reading a book's title and sorting it into the right bin. It is a bit jittery but works reasonably well zero-shot. They plan to open-source a smaller model 'sometime in July', and select partners can get the Mark1 and larger model weights."
    ],
    "takeaways": [
      "Don't use a VLA alone for multi-step tasks that need perception. Jointly training perception, embodied reasoning and control generalises better and is more robust to changes like a different table background or lighting, which break fine-tuned Chinese VLAs.",
      "Use learned token routing (data-sparse MoE) rather than hand-coded compression like patch averaging to cope with sparse video and long always-on context. Ideas like key frames versus delta frames are an earlier step Perceptron has since moved past.",
      "Plan data mixes early for spatial skills: internet crawls rarely label cardinalities such as left of or below, so embodied-reasoning data has to be added on purpose. The speaker credits this for beating Gemini ER with less compute.",
      "Weigh cheap video pre-training data against expensive teleop data (about $100/hr); the reported trade-off is 10x more video for 10x less teleop.",
      "Add online augmentations during training, such as a faked arm camera being off or simulated sunlight from one direction, for robustness. The speaker says the bigger gains come from bringing early fusion into robotics."
    ],
    "topics": [
      "embodied ai",
      "robotics",
      "multimodal models",
      "vla",
      "video understanding",
      "mixture of experts",
      "scaling laws",
      "computer vision"
    ],
    "tools": [
      "Perceptron",
      "Perceptron Mark1",
      "Gemini 3.1 Pro",
      "Gemini ER (embodied reasoning)",
      "Google DeepMind",
      "FAIR (Meta)",
      "AI2 Molmo",
      "Twitter"
    ],
    "quotes": [
      {
        "text": "you have 1 million tokens going in and you're essentially calculating the loss on something like you know 2% of all the tokens that are going in",
        "at": "04:15",
        "url": "https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=255s"
      },
      {
        "text": "you can actually start thinking of doing classical CV tasks as being an agentic task",
        "at": "09:50",
        "url": "https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=590s"
      },
      {
        "text": "you can essentially trade 10x uh less teleop data if you have 10x more video pre-training data",
        "at": "14:00",
        "url": "https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=840s"
      },
      {
        "text": "if you change the the background of the table, the policy will actually fail",
        "at": "18:45",
        "url": "https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1125s"
      }
    ],
    "words": 4412
  },
  "summary_url": "/#ZZcE0HeO-Hc",
  "transcript": {
    "html": "/transcripts/ZZcE0HeO-Hc.html",
    "txt": "/transcripts/ZZcE0HeO-Hc.txt",
    "vtt": "/transcripts/ZZcE0HeO-Hc.vtt"
  }
}