{
  "video": {
    "id": "A_I8mw8yfns",
    "title": "The Best Models Still Reason Like Toddlers — Andrew Dai, Elorian",
    "duration": 1097,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "A_I8mw8yfns",
    "title": "The Best Models Still Reason Like Toddlers — Andrew Dai, Elorian",
    "one_liner": "Frontier models like Claude, ChatGPT and Gemini do visual pattern-matching, not visual reasoning. They hallucinate a full 32-square chessboard from a partial one and lose track of what a robot did in a long video, so keep production vision tasks simple or expect a lot of hallucinations.",
    "summary": "Andrew Dai, co-founder and CEO of Elorian (transcribed as 'Elorium'), argues that current frontier models are far from AGI on visual reasoning. They pattern-match well enough to identify plants or games, but they fail detailed spatial, counting and long-video questions. He says popular evals like ARC-AGI and MMMU don't measure this, and that the 'missing paradigm in visual AI is thinking'. His company's answer is visual reasoning models trained on proprietary data, a synthetic data flywheel, changes to the architecture and native visual chain of thought. They target robotics, construction safety, and architecture/mechanical design.",
    "key_points": [
      "Failure examples: shown a partial chessboard, models say it has 32 white squares because they hallucinate a complete board. Asked how many rows the blue player has in a Catan photo, a model reasoned from 10 pieces off the board that there must be 5 on it, when there are actually 7. He bets no frontier model can report where pieces sit on a chessboard set up at random.",
      "'Context amnesia' in video: in a robot-arm clip, the latest models miss that the arm lifted the lid and that it is turning on the stove, because they can't stay consistent across long videos.",
      "His test for understanding versus reasoning: if a human can answer after looking for about one second ('what game is this?', 'what flower is this?'), it is pattern recognition and models get it right. If it needs a detailed, System 2 look (Daniel Kahneman), frontier models fall apart.",
      "Eval critique: ARC-AGI images are only 32x32 or 64x64 pixels, so an 85-90% score says little about real-world visual tasks. In MMMU the images are a minor part, and many questions can be answered without looking at the image. The industry needs new benchmarks for geometric alignment, spatial intelligence and object permanence.",
      "The current landscape: generation models like ByteDance's Seedance look great but lack physical grounding and causal logic. Disasters come out cartoonish or Hollywood-style because they copy training data from films and game engines. Understanding tools (Google Lens, SAM 3, YOLO, Mask R-CNN) are robust but 'fundamentally passive', with no reasoning.",
      "Elorian's four-part approach: (1) collect and generate its own visual-reasoning data, which it says isn't available online; (2) a synthetic data flywheel using evals, agents, SFT and RL; (3) improvements to the transformer architecture; (4) native visual chain of thought. Example: for 'how many red hotels?' the model draws boxes around the hotels, then narrows them down to the red ones.",
      "Target use cases: robotics, with a model API planned by the end of the year that feeds action-relevant scene understanding into planning and control systems. Construction safety: matching OSHA policy language to camera feeds, such as helmet counts or whether work follows the plans, instead of retraining a brittle model for each rule. Architecture and mechanical design: one mechanical engineering company said a small part of a robot testing platform takes 100-200 hours to design and the whole platform 2,000-3,000 hours, and frontier models fail on blueprints and 3D CAD/CAM files.",
      "Team: about 20 people. Dai spent 12 years at Google Brain/DeepMind, was first author of the pre-training + fine-tuning paper that GPT papers cite, co-led GLaM, co-led PaLM 2 pre-training architecture and co-led the Gemini data area. His co-founder led research for Apple's MM1. Chief reasoning architect Dustin Tran was previously lead of post-training at xAI."
    ],
    "takeaways": [
      "When designing systems on frontier models, keep the visual tasks very simple, the kind a human could answer in about a second. Detailed counting, spatial or positional questions will produce a lot of hallucinations.",
      "Don't read high ARC-AGI or MMMU scores as evidence of real-world visual reasoning. Check what the benchmark data actually looks like (resolution, whether the image is even needed).",
      "Test your own visual use cases against frontier models with quick adversarial examples: partial boards, piece positions, counts in cluttered scenes, long videos with state changes. He says finding failures 'just takes a few minutes'.",
      "For design and engineering work, pair model output with programmatic or simulation validation, the way code is run against unit tests (e.g. mechanical simulators from Siemens).",
      "Look at visual-heavy industries (construction, mechanical design, CAD/CAM quality control) as an overlooked opportunity. AI uptake so far has concentrated on coding, maths and documents."
    ],
    "topics": [
      "visual reasoning",
      "multimodal",
      "evals",
      "hallucination",
      "robotics",
      "synthetic data",
      "chain of thought",
      "mechanical design"
    ],
    "tools": [
      "Elorian",
      "Claude",
      "ChatGPT",
      "Gemini",
      "ARC-AGI",
      "MMMU",
      "MMLU",
      "ByteDance Seedance",
      "Google Lens",
      "SAM 3",
      "YOLO",
      "Mask R-CNN",
      "GLaM",
      "PaLM 2",
      "GPT",
      "Apple MM1",
      "Google Brain",
      "DeepMind",
      "xAI",
      "Siemens",
      "OSHA"
    ],
    "quotes": [
      {
        "text": "So the pattern matching is actively hurting them in this case.",
        "at": "01:58",
        "url": "https://www.youtube.com/watch?v=A_I8mw8yfns&t=118s"
      },
      {
        "text": "I would challenge anyone uh to give me like a real world complex task that can be reduced to a 32x 32 pixel problem.",
        "at": "06:22",
        "url": "https://www.youtube.com/watch?v=A_I8mw8yfns&t=382s"
      },
      {
        "text": "And so the missing paradigm in visual AI is thinking.",
        "at": "08:03",
        "url": "https://www.youtube.com/watch?v=A_I8mw8yfns&t=483s"
      },
      {
        "text": "people are not coding up the next iPhone or coding up the next uh SpaceX rocket. It's all fundamentally very visual.",
        "at": "17:26",
        "url": "https://www.youtube.com/watch?v=A_I8mw8yfns&t=1046s"
      }
    ],
    "words": 3143
  },
  "summary_url": "/#A_I8mw8yfns",
  "transcript": {
    "html": "/transcripts/A_I8mw8yfns.html",
    "txt": "/transcripts/A_I8mw8yfns.txt",
    "vtt": "/transcripts/A_I8mw8yfns.vtt"
  }
}