{
  "video": {
    "id": "418t26CVz-w",
    "title": "Local Agentic Theory For Mobile Games — Shafik Quoraishee & Joanne Song, The New York Times",
    "duration": 1084,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "418t26CVz-w",
    "title": "Local Agentic Theory For Mobile Games — Shafik Quoraishee & Joanne Song, The New York Times",
    "one_liner": "Two New York Times Games engineers argue that game AI should move from cloud inference and reward-trained RL to on-device agentic loops, and demo agents that play Space Invaders and solve the Mini Crossword — then show how the same local agent can retune a game's accessibility dials live for the player in front of it.",
    "summary": "Shafik Quoraishee traces game AI from Pac-Man's finite state machines through AlphaGo and EfficientZero to agentic systems, drawing the key distinction that RL changes the model's weights while an agentic system reasons over the game's state space through in-context learning and tool calls with no reward to grind out. He shows two experimental NYT agents — an agentic Space Invaders player running a perceive/predict/decide/act loop on an on-device model, and a constraint-satisfaction agent that backtracks through a satisfaction graph to fill the Mini Crossword — and lays out the space/time/energy constraint graph that governs anything running inside a 16 ms frame on a phone. Joanne Song then reframes the payoff as accessibility: grounded in WCAG 2.2's four pillars and WCAG 3.0's draft bronze/silver/gold scoring, she argues static easy modes and toggles are deterministic and blind to real-time need, and that a local agent should continuously tune dials like input tolerance and step granularity. The talk is explicit that this is experimental work — NYT puzzles are made by people and the shipped games contain no AI features.",
    "key_points": [
      "Heavy up-front disclaimer: NYT puzzles are made by people, the games are 'pretty much AI free', Wordle Bot is not an AI feature, and everything shown is experimental work about solvability and playability — not game generation.",
      "The case for on-device: no cloud round trip (so no inference cost or latency), computation stays inside the device's security zone and isn't needed upstream for telemetry, it works in a subway tunnel with no HTTP calls getting through, and it enables real personalization.",
      "RL vs agentic, stated as the core distinction: reinforcement learning runs many iterations to change the model's weights and tunes it to one game; an agentic system uses language models doing in-context learning to reason over the game state, with 'no reward system to grind out', learning the system through tool calls and adapting dynamically.",
      "History and state of the art named: Pac-Man's ghosts as a finite state machine (conditional symbolic AI) from the '80s; AlphaGo/AlphaZero; EfficientZero and EfficientZero V2 as the sample-efficient RL state of the art up to 2024, learning fast on far less data than traditional RL.",
      "Demo 1 — an agentic Space Invaders built on a local agentic framework, with the agent's states visible on screen, running a simple loop: perceive the scene, predict the spaceships' next move, decide (is the bullet or ship too close), act, loop.",
      "Demo 2 — a Mini Crossword solver described as a constraint satisfaction agent using a satisfaction graph; it sometimes places the wrong word, and the graph is optimized so that when constraints get too crowded or incorrect it backtracks efficiently, with the agent using natural language from the graph to backtrack.",
      "Three device constraints form a constraint graph with different penalty weights: space (model weights, compressed agent state history, compressed planning artifacts, plus render headroom), time (60 Hz means a new frame every 16 ms and agent planning must fit inside it or you get jank), and energy (phones aren't optimized for agentic workloads yet — NPUs and AI chips are coming). Time is penalized hard because the agent must not disrupt the user experience; space gets soft constraining.",
      "Model intelligence is the ceiling: ARC-AGI scores matter because current models aren't yet capable of the complex decisions the most interesting future games would need. Also on the radar: generalist on-device models (Google Gemma, transcribed 'Seema'), phone-based visual estimation agents, and fully generative worlds.",
      "Accessibility reframed as dials, not toggles: traditional games are rigid hand-authored fixed-state models, deterministic and blind to real-time need; WCAG 3.0's draft move from binary pass/fail to graded bronze/silver/gold is the same shift, and the agent's whole job is to dynamically calculate dials like input tolerance (forgiveness around physical constraints) and step granularity (breaking complex tasks into more operations) against the player's live context.",
      "Concrete accessibility interventions shown: vision models tracking eye gaze to detect search friction, analyzing shaky taps, handwriting recognition; an agent that monitors focus path in a broken dialogue, flags a keyboard/switch focus trap loop and injects an exit route live; and a 'live layout auditor' that measures the interface on the fly, catches touch-target violations and resizes controls dynamically.",
      "Roadmap of what's still missing: plans and decisions inside 16 ms, predictive models of the game so an agent can see what a layout change does before making it, long-term memory of one specific person's habits, a shared game state language so one agent works across games instead of being rebuilt per release, and faster chips paired with honest benchmarks proving the agent actually helps."
    ],
    "takeaways": [
      "Budget an on-device agent against all three constraints explicitly — space (weights + compressed state history + planning artifacts + render headroom), time (16 ms at 60 Hz), energy — and weight the penalties asymmetrically: soft-constrain space, punish time overruns hard, because jank is what the player actually feels.",
      "Reach for an agentic loop rather than RL when the game state changes in ways you can't pre-train for: you skip reward engineering and per-game weight tuning, and get in-context adaptation through tool calls instead.",
      "Stop shipping accessibility as a static 'easy mode' or a settings toggle — model it as continuous dials (input tolerance, step granularity) that an agent recalculates from live signals, since a fixed mode can't fix a grid that remains blind to the player.",
      "Use on-device sensing as accessibility input, not just telemetry: gaze for search friction, tap jitter for motor difficulty, handwriting as an alternative input — and have the agent act on it (resize targets, break a focus trap) rather than just observe.",
      "Ground the work in WCAG 2.2's four pillars now and design for WCAG 3.0's graded bronze/silver/gold direction, and pair any agent with real benchmarks that prove it made things better."
    ],
    "topics": [
      "agents",
      "on-device-ai",
      "games",
      "accessibility",
      "reinforcement-learning",
      "mobile",
      "constraints",
      "wcag"
    ],
    "tools": [
      "The New York Times Games",
      "Wordle Bot",
      "NYT Connections",
      "NYT Mini Crossword",
      "Pac-Man",
      "Space Invaders",
      "AlphaGo",
      "AlphaZero",
      "EfficientZero",
      "EfficientZero V2",
      "Google Gemma (transcribed 'Seema')",
      "ARC-AGI",
      "WCAG 2.2",
      "WCAG 3.0",
      "NPUs (neural processing units)"
    ],
    "quotes": [
      {
        "text": "Our puzzles are made by people. They're not made by AI.",
        "at": "00:38",
        "url": "https://www.youtube.com/watch?v=418t26CVz-w&t=38s"
      },
      {
        "text": "So, RL changes the model itself and changes the weights in the model. Um the agentic system reasons over a space in the game, um and there's no reward system to grind out.",
        "at": "06:40",
        "url": "https://www.youtube.com/watch?v=418t26CVz-w&t=400s"
      },
      {
        "text": "A static easy mode can't fix a crossword grid that remains fundamentally blind to our players.",
        "at": "13:45",
        "url": "https://www.youtube.com/watch?v=418t26CVz-w&t=825s"
      },
      {
        "text": "And the future of AI doesn't have to be one giant centralized brain. It can be billions of small local brains, each running on a personal device, each shaped entirely by the individual it serves.",
        "at": "17:28",
        "url": "https://www.youtube.com/watch?v=418t26CVz-w&t=1048s"
      }
    ],
    "words": 3635
  },
  "summary_url": "/#418t26CVz-w",
  "transcript": {
    "html": "/transcripts/418t26CVz-w.html",
    "txt": "/transcripts/418t26CVz-w.txt",
    "vtt": "/transcripts/418t26CVz-w.vtt"
  }
}