{
  "video": {
    "id": "XWcXwnysmpY",
    "title": "The Spatial Harness: Bringing Agents to the Canvas — Max Drake, tldraw",
    "duration": 1132,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer"
  },
  "analysis": {
    "video_id": "XWcXwnysmpY",
    "title": "The Spatial Harness: Bringing Agents to the Canvas — Max Drake, tldraw",
    "one_liner": "tldraw's Max Drake shows why LLMs are terrible at 2D space and walks through the escalating harnesses tldraw built to fix that — a single-shot canvas-teaching prompt, an MIT-licensed agent starter kit, multiplayer \"fairies\" that coordinate as visible characters, and a desktop app that lets Claude Code write plain JavaScript against the live editor.",
    "summary": "The talk argues that coding agents work well because text-in/text-out is the medium they were trained in, while 2D space is something they are genuinely bad at and that takes real engineering to teach. tldraw's answer is a stack of increasingly capable canvas harnesses: teaching an LLM to read a canvas from screenshot + JSON and predict how its actions land, wrapping that in an agentic loop that sets its own to-dos and moves its own viewport, then making agents multiplayer characters ('fairies') whose animated state replaces reading a chat log. The final move is to stop trapping agents inside the canvas — the tldraw desktop app exposes its editor instance over a server so an outside agent like Claude Code can script it in 'code mode', producing things like a canvas window manager and Pong played with real desktop windows. The closing argument is that the canvas should be a *place* where humans and agents collaborate, the same way it already is for remote human collaboration.",
    "key_points": [
      "Coding agents work because the medium they operate in — writing code, text in and text out — is the medium they were trained in; ask them to align UI in 2D space and they fail, because 'agents are really really bad at working in 2D space' and getting them to do it takes a lot of engineering work.",
      "tldraw is three things: the free infinite-canvas whiteboard app, the London company, and — most importantly per the speaker — the infinite canvas SDK that powers it. It exists because people with a killer canvas app idea got stuck on selection, resizing and matrix math and never built their actual app. Replit's new agent canvas is built on top of it.",
      "The 'teach' project was a single-shot prompt that taught the LLM to interpret the canvas from a screenshot plus JSON data, and to understand how the actions it emits will affect the canvas. Live demo: 'make the mouse blow out the candle' produced correctly positioned wind and smoke out of ordinary canvas shapes — no special mouse shape.",
      "The tldraw agent starter kit (MIT licensed, on the website) wraps that single-shot capability in an agentic harness: the agent sets its own to-dos, and changes its own viewport to go find things elsewhere on the canvas — the spatial equivalent of a coding agent searching a codebase for a definition.",
      "'Fairies' makes agents multiplayer characters you can grab, throw, recolor and re-hat (including a leg slider). The customization is not a joke: with 10 agents running you need to tell which is which, and their animated state means 'I don't have to read a chat in order to know what's actually going on.' Selecting several fairies gives you a group chat; one becomes an orchestrator that writes a plan, assigns a task, waits, and gets prompted to review when it completes.",
      "Near a launch, tldraw abandons its task-tracking software and builds one massive dependency graph on plain tldraw.com — the fairies-launch graph was shown as a real artifact. The hackathon 'tech tree app' turns that graph into a working tool: each node is a coding agent you can kick off, PRs get opened and merged from it, you can sketch a prompt on the canvas and wrap it into a named task ('facial animation canvas control'), assign it to Claude and hit run. Positioned as similar to Conductor or OpenAI Symfony but multiplayer, so a colleague can join and add tasks.",
      "The limitation of fairies: they're trapped in the canvas, and building like that means your entire harness has to be a canvas harness. The tldraw desktop app fixes this by exposing the running editor instance over a server so any agent — the speaker's Claude Code — can write plain JavaScript against it. 'It's code mode... you can turn your tldraw desktop app into a scripting environment.'",
      "Because Claude Code has access to the actual computer, not just the canvas, a colleague used the desktop app as a window manager (rectangles on the canvas driving real windows, likely via AppleScript) and built Pong played with desktop windows — 'ephemeral UIs' that act on the real world.",
      "The opening live demo — ask Claude Code to find the Notion doc a colleague emailed and build it in the desktop app — did not finish in the talk's 18 minutes. It had correctly pulled the Gmail, opened the Notion doc and found the spec, but was still working on the fluid simulation after 13 minutes."
    ],
    "takeaways": [
      "Don't assume an LLM can see your canvas. Feed it both a screenshot and the structured data, and explicitly teach it how its output actions will change what's there — that translation layer is the actual work, not a prompt detail.",
      "Give a canvas agent viewport control as a first-class tool. Letting it pan and zoom to find things is the spatial analogue of codebase search, and is what turns a single-shot prompt into an agent.",
      "When running many agents, encode their state visually instead of in a chat log, and make them individually distinguishable — at 10 agents, reading transcripts to know what's happening stops scaling.",
      "Don't force the whole harness to live inside the canvas. Expose your app's editor instance over a server so an external agent (e.g. Claude Code) can drive it with plain JavaScript — that keeps the spatial primitives while letting the agent touch real data and the real machine.",
      "Start from the multiplayer artifact you already make by hand. tldraw's pre-launch dependency graph became the tech tree app; an existing shared diagram is a better spec for an agent interface than a new abstraction.",
      "Grab the MIT-licensed tldraw agent starter kit rather than rebuilding canvas-agent plumbing from scratch."
    ],
    "topics": [
      "agents",
      "infinite canvas",
      "multi-agent",
      "multimodal",
      "agent ux",
      "multiplayer",
      "coding agents",
      "tool design"
    ],
    "tools": [
      "tldraw",
      "tldraw SDK",
      "tldraw desktop app",
      "tldraw agent starter kit",
      "Fairies",
      "tech tree app",
      "Claude Code",
      "Claude",
      "Replit",
      "Miro",
      "Conductor",
      "OpenAI Symfony",
      "Notion",
      "Gmail",
      "AppleScript",
      "ChatGPT"
    ],
    "quotes": [
      {
        "text": "part of the reason why these apps are so good and why they work is because they're, you know, the medium in which they're working, writing code is essentially the medium in which they were trained.",
        "at": "04:24",
        "url": "https://www.youtube.com/watch?v=XWcXwnysmpY&t=264s"
      },
      {
        "text": "it turns out agents are really really bad at working in 2D space and understanding 2D space and actually requires like a lot of engineering work to get them to uh do it.",
        "at": "04:52",
        "url": "https://www.youtube.com/watch?v=XWcXwnysmpY&t=292s"
      },
      {
        "text": "I don't have to read a chat in order to know what's actually going on. I can look at the state of the agents.",
        "at": "10:40",
        "url": "https://www.youtube.com/watch?v=XWcXwnysmpY&t=640s"
      },
      {
        "text": "The fairies are trapped in the canvas.",
        "at": "12:07",
        "url": "https://www.youtube.com/watch?v=XWcXwnysmpY&t=727s"
      }
    ],
    "words": 4460
  },
  "summary_url": "/#XWcXwnysmpY",
  "transcript": {
    "html": "/transcripts/XWcXwnysmpY.html",
    "txt": "/transcripts/XWcXwnysmpY.txt",
    "vtt": "/transcripts/XWcXwnysmpY.vtt"
  }
}