The Spatial Harness: Bringing Agents to the Canvas — Max Drake, tldraw

AI Engineer · 18 min · 252 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:13Thank you for coming here to my talk to watch me talk about uh agents on the canvas.
00:19Um the first thing I'm going to do though is uh before I have to record my screen.
00:24Uh the first thing I'm going to do is I'm going to ask my agent to do something on the canvas.
00:30And what I'm going to do is say, "Hey, uh, my colleague Spencer just emailed me a link to a notion document, uh, for a really cool demo we could build with the TL desktop app.
00:40Can you like find that document and then can you build it on the desktop app?"
00:45Thank you.
00:50Okay, so that's going to build uh, and then we're going to come back to it later and hopefully it'll work.
00:54Um, hi everyone, my name is Max Reich.
00:56Thanks so much for coming.
00:57Um I work on uh agents on the canvas at uh TLA.
01:02I'm a product engineer there.
01:04Um so first things first, am I qualified to be giving this talk?
01:07I like to think so.
01:08I've been doing like agents on the canvas stuff since before chat GBT came out.
01:12Um I think it's really cool.
01:14I think there's like so much UX stuff you can do with when you get LM, you have them working in space.
01:19Uh and I think it's really interesting.
01:21I've been doing it for about as long as you can have been doing it.
01:23Um more recently I've been talking about this a lot.
01:27Um, here's some proof.
01:28Um, and yeah, so I work at this company called Teal Draw.
01:32Um, can I get a quick show of hands?
01:33Has anybody ever heard of or used Teal Draw before?
01:36Yeah.
01:37Okay.
01:38Um, awesome.
01:38So, yeah, the thing that you've probably used if you use Teal Draw is, um, this app right here.
01:44So, this is all Teal Draw.
01:46Uh, this is a free uh, infinite canvas whiteboarding app.
01:49Um, you know, we have selections and arrows and resizing and, you know, all the things that you need in a in a whiteboard.
01:56Um, Traw is also the company uh that makes this app.
02:00It's based in London.
02:01It's where I work.
02:02Um, but the last thing that TLAR is, which is, I think, in my opinion, the most important, is it's the Infinite Canvas SDKs that that powers this app.
02:10Um, and so what that means is that, you know, this is kind of the teal draw.
02:15The SDK is the engine that powers a lot of Infinite canvas experiences.
02:18Um because it turns out it's really hard to get that kind of stuff right.
02:22Um and uh so if you ever want to build a Muro competitor or a slide designer or if you're like Replet um Replet has their whole new uh agent canvas stuff built on top of TLR.
02:34Um, and so the reason we built TL Draw in the first place was that we were running into this issue or people were running into this issue where they had this idea for this like really great killer canvas app.
02:44Uh, and they went to go build it and everybody would run into the same problem where they would run into they would have trouble making the actual canvas part of of the app
02:53and they would, you know, trying to deal with resizing and selection and, you know, all the matrix math.
02:58And the issue is that they wouldn't be able to build their actual app itself.
03:01They would get stuck on the canvas.
03:02And so we built Teal Draw to kind of be the engine um that could power that could be the canvas so that they could focus on the actual app.
03:10Uh when LLM came out, we like a lot of other people saw that this is going to be this weird new type of software.
03:16I don't know if anyone, you know, I'm sure a lot of you were building in 2022 and it was really exciting.
03:20Um and a lot of people it was the exact same thing.
03:23People had the idea had an idea for this cool app that would, you know, involve LM on the canvas having them manipulating things in space.
03:31but then they would try to build it and they'd get stuck.
03:33Um there were no best practices.
03:34People didn't really know how to do it.
03:36And so at TL Draw we uh realized that we need to make it easy for people to build uh with agents with LLMs on the canvas.
03:44Um and also so Teal Draw the SDK as well as the app has multiplayer built in with like live sync.
03:49It's really nice.
03:50There's cursors.
03:51There's you can see your collaborators cursors and selections and viewports.
03:54And I think all of the things that make just the canvas in general a really great uh place for interacting with and collaborating with your colleagues also make it a really great place for interacting
04:05uh and collaborating with agents.
04:06Uh and I hope I'm going to be able to show you guys some of that um in the demos that come up.
04:11Um so before we talk about agents on the canvas, uh really quickly I want to talk about agents not on the canvas.
04:17Um I'm sure you guys have all used an app that looks like this, you know, claude code.
04:21And I'm going to really oversimplify here, but basically, uh, part of the reason why these apps are so good and why they work is because they're, you know, the medium in which they're working,
04:30writing code is essentially the medium in which they were trained.
04:32You know, it's it's text in, text out.
04:34That's how they were trained.
04:35Um, and when we work with them, we give them a prompt and they write code.
04:39It's text in, text out.
04:40Uh, you know, again, oversimplifying, but that's essentially how they work.
04:44I don't know if you guys have ever, you know, tried to get your agents to do like UI stuff and try to get them align to align something.
04:50found that they could not do that whatsoever.
04:52Um because it turns out agents are really really bad at working in 2D space and understanding 2D space and actually requires like a lot of engineering work to get them to
05:02uh do it.
05:02And that's kind of the project that we've been embarking on at Teal Draw recently.
05:06Um and so the first thing we had to do, this is an older project, but the first thing we had to do is get them to teach them teach the agents
05:13or at this point not agents LLM to understand the canvas and understand kind of what they're even looking at.
05:19Um, so we had this project called teach um where um we tal So I'm going to I'm going to prompt this really quick.
05:26I'm going to say, "Hey, make the mouse blow out the candle."
05:31Yeah.
05:32So that's going to take a second.
05:33This is this is an older uh older project, but basically what we had to do is we had to kind of like teach the LMS how to take the like the screenshot that we give it and the JSON and all of the other information about the canvas
05:45and Oh yeah, there we Okay.
05:48So yeah, that's some that's some wind.
05:50Uh it's is it sometimes it gives us smoke as well.
05:53Yeah, and we got a little smoke as well.
05:55So So we basically had to take it how to like and I want to be very clear this is not um this is not like a special mouse shape.
06:02These are just like uh you know the these are just shapes uh on the canvas.
06:07This is um and so the work behind this it's a singleshot prompt but we basically we tell the agent how to interpret uh both via screenshots and via the data
06:16um what is actually on the canvas like what it's looking at um which is actually you know it's not a trivial problem and then also how to we teach it how to actually act on the canvas and to understand how the actions that it produces will affect the canvas.
06:29So, you know, it got it, you know, it made the it made the smoke, it made the it made the wind, it got the positions right, and it understood what it was doing.
06:37Um, so we we got this we kind of figured out how the um like kind of we got we taught it what the canvas is.
06:44Uh, but this was like a single shot, single prompt kind of thing.
06:47Um, and so the next thing we built is the uh teal draw agent starter kit.
06:51Uh, which basically turns that and wraps in a harness that lets an agent work agentically on the canvas.
06:56Um, the code is also MIT licensed.
06:59You can find it on um that you can find on the website.
07:01So, here's a little here's a little cat.
07:03I'm going to make this a little bigger.
07:04Um but what I'm going to say is, "Hey, so somewhere else on the canvas uh there are uh some friends for the cat.
07:12Can you please uh bring one of them over to the cat?
07:15Um her favorite color is red."
07:18And so I'm going to zoom out.
07:20I'm going to show you guys what's actually going on.
07:21So you can see the the view of the agent.
07:23There's some there's some potential friends over here.
07:25And um you know if you read the All right.
07:29So and basically what's going on is that the agent has kind of like we've given it a prompt and uh using the information it has about the canvas.
07:37It's uh going to kind of like make some goals for itself.
07:40You can see there's some to-dos in the corner here.
07:42It's it changed its view in order to see um what was you know the other stuff that was on the canvas.
07:49same way that if you ask a coding agent um you know you ask it you know where where do we define this thing in the codebase it can go and it can search it can find it.
07:58So this is kind of like turning that singleshot uh prompting experience into this kind of like agentic thing that you can uh have you know it can autonomously set goals
08:09and and work towards them.
08:11The uh next thing we did um we did this this uh project called fairies and so we basically we had this agent experience but we realized that you know teal draw and you know the canvas in general is so collaborative it's so multiplayer and we wanted to basically we wanted people to be able to work together with
08:28their agents and we also wanted the agents to be able to work together so this is this is a ferry um there's also you guys want to scan this QR code you can actually this is multiplayer you can join if you want you it requires a Gmail signup But you don't need to pay for tokens.
08:43Um, this is this is what the link is.
08:44So, basically, this is a this is a fairy.
08:47Um, this fair's name is is Joan.
08:49They don't like being they don't like being grabbed.
08:51Um, you can you can throw them around.
08:53You know, we added a lot of really important stuff.
08:55You can um you can you can change its hat.
08:58Um, you can change the the color and and this seems silly, but it's actually really important.
09:03uh and I'll talk about this a little bit more later, but actually understanding uh when you get a high level view of when you see your agents working on the canvas, it's important to know which one is which.
09:11And so differentiating them is actually important, which is why, of course, we added the leg slider.
09:15Um but so, you know, I can say like, you know, I can I can I can say hey to it.
09:22Um and I can say, you know, something like draw a cat.
09:28Uh and I can have it work.
09:29But the most important thing here is that fairies have friends, right?
09:32and they can here we go.
09:34Uh
09:36and they can fairies can work together.
09:37And so we kind of designed this like multi- aent collaboration system that works on the canvas.
09:42Um and I'm going to actually I'm going to go to I think one of my Yeah, I think so.
09:49My my colleagues agents are here working uh making this this really great scene.
09:53Um I'm going to bring mine over um summon and I'm going to give them a slightly different prompt.
10:01So, I'm gonna select them all.
10:02And now I have a group chat of of the agents, right?
10:05And I'm going to say, "Hey, I have a board meeting coming up in like 10 minutes, and I don't have any of my my figures.
10:10Can you draw up like a little memo for all of my uh financial data for fiscal year 2025?
10:16Thank you."
10:18Okay.
10:19So, what what's going to happen there basically is this kind of like creates this uh multi- aent, you know, coordination thing.
10:25We have uh one of the fairies is writing uh writing out a plan.
10:30You can see it.
10:30Um, and again, the animations are kind of cute and funny, but it's actually really important.
10:35I don't have to read a chat or go through, you know, imagine if I have 10 agents working.
10:39I don't have to read a chat in order to know what's actually going on.
10:42I can look at the state of the agents.
10:44Uh, and I can actually, you know, I can can see what's happening.
10:47So, we we have a task here that's been defined.
10:52Um, it seems like, you know, the the ferry is is uh waiting for that to to finish.
11:00Yeah.
11:01So, that one's one's board.
11:02That one's waiting.
11:03So, this is the orchestrator ferry.
11:04What what it's done is it's assigned the it's assigned the task.
11:07Uh, and now it's waiting for the other ones to start and finish it.
11:10Um, and it's going to get notified.
11:12It's going to get prompted in order in order to review.
11:14Um, it seems like forever reason's not working, but thank Oh, never mind.
11:20So, yeah, we have one we have this one.
11:23So yeah, we have one fairy who made the made the task, one fairy who's working on it.
11:26And so this is this kind of um you know, multi- aent coordination system on the canvas.
11:31I have, you know, you can see my colleague has his agents over here.
11:34Uh they're they're working as well.
11:36Um and so you can kind of collaborate with people and with agents in this uh environment.
11:41And I don't know, I think that's really cool.
11:44the so the the next thing.
11:48So the problem with fairies is that um they're kind of trapped in the canvas and all the stuff you've seen before.
11:54This requires if you want to build something like this, this requires like the you to like opt in and have your entire harness be a like canvas harness.
12:02Um and the downside of that is that it makes it really hard to have any of this work with stuff like outside in the real world.
12:07The fairies are the fairies are trapped in the canvas.
12:10Um, and so I I I built this experiment.
12:15We had a little hackathon uh internally.
12:17Um, but first as a quick motivation for that, um, at Teal Draw whenever we have an whenever we're getting closer to a launch, we like abandon all of our task tracking software and we make just one massive
12:28dependency graph of how like so this is what an actual this is a real thing from when we launched fairies actually.
12:34Um, and so this is what it looks like when we're like really like when is hitting the fan at at tealraw when we're launching something.
12:41Um, and I really like this interface because it it kind of lets you This is not like a special app.
12:47This is still just tealraw.com.
12:48You can, you know, move your shapes around and things like that.
12:51Um, but I really like this because it both it lets you see like what depends on what.
12:56It lets you know what's coming next.
12:57It lets you get a highle overview.
12:59You know, these are all green because we finished them, but you know, you can imagine during the project, some of them are in process.
13:04And I really want I really wanted something like this.
13:07Um, but something that I could actually that could actually do the work itself.
13:11And so I um prototyped this thing.
13:15It's called the tech tree app.
13:16And basically, it's similar to this.
13:17It's a dependency graph, but each of these tasks um is a a coding agent that you can kick off and you can have your agent kind of like be running and doing them autonomously.
13:28Um the project itself that's working on it's this little um this is just kind of like a demo app.
13:35Um but this is Can I Yeah.
13:38So this is this is a little fun, you know, multimodal input thing.
13:43I haven't written any of the code for this.
13:45This is um this is all written by agents.
13:47But I can manage all of the work is being done in this desktop app uh or in this app here.
13:52And so I can do something like I can see this one has has finished uh building some gesture controls for the canvas.
13:58So I can open the PR uh and unfortunately sorry Jeffrey I am just going to merge this.
14:03I'm not going to uh have it be explained to me.
14:06But so this is
14:08[snorts]
14:08and so yeah great.
14:09Awesome.
14:10It looks good.
14:10Um and then you know eventually this is going to get marked uh as as complete and this is also multiplayer which is really cool and you can have people working together.
14:20Yeah.
14:20So that's finished.
14:21Um and I you can also prompt from like inside the the app you can draw and have a prompt.
14:28So I can basically I can just take all of this uh and I can draw a little like so this is my prompt and I can wrap it in a task
14:35um and I can you know call it facial animation canvas control and then I can assign that to cloud and I can just hit run and so now that's working as well and so this is kind of like you know this is kind of something similar to conductor or openai symfony where you're using a kind of like
14:50one abstracted interface above uh what the actual in order to like manage your your multi- aent uh coordination and things like that And the thing I like about this also is that because this is multiplayer,
15:01one of my colleagues can come and join and add tasks and uh edit things and see the work that's been going on.
15:06So, it's much more collaborative than um like your own instance of something.
15:10Um here's the moment of truth.
15:12Let's see if that demo that I had it build in the beginning worked.
15:18Um all right, it's it hasn't built the fluid simulation yet.
15:23Uh it's been working for 13 minutes.
15:25Um that's actually fine.
15:29So basically this is the TL Draw desktop app.
15:31Um something that's really cool here is that we have uh so this is running locally.
15:35It's working on files.
15:36We'll see if it finishes.
15:37We'll we'll let this run.
15:38Um but basically what this does is this uh this basically exposes the editor instance of the TLR app that's running here.
15:46And it has a server that lets any agent uh for example my cloud code uh write just plain JavaScript against the against the editor.
15:55and and it it basically it you know it's it's code mode if you've ever used code mode um but you can basically turn your teal desktop app into um a like scripting environment and
16:08the one of my colleagues actually is u I'm going to this is this is the kind of off the rails bit of the canvas here um or of the of the talk.
16:18So here's something my colleague made uh using the same thing.
16:22So this is he has the teal draw desktop app in the corner here and he's using it as his uh window manager.
16:28Um and what he did the way he did this was he just told claude code to because cloud code has access to your actual computer.
16:34It's not locked into the canvas.
16:36Uh it basically you know it it made some rectangles and it uh probably wrote some Apple script or something to actually re uh you know move the things around.
16:44And so you can kind of make all of these uh like ephemeral UIs and have them actually be doing things in the real world.
16:52Um, another really cool one that he did was, um, if this loads, it's Pong on the desktop.
16:59Uh, let's hit it with a little refresh there and see if it works.
17:04Yeah.
17:04So, this is he's got in the corner here, you know, you you have uh you have TLR running.
17:10This is the desktop app and it's using the Windows in order to uh play Pong.
17:14And so again, like kind of crazy, but there's uh you know, maybe it seems a little silly, but let's see if this this worked.
17:24Oh, it's still working.
17:25Man, it was usually much faster.
17:26Um, but I I think this stuff is so cool because this lets you kind of, you know, do all of the weird kind of like spatial interfaces that you can do uh on the canvas.
17:36You get all of like the primitives of the canvas.
17:38Um, but you can like you can have your agents working kind of like in the real world.
17:43uh it has access to real data.
17:45If I scroll up, I'll show that um if uh you know this is my cloud code and it found it it got the Gmail, it got the notion doc, it found the spec and it's going to to implement it.
17:58Um but yeah, so to sum up, I think that agents working on the canvas is so cool and I think that there's like so much we can do if we use like the agent the canvas as a a place to work with agents.
18:11And I think the place part of it is really important because you know when we do you know with remote work collaboration we do a lot of stuff online with each other and we collaborate with people on the canvas
18:22and I think that the yeah the canvas can be a place where we uh collaborate with agents and I'm I'm I'm I'm vamping because I'm trying to see if this is finished but I don't think it's going to finish.
18:31Um, but thank you so much
18:49[music]