Computer-Use 2.0: Agents Just Got Multi-Cursor — Francesco Bonacci, Cua

AI Engineer · 16 min · 138 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Thank you for taking the time for coming over here.
00:15Um I'm Franchesco.
00:17I'm the CEO of the company.
00:18Uh alongside me, a couple of other folks.
00:21Um my co Dylan and my chief of infra Rob.
00:24They're going to walk on the stage in a while.
00:27Uh but before we do that, who's excited for some computer using agent talk happening now?
00:32Are you guys excited?
00:33Lovely.
00:34Um if I were to ask like what what was a computer using agent like one year ago, probably half the crowd would say I don't have any idea what really computer use mean.
00:44Um so um today I'm going to take you to a journey.
00:48Um basically like from our vision where um we come from so far on computer user um like this new shape of like agents are talking um and uh up to model intelligence
01:06um so we're going to start like with the vision of uh quad driver where we're coming from and uh if you how many of you guys been been working with computer use for
01:20one year.
01:22How about like two years?
01:25Lovely.
01:25Okay.
01:25So, our team has plenty of experience like we go all the way back uh our time on Microsoft.
01:31We were working on this type of guey agents we were calling them uh back in the days.
01:36Um and uh there is a um there's an example of like old-fashioned human uh agent loop.
01:47Um we basically refer um refer this as a human uh loop where you will have like an agent loop.
01:55You will have a um uh you will take a screenshot that the agents will have to reason and plan through and then um you will basically work with an action space in terms of like clicking, typing, scrolling around.
02:07So this is what we refer as as um um the old fashioned like computer use 1.0 you know, just to set the tone for for um for this talk
02:17and uh we come like a long um a long like way since this type of like um computer using agents.
02:27So this is this again like I'm going to skim over these slides but that's like the old fashioned way of like representing these agents loop as a human would do.
02:34Um we um here we go.
02:41Um over like two months ago we released a project in open source.
02:45It's called quad driver and uh um we um we made it working like in the background.
02:55That means that your computer user will will not take over your uh screen as like the computer use 1.0 um kind of like agent loop was doing back in the days.
03:07And uh um it all like started from uh um from like uh um from Codex releasing their computer use um model two months ago.
03:20So we kind of like take the challenge because we were already like working with this uh this type of background computer user.
03:26So over one weekend we act something together.
03:30And uh the trick here is really um not like having your agents like take over your screen.
03:36So there is a lot of like dark magic like happening behind the wood just to uh give you some context.
03:42Uh there are like some undocumented API um living in u um in the Apple framework that basically ships with your laptop and as you can see here like is in the demo you have like an I agent that is not taking over
03:56uh control of your over your laptop.
03:58Um we made it working not only for Mac OS but also spanning like across Windows and Linux.
04:04where uh this is like the very first like driver that is uh living on your um laptop and uh it lets uh really any agents connect to the underlying operating system
04:18uh either like using accessibility trees or like a screenshot level approach.
04:22We kind of like take all um this is what really the agents see for what it concerns.
04:29Um you will have to install quad driver.
04:32Um the agents will uh will take a snapshot of the window state and uh you will have to observe um and uh we really like take take like one uh uh like different like action path to really make the ground computer use happening.
04:50So you really um have um to observe the space in this case just by calling like get window state you get a an accessibility tree representation plus a screenshot
05:02and then you will go and uh um try a background execution using accessibility tree and if that doesn't work we go all the way and uh make the heavy lifting for you and just try a pixel background click.
05:17This is like uh kind of like best step for background at this stage.
05:22It's not like behaving in the same way Mac OS, Windows and Linux.
05:26So uh we we do like some of the lifting for you so that your AI agent can uh can run and disturb on your uh on your MacBook.
05:35um how we manage like to not break anything between like release cycles.
05:40We have ex we have like a lot of investment happening uh behind the scene uh when we test like new releases.
05:49uh we have about like eight different uh application harnesses that are um that that that we that we use for making sure that we don't break anything uh among different releases.
06:05Um among our early adopters you can see like clicky mass queno h company and droid factory.
06:12uh like huge thanks to them for using quad driver and like um basically releasing a lot of like upstream contribution back in in our framework.
06:22Um without further ado, I'm just going to move to the next part of the presentation which is going to be intelligence.
06:28Um and I'm going to have our CTO Don cover that.
06:34Hello.
06:35So thank you Franchesco.
06:36Um with cooler driver we gave an agent hands but then the question becomes how can you trust the agent to use those hands correctly and not leave anything broken behind
06:46and to answer that we had to build kuab bench so for a show of hands who here has heard of terminal bench or harbor yeah so a few few of you have heard of it
06:58and uh if you've ever authored a task for terminal bench then this might look familiar but in kuabench a task is made of three pieces the setup setup function which sets up the machine to initial state.
07:10The oracle function which provides a golden trajectory for the task and the evaluator which probes the environment to check if the agent successfully completed the task.
07:21Uh unlike terminal bench the oracle here is guey actions.
07:26So it looks kind of like pile of gooey when you write that and writing environments takes scale and expertise.
07:34On desktop there's more than uh five platforms that we target and um we try to collapse that into a single Python file.
07:43So using the Kubaben SDK you can write a guey that works across every desktop platform in a single Python file and use the same SDK to probe that GUI to get usable agent data.
07:55Anyone or any agent can author one of these tasks and when you put that to work you get a real catalog.
08:02We have currently over 130 verifiable tasks, 42 environments, and across five platforms.
08:09And each of these are easily reproducible using our CLI.
08:13And the latest addition to our data sets is one that we're proud of.
08:16Uh with collaboration with Snorkel AI, we built KUBench Kyad, which tests computer use agents on electrical engineering tasks using software by real professionals and evaluator functions that actually simulate the circuits.
08:34But the results are humbling.
08:35The top agent that we tested only got a full pass on six out of 25 of these tasks.
08:41Of those six, 100% of them involved editing an existing schematic.
08:46And when we start the task from a blank schematic, the success rate drops to 0%.
08:54And across all the models that we tested, the leaderboard is flat.
08:57No model has achieved more than 30% reward.
09:04But once you can score something, you can improve it.
09:07If we take a look at the Kua bench basic data set scaled up to 4K resolution uh testing an agent they typically get around 62% pass rate but when you switch the agent computer tool from the built-in one to KU driver the pass rate jumps from 62%
09:24to 80% using 34% less tokens and this is primarily because KU driver focuses on a window rather than the entire desktop but our evals might say you can trust model XYZ
09:38at task whatever.
09:40But how can you know that the task how can you know that the eval can be trusted?
09:44So before we test a task against any agent, we first try to break the environment ourselves.
09:52We have a matrix of agents attempt to do reward hacking and attempting to break the environment and we take all that data and we compile it into a nice code rabbit style code review and only tasks that survive our pipeline can enter the data set.
10:06And if you ask us how we trust that agent, the answer is that it's just evows all the way down.
10:12But to measure the intelligence of an agent, you can't just measure its ability to successfully perform actions.
10:18You also have to measure its ability to understand the world that it's operating in.
10:22Every run that we record can be forked through any moment in its trajectory to give you the state of the computer at that moment.
10:31From there we can probe a model asking to predict the reward, the internal state or any other observation of the computer and compare it against the fork.
10:41And that prediction is the world model of the agent made measurable.
10:45And with that um I'll let Robert take the stage.
10:50Thank you Dylan.
10:52Uh hello everybody.
10:54Uh I am the chief infra officer at Kua and I'm here to talk to you about um how you're probably leaving a lot of money on the table uh with idle GPUs
11:04if you do RL training uh for computer use agents.
11:07So I kind of want to introduce this uh diagram to y'all.
11:11Uh could I get like is is there like general familiarity with this diagram or this like something that most of us haven't seen before like any anyone?
11:20Awesome.
11:21Very niche.
11:22Um Almost everything on this is not really important for what we're talking about, but the blue portions are um and what those basically represent are GPUs uh generating tokens
11:34for um RL uh training.
11:37And if you zoom in on this a little bit, you can kind of see like how this typically looks like with a sandbox environment is you're going to be generating some tokens
11:48um and then you finish your task on a sandbox and then you're waiting for either like a new sandbox to spin up or for your existing one to reset.
11:58Uh the problem here is that like this is just pure cost.
12:02um your GPU really isn't doing anything useful here and you know I don't know if you've heard but GPU time is pretty expensive right now.
12:10So um as you're scaling this cost really compounds a lot and you really want to focus on minimizing this if possible.
12:18So one thing that you might try to do is uh minimize the startup time of your sandbox.
12:24And I mean you should do that like that's a great thing to do but uh you know especially for computer use style environments sometimes this can be a little bit impractical.
12:35Um you know your researchers might give you like a 40 gigabyte environment and that might just be necessary and it takes a long time to pull that down and start it up.
12:44So you know how do you how do you design your training infrastructure so that you can minimize the GPU startup or the minimize the startup time of the sandbox?
12:54uh even when the sandbox is like not well designed to be start up quickly.
13:00Um so the way we do the oh man is it not so the way we do this is a pool and this is supposed to be animated but it's not animating.
13:12So um I guess I'll just explain to you orally and what what that is is uh so we have like a a set of GPUs here which all want to use
13:24a sandbox and what we will do is that we use a demandbased autoscaler to detect um how many GPUs like currently need a sandbox and we can grow the pool to be that size
13:37uh on demand.
13:38And what that means is that uh if you have let's say like you have a warm pool that you want to allocate to your GPU cluster, you don't actually need to know upfront what that warm pool size is.
13:50We can figure out what that warm pool size should be for you um on demand.
13:55And that might even change over the course of your multi-day training run.
13:58Uh you might start needing a lot of sandboxes, but then as your generations get longer, you might need less.
14:05So these also could be like, you know, easily uh two to four times cheaper than your GPUs.
14:14So having a little bit of redundancy here, uh you still wind up saving money because you're maximizing the use of your GPU time.
14:21Um yeah, come see me after if you want to see the animation because it's it's cool.
14:26Um so yeah so now when you have like this like uh redundancy in your pool you're paying the cost of that startup time on the infrastructure side not on the GPU side so your GPU workers have full utilization
14:43um yeah and then because we use this we can give you instant sandboxes for your GPUs for Windows Windows Linux Android uh and Mac OS is coming up Um,
14:56and I'm going to hand it back to Franchesco to uh close it out for us.
15:02Lovely.
15:03Uh, thank you Don, thank you Rob for taking this over.
15:06Um, we do have like plenty of time for Q&A.
15:10So if you guys like have any questions like happy to um take them either for quad driver qua bench basically what Dylan presented or qua fleet uh which is like what uh Robert covered
15:30any questions otherwise we can wrap this up.
15:34Oh, I see.
15:44Um, so the story for mobile Android there is very far you can go.
15:49Um, we are talking with the arm team because they do have like an arness that runs on Android.
15:55I guess like if you're talking about background there is some level of like background that can happen if you containerize a workload and basically on Android you can even like run your own container or like
16:07sort of like Ubuntu or like GUI docker container uh within Android um but yeah the Android ecosystem especially compared to iOS is more inclined to that form of like background
16:21uh computer use but it's more towards like tool use than really like controlling GUI interface.
16:27Um we work with the activity framework and uh do tool use in the background.
16:38Cool.
16:38Thank you guys.