WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=ZSQb5fzRFPw NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/ZSQb5fzRFPw.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.639 --> 00:00:15.839 Thank you for taking the time for coming over here. s3 00:00:15.839 --> 00:00:17.760 Um I'm Franchesco. s4 00:00:17.760 --> 00:00:18.880 I'm the CEO of the company. s5 00:00:18.880 --> 00:00:21.279 Uh alongside me, a couple of other folks. s6 00:00:21.279 --> 00:00:24.720 Um my co Dylan and my chief of infra Rob. s7 00:00:24.720 --> 00:00:27.039 They're going to walk on the stage in a while. s8 00:00:27.039 --> 00:00:32.160 Uh but before we do that, who's excited for some computer using agent talk happening now? s9 00:00:32.160 --> 00:00:33.280 Are you guys excited? s10 00:00:33.280 --> 00:00:34.239 Lovely. s11 00:00:34.239 --> 00:00:44.320 Um if I were to ask like what what was a computer using agent like one year ago, probably half the crowd would say I don't have any idea what really computer use mean. s12 00:00:44.320 --> 00:00:48.800 Um so um today I'm going to take you to a journey. s13 00:00:48.800 --> 00:01:06.560 Um basically like from our vision where um we come from so far on computer user um like this new shape of like agents are talking um and uh up to model intelligence s14 00:01:06.560 --> 00:01:20.640 um so we're going to start like with the vision of uh quad driver where we're coming from and uh if you how many of you guys been been working with computer use for s15 00:01:20.640 --> 00:01:22.400 one year. s16 00:01:22.400 --> 00:01:25.040 How about like two years? s17 00:01:25.040 --> 00:01:25.600 Lovely. s18 00:01:25.600 --> 00:01:25.920 Okay. s19 00:01:25.920 --> 00:01:31.520 So, our team has plenty of experience like we go all the way back uh our time on Microsoft. s20 00:01:31.520 --> 00:01:36.720 We were working on this type of guey agents we were calling them uh back in the days. s21 00:01:36.720 --> 00:01:47.520 Um and uh there is a um there's an example of like old-fashioned human uh agent loop. s22 00:01:47.520 --> 00:01:55.119 Um we basically refer um refer this as a human uh loop where you will have like an agent loop. s23 00:01:55.119 --> 00:02:07.520 You will have a um uh you will take a screenshot that the agents will have to reason and plan through and then um you will basically work with an action space in terms of like clicking, typing, scrolling around. s24 00:02:07.520 --> 00:02:17.920 So this is what we refer as as um um the old fashioned like computer use 1.0 you know, just to set the tone for for um for this talk s25 00:02:17.920 --> 00:02:27.200 and uh we come like a long um a long like way since this type of like um computer using agents. s26 00:02:27.200 --> 00:02:34.640 So this is this again like I'm going to skim over these slides but that's like the old fashioned way of like representing these agents loop as a human would do. s27 00:02:34.640 --> 00:02:41.920 Um we um here we go. s28 00:02:41.920 --> 00:02:45.920 Um over like two months ago we released a project in open source. s29 00:02:45.920 --> 00:02:55.440 It's called quad driver and uh um we um we made it working like in the background. s30 00:02:55.440 --> 00:03:07.519 That means that your computer user will will not take over your uh screen as like the computer use 1.0 um kind of like agent loop was doing back in the days. s31 00:03:07.519 --> 00:03:20.400 And uh um it all like started from uh um from like uh um from Codex releasing their computer use um model two months ago. s32 00:03:20.400 --> 00:03:26.720 So we kind of like take the challenge because we were already like working with this uh this type of background computer user. s33 00:03:26.720 --> 00:03:30.000 So over one weekend we act something together. s34 00:03:30.000 --> 00:03:36.959 And uh the trick here is really um not like having your agents like take over your screen. s35 00:03:36.959 --> 00:03:42.239 So there is a lot of like dark magic like happening behind the wood just to uh give you some context. s36 00:03:42.239 --> 00:03:56.080 Uh there are like some undocumented API um living in u um in the Apple framework that basically ships with your laptop and as you can see here like is in the demo you have like an I agent that is not taking over s37 00:03:56.080 --> 00:03:58.720 uh control of your over your laptop. s38 00:03:58.720 --> 00:04:04.080 Um we made it working not only for Mac OS but also spanning like across Windows and Linux. s39 00:04:04.080 --> 00:04:18.160 where uh this is like the very first like driver that is uh living on your um laptop and uh it lets uh really any agents connect to the underlying operating system s40 00:04:18.160 --> 00:04:22.400 uh either like using accessibility trees or like a screenshot level approach. s41 00:04:22.400 --> 00:04:29.759 We kind of like take all um this is what really the agents see for what it concerns. s42 00:04:29.759 --> 00:04:32.080 Um you will have to install quad driver. s43 00:04:32.080 --> 00:04:50.639 Um the agents will uh will take a snapshot of the window state and uh you will have to observe um and uh we really like take take like one uh uh like different like action path to really make the ground computer use happening. s44 00:04:50.639 --> 00:05:02.960 So you really um have um to observe the space in this case just by calling like get window state you get a an accessibility tree representation plus a screenshot s45 00:05:02.960 --> 00:05:17.199 and then you will go and uh um try a background execution using accessibility tree and if that doesn't work we go all the way and uh make the heavy lifting for you and just try a pixel background click. s46 00:05:17.199 --> 00:05:22.720 This is like uh kind of like best step for background at this stage. s47 00:05:22.720 --> 00:05:26.880 It's not like behaving in the same way Mac OS, Windows and Linux. s48 00:05:26.880 --> 00:05:35.120 So uh we we do like some of the lifting for you so that your AI agent can uh can run and disturb on your uh on your MacBook. s49 00:05:35.120 --> 00:05:40.240 um how we manage like to not break anything between like release cycles. s50 00:05:40.240 --> 00:05:49.039 We have ex we have like a lot of investment happening uh behind the scene uh when we test like new releases. s51 00:05:49.039 --> 00:06:05.520 uh we have about like eight different uh application harnesses that are um that that that we that we use for making sure that we don't break anything uh among different releases. s52 00:06:05.520 --> 00:06:12.800 Um among our early adopters you can see like clicky mass queno h company and droid factory. s53 00:06:12.800 --> 00:06:22.400 uh like huge thanks to them for using quad driver and like um basically releasing a lot of like upstream contribution back in in our framework. s54 00:06:22.400 --> 00:06:28.880 Um without further ado, I'm just going to move to the next part of the presentation which is going to be intelligence. s55 00:06:28.880 --> 00:06:33.440 Um and I'm going to have our CTO Don cover that. s56 00:06:34.800 --> 00:06:35.199 Hello. s57 00:06:35.199 --> 00:06:36.880 So thank you Franchesco. s58 00:06:36.880 --> 00:06:46.400 Um with cooler driver we gave an agent hands but then the question becomes how can you trust the agent to use those hands correctly and not leave anything broken behind s59 00:06:46.400 --> 00:06:58.160 and to answer that we had to build kuab bench so for a show of hands who here has heard of terminal bench or harbor yeah so a few few of you have heard of it s60 00:06:58.160 --> 00:07:10.000 and uh if you've ever authored a task for terminal bench then this might look familiar but in kuabench a task is made of three pieces the setup setup function which sets up the machine to initial state. s61 00:07:10.000 --> 00:07:21.599 The oracle function which provides a golden trajectory for the task and the evaluator which probes the environment to check if the agent successfully completed the task. s62 00:07:21.599 --> 00:07:26.800 Uh unlike terminal bench the oracle here is guey actions. s63 00:07:26.800 --> 00:07:34.639 So it looks kind of like pile of gooey when you write that and writing environments takes scale and expertise. s64 00:07:34.639 --> 00:07:43.199 On desktop there's more than uh five platforms that we target and um we try to collapse that into a single Python file. s65 00:07:43.199 --> 00:07:55.440 So using the Kubaben SDK you can write a guey that works across every desktop platform in a single Python file and use the same SDK to probe that GUI to get usable agent data. s66 00:07:55.440 --> 00:08:02.080 Anyone or any agent can author one of these tasks and when you put that to work you get a real catalog. s67 00:08:02.080 --> 00:08:09.120 We have currently over 130 verifiable tasks, 42 environments, and across five platforms. s68 00:08:09.120 --> 00:08:13.199 And each of these are easily reproducible using our CLI. s69 00:08:13.199 --> 00:08:16.800 And the latest addition to our data sets is one that we're proud of. s70 00:08:16.800 --> 00:08:33.840 Uh with collaboration with Snorkel AI, we built KUBench Kyad, which tests computer use agents on electrical engineering tasks using software by real professionals and evaluator functions that actually simulate the circuits. s71 00:08:34.000 --> 00:08:35.839 But the results are humbling. s72 00:08:35.839 --> 00:08:41.120 The top agent that we tested only got a full pass on six out of 25 of these tasks. s73 00:08:41.120 --> 00:08:46.720 Of those six, 100% of them involved editing an existing schematic. s74 00:08:46.720 --> 00:08:53.519 And when we start the task from a blank schematic, the success rate drops to 0%. s75 00:08:54.240 --> 00:08:57.839 And across all the models that we tested, the leaderboard is flat. s76 00:08:57.839 --> 00:09:03.200 No model has achieved more than 30% reward. s77 00:09:04.959 --> 00:09:07.519 But once you can score something, you can improve it. s78 00:09:07.519 --> 00:09:24.000 If we take a look at the Kua bench basic data set scaled up to 4K resolution uh testing an agent they typically get around 62% pass rate but when you switch the agent computer tool from the built-in one to KU driver the pass rate jumps from 62% s79 00:09:24.000 --> 00:09:38.000 to 80% using 34% less tokens and this is primarily because KU driver focuses on a window rather than the entire desktop but our evals might say you can trust model XYZ s80 00:09:38.000 --> 00:09:40.080 at task whatever. s81 00:09:40.080 --> 00:09:44.320 But how can you know that the task how can you know that the eval can be trusted? s82 00:09:44.320 --> 00:09:52.160 So before we test a task against any agent, we first try to break the environment ourselves. s83 00:09:52.160 --> 00:10:06.080 We have a matrix of agents attempt to do reward hacking and attempting to break the environment and we take all that data and we compile it into a nice code rabbit style code review and only tasks that survive our pipeline can enter the data set. s84 00:10:06.080 --> 00:10:12.640 And if you ask us how we trust that agent, the answer is that it's just evows all the way down. s85 00:10:12.640 --> 00:10:18.640 But to measure the intelligence of an agent, you can't just measure its ability to successfully perform actions. s86 00:10:18.640 --> 00:10:22.800 You also have to measure its ability to understand the world that it's operating in. s87 00:10:22.800 --> 00:10:31.440 Every run that we record can be forked through any moment in its trajectory to give you the state of the computer at that moment. s88 00:10:31.440 --> 00:10:41.440 From there we can probe a model asking to predict the reward, the internal state or any other observation of the computer and compare it against the fork. s89 00:10:41.440 --> 00:10:45.920 And that prediction is the world model of the agent made measurable. s90 00:10:45.920 --> 00:10:50.560 And with that um I'll let Robert take the stage. s91 00:10:50.560 --> 00:10:52.560 Thank you Dylan. s92 00:10:52.560 --> 00:10:54.160 Uh hello everybody. s93 00:10:54.160 --> 00:11:04.480 Uh I am the chief infra officer at Kua and I'm here to talk to you about um how you're probably leaving a lot of money on the table uh with idle GPUs s94 00:11:04.480 --> 00:11:07.760 if you do RL training uh for computer use agents. s95 00:11:07.760 --> 00:11:11.279 So I kind of want to introduce this uh diagram to y'all. s96 00:11:11.279 --> 00:11:20.640 Uh could I get like is is there like general familiarity with this diagram or this like something that most of us haven't seen before like any anyone? s97 00:11:20.640 --> 00:11:21.200 Awesome. s98 00:11:21.200 --> 00:11:22.160 Very niche. s99 00:11:22.160 --> 00:11:34.399 Um Almost everything on this is not really important for what we're talking about, but the blue portions are um and what those basically represent are GPUs uh generating tokens s100 00:11:34.399 --> 00:11:37.279 for um RL uh training. s101 00:11:37.279 --> 00:11:48.480 And if you zoom in on this a little bit, you can kind of see like how this typically looks like with a sandbox environment is you're going to be generating some tokens s102 00:11:48.480 --> 00:11:58.160 um and then you finish your task on a sandbox and then you're waiting for either like a new sandbox to spin up or for your existing one to reset. s103 00:11:58.160 --> 00:12:02.240 Uh the problem here is that like this is just pure cost. s104 00:12:02.240 --> 00:12:10.079 um your GPU really isn't doing anything useful here and you know I don't know if you've heard but GPU time is pretty expensive right now. s105 00:12:10.079 --> 00:12:18.480 So um as you're scaling this cost really compounds a lot and you really want to focus on minimizing this if possible. s106 00:12:18.480 --> 00:12:24.959 So one thing that you might try to do is uh minimize the startup time of your sandbox. s107 00:12:24.959 --> 00:12:35.120 And I mean you should do that like that's a great thing to do but uh you know especially for computer use style environments sometimes this can be a little bit impractical. s108 00:12:35.120 --> 00:12:44.000 Um you know your researchers might give you like a 40 gigabyte environment and that might just be necessary and it takes a long time to pull that down and start it up. s109 00:12:44.000 --> 00:12:54.639 So you know how do you how do you design your training infrastructure so that you can minimize the GPU startup or the minimize the startup time of the sandbox? s110 00:12:54.639 --> 00:13:00.320 uh even when the sandbox is like not well designed to be start up quickly. s111 00:13:00.320 --> 00:13:12.720 Um so the way we do the oh man is it not so the way we do this is a pool and this is supposed to be animated but it's not animating. s112 00:13:12.720 --> 00:13:24.720 So um I guess I'll just explain to you orally and what what that is is uh so we have like a a set of GPUs here which all want to use s113 00:13:24.720 --> 00:13:37.680 a sandbox and what we will do is that we use a demandbased autoscaler to detect um how many GPUs like currently need a sandbox and we can grow the pool to be that size s114 00:13:37.680 --> 00:13:38.800 uh on demand. s115 00:13:38.800 --> 00:13:50.079 And what that means is that uh if you have let's say like you have a warm pool that you want to allocate to your GPU cluster, you don't actually need to know upfront what that warm pool size is. s116 00:13:50.079 --> 00:13:55.440 We can figure out what that warm pool size should be for you um on demand. s117 00:13:55.440 --> 00:13:58.880 And that might even change over the course of your multi-day training run. s118 00:13:58.880 --> 00:14:05.519 Uh you might start needing a lot of sandboxes, but then as your generations get longer, you might need less. s119 00:14:05.519 --> 00:14:14.560 So these also could be like, you know, easily uh two to four times cheaper than your GPUs. s120 00:14:14.560 --> 00:14:21.600 So having a little bit of redundancy here, uh you still wind up saving money because you're maximizing the use of your GPU time. s121 00:14:21.600 --> 00:14:26.000 Um yeah, come see me after if you want to see the animation because it's it's cool. s122 00:14:26.000 --> 00:14:43.040 Um so yeah so now when you have like this like uh redundancy in your pool you're paying the cost of that startup time on the infrastructure side not on the GPU side so your GPU workers have full utilization s123 00:14:43.440 --> 00:14:56.639 um yeah and then because we use this we can give you instant sandboxes for your GPUs for Windows Windows Linux Android uh and Mac OS is coming up Um, s124 00:14:56.639 --> 00:15:02.240 and I'm going to hand it back to Franchesco to uh close it out for us. s125 00:15:02.880 --> 00:15:03.519 Lovely. s126 00:15:03.519 --> 00:15:06.720 Uh, thank you Don, thank you Rob for taking this over. s127 00:15:06.720 --> 00:15:10.480 Um, we do have like plenty of time for Q&A. s128 00:15:10.480 --> 00:15:23.120 So if you guys like have any questions like happy to um take them either for quad driver qua bench basically what Dylan presented or qua fleet uh which is like what uh Robert covered s129 00:15:30.320 --> 00:15:34.160 any questions otherwise we can wrap this up. s130 00:15:34.160 --> 00:15:36.720 Oh, I see. s131 00:15:44.480 --> 00:15:49.759 Um, so the story for mobile Android there is very far you can go. s132 00:15:49.759 --> 00:15:55.759 Um, we are talking with the arm team because they do have like an arness that runs on Android. s133 00:15:55.759 --> 00:16:07.920 I guess like if you're talking about background there is some level of like background that can happen if you containerize a workload and basically on Android you can even like run your own container or like s134 00:16:07.920 --> 00:16:21.040 sort of like Ubuntu or like GUI docker container uh within Android um but yeah the Android ecosystem especially compared to iOS is more inclined to that form of like background s135 00:16:21.040 --> 00:16:27.680 uh computer use but it's more towards like tool use than really like controlling GUI interface. s136 00:16:27.680 --> 00:16:33.920 Um we work with the activity framework and uh do tool use in the background. s137 00:16:38.560 --> 00:16:38.959 Cool. s138 00:16:38.959 --> 00:16:41.360 Thank you guys.