WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=CTLa_p6iOiY NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/CTLa_p6iOiY.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.880 --> 00:00:29.840 Hi everyone, I am Pugadoro and I'm the founder at Programmabs and today I'm going to talk about computer use agency evaluation and most of the work um and the details about it are in a in a paper with this title and I did this work while s3 00:00:29.840 --> 00:00:35.520 Meta super intelligent labs with the collaborators you see on this slide. s4 00:00:36.000 --> 00:00:44.160 And so to start I want to introduce this type of uh agent is a weird type of agent that I call a replay agent. s5 00:00:44.160 --> 00:00:54.960 So imagine we run this process um we run our frontier a frontier model a good one uh on a benchmark we like and then for every task we collect s6 00:00:54.960 --> 00:01:01.280 uh a successful trace or a successful trajectory and we have a recorded tape of this type. s7 00:01:01.280 --> 00:01:14.960 So the actions might be tapping uh typing scrolling and we record this and then we do this for all the tasks in the benchmark and we com sort of compile this into s8 00:01:14.960 --> 00:01:22.159 a replay agent that just when the tasks arrive replace that sequence of actions blindly. s9 00:01:22.159 --> 00:01:28.960 So if you do this for common benchmark with hundreds of tasks, this is going to be a script that is like less than a megabyte. s10 00:01:28.960 --> 00:01:34.479 And this is a completely valid like type of agent that you can evaluate on the benchmark. s11 00:01:34.479 --> 00:01:48.479 And if you try to evaluate this uh this kind of agent on standard benchmarks such as OSW word or mobile word, you will see that the success rate of this agent compared to the uh frontier model from which the agent was extracted s12 00:01:48.479 --> 00:01:52.240 is actually uh the same or even better. s13 00:01:52.240 --> 00:02:09.200 So this is a kind of a weird but maybe trivial phenomenon but I would argue that we shouldn't accept this kind of blind scripts beating the frontier models and the trick here or like the reason for why this happens is that um determinism s14 00:02:09.200 --> 00:02:27.040 of of the existing of most existing benchmarks and so uh if the benchmark is static is deterministic then it is somehow gameable by this sort of strategy and it goes even deeper than this like if you look at one of the metrics that people have been using in the past for evaluating s15 00:02:27.040 --> 00:02:42.959 computer use agents um and it's pass at K this metric is defined as you know the probability of at least uh one of K attempts succeeding um but if you look into like the details of how this metric works on a deterministic s16 00:02:42.959 --> 00:02:53.680 environment you will see that is literally um and we prove it like formally in the paper uh like evaluating the success rate of the replay agent that I've shown to you. s17 00:02:53.680 --> 00:03:04.480 Um so so basically if that replay agent felt weird uh to you also pass at K on computer use tasks should somehow feel weird to you. s18 00:03:04.480 --> 00:03:12.159 Um or in other words pass K is sort of a metrification of that uh of that exploit of the replay agent. s19 00:03:12.159 --> 00:03:24.800 So these are two problems specific problems but they point at two general classes of problems in in Kua benchmarks and uh these problems are around environments. s20 00:03:24.800 --> 00:03:31.599 So building environments and evaluation building good metrics to know if your agent is good or not. s21 00:03:31.599 --> 00:03:42.959 uh and in particular we we want to have environments that don't have exploitable structure and we want to have metrics that are not fragile or they are not based on fragile statistics. s22 00:03:42.959 --> 00:03:46.480 Uh and so I'm going to talk about the both of the aspects now. s23 00:03:46.480 --> 00:03:50.640 So let's talk about building principle environments first. s24 00:03:50.640 --> 00:04:04.159 So the first aspect um that I uh that I worked on while uh you know working on environments is to try to design a set of principles that could be guiding principle when s25 00:04:04.159 --> 00:04:09.360 building environments so that build like robust environments and trustworthy environments. s26 00:04:09.360 --> 00:04:20.720 Uh and so if you think about the problem that I was describing with uh replay agents, uh the first thing that you could think about as a solution not to have a replay agent to like hack your benchmark s27 00:04:20.720 --> 00:04:24.400 is uh to have your benchmark to be multiffactorial. s28 00:04:24.400 --> 00:04:30.240 So that means uh varying um generating variation for your benchmark. s29 00:04:30.240 --> 00:04:41.919 So having stockasticity into the benchmark and for computer use environment that means uh varying stuff like uh data or appearance or or simply the initial state. s30 00:04:41.919 --> 00:04:47.440 But if you do that you want to make sure that all the combinations that you generate are valid. s31 00:04:47.440 --> 00:04:59.040 Uh and so you want to have as a design principle in your environment also a system for checking and verifying that everything is working as as intended for every combination. s32 00:04:59.040 --> 00:05:03.840 And of course you want uh the usual uh the usual things for your environment. s33 00:05:03.840 --> 00:05:10.479 So you want your environment to be sandboxed and you want uh your environment to support like verifiers. s34 00:05:10.479 --> 00:05:15.280 So privileged information and you want of course your environment to be realistic. s35 00:05:15.280 --> 00:05:25.199 So if it's a reproduction of a real system, you want uh that reproduction to to be faithful so that the score that you get out is um is a good one. s36 00:05:25.199 --> 00:05:33.919 Um and so if you sort them out uh you can remember these sort of principles as the prism principles for for environment design. s37 00:05:33.919 --> 00:05:40.720 And we tried that method to build uh a benchmark that would be satisfying all of these principles. s38 00:05:40.720 --> 00:05:45.360 And if you look at existing benchmarks, some of them do some things uh in a good way. s39 00:05:45.360 --> 00:05:48.400 Some of uh some others do other things in a good way. s40 00:05:48.400 --> 00:05:52.639 But there is no unified benchmark that sort of matches all of these boxes. s41 00:05:52.639 --> 00:05:55.680 And we built one that is called DGword. s42 00:05:55.680 --> 00:06:03.440 So the way DG word uh in practice was built is as a set of like mobile apps um for for Android devices. s43 00:06:03.440 --> 00:06:14.160 So it's 15 apps on spanning different domains uh with 387 uh verified scenarios and uh a number of configurations. s44 00:06:14.160 --> 00:06:19.039 So uh these configurations they are in a large number of 3.2 million. s45 00:06:19.039 --> 00:06:22.319 Uh but the important thing is that they are verified. s46 00:06:22.319 --> 00:06:27.759 Um and indeed uh the axis are the ones that I was mentioning before. s47 00:06:27.759 --> 00:06:33.600 So you can imagine for each one of the tasks you can vary um things like the the instance. s48 00:06:33.600 --> 00:06:46.000 So what is the exact amount of money that you're sending for instance or the data profile like which kind of contacts or emails you have in the data for your uh for your task or like the theme s49 00:06:46.000 --> 00:06:47.520 or the starting screen. s50 00:06:47.520 --> 00:06:51.680 So do you start from the login page or do you start from another valid page? s51 00:06:51.680 --> 00:07:02.240 So if you do the math, even if you start from a relatively low number of uh base cases for each one of these variables, you end up having many many combinations. s52 00:07:02.240 --> 00:07:10.160 So you you can get to like millions of combinations and if you scale this up, you can get to easily to billions of of combinations. s53 00:07:10.160 --> 00:07:19.039 Um and all of these uh you know different axis can be can be manipulated by coding agents because in the end they are like forms of software. s54 00:07:19.039 --> 00:07:24.319 So you can have a coding agents to generate different instances different themes and such. s55 00:07:24.319 --> 00:07:27.759 Um, so you might think maybe it's easy to build an environment. s56 00:07:27.759 --> 00:07:34.720 You just generate as much software as you can uh with a coding agent and then you have like a diverse environment. s57 00:07:34.720 --> 00:07:36.880 But it's a little bit trickier than that. s58 00:07:36.880 --> 00:07:45.199 And indeed coding agents can generate a lot of software, but a lot of software is not the same as an effective cool environment. s59 00:07:45.199 --> 00:07:53.039 And the reason for this is that you need to verify the correctness of of your uh combination, right? s60 00:07:53.039 --> 00:08:02.240 Um and so the the the key to scale these up is to have a verification strategy for the variations of of your tasks. s61 00:08:02.240 --> 00:08:06.720 Um and so the kind of verification strategy to follow is is this one. s62 00:08:06.720 --> 00:08:20.479 So you you can generate many configs all the combinations uh of the different factors that I that I've explained before h and you can then have a system that rejects the broken ones the ones that are not valid s63 00:08:20.479 --> 00:08:32.479 and just keeps the the valid configs and so in the case of dig uh we did this by building a system that looks a little bit like a compiler and that works in the in the following way. s64 00:08:32.479 --> 00:08:35.839 So you start from a parameterized task template. s65 00:08:35.839 --> 00:08:39.599 Um and so uh this might look like something like this. s66 00:08:39.599 --> 00:08:50.720 So you have send a certain amount or certain recipient and then you have a verifier that corresponds to that template and then you have mock data uh for that task. s67 00:08:50.720 --> 00:08:55.040 So data that you need for that specific task to happen. s68 00:08:55.040 --> 00:09:04.320 uh and then we have a system that is like the dig compiler that takes all of these and given a base case of data base case of UI state s69 00:09:04.320 --> 00:09:20.800 puts all of these together and creates like a valid configuration and so you can build systems like this in which the the main craft is good software engineering uh to make sure that actually the combinations that you have are both diverse and valid. s70 00:09:21.760 --> 00:09:33.279 And so if you build a benchmark in this way um and you follow the the prism principles that I was talking about before you actually can have a benchmark that is not hackable in that way. s71 00:09:33.279 --> 00:09:45.440 Uh and so if you do the same process we did before you evaluate your frontier model and then you evaluate the corresponding replay agent uh you will see that the replay agent doesn't get a lot of performance. s72 00:09:45.440 --> 00:09:49.120 Um it gets a little bit of performance that is probably what you want. s73 00:09:49.120 --> 00:09:58.560 Sometimes some tasks maybe are repeatable by nature, but on average you you shouldn't expect a replay agent to have good performance on the benchmark. s74 00:09:59.440 --> 00:10:14.000 Once you build like this diverse um combinations, you also can do other things like um measuring the um robustness of frontier models on different over different axis of variation. s75 00:10:14.000 --> 00:10:19.440 So the axis of variation uh I described before are here represented there. s76 00:10:19.440 --> 00:10:27.519 Um and you can see that in the worst case frontier models are pretty bad actually at um being robust to these variations. s77 00:10:27.519 --> 00:10:36.720 And so for instance if you have a model uh that seems to be good at a given task, you would expect that if you just very you know which screen s78 00:10:36.720 --> 00:10:42.880 the task is starting from or like what is the team of the app, the model should pretty much have the same performance. s79 00:10:42.880 --> 00:10:46.160 But this is actually not the case for most frontier models. s80 00:10:46.160 --> 00:10:56.560 And so if you have infrastructure like this, you can actually measure measure that and like tailor your expectation uh about this kind of uh robustness. s81 00:10:56.880 --> 00:11:07.600 So this was about the first aspect that was building an environment that supports um diversity uh and that is robust enough to evaluate models. s82 00:11:07.600 --> 00:11:14.399 But the second aspect is as important as the first one is to measure uncertainty honestly. s83 00:11:14.399 --> 00:11:26.560 Uh so once you have all of this variation, how do you handle like computing um the real performance of your agent and basically there are two sources of of stoasticity s84 00:11:26.560 --> 00:11:40.320 of variation and uh they are not exactly the same but they are equally important and so the one that we usually think about is the one about the actions right and so you you run your model multiple times s85 00:11:40.320 --> 00:11:48.000 in many cases you can have even quite different trajectories out of it because the action at each step would be different. s86 00:11:48.000 --> 00:12:00.240 uh but if you have a benchmark like the one that I've described with multiple combinations with multiple variations then also the variability from the environment becomes important and we want to capture that s87 00:12:00.240 --> 00:12:10.399 because that is what we are going to find uh in the real world and so we need a methodology that captures uh both of these types of variation and s88 00:12:10.399 --> 00:12:23.519 uh in the paper there are the details but but basically we build a meth we built a methodology that can accurately um capture these two types of variation taking into account the structure of the benchmark s89 00:12:23.519 --> 00:12:29.200 and so if you start like in in practice it is useful to use this concept of coverage. s90 00:12:29.200 --> 00:12:39.120 Uh so when you when you compute a confidence interval uh basically you have some confidence that the performance of the model is inside of that range. s91 00:12:39.120 --> 00:12:49.120 Um and so you would expect that a 95% confidence interval would say that you know 95% of the time the performance of the model is on that range. s92 00:12:49.120 --> 00:13:01.040 But if you only use rollouts, so you only use the base case and what people would use normally actually in realistic cases you have something like 17% on 20% coverage. s93 00:13:01.040 --> 00:13:09.040 And that means that basically you only 20% of the time you uh guess the right performance of the agent which can be pretty bad. s94 00:13:09.040 --> 00:13:18.880 But like if you take it into into account the hierarchy and you use the proper way of computing confidence intervals, you can get to the to the full confidence interval and be 90 s95 00:13:18.880 --> 00:13:20.800 95% accurate. s96 00:13:20.800 --> 00:13:35.920 And so if this seems quite abstract uh you know in practice that means that if you want to make a decision about which models to deploy maybe you have model A model B and you do an eval for for those two models s97 00:13:35.920 --> 00:13:48.079 um you can have cases in which the confidence intervals seem really really small uh and so you make a decision based on those small confidence intervals but they are actually overconfident s98 00:13:48.079 --> 00:13:50.320 and so uh this was the wrong decision. s99 00:13:50.320 --> 00:13:54.160 So the the orange bars are the real performance here. s100 00:13:54.160 --> 00:14:04.320 Uh so you make this decision and if a mistake is pretty costly for you and you have many tasks like if you have one million tasks and there is a 4% s101 00:14:04.320 --> 00:14:12.160 mismatch in performance for real in the models and each mistake is like $20 uh like $12 on average. s102 00:14:12.160 --> 00:14:17.600 Uh it can cost you like hundreds of thousands of dollars uh in a single month. s103 00:14:17.600 --> 00:14:24.000 So it can be like super costly as a mistake just for a confidence interval being overconfident. s104 00:14:24.000 --> 00:14:33.279 But if you have like a reliable way of computing the confidence interval, the method would tell to you um I'm not confident enough to make an informed decision. s105 00:14:33.279 --> 00:14:40.320 And so you can choose like to spend more money to spend more time on evaluating models and avoid the co costly mistake. s106 00:14:40.320 --> 00:14:49.440 So we don't want to like elude ourselves with like wrong confidence intervals because there's uh you know money on the table essentially. s107 00:14:50.079 --> 00:14:57.680 And so this is sort of a final checklist of uh the things that I've uh that I've discussed so far. s108 00:14:57.680 --> 00:15:06.720 Uh so again to recap some of the things that are important in building on benchmark are about the environment and some other things are about the metrics. s109 00:15:06.720 --> 00:15:14.160 So on the environment you can follow uh the principles uh that I described before like prism principles. s110 00:15:14.160 --> 00:15:25.839 So some of these things are rather common but some things like um varying initial state across runs they are pretty rare across existing benchmarks but they are very important s111 00:15:25.839 --> 00:15:32.639 and so I would suggest you to try to incorporate these into your evals and and things about the metrics. s112 00:15:32.639 --> 00:15:48.639 Uh you can of course read the paper for the details but essentially um it's very important to avoid uh replayability as something that uh you can have in your benchmark and also to focus on having accurate confidence intervals. s113 00:15:48.639 --> 00:15:56.560 Uh so respecting the benchmark structure and trying to avoid um underestimating the uncertainty overall. s114 00:15:57.360 --> 00:16:00.160 And so I've heard many times sentences like this. s115 00:16:00.160 --> 00:16:06.720 this benchmark can be gamed but everybody's still using it or like there is no error bar but I don't see people using them. s116 00:16:06.720 --> 00:16:16.000 So these are sort of uh things that we can think when we don't have enough time but actually a non-rigorous benchmark is misleading. s117 00:16:16.000 --> 00:16:25.920 Um you know it can be misleading for the field because everybody could be seeking you know um maximizing a score on a benchmark that maybe is not capturing what we care about s118 00:16:25.920 --> 00:16:31.440 but especially it can be misleading for uh you know your own decisions. s119 00:16:31.440 --> 00:16:43.920 And so if you um are deluding yourself on thinking that a score is like confident and that is confidently telling that your model is good actually you are going to pay for those mistakes s120 00:16:43.920 --> 00:16:52.880 and so I think it's very good uh usually to be honest with with yourself and to try to be rigorous in the evaluations that you have. s121 00:16:52.880 --> 00:17:07.760 Um as the last slide uh I just started uh this company programmer and we are building uh the best infrastructure for ku enable verification uh and so we are hiring if you're interested or want to chat and this is our website. s122 00:17:07.760 --> 00:17:09.671 Thank you very much. s123 00:17:09.671 --> 00:17:11.671 [applause] s124 00:17:24.736 --> 00:17:26.736 [music]