The Autonomous Computer: Infrastructure for Computer Use Agents — Ang Li, Simular

AI Engineer · 72 min · 676 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12a little bit about my background.
00:14Um, I'm the co-founder and CEO of a similar and this year is pretty much the like the 30th years that I've interacted with computers and now I'm trying to build a computers that's fully autonomous so I don't have to interact with that.
00:30And if I look look back to my history, the first 10 years I'm I I I basically get to know computer when I was pretty young.
00:37The first 10 years I focus on comparative programming like basically writing code every day.
00:42And the the second decade I spent a lot of time uh training neuron nets basically published a lot of AI research papers uh during my PhD and later on in deep mind and the third
00:54um uh the third decade so I joined deep mind in 2017 that's the beginning of the third decade so this year is 20 uh 26 almost 10 years so so when when I first joined deep mind we really try to solve this problem called AGI
01:10artificial general intelligence And back then everyone is still confused.
01:14Even now it's everyone's still confused about what's the definition of AGI but in general it's kind of like a general system kind of like like human uh humans like like a machine
01:24can do all different kinds of things in there and uh hopefully next year we can solve the problem.
01:30So I I can think about what to do what kind of problem to solve uh for the for the first decade.
01:35Okay.
01:36So I want to u start my talk now.
01:39Um basically the reason I want to introduce that is because I spent 10 years on coding 10 years on neuronets 10 10 years on solving AGI.
01:48So actually that gave me this this kind of bias that we we try to inject to our current company that we have this uh this kind of principle in mind
01:58the general AI system needs to be neurosyolic basically code combined with neuronets.
02:04So, so which I will I will show you quickly why uh like we have this kind of intuition in there.
02:11So, the goal for this workshop I know we we're kind of short on time and I do want to give uh three things back to you.
02:18First is basic knowledge about computer use u I understand 50% of you are still new to this domain and second is the tools we we have been working on as part of the infrastructure.
02:30There are four different type type of components that we believe would be helpful to your future development of computers uh applications and I'll also show some applications that other people has been working on or other developers
02:42and third which is important uh since uh what we have a product called sigh and this product is still closed beta right now so I want to give everyone invitation code at the end of the the workshop so you have a you know opportunity to play with the product also do let me know uh Feel free to
03:00follow me on uh Twitter and join Discord.
03:03Let me know if you need more credits.
03:05So, so I definitely like give more credits in there.
03:08So, the first question is what is a computer use agents, right?
03:13So, uh normally what I do with the with my talk is I can take a selfie here with everyone.
03:23I'll take half.
03:25Yeah.
03:25So, so this is what I do every day now.
03:29So actually I'm I'm the CEO, so I don't use computer all day long.
03:32I have my agents on my phone.
03:34So I can text my agent, send a photo there and say post the photo to LinkedIn.
03:49Yeah.
03:50And then the agent will will be working on it.
03:52So let's see.
03:54So I do want to see if the agent is responding there.
04:05I think I got some.
04:13Okay.
04:13So, let's see.
04:17I do have my message there.
04:19Okay.
04:19The agent is responding.
04:21So, he has my photo there.
04:24And you can see the agent is starting asking me questions there.
04:27So uh I uh I won't interact with the agent right now but I want to show you the basic interface.
04:32So computer is agents means mean means the agent that has a computer which is a full Windows desktop in this scenario and if I continue interact with the computer the mouse and keyboard
04:43is know it's going to work on this desktop and open a LinkedIn later and I'll go back to the presentation.
04:53I just show you an example of having one agent.
04:56So I can talk to the agent agent working on applications but actually you can have multiple that's what we are actually working on.
05:05So uh the these are four agents you can see they have different names right so QA agents is basically like uh doing QA on the products testing product is one big use case in there they also research customer support go to the score channel and looking at all of the information there.
05:22So the the actual agents are slower.
05:24So it's actually this is recording for like hours.
05:27So I just compressed that into 30 seconds for you to basically grab the you know the basic understanding of of how computer us agents is working in there.
05:36So the general idea this is very different from um most of the agents that we talk about right now like most of the agent they're based on APIs and you don't actually see
05:46visually what the agent is doing.
05:49Computer use agents is another category and we believe is a is a is another bigger category which can do a lot more things that you you don't have APIs you can still let agents to work on it
06:01and then I want to do a little bit ritual because we started a company in 2023 right and that was uh right after GT 3.5 and uh GT4V so I want to show you one example that two years ago we kind of we're kind of hopeless
06:15two years ago on this problem in there.
06:18I actually post it on on Twitter.
06:21So this is is my tweet and let me pause a little bit.
06:26This is my tweet two years ago.
06:28So the the problems is like can we actually use GP4V to understand the screen and then click on the iMessage icon on the screen.
06:38Very very simple trivial trivial problem.
06:41And I basically use the model build up a iterative refinement of the the grounding.
06:50You see the the most the first time is is not correct.
06:52The second time is go to the mailbox icon and the third time is correct on the iMessage.
06:59So this is this is the only one successful demo I did.
07:04[laughter]
07:04So so I asked I asked the agent to to click on Zoom.
07:08It doesn't work at all.
07:10So two years ago this problem is basically hopeless but we kind of have the intuition that uh financial model is going to be better and better over time eventually
07:18this problem will be solvable but it's just like it's not possible within two years back our um that actually motivates us to think about what's the general strategy for solving this computer use agents uh problem
07:32and we do have a name for that it's called autonous computer because you know like if the computer can do all different kind of things it's autonomous right and then The question is how do we solve autonomous computer?
07:44So when I when I talk about it there are only two words autonomous and computer.
07:48So it's become so natural two directions you think about one is the autonomy the other is computer right autonomy meaning like do we have a system that can actually understand
07:57the context the environment of the of the computer and do all the reasoning planning can recover from its own mistake um like doesn't require human to babysitting the the agent and the second part is computer like what kind of computer
08:12uh the agent can use so this is a core question but back then two years ago we feel computer is actually easy problem because everyone already have computers right so that's why we focus a lot on the autonomy part
08:24so when we talk about autonomy this goes back to the the underlying principle we initially have is called neuro symbolic neuro means this also happens to self-driving car it's basically the the same concept for all different kinds of autonomous system they all have the same concept
08:40you you you have a component that's neuro here is a VM vision language model VM agent and the Second part is symbolic and here is we call it coding agent back then
08:51and basically we need to solve these two technology in order for us to make autonomy works and I just show you a demo two years ago VLM doesn't work at all
09:01right so that's why we actually spent a lot of time two years ago working on the coding layer to ensure okay like we can actually control the computer in a more reliable way but the second problem is we we set up a research team two years ago to think about okay how can we make this
09:16craft app VM better in terms of using the uh using the computers.
09:23Um before I start talk about the the trajectory our journey on the VM agents I want to uh briefly talk about the the signals like you know like self-driving cars uh like we running on the road they have signals lighter
09:36radar and uh and like camera right three different kind of signal computers had the same so I often tell people like okay we are we actually working on self-driving cars in the in computer in a digital world basically changing the steering wheel to your mouse and keyboard so we still three different kinds of signals.
09:54One is screenshot, the other is accessibility tree.
09:56So accessibility tree is very unique.
09:59It's it's kind of rare, you know, in in our in our daily life.
10:02DOM tree is pretty popular because we use browsers.
10:05Accessibility tree is a kind of a uh still kind of like a dome tree, but this is this tied to each of the operating system.
10:13Mac has it its own accessibility tree format and Windows has it like a separate protocol for accessibility tree.
10:20But this is the basic the the the layer that operating system use to render all of the windows all of the icons on on your screen in there.
10:30So this this part is very important.
10:31It's because um yeah the these are the the effort we've been working on is called a so the whole idea is can we build out a harness.
10:42So two years ago we don't call it harness.
10:44Now day everyone call it harness.
10:45So two years ago we we build out we try to finish this project called agents build out a harness b uh build upon all of the foundation models not just one so you can easily switch models it's model agnostic framework that allows you to produce the best uh computers capability in there
11:04and the first generation was done in late 2024 and it's called agents uh we just call it agents because that that's the beginning and This agent achieved 20% success rate on OS word 20%.
11:20And that was the state of art state-of-the-art back then in 2024 which is very interesting.
11:26And this paper this paper also won the best paper in I clear 2025 agenda AI workshop you know that the the whole industry was moving so fast over the past year
11:37and which is crazy and this the the choice for this harness is because first we know vision language model is so bad it's not going to work.
11:47That's why we chose to use accessibility tree in the first generation.
11:51Basically, we look at all of the the underlying protocols of the tree structure to describe the current environment and pass it to a large language model instead of vision language model
12:02and use ID ID based grounding basically linearize all of the all of the buttons put them IDs and ask LM like which button what's the ID of the button where I need to click on right now.
12:13It kind of works.
12:14There's 20%.
12:15which is super exciting because in the beginning this benchmark is like 5% success rate in the early
12:22[laughter]
12:222024 in there and then early 2025 we started to see okay vision language model is getting better so the second generation of agents we actually uh completely dis discard the accessibility tree so it's a vision only
12:38approach and the interesting part is this architecture we have multiple layers like managers and workers.
12:47So it's kind of like human organization and we showed it kind of works and this work specifically was um a a lot of friends in summary actually talked to me about this work.
12:58They they feel pretty excited about this work.
12:59The reason is this master demonstrate that if you have two models you are better than one models.
13:08If you have two foundation models within the same harness it's better than one uh foundation models.
13:14And the second foundation model is a seven billion open source model.
13:19So yeah, so all of the startup friends, they are pretty excited about it because okay, there's a there's our journey like I can train some small models and we can be 20%
13:28better than uh like cloud or openi uh computer agents there and late 2025 we have our third generation agents and the whole idea of this part is can we actually do
13:43test time scaling it's called inference time scaling is when you have a agent can you have multiple agents to try to attack uh solve the same task and eventually you have better performance
13:54and with demonstrator it works.
13:55So this is uh the figure for the total resource budget meaning the total number of workers versus uh times the total number of steps and this for like fair comparison
14:06and you can see if you have 10 agents you actually scale better in terms of success rate.
14:11So then in December we break the human performance uh record in OS world benchmark and these are comparison of all the trajectory between among uh agents and anthra and openi
14:25uh both big labs they achieve the same human performance uh in March this year and that also uh it's uh last December was an exciting moment like we feel okay like computer user agents is finally
14:39start to be somewhat useful but you know this benchmark is kind of research benchmark it doesn't mean like in real life it's already 100% it's already better than human
14:48um a few days ago OS word has uh has released their second generation called OS world 2 I'm specifically excited about this new benchmark so um in the new benchmark the best performing model is only 20% in that benchmark
15:04so so we are we are entering the second phase of computers uh agents in there I want to show show you one video.
15:12Uh there was a very interesting um demo that one of the developer used.
15:18So this this project actually won the cow hacks last year.
15:21So it's it's a it's a one of the biggest hackathon in uh hosted in in Berkeley and they they built their applications based on agents the open source one.
15:38I guess uh you probably don't I'll hear the voice.
15:43What if you can control your computer from anywhere in the world just by it?
15:48We built a multimodal agent with full control of the computer.
15:51So instead of calling your mom to debug your two out of memory errors, you can just FaceTime the Mac instead.
15:56Let me show you how it works.
15:58Okay, so suppose that you're training an A model at home, but you already left and you just realized that you had a CUDA out of memory error.
16:04So what's the easiest way to fix this?
16:06Well, instead of driving all the way back home, you can just call your computer and tell it to do itself.
16:11So, we can call it and text it to FaceTime me and then this will initiate the computer use agent right here to uh start a FaceTime call so that I can tell it what I want to do.
16:24Yeah, they they did a very interesting thing.
16:29Let me
16:30So now I can see what the thing says.
16:32Can we fix my CUDA out of memory error in VS Code?
16:36Okay, the task is to fix the CUDA out of memory error in VS Code.
16:39I will hit command space VS Code to open the application.
16:46So these are all automatic.
16:47Okay, I'm in VS Code.
16:48I'm looking for the CUDA out of memory error.
16:50It looks like this file has a CUDA out of memory error.
16:53Based on the comments, it looks like the tensor size is too large.
16:56So I should try to decrease this to something reasonable.
17:07Yeah, I think I think you kind of get the idea.
17:09So the idea is build a application that you can control your computer using FaceTime and use FaceTime to debug in VS code.
17:18That was happening in October last year.
17:21That's before like even open claw was started as a project in there.
17:26Then we had to think about what's the second uh you know second phase of computer use agents in there.
17:32So we already surpass human performance in the benchmark like what's next?
17:36Then we realized okay there's actually a very interesting concept to called pass to the power of K. And in the past, everyone was measuring pass ad meaning is is pretty popular in in VIP coding like you you have a task you add you ask the agent to
17:53uh produce the code right if the code doesn't work and then you you put some prompts in there and produce another another uh code in there one of the code
18:03becomes successful is successful because you don't care about what's happening in the past you only care can I get a code that works right in computer use is very different because
18:12Like we we started to get customers two years ago and all of the customer talk to me about this question like can I actually put your agents hosted in a server and run this workflow every day
18:26and I don't have to you know keep looking at looking at looking at what workflows I don't have to debug in there.
18:33So basically everyone was telling me about this metric pass to the power k. So this means I run this agent K times every time it needs to be successful
18:45and this metric is not captured by any benchmark in there and then we started to research this problem.
18:52So we took the best performing agents in OS word and their performance is pretty good in three different categories and then we run the stem agents three times.
19:02We measure the performance of pass to about to about three right from this figure you can see all the agents has degraded performance in there they almost dropped 10% success rate even even the best performing agent in there
19:19and the second part is on perturb environment meaning my querics could be a little bit different and environment could be a little bit different each time and it get even worse
19:28so this this is still a problem in the computers agents nowadays Because you can see uh Claude has Codex has uh agents, claw has agents.
19:39None of the agents actually produce reliable repeatable process for you.
19:46So this is a recent paper we published uh like two months ago.
19:50So basically discussing this problem and there's a little simple trick that we proposed in this paper that everyone can you know can borrow the ideas in the future if you want your agents to be more reliable.
20:01The trick is basically ask the agent to to ask you clarification clarification questions when you give them the instruction.
20:10So, so basically this paper was saying like suppose the agent always ask you some questions about ambi ambiguous part in your in your prompt the reliability of the agent will be better
20:24and there's a prompt in the in a paper that everyone can look at it like basically it's a it's it's pretty simple it's just like okay like your goal is to minimize
20:33u uh have minimum uh clarification that still preserving the nature this line is very important You put this prompt to the agent and the agent can perform better.
20:45After you put this clarification prompt to the agent, you can see like the overall um reliability of the agent is getting better like dash line is the original performance
20:57with no clarification and the solid line is the one with clarification.
21:01Right?
21:01You can see cloud is getting better and agent S with GPT5 which is interesting has the biggest improvements in there.
21:10So, so the the purple line is model agnostic framework and the green line is is the you know like big labs computer agents in there.
21:22Okay.
21:22So, so basically this is our journey for solving the VM agent which seems to be work work pretty well and still like nowadays still people using the agents framework because model agnostic you can plug in any open source foundation models.
21:35I'm also like looking forward to getting like results with the latest open source big model like GLM uh 5.2 tool and see how the performance looks like in there.
21:46And then the second question is okay, it doesn't solve the reliability problem, right?
21:51It like these these agent doesn't seem to be have as long as you're using a neuronet, as long as you use a VM, it's in the end at the end of the day, it's a probabilistic model.
22:02It produce a probability each time.
22:05So it's not going to give you the same performance uh if you run it for like 247 every day.
22:12Then we had to think about what's the next thing to think about is basically symbolic layer.
22:17We call it coding agent and we open source this uh language we call it simulan.
22:23Uh it's a language for similar.
22:25So we call it simulan there.
22:27So this language is kind of like if you're familiar with RPA industry it's kind of like what RPA industry used to do like they ask engineers to write a code.
22:36uh but the common problem for uh RPA is there's no single protocol that's shared across the whole community that everyone can use and we try to create this uh uh in browser automation we have playright
22:48which is in a standard way but in computer use there's no really no uh industry standard there so that's why we we open source this we hope it's helpful to all the developers so you can
22:58u actually like build your own square pin layer for different workflows that's reliable deterministic each time.
23:08Yeah.
23:08So, basically the goal is to allow you to run to create uh source code for Simulan and run this code you know you know terminal and you can also use it in claw code.
23:20So I I do want to give you a quick demo there.
23:24So let me Okay.
23:36So now I'm in a cloud code.
23:41So if you install um maybe I should show you simul Yeah.
23:51So these are the the landing page and if you go there there's a simple oneline installer npm installer.
23:57Once we install, you should be able to use it on your own terminal.
24:02And I'm a claw code user.
24:04So I like to see because no one is writing code nowadays, right?
24:08It's important for you to use V coding to to do this kind of automation.
24:13So once you install there's a skill that you can is paired with Simulan.
24:17So you can just uh you know use the skill.
24:20So I can say go to AIE website and summarize the events.
24:34Okay.
24:34So let's see how it works.
24:41So now it's checking checking version.
24:44So these are the logistic stuff.
24:46But it does understand it's going to use simulan to open AI engineer words fail website continue.
25:02So you can notice is it's try to use a simulant JS is a JavaScript uh language.
25:13And now clock is gonna try to write some code using the simul protocol.
25:26Yeah, I think it's because the the clock is new.
25:29So u it needs to do lots of uh logistics stuff.
25:35I allow reading for everything.
25:44So you can still think about this this uh this is approach.
25:47It's a different approach for computers.
25:49It's basically reduce computers problem into a uh coding problem in there.
25:55I have what I need.
25:56Let me write a script that opens the website extract the content in there and then you can see the the agent is writing the code.
26:06After that it opens the website automatically on your so so similar line is running on your own computer.
26:12If you have a virtual machine you can also deploy virtual machine there.
26:20So there's a and then you know after after clock run the code it recognize some errors and after it gets the errors it try to revise the code.
26:32So this is pretty much if you enable you know like auto approval this is pretty much already um a self-improving uh computer agents there some level of self self-improving
26:43there I do want to see the it's not there and you can see claw code is decided to use accessibility snapshot instead of the screen screenshot because you know it's a summarization
27:08task.
27:12Okay, it's coming.
27:14It gets uh it actually summarize it did.
27:19Yeah, it summarized all of the events for AI engineer in there.
27:23Okay.
27:24So, I'll pause here and I do want to show you the full code which could be interesting to look at.
27:35Yeah.
27:35So, these are the full code.
27:36It looks like JavaScript uh just like any any other JavaScript and it does open a window take the snapshot and write to the file.
27:51Okay.
27:51So, let me go back to my presentation.
28:00Okay.
28:00So, so I I've been talking about like two completely different approach.
28:04They're orthogonal.
28:05They're not the same same approach.
28:07But the question is do we actually need two frameworks?
28:11Right?
28:11So, so they they try to compete with each other.
28:16And this goes back to like a like a very uh interesting and important topic in in the whole deep learning because uh when I was in deep mind I specifically focusing on this problem called continual learning.
28:30So that's why we did a lot of study on like uh AI systems in there but in the end we realized okay there's no silver bullet like yeah the the
28:39when whenever technology you have uh like you have still have to solve the two problems as a trade-off like the the learning any learning system needs to learn to new things and any of the learning
28:51learning system needs to you know make sure you can accomplish the the task that you you you're already successful uh repeatedly.
28:59So this become it's called stability and plasticity dilemma.
29:03So you you want the learning system to be stable but at the same time you want the learning system to to have this capability to adapt uh to adapt to new environments in there.
29:13It's never be the case that you you are both you you're good at both.
29:18So that's the conclusion.
29:19Yeah, it's always trade-off.
29:20Then then the problem we can think about is how do we balance the two in it?
29:25And if you look at the two system I just demonstrated these two systems are coding agent and VM agents right they're basically representing the two extreme of this learning system the coding engine produce the code and it's determinance all the time but it's kind of stubborn
29:42right if you run a code every day if the environment changes then the code breaks so that that's the problem and the V engine is kind of adaptive but the problem is you cannot rely on the V agents to
29:53uh you know like repeat the process all the So the core question is okay can we find a middle ground between the two because the north star is basically the best adaptivity
30:04and the best reliability right can we also call this this curve we call this curve parental front and can can we actually find a the sweet spot there so this becomes the current
30:17direction that our company is specifically focusing on like basically try to find a unified framework that we can combine these two into one single single principle which which we call neurosyic
30:29right because there's a neuro then symbolic is there a way that we can actually like fuse these two technology into one single framework that it becomes self-improving all the time and becomes robust when when
30:40reliable when it's needed becomes uh adaptive when environment changes so I wouldn't say we solved the problem I wouldn't say we solved the problem but uh I would say like we we are already at the initial phase of moving towards this direction
30:55because there's a clear paradigm that we can do right now because we have we have a VM agent that's better than human in a small benchmark and we have
31:05a coding protocol that we can actually write a code using coding agent and the protocol can produce deterministic workflows in there.
31:15This is a simple way for us to do this new experiment.
31:19So basically we we let the VM agent to run in the environment and the agent keep writing the code to a file and once this workflow is successful you have a you have a you have code that's that's that's
31:34successful and the next time you run the agent you can run a code first instead of run a VM agent right if it doesn't work if the environment changes like when it breaks
31:45uh as as I just demonstrated there's actually a healing loop that you can do with the coding agent.
31:53So this this become very interesting and uh yeah I added this slide.
31:59So basically another interesting uh story is this year's cow hacks the the Sam hackathon the first place also use our uh our agent but this time they they're not using the agent to build a product they actually use the agent to test the product
32:15which is another interesting category that everyone can look at because all of us are developers right we write code and we know like nowadays there are more and more PRs more and more code then who's going to do the testing then computer
32:27use agency is a solution for that So basically this is the agent that can just move in the mouse click on the uh on the dashboard and doing a testing for you.
32:35I'm not sure if I have time.
32:36If I have time I really want to uh show a demo about like the the testing part but I can I can try let me try um let me try to show you the loop
32:52that we have in the product.
32:54It's a it's an initial effort.
32:57It works in a lot of cases, but sometimes there are still some hiccups in there.
33:02So, um I do look I do look forward to everyone like testing the product and give us feedback in there.
33:12Okay.
33:12So, let me I'm not sure if this one is doing Oh, this one.
33:21Okay.
33:23So this is the value VM.
33:26This is the agent is not you see I have I have a lot of VMs here and I I do put different names to to them.
33:43Um so let me try this one though.
33:50So in order for you to enable the neuros symbolic mode you need to come to this dashboard click on agent there are a lot of special there are a lot of special algorithm we do uh here like humanized
34:03control like the trajectory is different uh clical integration so you need to look at the sim flow code replay so this is a specific experiment we're doing right now
34:13so the whole idea is uh basically the diagram I just showed uh if you enable that we can try something.
34:21Let me ensure the desktop in there.
34:23Okay.
34:24So, we can try this.
34:25Yeah, we can still try the the simple thing.
34:27Just want to give you a rough idea of how this works.
34:31Go to AIE web page and um find the first speaker on Tuesday.
34:51There's a always has problem during live demo in let me restart agent.
35:12Let me see if my VPM is awake.
35:21Yeah, while we're waiting.
35:22So, let me show you another demo because I al I have a backup in there if live demo needs no longer time.
35:30So, we can show another one.
35:32So, this one is a collaboration with Microsoft.
35:35Uh so Microsoft has uh Microsoft is also built in this computer infrastructure uh because because they are the creator of Windows.
35:43So these are the VMs from Microsoft.
35:45It's called Microsoft Windows 365 for agents and the idea is from this you can see like on the left side the chart.
35:54So the idea is okay like use uh use our agents on Microsoft virtual machines to to fill out the form on a traditional ERP.
36:03So this is one of the biggest area for computer computer use agents because uh there's still so many traditional industries and they don't have modern APIs they don't have like you know they still running all the old
36:16um softwares on their Windows machines right so if you have if you build out any like applications that allow you to go to certain verticals because we are not vertical company and we are horizontal infrastructure if you're interested in building like towards different verticals
36:31these are good opportunities Um like for example some of our customers they uh they work with insurance website they specifically working on insurance.
36:40So basically they use the agents go to insurance website and uh fetch the data and some of them are healthcare u they work with doctors they have uh like doctors has windows uh machines with ERP system very old
36:53and you can use computeriz agents to automate all the data entry.
36:56So there's lots of ideas that uh people has been uh exploring in this direction.
37:02So, um, yeah.
37:04So, those are hopefully that's that that'll give you some insights there.
37:12Oh, it works.
37:13Yeah.
37:13So, so let me because I guess I have some skills that's already there.
37:25So, there's a skill here.
37:28Let me remove it.
37:34go to AIE website and see who's speaking first on Tuesday.
37:51I already have website.
37:52So, let me make a challenge.
37:55I create a new one.
38:02And you can see all of the logs here.
38:09And the interesting part you can see this agent is writing uh almost like line by line simulan code.
38:16And after that it's going to com compose all of the code into one big file.
38:24Navigate into the schedule page.
38:28Okay, so we get the get the information and if you notice the agent is actually it's not a screenshot.
38:38So whenever you try to find information the agent actually decide should I take a screenshot or should I take accessibility tree this time actually take access accessibility tree.
38:47So you can see all the information that beneath in one shot in there.
38:51And what I can do now because I want to show you that uh save to a skill.
39:01So if I enable that neurosy mode and I say save to skill, this skill will have all of the code that you just run in a in a previous session.
39:13And then we can look at AI speaker on day.
39:19Let me find a skill there.
39:30I didn't find it though.
39:37Okay.
39:37So this skill is saved.
39:40So I'll try a different query.
39:43on a different day.
39:47Let's try on Wednesday.
39:53And you can see there's a replay mode showing up.
39:56The interface a little bit different because it it does have the code.
40:00It has the code and it changed the parameter of the code.
40:04And if you click on the activity log, you can see the agent is actually producing a lot more code in the first step.
40:12So these are the the code.
40:14It has like different parameters in there.
40:16U by default is Tuesday, right?
40:21The default is Tuesday.
40:22If you set it Wednesday and it's going to uh change the parameter there and there's a hiccup.
40:30You can see there's a hiccup for the the agent need to filter in there.
40:37It does find the information and replay successful in there.
40:41there's a little hiccup and then after the hiccup the agent uh convert it to the neural mode which is a VM agent.
40:49So the first time it always try to produce all of the code as much as possible because if you have a pretty good hit rate it's going to have a very determin
41:00successful overflow and then the agent also have verification in the middle.
41:04So every time it runs the code, you have verification.
41:06If there's something wrong, stop the it's kind of like it's kind of when back when I was uh when I was in school, we we write everything that teacher says in the in the memo as a playbook, right?
41:19And next time when I want to do something, I look at my memo and try to do all the, you know, the steps step by step.
41:26And if the environment is different, I notice, okay, I I try to do this action, but the environment doesn't allow me to do that.
41:34And then I have to like think more spend more time on you know like exploring the new environment and try to figure out like what's the best strategy to do.
41:44Okay.
41:44So let's move on.
41:46Uh I believe that's a good direction for us to for even for the whole industry to think about like what's the what's the better um agents that we can produce with
41:57this kind of balance of flexibility and reliability.
42:02And we do see pretty good uh success in this this type of experiments.
42:07It's like okay I don't I think the I think the main point is you don't have to spend the same amount of tokens each time anymore with this kind of technology because
42:20uh because the second time you run the workflow maybe 50% of the effort is done by code without any LM request in there right and uh you can you can dramatically reduce your cost so that's why we also experimented in this product we also experiment with people
42:38on like $500 unlimited task plan.
42:42So the price and plan may also change.
42:44It's not going to be like uh usage based forever.
42:47Uh because you think about like as a human, I'm I'm a human.
42:51When I try to do a task a 100 times, I become so familiar with that.
42:56I can pretty much do all the things with my eyes closed, right?
43:01But today's agents, they don't have this capability.
43:04every time you do the same task, even you do a thousand times or 10,000 times for the same task, even the same query, you still pay the same amount of money to anthrop and open,
43:14which doesn't seems to be like the the reasonable uh is it's not economical as well.
43:21And yeah, so uh hopefully that that gave you a pretty good sense about what direction we're we're going forward.
43:26And uh I do encourage everyone to try try it out because we we actively rapidly improving that neuros symbolic agents uh in in a in the coming months and
43:36uh hopefully we can achieve some like a breakthrough in that direction.
43:40And the second second problem we have to think about is okay like what's the what do we do with with the computer?
43:46So two years ago we we work on this autonomy uh technology on your computer.
43:52So basically we have a we we used to have a product that runs the agent kind of like called co-working on your own computer but we receive a lot of complaint from users because everyone has only one computer
44:03and then when the agent is working on it it's in foreground you cannot you cannot do anything.
44:08Yeah.
44:08So so that actually motivates us to think about okay suppose we have a human level agent uh that works.
44:16Suppose we have something that works.
44:18It's not it's not there yet.
44:19We we're very close but when that day comes we have agent that works then we have to think about what do we do with the computer like do you still need one computer or maybe you need 10.
44:33So the current observation is whenever you have something that works, users always ask us to can you can you actually host this agent on a separate machine?
44:42Can you host on a server?
44:44So this is always always the same conclusion from most of the users.
44:49And then they will start to ask for more machines.
44:53So there's a trajectory.
44:54There's a trajectory.
44:55First we are in the in a in a phase that we really try to make agents works.
45:00Once it works, you need 10 machines, 100 machines.
45:03That's why it motivates us to work on the cloud computers.
45:07Um I I briefly talk I already briefly talked about it.
45:11I think that the whole point is if we have the cloud managed desktop uh it becomes so scalable and everyone in the end I hope everyone can have a 100 computers so you become the CEO of your own
45:22AI organization, right?
45:24So you have 100 AI employees in there and everyone become 100x productive.
45:31So yeah, basically this year uh we we launched a product side in March.
45:35Uh it's actually a pretty new product and together with that is called side box.
45:40is the virtual computer cloud in there and from the from from that demo you can already see I have personally I have 20 machines uh running in the background in there and uh this is a general diagram s is actually the interface and sausent
45:57which uh do which does is auton autonomy part does the planning and decision and then doing operations on the sidebox which is a cloud computer and okay and then we have a lot of friends who developers.
46:10So they keep asking us so can you actually expose the API.
46:12So today we actually uh releasing the API for uh accessing the uh the cloud computer.
46:18I don't think I have time to demo but it's actually pretty simple.
46:21It's called SAPI S API SAPI and basically you can do everything that I did in the web portal through your terminal using the CLI tools and you can also use there's also API tool in there
46:35and uh we just released this website so feel free to check it out there's a lots of APIs CLI and once you sign up you have a machine in the remote
46:46uh remote setting you can use this command line and API to uh control that computer and integrate to your own product.
46:57[snorts]
46:58Yeah, I'll skip the live demo.
47:00So the the general vision for this direction is we want we want everyone to have one single agent and this agent can access multiple platforms not just Windows.
47:09So we also have uh uh support on Mac OS.
47:12The NUX is coming soon.
47:14Lux is very uh economically friendly.
47:18is it's pretty cheap uh in terms of our pricing uh like based on our internal evaluation and the interesting part is we also have Android.
47:26So Android is also coming soon.
47:27So I I do hope like in the future you have a couple Android phones in the cloud you have a couple windows and the agent actually can orchestrate among all of the device to doing all different task
47:38uh in there and I want to show some interesting demos for the Android.
47:42So these are all internal uh like evaluation.
47:46So this one is a play in the Angry Bird game.
47:50And the interesting part is actually drag.
47:53Let me see.
47:54Yeah.
47:59Yeah, you actually drag, you know, it has a drag actions.
48:03He drag drag the screen and doing a task in there.
48:07And another demo I feel pretty cool is authenticator.
48:11So you actually go to authenticator and uh it's the same interface inside.
48:17Basically we swap Windows VM to Android one.
48:22So let me make it faster.
48:26Yeah, it goes to this M MFA authentication challenge website and fill out the forms there and then get the token.
48:45Yeah.
48:45And then success.
48:46So, so this is this is one of the problem that people keep asking us like okay like can we do 2FA?
48:51still possible and I do I do see there's a there's some value but we have to use it in with like uh c caution there and this is a example for
49:02I believe the social media on the phone u like you can actually do different kind of task there yeah it's pretty exciting I I feel the Android is a
49:14interesting area that like the current industry hasn't really like explored much you know there there should be a lot of applications that we can do with Android virtual machines.
49:23So basically everything is converged right now.
49:25So I started with autonomous computer and I decompose everything into autonomy and computer and then autonomy becomes two parts the neural part and symbolic part.
49:36Computer become local machines and cloud machines.
49:39So s actually serve for both.
49:41So you can actually like integrate your own machines to be part of the you know the fleet in there and you can also buy our cloud computers and then everything become
49:51site which is released uh in March this year.
49:53I'm pretty excited about this new product and I hope uh that will bring a lot of new opportunities uh u because this is something like I didn't imagine like
50:03two years ago like we work on this technology we don't know much about like what the vertical application we can do but we actually realized okay like for example there's a YC startup
50:13uh saw me in one of the event and they tell me they actually use agents for their own vertical applications so I do see there's a huge opportunity that within like in the Next year
50:24could be in the next year the tech technology will be uh relatively based on the the past estimation the next year will be very interesting to to see what's what's going to happen and then what I can imagine there are a lot of uh new applications
50:39like blossom in in in this this domain in within a year yeah basically to summary we have four uh offering for all the developers side work is the basically the whole interface you can explore you go to the different algorithms in the developer mode
50:58give it a try let us know how it works and we actively developing new technology in there agents simulan which is open sourced and everyone can use it's pretty much free uh free to use and sidebox is the API for the for the virtual desktop
51:11and we just released today so expect some bugs uh do let us know uh and I I'll give you the invitation code uh for uh today is aie EWF
51:232026 and please give it a try.
51:26Let me know uh the feedback.
51:28We do have a credits program like uh happy to collaborate in a diff in different way different capacity and see how to you know like uh make your development better
51:38uh in the future.
51:39All right.
51:40Thank you so much.
51:41Uh I I want to leave some time for questions.
51:44So feel free to uh ask me questions in there and thank you for your patience.
51:49[applause]
51:50I I'll make I'll make sure the photo is posted.
51:53So, let me know if you want to be mentioned in there.
52:01Yeah, feel free to ask any I I'll stay here like for a few minutes.
52:05I understand like uh there's another events going there.
52:10Yeah.
52:11Question memory anyway.
52:15Spatial memory.
52:16Yeah.
52:17for VM how it visually looks that
52:23for now more most of the memory are reduced to text in there yeah and we do have as I mentioned in agent S2 we do have a model separate model so it's actually not a simple wrapper of a large
52:36vision language model so we have other 7 billion open source models sometimes we also like fine-tuning that model and that model is specifically responsible for recognizing the screen and the interesting part is we know the foundation model is progressing so so fast and so good but it's still not very good at understanding the screen.
52:55So that's an interesting yeah interesting conclusion.
52:58Probably it's because the internet doesn't have many like screenshot data uh available.
53:05So so it's it's not available for them to train the models in there.
53:09Um but I can imagine the model get getting better and better.
53:13But it's just like there's always a gap.
53:16If if you have a model that can specifically for your own application, you can fine-tune that model.
53:21is always going to be better than like uh like a general model from big labs in there.
53:28Yeah.
53:28Question
53:29couple questions.
53:30Yeah.
53:44Have you experienced the process?
53:50Yeah.
53:50So, so let me answer answer your first qu first question first and uh the demo I showed is LM right generating the code and this is this is the approach I call a stateless programming.
54:04So all of the past 20 years we all do stateless programming and the neurosyolic approach I call it stateful programming meaning when you every time you run one line of code you look at the environment
54:16look at the state and the state actually change.
54:18So this is I I believe this is a new paradigm for programming in the future.
54:22So program will be more stable in the future and uh we we tried LM and the problem is this problem is essentially multimodel right so accessibility can have some cap capability of you know like some
54:37but it's not it's still the same there's no silver bullet so some people argue you can do computer agent just for screenshots just using screenshots some people say accessibility
54:48is good enough some people say DOM is good enough but in the And after all the evaluation there's no silver bullet on everything.
54:54So you always has a corner cases of one which is not corner case of the second.
54:59So, so in the end I think we are actually actually taking a whimo approach but you have lighter radar and camera at the same time right and some some company they may take the Tesla approach like I I just have camera and yeah but if you look at today's self-driving car
55:16uh we do see a lot of whimo uh running on the road with better reliability there it's just that you have multiple systems that help each other to to make sure they don't make mistakes
55:27like especially fatal mistake you don't want You don't want agent go to your bank and add another zero when you send a check to other people.
55:35Second.
55:36Yeah.
55:36Second question.
55:53Yeah.
55:53So uh the good part for accessibility tree is that information is used to render the screen.
56:01So basically that information has pretty much all of the information that's needed to render a screen and if you can use that information is definitely more reliable.
56:10But there are some uh there still corner cases like the developer of that application can intentionally say this part do not expose in the in a tree.
56:22So, so you can still capture that in a screenshot, but you don't capture in accessibility.
56:27But you can imagine some applications like uh for example, LinkedIn has a lot of B detection there pretty much the I used to tell a joke with my friends like LinkedIn is is the Turing test for AGI.
56:39Yeah.
56:40[laughter]
56:40Because Yeah.
56:41If if LinkedIn doesn't recognize your agent as agent, then you pretty much achieve AGI because you pass the Turing test in because they they're in adversarial setting and uh they have a lot of techniques to detect
56:54and like this kind of accessib could be one of them in there.
56:59Yeah.
57:01So that's why you still need multiple you know combination of the modality.
57:05So yeah, there there's one person I I'll yeah I'll let that person ask question first.
57:12Yeah, thank you.
57:13Yeah, thank you very much.
57:14First of all,
57:17so we're from Germany.
57:21We have a use case in Germany where you take information and then a lot of things with them.
57:36But the question is how reliable is all of that that you have shown in the real world today when you compare it with the healthare sector for example.
57:50So you need very accurate information and very accurate processes
57:56right
57:57what is your opinion on that?
57:59Yeah.
57:59So, so basically the question is so we need a very accurate information.
58:04So basically we need everything to be 100%.
58:06Right.
58:07So even when you do data entry you need to be 100%.
58:10So uh that problem is the biggest problem of the current foundation models but we do have a solution for that.
58:16So it's basically uh as you can see everything is a loop right.
58:22So all the all the fundamental principles everything is a loop.
58:24when you do something you always verify criticize yourself and it's the same for humans.
58:30So when I when I propose some ideas to to my friends they they criticize so I have the loop so I can like reason with less bios in there.
58:38So the the accuracy problem can also solve by this approach.
58:42So we we do have some experiments that I can tell you one solution we did is uh uh whenever you have something you fill out the form you take a form
58:51reverse it and ask the model does that make sense right so ask the model to verify if the mo most of the time the model recognize something wrong in there
59:02because verification is is much easier than generation generation general action and if there's something wrong you take the results back and ask the agent to do the same task again.
59:13It actually improved the performance a lot.
59:16And um another thing you can think about is the accessibility tree like for us to write the code and for the code to be deterministically replayed you can only do in accessibility tree.
59:29You cannot do in a screenshot right because the tree is text can you can do 100% matching there.
59:36So that kind of a 100% text matching is also important to ensure everything is uh reliable.
59:42So in the end the conclusion is you have to have a system.
59:46It's it's not going to be one model.
59:47You put it there and then you can hope that's 100% it's not going to happen in there.
59:52And uh but there's definitely a solution for your use case in there.
59:57Yeah.
59:58Sorry.
59:59Your turn.
60:00Yeah.
60:13How do you kind of keep it?
60:30Yeah.
60:30So, so in the product side we do have a guard rail.
60:32So whenever there's a action that recognize that irreversible action the guaro will say stop there.
60:38So it's another system another system monitoring this agent.
60:41If the agent want to do a certain action it'll basically cut off and say ask for approval.
60:47But I believe your your problem is even more challenging is more at the low level of the operating system level like for example the agent will say open this applications
60:57and you do want this application to be blocked in there right so this requires another layer uh on the OS level that's why we believe have the claw managed uh Windows machine is a better direction because we have we we have the VM
61:12so if you need we can also produce uh the image the VM image for you so So in a VM image you have lots of opportunity to uh change the underlying
61:23you know structure of the operating system.
61:25So you can actually block some of the actions have the white list in there.
61:29So I believe that's the that's the final direction.
61:32Uh we are still early on the VM part.
61:34I we we try to explore that direction and uh like yeah if you're interested like we can also think about some partnership in the future uh on how to have better guard rails.
61:45Yeah, it's definitely like a security security guard is definitely like one important aspect in there.
61:51Yeah.
61:52Thanks.
61:53Yeah.
61:53Just the protection of an irreversible something.
62:02Yeah.
62:02Yeah.
62:03So, so that that's that goes to the same principle that the first line I put in the so basically everything is neurosymbolic.
62:09If you want everything to be reliable, everything needs to be neuros symbolic, you always go to the code first because code can guarantee 100%.
62:17Right?
62:17When the code is confusing, you always go to the flexible agent and then come back come back to the code.
62:24So that's why the code is still super important.
62:26It's the same principle for the design for the guar.
62:30So the guar is also like code first.
62:32there lots of rules happening there and if the rule doesn't capture it goes to agent and then goes to human for verification and then comes back uh it's a it's a design principle that we try to follow in all of the system inside uh even every components uh in the in the system
62:49thanks question
62:50yeah so I mean you know
62:58repeating cycles Does it get more?
63:06Right.
63:07Right.
63:07Right.
63:07So, so that that's the trajectory.
63:09So, so basically we want the token consumption to eventually become zero.
63:15Yeah.
63:15event should become zero and you can see sometimes you can see the code actually start to have more branch
63:22show showing up when you will see different scenarios you see more it's just like human learning right when you see uh you have more data samples then you are
63:32less biased in terms of decision making yeah it's the same for this approach yeah
63:40oh so question has proprietary program that they for legacy programs, right?
64:02On the initial prompt, you have to be very specific because it's never seen this program.
64:06Go to go to this check this box in how do you like especially a long process is that a long promp or do you actually show it how to do something?
64:23Yeah, for for this version you can see there's a prompt and the agent doing the task uh well maybe in a couple months we we also think about it's it's uh it's pretty common in research community.
64:35So basically if you can replicate like people talking about word models right if you can replicate this applications uh that you have better control on not doing if you have guard rails that the the agent is not going to do like bad things and then you can do exploration we call exploration in the environment.
64:51So basically you let the agents do free exploration on the envir environment for different uh like maybe a 100 times and then the agent become more familiar with the environment
65:02and then your basically your prompt will be less you you don't need very accurate problem like whether this button or that button you you can also al also like succeed in terms of like
65:12uh finding the first basically the goal is to finding the first successful trajectory once you found it it can be successful all the And you already know that process you can already tell
65:24the exact process.
65:25Yeah.
65:26Yeah.
65:27Yeah.
65:27Yeah.
65:27Yeah.
65:27Yeah.
65:28For for many companies they call SOP.
65:31So yeah they have internal they have SOP.
65:33So we we actually did a interesting experiment.
65:35We feed one SOP directly to the uh coding agent right and one shot in one shot everything and it's 100% successful directly.
65:47So it's pos is there definitely possible but I I hold the uh the possibility that like everything now is still kind of demo phase.
65:56We we want a technology that's general enough that every customer coming here and we don't have to worry about anything breaks in there.
66:03So there's still lots of corner case that we have to solve going forward.
66:08I remember there's one gentleman raised hand before.
66:15Oh okay.
66:15Yeah.
66:16All right.
66:17All right.
66:17Thank you so much.
66:18Oh, one more.
66:32And actually your opinion on that actually software but you can see like we are actually the pixels certain things at the end.
67:09So this computer don't even see it working on it.
67:28I understood that you want to agent working but is it something you Yeah.
67:42Um uh I I so uh my impression is kind of like a map and then you try to let the agent to walk through the map.
67:51So he can just go if you ask us to read screenshot go there figure out just how we should go like what it doesn't f
68:10yeah yeah yeah so that that's the whole idea of the coding layer so the coding layer basically encode all of the trajectory into the code right and I think I think the problem we try to solve is
68:22uh suppose there's legacy software then it's not going to change then this this approach work but if we talk about like websites right for social media or like different websites
68:33and they actually change and when they change then your path is not going to work in there is there a way for us to actually adapt adapt the agent
68:41uh during that scenario
68:43I mean my answer to that problem was if your main agent again like check it out.
69:03So you have continuous improving.
69:13Yeah.
69:13Yeah.
69:13Yeah.
69:13I get your point.
69:14So, so basically his solution is okay, every time there's a change, I have a swarm of agents doing the exploration in there.
69:21So, I always know the the best uh
69:25Yeah.
69:25the path in there.
69:26You have to like actually
69:30Yeah.
69:30Yeah.
69:30Yeah.
69:31That that's exactly my response to that gentleman.
69:34So, basically the idea is if you have legacy system and you have is the zero cost for exploration, right?
69:41Because for example, it's if it's a bank, then there's a cost for exploration.
69:45I cannot explore for all of the actions.
69:48But if you don't you don't have any like cost for doing exploration.
69:51You just do exploration like put agents in there and like let them try different things and map all of the trajectory there and it's actually a simpler problem to solve.
70:00So basically I I think 80% of the the applications is actually living in the legacy system right now.
70:07And if we have you can like it's it's actually easier problem to the general uh system.
70:13So that can also already capture a lot of economic value in the current industry there.
70:18Yeah.
70:19Thanks for your feedback.
70:20Yeah.
70:21Further furthermore
70:22you think enough data you can signal enough data.
70:31Yeah.
70:31So, so that the question is suppose we have enough data can we actually train uh trend the agents to be very successful.
70:39So there was a there was a lot of discussion around that.
70:44So basically my conclusion is you need more exponentially more money than open eye and anthropobic to do it
70:52[laughter]
70:53because uh vision data is two dimensional tax is one dimensional so we are talking about like something like much bigger right so so you uh if you have if you can give me like
71:06couple trillion dollars we can we can certainly consider that yeah but it's challenging maybe it's possible with the with a new paradigm of but given the current paradigm of deep learning
71:18it's not something that's uh like I don't think it's achievable in the near term right now.
71:24Yeah.
71:25Question final question.
71:29Right.
71:41There's a lot of Right.
71:45How does it do?
71:57Oh, eventually your code will have a lot of eels logic in there.
72:02So, so you can imagine the code is the code is not going to be manageable by humans for sure.
72:07Uh but the code is still be able the agent is still be able to write that code with lots of eels while loop uh complex logics in there and it'll still work
72:19and yeah that that's the eventual goal of this system in there.
72:23Yeah.
72:24But you can also chop it chop it to make a simple problems each time.
72:27But then you have a boundary problem like okay how do you define a smaller problem like what's the boundary?
72:32What's the input and output?
72:34Right?
72:34So sometimes you if you have a simple solution that's that's actually even better.
72:40All right.
72:40Thank thank you everyone.
72:56[music]