WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=fjF8EKnxKCU NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/fjF8EKnxKCU.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.480 --> 00:00:13.360 Hi everyone. s2 00:00:13.360 --> 00:00:14.320 Uh thank you for coming. s3 00:00:14.320 --> 00:00:20.000 I know it's the fourth day, last session before the keynote starts again and we are going to do something fun. s4 00:00:20.000 --> 00:00:24.480 Uh we're going to look into how files are basically replacing Python. s5 00:00:24.480 --> 00:00:30.400 And before we begin, I would like to start with my favorite definition of what is an agent from Simon. s6 00:00:30.400 --> 00:00:34.480 An LLM agent runs tools uh in a loop until it achieves a goal. s7 00:00:34.480 --> 00:00:41.200 And what we are going to do is we are going to build uh the same agent, the same GitHub PR review agent in three different ways. s8 00:00:41.200 --> 00:00:43.840 And we are going to delete code on the way. s9 00:00:43.840 --> 00:00:47.440 Each new version, less code, more files basically. s10 00:00:47.440 --> 00:00:54.399 Um before we begin, I would like to quickly introduce you to the interactions API, which is our new Gemini API. s11 00:00:54.399 --> 00:00:58.559 It's a unified interface for um running models and agents. s12 00:00:58.559 --> 00:01:06.320 So you can use the interactions API to call the Gemini models directly or to call our new agents, which also comes with sandbox. s13 00:01:06.320 --> 00:01:09.680 It supports serverside state management, background execution. s14 00:01:09.680 --> 00:01:13.760 So it's perfectly suited for all that's coming in the next years. s15 00:01:13.760 --> 00:01:35.119 and um the capabilities it's the same API for tool call multimodality understanding multimodality generation so you always have the same interface might look very familiar if you're using other LLM applications we really try to build something for developers which you like to use uh to build and that's something we are going to do so s16 00:01:35.119 --> 00:01:46.399 something little bit different in the interactions API to other LLM applications or APIs is that we moved away from this term based based uh conversation history to steps. s17 00:01:46.399 --> 00:01:51.520 So until I would say a few months ago, most of the applications were really turnbased. s18 00:01:51.520 --> 00:01:58.560 Normally you had a user in input and then a model output, a user input, a model output, which definitely works for normal chat application. s19 00:01:58.560 --> 00:02:01.600 But as soon as you start to build agents, use reasoning model. s20 00:02:01.600 --> 00:02:04.960 We have more than just a user role and a model role, right? s21 00:02:04.960 --> 00:02:08.879 So we have like different inputs, we have different types, we have reasoning. s22 00:02:08.879 --> 00:02:24.720 So we decided to like make a cut, make a change and build something really for agents and that's what you see on the flat steps timeline on the right where you have a user input then you have reasoning you have a function call you have a function result and you no longer need to like abuse the user s23 00:02:24.720 --> 00:02:28.160 role for passing back data from an environment. s24 00:02:28.160 --> 00:02:35.760 So roughly a year one and a half years ago writing agents mostly meant writing a loop in Python. s25 00:02:35.760 --> 00:02:37.760 You needed to define a JSON schema. s26 00:02:37.760 --> 00:02:39.680 You needed to define Python functions. s27 00:02:39.680 --> 00:02:42.560 You needed to look at the output from the LLM. s28 00:02:42.560 --> 00:02:46.160 Need to check if it was a function call or if it was a text response. s29 00:02:46.160 --> 00:02:55.440 And then needed to match it against um the type and then like call the tool look of if you get an error and then like go back and forth and s30 00:02:55.440 --> 00:03:03.680 let's look at some some code example on how this would look and also run it and hope that uh the demo gods are great to us. s31 00:03:03.680 --> 00:03:09.440 So I built or I let Gemini build a basic implementation of this Python loop. s32 00:03:09.440 --> 00:03:11.920 So we have our uh class. s33 00:03:11.920 --> 00:03:15.280 We have a run function which uses the interactions API. s34 00:03:15.280 --> 00:03:24.480 We have all of the weird complex passing with function calling with uh appending the errors checking if we get an error and then we have uh the result again. s35 00:03:24.480 --> 00:03:28.000 And what we need of course for an agent is we also need a system instruction. s36 00:03:28.000 --> 00:03:30.720 So there's a separate file for the system instruction. s37 00:03:30.720 --> 00:03:31.360 Very basic. s38 00:03:31.360 --> 00:03:47.440 QR GitHub PR reviewer and then of course we need tools and for tools we needed to write those um JSON schemas specifications of description exactly define which uh actions the agent can take and then of course we need the implementation s39 00:03:47.440 --> 00:03:53.040 in this case using the the basic uh GitHub API just sending some some requests. s40 00:03:53.040 --> 00:04:11.519 So we can run this um in and basically the main main implementation is a very simple uh input interface and we can say something like hello and yes we get back hey I'm an agent and then we s41 00:04:11.519 --> 00:04:26.320 yes ask it to review a pull request on the Gemini skills repository and what we should see is like the agent should hopefully start soon sending function calls function results function calls function results but it's very limited to yes s42 00:04:26.320 --> 00:04:38.320 uh great it works very limited to the tools we define so if we ask the agent to do something which it does not have the capabilities to it just says hey I cannot do this um which is unfortunate s43 00:04:38.320 --> 00:04:49.360 but that's how we were building agents um raw Python code a lot of files a lot of things which can go wrong a lot of code to manage so what happened afterwards s44 00:04:49.360 --> 00:04:56.400 um or what we we need to do we have like a token generation We have the native function calling and we must execute the loop. s45 00:04:56.400 --> 00:04:57.919 We must handle the tool routing. s46 00:04:57.919 --> 00:04:59.600 We must create a JSON schemas. s47 00:04:59.600 --> 00:05:01.360 We must write the Python code. s48 00:05:01.360 --> 00:05:03.199 We need to execute the Python code. s49 00:05:03.199 --> 00:05:04.240 We need to manage the state. s50 00:05:04.240 --> 00:05:07.680 So there's a lot of things we need to do to get an agent running. s51 00:05:07.680 --> 00:05:10.160 And then we got agent frameworks. s52 00:05:10.160 --> 00:05:14.479 There were many different agent frameworks which abstracted away some of that complexity. s53 00:05:14.479 --> 00:05:24.639 One example here is the ADK framework um where you have an agent class now which handles all of the tool loops, the function calling, the retries, the error handling s54 00:05:24.639 --> 00:05:26.720 and it made it a little bit easier. s55 00:05:26.720 --> 00:05:36.160 We basically removed all of the boiler plate code which we always needed to write for agents put it into a framework and help people build with it. s56 00:05:36.160 --> 00:05:41.759 So back to the demo and um same example. s57 00:05:41.759 --> 00:05:47.759 So we go into the CR2 and what is very interesting if you let me open both. s58 00:05:47.759 --> 00:05:51.440 So we still have our we don't have our agent file anymore. s59 00:05:51.440 --> 00:05:53.039 So the agent went away. s60 00:05:53.039 --> 00:05:56.479 We still have our prompt same system prompt. s61 00:05:56.479 --> 00:06:10.000 We still have our tools in this case also no JSON definitions anymore because those agent frameworks now use the uh signature of our functions to create those JSON schemas on the fly to provide the model. s62 00:06:10.000 --> 00:06:13.440 So let's stop our um agent. s63 00:06:13.440 --> 00:06:17.520 Now let's run our second agent. s64 00:06:17.520 --> 00:06:25.280 Similar interface, similar prompt and we should see a similar expected behavior where we have function calls. s65 00:06:25.280 --> 00:06:27.120 We try to get the PR data. s66 00:06:27.120 --> 00:06:36.720 We try to get the diff, we try to get all of the code we need and it works and we wait for for the agent to yes continue. s67 00:06:36.720 --> 00:06:38.240 But similar difficulty here. s68 00:06:38.240 --> 00:06:59.120 If I ask it like what's the weather in San Francisco um we should get back hopefully a result like hey I cannot do this I don't have access to the weather API which obviously makes sense because we did not define any tool still very unfortunate because we need to be very explicit on what our agent can do s69 00:06:59.120 --> 00:07:05.840 and we all know nowadays that we just want to prompt something and we wanted the agent to do whatever it takes to to achieve that goal. s70 00:07:05.840 --> 00:07:08.479 So what is left for us to do? s71 00:07:08.479 --> 00:07:09.680 What does the framework solve? s72 00:07:09.680 --> 00:07:19.840 The framework solves the turn taking loops, the routing, the execution mapping, the JSON schema creation for like the different function calls, but we still own the Python plumping. s73 00:07:19.840 --> 00:07:22.960 So we still need to write those tools with Python code. s74 00:07:22.960 --> 00:07:35.520 We still need to add specific rules or requirements to like make sure whatever we want the agent to do and we need to provide the environment where all of the tools are running, where we want to host it. s75 00:07:35.520 --> 00:07:37.919 So what comes afterwards? s76 00:07:37.919 --> 00:07:45.280 Afterwards hopefully comes remote agents and at Google IO we launched the anti-gravity remote agent on the Gemini API. s77 00:07:45.280 --> 00:07:51.599 The anti-gravity agent uh is powered by the same agent harness which powers the anti-gravity IDE. s78 00:07:51.599 --> 00:08:01.759 Here the same harness very important does not mean the same agent because the anti-gravity agent is a coding agent at the moment and the uh agent available in the Gemini API is a general purpose agent. s79 00:08:01.759 --> 00:08:08.000 So there might be different system instruction, there might be slightly different tools because the Gemini API already has a Google search tool. s80 00:08:08.000 --> 00:08:19.759 So we use that what we have built and but very importantly it comes with this new environment parameter and this environment parameter here allows the agent to get access to a hosted isolated s81 00:08:19.759 --> 00:08:25.120 cloud sandbox where it can run tools, where it can run bash commands and where it can save files. s82 00:08:25.120 --> 00:08:28.720 And those environments can be um configured. s83 00:08:28.720 --> 00:08:40.640 So you can provide sources and sources can be a GitHub repository, it can be a GCS bucket, it can be inline files and of course very important we want to make sure that those agents are secured s84 00:08:40.640 --> 00:08:43.599 and cannot use our credentials in any way possible. s85 00:08:43.599 --> 00:08:54.160 So we created a network proxy around the um agent sandbox which basically injects the credentials when the agent makes a request from inside the sandbox to outside the sandbox. s86 00:08:54.160 --> 00:08:56.640 So the agent never really sees your credential. s87 00:08:56.640 --> 00:09:06.640 It just knows hey I can call the GitHub API and then on the fly we make sure that it received the correct token which you define and you can also limit which domains the agent has access to. s88 00:09:06.640 --> 00:09:13.519 So if you want to restrict the agent completely on which network access it can or which website it can access you just leave it blank. s89 00:09:13.519 --> 00:09:19.200 By default the agent can access all because I mean it's a hassle if you first need to define where to go. s90 00:09:19.200 --> 00:09:28.560 So we tried to stay simple and of course making an API call is nice but we thought hey people want to reuse their configuration want to reuse their agents. s91 00:09:28.560 --> 00:09:44.000 So we added the agents API where you can define your own custom ID you the same system instruction the same base agent the same base environment and then you can create that agent and then you can use that agent in the same exact way as you use Gemini models or as you use the anti-gravity agent by providing s92 00:09:44.000 --> 00:09:44.720 the ID. s93 00:09:44.720 --> 00:09:54.800 So all of the existing code can be reused with your own custom agent, with your own custom tools, with your own custom uh credentials, environments, whatever you need for it to to run. s94 00:09:54.800 --> 00:09:59.680 So let's look at how this will look for SS code and as a demo. s95 00:09:59.680 --> 00:10:03.360 And okay, now 03. s96 00:10:03.360 --> 00:10:08.880 And what might be very obvious is that we no longer have a source directory. s97 00:10:08.880 --> 00:10:11.120 So the code went away. s98 00:10:11.120 --> 00:10:17.360 We have now an agents M uh folder with an agents MD file with system instructions. s99 00:10:17.360 --> 00:10:19.680 So very similar system instruction. s100 00:10:19.680 --> 00:10:25.519 The only difference here is that we tell the agent, hey, you have access to the GitHub CLI. s101 00:10:25.519 --> 00:10:32.880 So we no longer create specific tools for reading files from a GitHub pull request, for accessing a GitHub pull request. s102 00:10:32.880 --> 00:10:38.560 We just tell the agent, hey, you have a GitHub CLI, you have a bash tool, you have file systems. s103 00:10:38.560 --> 00:10:41.120 try to use it whenever you think it's important. s104 00:10:41.120 --> 00:10:48.720 And since we don't have the CLI installed, we have a very basic bash script in this case which checks, hey, if the GitHub CLI is installed, please use it. s105 00:10:48.720 --> 00:10:51.279 If not, download it and install it on the first turn. s106 00:10:51.279 --> 00:10:56.560 So, we go into our terminal and we run our agent here. s107 00:10:56.560 --> 00:11:05.279 In this case, maybe important I use a stream version because otherwise we would wait like a few seconds and we not get back any we would not get back any anything back. s108 00:11:05.279 --> 00:11:13.760 So same prompt and we should soon see um our function calls and function results coming in. s109 00:11:13.760 --> 00:11:14.000 Yes. s110 00:11:14.000 --> 00:11:23.360 So in this case since we run inside a sandbox the agent first like explores the sandbox to really make sure hey do we have this GitHub CLI installed and then tries to run it. s111 00:11:23.360 --> 00:11:25.519 It did not find it on the first turn. s112 00:11:25.519 --> 00:11:28.880 So it installs it and then we can see the agent doing its work. s113 00:11:28.880 --> 00:11:31.680 And in this case it's not using the predefined function calls. s114 00:11:31.680 --> 00:11:36.720 It's using the GitHub CLI and it's already existing knowledge about how it works. s115 00:11:36.720 --> 00:11:37.760 I have a bash tool. s116 00:11:37.760 --> 00:11:44.560 I have like access to the file system and I do all of that work to see or to like review the the pull request. s117 00:11:44.560 --> 00:11:47.360 Let's wait a little bit. s118 00:11:47.360 --> 00:11:48.640 Okay. s119 00:11:48.640 --> 00:11:59.120 And I think the the amazing part here is like if we ask the same question as before, what's the weather in San Francisco? s120 00:11:59.600 --> 00:12:06.560 We should hopefully see that the agent tries to use ah it uses Google search in this case on 2nd of July. s121 00:12:06.560 --> 00:12:07.440 Let me quickly check. s122 00:12:07.440 --> 00:12:09.120 Yeah, that's today. s123 00:12:09.120 --> 00:12:12.639 And we have around 20° Celsius and it works. s124 00:12:12.639 --> 00:12:18.399 So the agent became more of a general purpose agent and we don't need to like specify all of the tools. s125 00:12:18.399 --> 00:12:27.680 We basically trust the model on understanding hey I have a specific set of very atomic general purpose tools to solve my task or the task for the user. s126 00:12:27.680 --> 00:12:36.959 And if we look at the the code uh for like the the input or like the the sorry the the interface we have our sources here. s127 00:12:36.959 --> 00:12:40.639 So we have the the bash script which install the GitHub CLI. s128 00:12:40.639 --> 00:12:47.120 We have the agents MD file and then we say hey you can use the GitHub API with credentials. s129 00:12:47.120 --> 00:12:50.480 So I want to access or use GitHub credentials in a secure way. s130 00:12:50.480 --> 00:12:55.760 So I created a token for the API and also for github.com since you need both URLs. s131 00:12:55.760 --> 00:12:58.079 one uses is used for the git uh commands. s132 00:12:58.079 --> 00:13:11.120 The other one is used for HTT commands and then domain all is basically hey in addition to the GitHub URLs you can use all of the web but you don't have credentials for it and then it's a it's a simple single API call to the anti-gravity s133 00:13:11.120 --> 00:13:24.240 agent with your or user input with the environment and then also with the previous interaction ID that we keep the multi-turn going and that that's all it takes and it's a single API call on the backend side we start that cloud sandbox s134 00:13:24.240 --> 00:13:39.440 we load the agents MD file and the skills from the environment provided to the model and then the model between the API and the sandbox does all of the the looping calling the function returning the function results calling the function returning the function results and that is all it takes. s135 00:13:39.440 --> 00:13:42.160 So where does it leave us? s136 00:13:42.160 --> 00:13:44.720 We no longer need to execute loops. s137 00:13:44.720 --> 00:13:47.440 We no longer need to do two routing. s138 00:13:47.440 --> 00:13:50.000 We have a serverside conversation and session state. s139 00:13:50.000 --> 00:13:52.160 So we only need to provide new inputs. s140 00:13:52.160 --> 00:13:56.720 The context window and the compaction is also automatically managed by the agent. s141 00:13:56.720 --> 00:14:09.040 So if we continue our conversation at a certain point the context is compacted and we can continue without the need to manage anything and we also get an isolated remote Linux sandbox which we can use to run our code. s142 00:14:09.040 --> 00:14:11.040 So what is still left for us? s143 00:14:11.040 --> 00:14:12.880 We need to define instructions. s144 00:14:12.880 --> 00:14:16.160 We need to define rules behaviors in an agent MD file. s145 00:14:16.160 --> 00:14:20.880 We need to provide capabilities or context and skills MD and we need to own the evils. s146 00:14:20.880 --> 00:14:30.079 So all of the heavy lifting, the infrastructure management, all of the same code which probably every one of us has written of us here like 20 times is no longer needed. s147 00:14:30.079 --> 00:14:36.240 And you can start really building your product instead of like needing to rewrite the same code over and over again. s148 00:14:36.240 --> 00:14:40.480 And very important is like, hey, that's great, but what about extending? s149 00:14:40.480 --> 00:14:46.800 And I think looking into how extending previous agents to like those new agents work. s150 00:14:46.800 --> 00:14:55.199 It's very obvious that previously if we want to do like some kind of security scanning on a pull request, we would need to define or write a Python function. s151 00:14:55.199 --> 00:14:58.240 We would need to understand okay which CLI tools do we need to use? s152 00:14:58.240 --> 00:15:07.839 We need to define a new function schema and then we needed to add it to our tools need to run it and then so there's a lot of things we need to do on on agents powered by files. s153 00:15:07.839 --> 00:15:16.079 We write a skills MD file maybe with some additional information on which CLI tool to use or maybe provide the CLI tool inside the environment and then we extended the capabilities. s154 00:15:16.079 --> 00:15:17.519 we don't need to change our code. s155 00:15:17.519 --> 00:15:22.560 We just provide more files to the agent and the agent decides on what we want to do. s156 00:15:22.560 --> 00:15:25.199 And I like to bring up some very good examples. s157 00:15:25.199 --> 00:15:35.839 So at a engineer in Europe, Cursor did a great talk on how they replaced uh roughly 12,000 lines of TypeScript code with a 200 lines agent files to create something similar. s158 00:15:35.839 --> 00:15:49.839 So they had a very hard-coded code um orchestration for doing git work trees and they were m able to replace it with just a skill and markdown files and there are more I would say bitter lessons of ancient engineering s159 00:15:49.839 --> 00:16:08.480 manos has refactored their harness five times in six months last year langen has rearchitected their open deep research three times a year and then also worsel has removed 80% of their tools to achieve fewer steps faster responses and better accuracy so there's an obvious trend that with better model capabilities, s160 00:16:08.480 --> 00:16:11.199 we can remove orchestration code. s161 00:16:11.199 --> 00:16:17.839 But if your harness is getting more complex as the model improves, you are most likely overengineering your harness. s162 00:16:17.839 --> 00:16:28.240 So if you struggle with model improvements and adding new capabilities which lead to more complexity and more code, you might need to rethink a little bit on how your agent harness looks. s163 00:16:28.240 --> 00:16:30.880 And so where does it end up? s164 00:16:30.880 --> 00:16:32.399 Agents are just files. s165 00:16:32.399 --> 00:16:35.920 We write markdown files to extend capabilities. s166 00:16:35.920 --> 00:16:39.839 Agents can learn from those um can create their own files. s167 00:16:39.839 --> 00:16:49.920 So if you have a session and tell the agent to remember something to take notes of rules of preferences, the agent just writes it to this and then can reuse it in the later session s168 00:16:49.920 --> 00:16:52.320 and you can also externalize context. s169 00:16:52.320 --> 00:17:05.120 So if you have a very long running session and during that session you notice hey maybe I want to additionally work on another feature you can like just write that information that hand off to a file and like tell the agent to later pick it up. s170 00:17:05.120 --> 00:17:07.039 Uh so what are the takeaways? s171 00:17:07.039 --> 00:17:16.480 We should not fight the model like we should stop micromanaging the execution paths provide general tools to the agent and let the model explore reason and discover the right solution. s172 00:17:16.480 --> 00:17:20.400 Own what is yours meaning focus on your domain instructions. s173 00:17:20.400 --> 00:17:21.600 Focus on the workflows. s174 00:17:21.600 --> 00:17:27.760 Focus especially on the evals, define clean tools and verify the outcomes and really build to delete. s175 00:17:27.760 --> 00:17:37.520 Like we have seen in the past many many times, the better the model get, the more code we can remove and the more things we need to change and obviously we all want to benefit from better models. s176 00:17:37.520 --> 00:17:47.840 So what the things for you to get to do on Monday, you can scan that QR code which brings you directly to EI studio where you can immediately try out the anti-gravity s177 00:17:47.840 --> 00:17:48.559 harness. s178 00:17:48.559 --> 00:17:50.080 So you can already start prompting it. s179 00:17:50.080 --> 00:17:53.039 it will start your own custom sandbox. s180 00:17:53.039 --> 00:17:55.840 If not, um, start or create your API key. s181 00:17:55.840 --> 00:17:59.039 We are currently working on a free tier for the API. s182 00:17:59.039 --> 00:18:05.039 So hopefully you can start exploring faster soon and then definitely start building files and skills. s183 00:18:05.039 --> 00:18:06.480 And that's it. s184 00:18:06.480 --> 00:18:09.277 Thank you for for coming. s185 00:18:09.277 --> 00:18:11.277 [applause] s186 00:18:24.342 --> 00:18:26.342 [music]