Agents Without Code: Skills, YAML, and Filesystems Replaced Python — Philipp Schmid, Google DeepMind

AI Engineer · 18 min · 186 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:12Hi everyone.
2. 00:13Uh thank you for coming.
3. 00:14I know it's the fourth day, last session before the keynote starts again and we are going to do something fun.
4. 00:20Uh we're going to look into how files are basically replacing Python.
5. 00:24And before we begin, I would like to start with my favorite definition of what is an agent from Simon.
6. 00:30An LLM agent runs tools uh in a loop until it achieves a goal.
7. 00:34And what we are going to do is we are going to build uh the same agent, the same GitHub PR review agent in three different ways.
8. 00:41And we are going to delete code on the way.
9. 00:43Each new version, less code, more files basically.
10. 00:47Um before we begin, I would like to quickly introduce you to the interactions API, which is our new Gemini API.
11. 00:54It's a unified interface for um running models and agents.
12. 00:58So you can use the interactions API to call the Gemini models directly or to call our new agents, which also comes with sandbox.
13. 01:06It supports serverside state management, background execution.
14. 01:09So it's perfectly suited for all that's coming in the next years.
15. 01:13and um the capabilities it's the same API for tool call multimodality understanding multimodality generation so you always have the same interface might look very familiar if you're using other LLM applications we really try to build something for developers which you like to use uh to build and that's something we are going to do so
16. 01:35something little bit different in the interactions API to other LLM applications or APIs is that we moved away from this term based based uh conversation history to steps.
17. 01:46So until I would say a few months ago, most of the applications were really turnbased.
18. 01:51Normally you had a user in input and then a model output, a user input, a model output, which definitely works for normal chat application.
19. 01:58But as soon as you start to build agents, use reasoning model.
20. 02:01We have more than just a user role and a model role, right?
21. 02:04So we have like different inputs, we have different types, we have reasoning.
22. 02:08So we decided to like make a cut, make a change and build something really for agents and that's what you see on the flat steps timeline on the right where you have a user input then you have reasoning you have a function call you have a function result and you no longer need to like abuse the user
23. 02:24role for passing back data from an environment.
24. 02:28So roughly a year one and a half years ago writing agents mostly meant writing a loop in Python.
25. 02:35You needed to define a JSON schema.
26. 02:37You needed to define Python functions.
27. 02:39You needed to look at the output from the LLM.
28. 02:42Need to check if it was a function call or if it was a text response.
29. 02:46And then needed to match it against um the type and then like call the tool look of if you get an error and then like go back and forth and
30. 02:55let's look at some some code example on how this would look and also run it and hope that uh the demo gods are great to us.
31. 03:03So I built or I let Gemini build a basic implementation of this Python loop.
32. 03:09So we have our uh class.
33. 03:11We have a run function which uses the interactions API.
34. 03:15We have all of the weird complex passing with function calling with uh appending the errors checking if we get an error and then we have uh the result again.
35. 03:24And what we need of course for an agent is we also need a system instruction.
36. 03:28So there's a separate file for the system instruction.
37. 03:30Very basic.
38. 03:31QR GitHub PR reviewer and then of course we need tools and for tools we needed to write those um JSON schemas specifications of description exactly define which uh actions the agent can take and then of course we need the implementation
39. 03:47in this case using the the basic uh GitHub API just sending some some requests.
40. 03:53So we can run this um in and basically the main main implementation is a very simple uh input interface and we can say something like hello and yes we get back hey I'm an agent and then we
41. 04:11yes ask it to review a pull request on the Gemini skills repository and what we should see is like the agent should hopefully start soon sending function calls function results function calls function results but it's very limited to yes
42. 04:26uh great it works very limited to the tools we define so if we ask the agent to do something which it does not have the capabilities to it just says hey I cannot do this um which is unfortunate
43. 04:38but that's how we were building agents um raw Python code a lot of files a lot of things which can go wrong a lot of code to manage so what happened afterwards
44. 04:49um or what we we need to do we have like a token generation We have the native function calling and we must execute the loop.
45. 04:56We must handle the tool routing.
46. 04:57We must create a JSON schemas.
47. 04:59We must write the Python code.
48. 05:01We need to execute the Python code.
49. 05:03We need to manage the state.
50. 05:04So there's a lot of things we need to do to get an agent running.
51. 05:07And then we got agent frameworks.
52. 05:10There were many different agent frameworks which abstracted away some of that complexity.
53. 05:14One example here is the ADK framework um where you have an agent class now which handles all of the tool loops, the function calling, the retries, the error handling
54. 05:24and it made it a little bit easier.
55. 05:26We basically removed all of the boiler plate code which we always needed to write for agents put it into a framework and help people build with it.
56. 05:36So back to the demo and um same example.
57. 05:41So we go into the CR2 and what is very interesting if you let me open both.
58. 05:47So we still have our we don't have our agent file anymore.
59. 05:51So the agent went away.
60. 05:53We still have our prompt same system prompt.
61. 05:56We still have our tools in this case also no JSON definitions anymore because those agent frameworks now use the uh signature of our functions to create those JSON schemas on the fly to provide the model.
62. 06:10So let's stop our um agent.
63. 06:13Now let's run our second agent.
64. 06:17Similar interface, similar prompt and we should see a similar expected behavior where we have function calls.
65. 06:25We try to get the PR data.
66. 06:27We try to get the diff, we try to get all of the code we need and it works and we wait for for the agent to yes continue.
67. 06:36But similar difficulty here.
68. 06:38If I ask it like what's the weather in San Francisco um we should get back hopefully a result like hey I cannot do this I don't have access to the weather API which obviously makes sense because we did not define any tool still very unfortunate because we need to be very explicit on what our agent can do
69. 06:59and we all know nowadays that we just want to prompt something and we wanted the agent to do whatever it takes to to achieve that goal.
70. 07:05So what is left for us to do?
71. 07:08What does the framework solve?
72. 07:09The framework solves the turn taking loops, the routing, the execution mapping, the JSON schema creation for like the different function calls, but we still own the Python plumping.
73. 07:19So we still need to write those tools with Python code.
74. 07:22We still need to add specific rules or requirements to like make sure whatever we want the agent to do and we need to provide the environment where all of the tools are running, where we want to host it.
75. 07:35So what comes afterwards?
76. 07:37Afterwards hopefully comes remote agents and at Google IO we launched the anti-gravity remote agent on the Gemini API.
77. 07:45The anti-gravity agent uh is powered by the same agent harness which powers the anti-gravity IDE.
78. 07:51Here the same harness very important does not mean the same agent because the anti-gravity agent is a coding agent at the moment and the uh agent available in the Gemini API is a general purpose agent.
79. 08:01So there might be different system instruction, there might be slightly different tools because the Gemini API already has a Google search tool.
80. 08:08So we use that what we have built and but very importantly it comes with this new environment parameter and this environment parameter here allows the agent to get access to a hosted isolated
81. 08:19cloud sandbox where it can run tools, where it can run bash commands and where it can save files.
82. 08:25And those environments can be um configured.
83. 08:28So you can provide sources and sources can be a GitHub repository, it can be a GCS bucket, it can be inline files and of course very important we want to make sure that those agents are secured
84. 08:40and cannot use our credentials in any way possible.
85. 08:43So we created a network proxy around the um agent sandbox which basically injects the credentials when the agent makes a request from inside the sandbox to outside the sandbox.
86. 08:54So the agent never really sees your credential.
87. 08:56It just knows hey I can call the GitHub API and then on the fly we make sure that it received the correct token which you define and you can also limit which domains the agent has access to.
88. 09:06So if you want to restrict the agent completely on which network access it can or which website it can access you just leave it blank.
89. 09:13By default the agent can access all because I mean it's a hassle if you first need to define where to go.
90. 09:19So we tried to stay simple and of course making an API call is nice but we thought hey people want to reuse their configuration want to reuse their agents.
91. 09:28So we added the agents API where you can define your own custom ID you the same system instruction the same base agent the same base environment and then you can create that agent and then you can use that agent in the same exact way as you use Gemini models or as you use the anti-gravity agent by providing
92. 09:44the ID.
93. 09:44So all of the existing code can be reused with your own custom agent, with your own custom tools, with your own custom uh credentials, environments, whatever you need for it to to run.
94. 09:54So let's look at how this will look for SS code and as a demo.
95. 09:59And okay, now 03.
96. 10:03And what might be very obvious is that we no longer have a source directory.
97. 10:08So the code went away.
98. 10:11We have now an agents M uh folder with an agents MD file with system instructions.
99. 10:17So very similar system instruction.
100. 10:19The only difference here is that we tell the agent, hey, you have access to the GitHub CLI.
101. 10:25So we no longer create specific tools for reading files from a GitHub pull request, for accessing a GitHub pull request.
102. 10:32We just tell the agent, hey, you have a GitHub CLI, you have a bash tool, you have file systems.
103. 10:38try to use it whenever you think it's important.
104. 10:41And since we don't have the CLI installed, we have a very basic bash script in this case which checks, hey, if the GitHub CLI is installed, please use it.
105. 10:48If not, download it and install it on the first turn.
106. 10:51So, we go into our terminal and we run our agent here.
107. 10:56In this case, maybe important I use a stream version because otherwise we would wait like a few seconds and we not get back any we would not get back any anything back.
108. 11:05So same prompt and we should soon see um our function calls and function results coming in.
109. 11:13Yes.
110. 11:14So in this case since we run inside a sandbox the agent first like explores the sandbox to really make sure hey do we have this GitHub CLI installed and then tries to run it.
111. 11:23It did not find it on the first turn.
112. 11:25So it installs it and then we can see the agent doing its work.
113. 11:28And in this case it's not using the predefined function calls.
114. 11:31It's using the GitHub CLI and it's already existing knowledge about how it works.
115. 11:36I have a bash tool.
116. 11:37I have like access to the file system and I do all of that work to see or to like review the the pull request.
117. 11:44Let's wait a little bit.
118. 11:47Okay.
119. 11:48And I think the the amazing part here is like if we ask the same question as before, what's the weather in San Francisco?
120. 11:59We should hopefully see that the agent tries to use ah it uses Google search in this case on 2nd of July.
121. 12:06Let me quickly check.
122. 12:07Yeah, that's today.
123. 12:09And we have around 20° Celsius and it works.
124. 12:12So the agent became more of a general purpose agent and we don't need to like specify all of the tools.
125. 12:18We basically trust the model on understanding hey I have a specific set of very atomic general purpose tools to solve my task or the task for the user.
126. 12:27And if we look at the the code uh for like the the input or like the the sorry the the interface we have our sources here.
127. 12:36So we have the the bash script which install the GitHub CLI.
128. 12:40We have the agents MD file and then we say hey you can use the GitHub API with credentials.
129. 12:47So I want to access or use GitHub credentials in a secure way.
130. 12:50So I created a token for the API and also for github.com since you need both URLs.
131. 12:55one uses is used for the git uh commands.
132. 12:58The other one is used for HTT commands and then domain all is basically hey in addition to the GitHub URLs you can use all of the web but you don't have credentials for it and then it's a it's a simple single API call to the anti-gravity
133. 13:11agent with your or user input with the environment and then also with the previous interaction ID that we keep the multi-turn going and that that's all it takes and it's a single API call on the backend side we start that cloud sandbox
134. 13:24we load the agents MD file and the skills from the environment provided to the model and then the model between the API and the sandbox does all of the the looping calling the function returning the function results calling the function returning the function results and that is all it takes.
135. 13:39So where does it leave us?
136. 13:42We no longer need to execute loops.
137. 13:44We no longer need to do two routing.
138. 13:47We have a serverside conversation and session state.
139. 13:50So we only need to provide new inputs.
140. 13:52The context window and the compaction is also automatically managed by the agent.
141. 13:56So if we continue our conversation at a certain point the context is compacted and we can continue without the need to manage anything and we also get an isolated remote Linux sandbox which we can use to run our code.
142. 14:09So what is still left for us?
143. 14:11We need to define instructions.
144. 14:12We need to define rules behaviors in an agent MD file.
145. 14:16We need to provide capabilities or context and skills MD and we need to own the evils.
146. 14:20So all of the heavy lifting, the infrastructure management, all of the same code which probably every one of us has written of us here like 20 times is no longer needed.
147. 14:30And you can start really building your product instead of like needing to rewrite the same code over and over again.
148. 14:36And very important is like, hey, that's great, but what about extending?
149. 14:40And I think looking into how extending previous agents to like those new agents work.
150. 14:46It's very obvious that previously if we want to do like some kind of security scanning on a pull request, we would need to define or write a Python function.
151. 14:55We would need to understand okay which CLI tools do we need to use?
152. 14:58We need to define a new function schema and then we needed to add it to our tools need to run it and then so there's a lot of things we need to do on on agents powered by files.
153. 15:07We write a skills MD file maybe with some additional information on which CLI tool to use or maybe provide the CLI tool inside the environment and then we extended the capabilities.
154. 15:16we don't need to change our code.
155. 15:17We just provide more files to the agent and the agent decides on what we want to do.
156. 15:22And I like to bring up some very good examples.
157. 15:25So at a engineer in Europe, Cursor did a great talk on how they replaced uh roughly 12,000 lines of TypeScript code with a 200 lines agent files to create something similar.
158. 15:35So they had a very hard-coded code um orchestration for doing git work trees and they were m able to replace it with just a skill and markdown files and there are more I would say bitter lessons of ancient engineering
159. 15:49manos has refactored their harness five times in six months last year langen has rearchitected their open deep research three times a year and then also worsel has removed 80% of their tools to achieve fewer steps faster responses and better accuracy so there's an obvious trend that with better model capabilities,
160. 16:08we can remove orchestration code.
161. 16:11But if your harness is getting more complex as the model improves, you are most likely overengineering your harness.
162. 16:17So if you struggle with model improvements and adding new capabilities which lead to more complexity and more code, you might need to rethink a little bit on how your agent harness looks.
163. 16:28And so where does it end up?
164. 16:30Agents are just files.
165. 16:32We write markdown files to extend capabilities.
166. 16:35Agents can learn from those um can create their own files.
167. 16:39So if you have a session and tell the agent to remember something to take notes of rules of preferences, the agent just writes it to this and then can reuse it in the later session
168. 16:49and you can also externalize context.
169. 16:52So if you have a very long running session and during that session you notice hey maybe I want to additionally work on another feature you can like just write that information that hand off to a file and like tell the agent to later pick it up.
170. 17:05Uh so what are the takeaways?
171. 17:07We should not fight the model like we should stop micromanaging the execution paths provide general tools to the agent and let the model explore reason and discover the right solution.
172. 17:16Own what is yours meaning focus on your domain instructions.
173. 17:20Focus on the workflows.
174. 17:21Focus especially on the evals, define clean tools and verify the outcomes and really build to delete.
175. 17:27Like we have seen in the past many many times, the better the model get, the more code we can remove and the more things we need to change and obviously we all want to benefit from better models.
176. 17:37So what the things for you to get to do on Monday, you can scan that QR code which brings you directly to EI studio where you can immediately try out the anti-gravity
177. 17:47harness.
178. 17:48So you can already start prompting it.
179. 17:50it will start your own custom sandbox.
180. 17:53If not, um, start or create your API key.
181. 17:55We are currently working on a free tier for the API.
182. 17:59So hopefully you can start exploring faster soon and then definitely start building files and skills.
183. 18:05And that's it.
184. 18:06Thank you for for coming.
185. 18:09[applause]
186. 18:24[music]