Zero Cold Starts: Serverless AI Agents on Akamai Functions — Thorsten Hans

AI Engineer · 22 min · 217 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01[music]
2. 00:12Welcome everybody.
3. 00:13Thanks for joining my talk.
4. 00:14In the upcoming 20 minutes, I want to give you an introduction on how you can build ultraast agents and MCPs with spin and run them globally distributed.
5. 00:24So my name is Tossen.
6. 00:25I work as a senior developer advocate with Ecomi and um I guess all of you have already built either MCPS or agents and they are no different than any other kind of application.
7. 00:38So they obviously demand super fast response times so that we can deliver the best possible user experience.
8. 00:46However, traditional regionbound architectures or infrastructures fall short when you want to address a global audience and they they come with some frictions which lead to the inception of edge native
9. 01:03architectures.
10. 01:04So obviously latency is one of the biggest concerns depending on where you deploy your MCPs or your agents.
11. 01:12It might be fast for a subset of your global audience, but others might experience a way lower latency and resulting in a way worse user experience.
12. 01:25Um, depending on your distributable unit, right?
13. 01:27Even so, if you even either choose containers or VMs, you may experience a cold start delay that might range from several seconds up to multiple minutes, especially when you have to scale
14. 01:42dramatically fast to meet unexpected user demand.
15. 01:46However, we can overcome that obviously uh by overprovisioning the infrastructure.
16. 01:52However, keeping around many many containers or servers and actually not utilizing them results in obviously higher cloud spendings and is not sustainable from an environmental footprint.
17. 02:06So there are higher level solutions uh that we can use to run MCPS or agents but they usually or often come with proprietary SDKs or uh deployable units or have their own way on how you should treat them in order to make them work in a production grade environment.
18. 02:28Once you figured out about those constraints, moving away or migrating your workloads, your agents to another hyperscala to another kind of infrastructure, it's often rather complex and extremely costly.
19. 02:43So, but there's a solution to that and that solution involves a new way of how we as engineers could build backend or serverside applications and this is done by adopting web assembly on the server side.
20. 02:59You can think of web assembly as um as containers relate to virtual machines.
21. 03:05So, they are binaries super tiny in size, easy to distribute, super fast to run and portable.
22. 03:12What I refer to as the edge native AI stack is a combination of a CNCF project called spin and echcomi functions.
23. 03:21One runtime where you can run those web assembly applications.
24. 03:25So let's dive into spin for a minute.
25. 03:28Uh spin as I said is a CNCF project that is vendor agnostic and it allows you to build web assembly applications that are meant to be executed on the server or in the cloud.
26. 03:41Spin itself consists of three major pillars.
27. 03:44So it is a CLI that we as developers use to address all our day-to-day concerns.
28. 03:48So everything you know in the realm of the inner loop.
29. 03:52It is a runtime.
30. 03:53So it's based on WM time um that allows us to run these kinds of applications either locally or on microcontrollers or constrained environments.
31. 04:05Last but not least, Spin gives us language specific SDKs to boost the developer productivity when it comes to day-to-day jobs like for example interacting with HTTP services, integrate interacting with databases
32. 04:18or different kind of things.
33. 04:22Although we have language specific SDKs, spin itself is language agnostic and you can choose from all the languages that are able to be compiled down to web assembly including the web assembly system interface or shortwi
34. 04:37and you can learn more about spin on spin framework.dev.
35. 04:41So scan this QR code and you will end up on the spin documentation.
36. 04:46So spin is the developer tool that we use for building these kinds of applications.
37. 04:52Echi functions is the first truly serverless computing platform that is globally distributed and has zero cold start at all.
38. 05:02So every application that you deploy to Ecomi functions is cold started per request that hits your endpoint and on average we can cold start in less than half a millisecond.
39. 05:13It's fully managed and it's again developer centric.
40. 05:17Your developers deploy the workload.
41. 05:19No knobs to turn, no terraform to write, just a simple oneliner and your app is globally distributed at no extra cost.
42. 05:30Security is um is observed by web assembly's sandbox.
43. 05:34So there's no way a workload could interact with external resources without us as developers explicitly granting those permissions to our workloads.
44. 05:45And to be a full-fledged serverless computing platform at the edge, we need some kind of persistence layer.
45. 05:52So Echami functions comes with a built-in key value store that is again multi-tenant.
46. 05:57You don't have to roll credentials.
47. 05:59It is automatically associated to your app if your intent is to use it.
48. 06:05Again, there is a QR code.
49. 06:07Scan that and you can find out more about Echo My Functions.
50. 06:10Sign up for a trial or a free a free account there and you know explore it.
51. 06:17So these are the technical underpinnings that I want to use in order to build first MCP servers.
52. 06:25So instead of handrolling a new MCP server, I mean we can implement the HTTP protocol or the uh the specified API on top of HTTP to build an MCP.
53. 06:36There is a smarter way of doing that and a smarter way is WMCP.
54. 06:40WMCP again is a CLI that integrates seamlessly with spin.
55. 06:45So what it what this allows us to do is basically to focus just on the business logic and in in the in the realm of an MCP business logic basically means we implement our tools
56. 06:57we implement our resources or prompts and we are good to go everything else all those non-functional requirements that we have to tackle right like for example authenticating users using of 2 and OIDC
57. 07:12that's done by WMCP and we then take readyto-use components off the shelf and compose a final MCP server component together and run that on spin or on a myi functions
58. 07:28and that's what we are going to do right now.
59. 07:30So I have installed wmcp on my machine it's github.com wmcpwasmcp and it gives you a nice little CLI.
60. 07:41So we can do womcp new.
61. 07:43We can choose a language, Rust, TypeScript, uh, Python.
62. 07:48And we can choose from templates.
63. 07:49The default template is me only caring about tools for now.
64. 07:54And let's call this new MCP server hello.
65. 07:56We get instructions.
66. 07:57Hey, move into hello.
67. 07:59Uh, hello.
68. 08:02Then let's open up a new terminal on this side.
69. 08:06CD into hello over here as well.
70. 08:08And let's run npm install because that takes a few seconds.
71. 08:14And let's open this one in the editor and let's do this full screen.
72. 08:20So what do we get over here?
73. 08:22Uh we get a boiler plate that's uh ready to use and those squirrel lines will go away once mp npm install is done.
74. 08:31So all we have to do is we have to express the scheme on which inputs we expect.
75. 08:37We have obviously to provide the list of our tools.
76. 08:40So again we have to be good citizen over here and have to provide precise descriptions so that humans and LLMs know how to handle these kind of tools.
77. 08:51So we list them in a simple array and then there's one ingestion function that's called call tool and depending on uh the the payload that we receive we have to decide which tool we have to execute.
78. 09:05If we jump into handle example tool over here, we can see uh all incoming arguments are validated and if everything's fine, then we basically uh concaten concatenate the received input with a prefix
79. 09:21and return that back to the user.
80. 09:23Nothing fancy.
81. 09:25So let's close the editor over here and follow the instructions.
82. 09:28Next on list is make.
83. 09:30make goes on and takes the JavaScript code, compiles that down to web assembly.
84. 09:36So we end up with a tools web assembly component which is not yet a full-fledged uh MCP server.
85. 09:43Next in line would be where MCP compose.
86. 09:46Let's compose a full-fledged MCP by taking our tools and saying hey please give me an MCP for the sake of this demo.
87. 09:54I don't care about authentication.
88. 09:58Okay, it pulls all the necessary uh other components and composes together an MCP server and gives me instructions on how to run that.
89. 10:06I can do a spin up -f server.wazm and we should see that running on localhost 3000.
90. 10:14Okay, let's bring back the other terminal and let's run np npx the uh mcp inspector over here.
91. 10:22That should fire up in the browser.
92. 10:24There we go.
93. 10:25Uh it's already pointing to localhost 3000 right here.
94. 10:29So let's connect.
95. 10:31Let's list tools.
96. 10:32There's our example tool.
97. 10:33We provide fu as an input parameter.
98. 10:35We run the tool and we should get back foo.
99. 10:39Great.
100. 10:39Nothing fancy cool works.
101. 10:41But that's the local experience.
102. 10:43So how can we take that to become a glo or how can we turn that into a globally distributed MCP server?
103. 10:52Well, instead of globally distributing uh the hello world MCP server, I have a decision lock MCP server and let me open up this one first.
104. 11:02So there is this one is implemented in Rust for the sake of implementing it in Rust.
105. 11:08Um so let's go into lip RS over here.
106. 11:11There is an equivalent class or structure in Rust.
107. 11:15As you can see, we're looking at the call tool function where we branch out and either use one of the CRUD APIs or tools that is provided by my MCP server.
108. 11:26If we dive into the actual implementation um let me no this is wrong.
109. 11:33I want this and then I want to do that uh decision.
110. 11:38RS if we go to the bottom of the file you can see this is slightly different syntax but the same meaning right I have to define my tools and if we go to line 15 or so yes list decisions
111. 11:50you can see over here that I use a store so on my local machine spin recognizes hey toen wants to use the key value store so it automatically creates a key value store using SQLite
112. 12:06on my machine for development time purposes and on ecom myi functions it's leveraging the globally distributed um key value store without me having to configure anything from here it's actually the same story right I have to use that make file and compose the server
113. 12:26however I think it's smarter to update the application manifest and concatenate those tasks so that I can remain in the spin developer experience and basically do a spin build
114. 12:40to compile compose and compile my MCP server.
115. 12:44However, I can also combine this one with spin ecomi deploy.
116. 12:52So this right now composes the server asks me hey how do you want to call that MCP server on a my functions?
117. 13:01I call it ADR MCP.
118. 13:03I allow um my terminal to interact with the service registry.
119. 13:09Now it's taken my Wom file and my application uh manifest generating OCI layers from those files uploading them to Echcomi functions on the server side or on the service side.
120. 13:22We receive those two files and we distribute them across the globe through all the service regions.
121. 13:30Every region has to confirm a that it received the workload that is capable of running the workload.
122. 13:38Then we instantiate the workload once and do a internal health check.
123. 13:43And once that came back for all the service regions with an HTTP 200 then we roll a generic subdomain on your behalf so that you that your MCP server
124. 13:56will be on the public internet.
125. 13:58Obviously you can shield that with other products and services uh like the property manager to shield it behind whatever you want to enforce before uh forwarding users to the MCP server.
126. 14:11So we get that URL.
127. 14:13So we are globally distributed in roughly 60 seconds.
128. 14:17Let's move over to an uh to a client.
129. 14:20I use Z over here but that works with any client that basically supports MCP.
130. 14:26I add a server.
131. 14:29I go to remote over here and let me replace this one with the URL that we received.
132. 14:35Add the server.
133. 14:36And we can see I have five tools already.
134. 14:39I can go here and say view tools.
135. 14:42And these are the CRUD tools that my MCP defined.
136. 14:46So I can go back and chat with my IDE in the context of my project so that it could leverage uh my MCP server.
137. 14:54track a decision to use or of 2.0 Hello for awful.
138. 15:04Okay, so it should now identify my intent and ask permission to run the insert decision tool.
139. 15:10We can see the raw input.
140. 15:12Looks good.
141. 15:12We allow it.
142. 15:13It now calls out to echo my functions has already has already tracked my decision and I can say give me all the decisions please.
143. 15:25Now the intent is to use the list decisions tool and again we are uh yes it might want to read uh details to give me everything that it tracked
144. 15:40or and uh with that we have a globally distributed um MCP server that's routing all the requests to the closest service region no matter where your users are and again you can write those tools or prompts or resources using TypeScript, Python or Rust templates.
145. 16:00Cool.
146. 16:00Uh let's move on with agents.
147. 16:02Um so, oops, nope, this one.
148. 16:06So, for AI agents, the story is no different.
149. 16:09However, instead of using another CLI, uh we can simply pick any kind of SDK that's out there and that's popular.
150. 16:18Um I I think the Verscel AI SDK is pretty nice to use.
151. 16:23So this allows us basically to create a new spin app using TypeScript.
152. 16:28Just bring in the Versal AI dependencies and to build custom agents.
153. 16:34Again, let's move back to the terminal to see how that looks like.
154. 16:38Uh so let's move one up and let's CD into a simple game agent over here.
155. 16:44And what's going on over here?
156. 16:48Obviously, I have created a new spin app using a template so that I can get started pretty quickly.
157. 16:54I'm leveraging a large language model deployed to a Lenode instance with a dedicated GPU.
158. 17:01And all my spin app does it defines the tools.
159. 17:04So, this is a simple game agent for the sake of demonstration purposes that can flip co flip coins that can roll a dice where you can specify how many sides that dice should have.
160. 17:15And over here in the actual HTTP handler, we load the config and we define our agent, you know, pointing it to the old lama instance running on the node.
161. 17:28We give our agent an identity and an instruction on what its intent actually is.
162. 17:34We register our tools and we give it some constraints of of how many steps it might take.
163. 17:40We generate a response and return it back to the collie.
164. 17:44Okay, let's uh go let's do a quit all over here and let's do another spin aka deploy no confirm build and it asks for the name that's the game agent.
165. 17:59So while it is uploading let me explain what happens when we compile a lang an interpreted language.
166. 18:07So during compilation time we load the interpreter, we load your custom JavaScript and then we evaluate the global scope.
167. 18:15This has one advantage that not just the runtime is hot also all the dependencies are loaded and then we take a memory snapshot write that to WM store that on disk meaning
168. 18:27that every web assembly application written in JavaScript is faster than a native JavaScript application when it comes to cold starts and runtime performance.
169. 18:37The only trade-off though is that we have to ship the JavaScript runtime along with our um with our user code.
170. 18:46But we use an optimized JavaScript runtime called stalling monkey which is roughly 10 megabytes in size.
171. 18:52So we get back an URL.
172. 18:55Let me replace this URL over here.
173. 18:58And let's say toss a coin.
174. 19:02And we see um we see a response back.
175. 19:05And if we look at these headers, we can see uh the point of presence where we enter the Ecomi network is Santa Clara and the closest service region for our
176. 19:15AI agent is LAX.
177. 19:18Another interesting header is X envoy upstream service time.
178. 19:22And this tells me and you right now as well that it took 33 34 milliseconds to receive the request to run through the business logic to reach out to the large language model to generate a response and to go all the way back to the user.
179. 19:40Oh, there's an internal server error.
180. 19:42Great.
181. 19:42Awesome.
182. 19:42Let's try this again.
183. 19:44Oh, yeah.
184. 19:45That was an user error.
185. 19:47Sorry.
186. 19:48So, the error was staying 30 cmters in front of the screen.
187. 19:51Um looking at it now we are 866 milliseconds.
188. 19:55Um that is more realistic because we have to go through the LLM to basically see that uh our to our toss resulted in tails.
189. 20:04Let's do that again.
190. 20:05Tails tails tails tails.
191. 20:08Oh, there's a hat.
192. 20:09Nice.
193. 20:10So roughly 820 milliseconds is what we could achieve with a full loop through the LLM by taking our agent and deploying it to a functions.
194. 20:21Okay, so the developer workflow is always the same.
195. 20:24No matter if you build any kind of line of business application, if you build an MCP or if you build an agent, you implement your application using any language that you could compile down to web assembly.
196. 20:38Rust, TypeScript, Python, Go are the most prominent ones these days.
197. 20:42You can test it locally with the spin CLI.
198. 20:46You deploy it using a single command called spin a deploy.
199. 20:49And finally, your you point your clients to your MCP or to your agent.
200. 20:56So final or key takeaways from my from my point are portability.
201. 21:02You can take those spin apps and run them on any Wii compliant runtime.
202. 21:07Echi functions is one runtime and it's the globally distributed runtime.
203. 21:12Everything is based on open standards.
204. 21:14So spinners, CNCF, Wesi, Web Assembly are governed by the bite code alliance.
205. 21:19But ultimately you gain speed, right?
206. 21:21You have fast response times.
207. 21:23Even the time to market for your developers to build MCPs and to build agent goes down because you don't have to roll the infrastructure.
208. 21:31You don't have to implement non-functional requirements.
209. 21:34That's all taken care of.
210. 21:36And in terms of infrastructure, there's no ops.
211. 21:40It's spin aka deploy is all you need.
212. 21:42There are other commands in the box so that you can g gather metrics that you can retrieve logs that you can upgrade and so on and so forth.
213. 21:50But developer experience is our our highest metric when we started spin and while we continue in that direction.
214. 22:01Um we have a new website that's called the Ecomi developer hub where we share stories and experiments and and experiences like these.
215. 22:09So go check out developers.accomi.com.
216. 22:11ecomi.com and if you have any further question find me at the Akcomi booth and with that thank you very much.
217. 22:35[music]