WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=hD9-V56FNRI NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/hD9-V56FNRI.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.559 --> 00:00:13.440 Hello everyone. s3 00:00:13.440 --> 00:00:14.639 Good good afternoon. s4 00:00:14.639 --> 00:00:20.400 Uh today uh I will be talking about AI agents are also distributed systems. s5 00:00:20.400 --> 00:00:25.840 So as uh the models have started to become more complex. s6 00:00:25.840 --> 00:00:39.920 Initially the uh LLM models were just text in text out without performing any actions and uh the effect that they can produce was just a wrong model output. s7 00:00:39.920 --> 00:00:56.160 However, with now the capability of agent the agent the rise in agent capabilities where the systems can now talk to external systems uh it has turned into a distributed systems and it is important s8 00:00:56.160 --> 00:01:02.239 to incorporate distributed systems thinking and concepts when building AI agents. s9 00:01:02.239 --> 00:01:07.680 So I will be going over uh that uh in this talk. s10 00:01:07.680 --> 00:01:14.640 So you guys might have uh heard about incidents being caused by a AI agents. s11 00:01:14.640 --> 00:01:28.560 Uh for instance, the replicate AI agent deleting a production incident production database or Air Canada chatbot basically making an uh an incorrect refund. s12 00:01:28.560 --> 00:01:41.200 And both of these uh incidents or a lot of these incidents could have been prevented uh by good systems thinking when building these uh AI agents. s13 00:01:41.200 --> 00:01:53.040 So for instance for the replet a uh uh uh incident we could have good uh uh we could have robust backups. s14 00:01:53.040 --> 00:01:54.399 We could have scoped authority. s15 00:01:54.399 --> 00:02:00.719 we should we shouldn't ideally have uh allow AI agents to delete production databases. s16 00:02:00.719 --> 00:02:17.920 Uh moreover for Air Canada chatbot it would have been uh a good idea to have uh authoritative source of truth retrieval so that it's not making uh uh decisions based on stale or incorrect policies. s17 00:02:19.120 --> 00:02:23.599 So let's uh go over the transition from chatbot to production system. s18 00:02:23.599 --> 00:02:38.800 Uh so uh initially when we were in the uh in the uh age where LLMs were just chat bots uh we had prompt in and we were outputting text s19 00:02:38.800 --> 00:02:44.160 there were no side effects the agent was not interacting with any other system. s20 00:02:44.160 --> 00:03:01.920 However, uh due to agentic uh in the agentic era in the agentic revolution, now those agents uh by ingesting prompt can uh run an agent loop, call external services, call tools and also perform state changes. s21 00:03:01.920 --> 00:03:08.800 The architectural boundary now has moved uh way beyond an LLM model. s22 00:03:08.800 --> 00:03:13.200 And the difference is that it can now cause side effects in the outside world. s23 00:03:13.200 --> 00:03:25.519 So when basic when building AI agents, it is important to recognize the external systems that it is talking to, [clears throat] the states that uh it is interacting with s24 00:03:25.519 --> 00:03:31.200 and what credentials does it have and the actions that it can perform. s25 00:03:32.799 --> 00:03:42.560 uh I ideally like to think about it as uh uh AI agents as basically having a probabilistic coordinator. s26 00:03:42.560 --> 00:03:48.879 In distributed systems as well, we used to have services which were coordinating uh multi-step workflows. s27 00:03:48.879 --> 00:03:51.440 However, they were deterministic in nature. s28 00:03:51.440 --> 00:03:56.959 But in the case of AI agent, the AI acts as a probabilistic coordinator. s29 00:03:56.959 --> 00:04:01.760 The amount of action, the kind of actions that it can take can vary quite a lot. s30 00:04:01.760 --> 00:04:18.720 It is not just a decision tree that uh we typically in traditional systems would have mapped out and those uh actions can have severe consequences uh ba uh if they are not confined s31 00:04:18.720 --> 00:04:23.520 by our determinist by having deterministic controls in place. s32 00:04:23.520 --> 00:04:39.040 So it is important to ensure uh that we have deterministic controls in place to ensure that agent or the AI agent is not performing any uh any actions that might be s33 00:04:39.040 --> 00:04:42.000 uh uh problematic. s34 00:04:42.720 --> 00:04:51.680 So uh let's uh discuss the how a typical agent loop might look like. s35 00:04:51.680 --> 00:04:56.400 So at first it might uh do some planning. s36 00:04:56.400 --> 00:05:12.639 Then based on that plan it will it will perform an action and it will then observe the results of those actions and uh it might persist that into some d some data store and then decide what to do next. s37 00:05:12.639 --> 00:05:16.639 Each step in this loop is basically crossing a a boundary. s38 00:05:16.639 --> 00:05:22.160 During planning, it can interact with data sources to retrieve some data. s39 00:05:22.160 --> 00:05:30.639 Uh during action, it can call external APIs, tools, uh databases and perform any actions. s40 00:05:30.639 --> 00:05:42.800 During observation phase, it can perform it can get partial results and basically plan or make subsequent actions based on those partial results. s41 00:05:42.800 --> 00:05:59.280 It can persist incorrect data or uh and uh when deciding it might also decide to uh perform an incorrect action or uh worse it can also do a retry s42 00:05:59.280 --> 00:06:01.039 storm. s43 00:06:01.039 --> 00:06:07.840 So it is very important when building an agent loop to persist every step of the process. s44 00:06:07.840 --> 00:06:20.800 Whatever actions the agent is doing, whatever context it is retrieving, it is important to uh persist that so that if anything fails, the agent is able to recognize where it failed and it can perform s45 00:06:20.800 --> 00:06:23.039 uh a reversible action. s46 00:06:23.039 --> 00:06:25.440 Uh it can perform undo operations. s47 00:06:25.440 --> 00:06:31.120 Similarly, there should be explicit transactions uh identified for each step. s48 00:06:31.120 --> 00:06:36.479 So for instance, if an agent is making a call, if it fails, what it should do? s49 00:06:36.479 --> 00:06:43.280 What should be the transaction to compensate for a uh for a irreversible or unsafe operation? s50 00:06:43.280 --> 00:06:52.479 For instance, if an agent makes sends an email to a a wrong email to a customer, what should it do to compensate for that? s51 00:06:53.600 --> 00:07:06.560 So, uh tool calls are just wrappers around uh external external APIs, databases, cues, uh and so on. s52 00:07:06.560 --> 00:07:21.680 And uh with uh when calling the when making these remote calls, there are some failures that you incorporate uh such as network delays, timeouts, uh you can make duplicate requests s53 00:07:21.680 --> 00:07:26.080 or worse the server side request uh might succeed. s54 00:07:26.080 --> 00:07:30.960 However, the client however the client might be reported an error. s55 00:07:30.960 --> 00:07:38.479 We have we have seen uh instances where uh a data by base might have written the data. s56 00:07:38.479 --> 00:07:52.160 However, due to some other errors, the server might have reported uh uh to us the error and uh with humans in the loop we can make we can basically s57 00:07:52.160 --> 00:07:58.400 perform correct corrective actions based on uh by seeing uh the database and actual source of truth. s58 00:07:58.400 --> 00:08:04.080 But in agent's case, we need to ensure that we have uh we have proper guardrails in place. s59 00:08:04.080 --> 00:08:12.639 So for instance, an agent calls refund customer uh tool call which basically performs a refund to the customer. s60 00:08:12.639 --> 00:08:18.400 The request times out uh that did the refund happen or not? s61 00:08:18.400 --> 00:08:22.319 What will the agent uh infer from that? s62 00:08:22.319 --> 00:08:25.199 Would it retry uh refunding to the customer? s63 00:08:25.199 --> 00:08:30.639 you basically the the timeout does not actually mean that there a failure had occurred. s64 00:08:30.639 --> 00:08:32.000 It means unknown. s65 00:08:32.000 --> 00:08:47.839 And it is important to have uh when designing these tools, it is important to have request ids, item potency keys so that when making duplicate requests, they are not causing duplicate side effects. s66 00:08:47.839 --> 00:08:59.040 uh and the system can always do a status lookup like what the previous request was and what was the status of that so that it is not making side effect s67 00:08:59.040 --> 00:09:04.399 it is not making side effects with duplicate a with duplicate requests. s68 00:09:05.839 --> 00:09:20.720 So [clears throat] AI agents when they whenever they uh uh whenever they uh they face failures they retry the their first uh action is to perform retries. s69 00:09:20.720 --> 00:09:25.440 So it is really important to have item potency baked in. s70 00:09:25.440 --> 00:09:38.640 uh it if a same request is coming in to an external API or the tool it should recognize that this is a duplicate request and ensure that no side effects are being take are taking place. s71 00:09:38.640 --> 00:09:48.160 Moreover, uh we should also prevent uh AI agents to perform retry storms to external APIs because this can cause cascading failures. s72 00:09:48.160 --> 00:10:00.480 uh we we should have max turns budget spend and max parallel calls to prevent uh to uh to ensure that the fan out is not that large. s73 00:10:00.480 --> 00:10:17.040 Moreover, we should have exponential back back off in place to ensure that uh the downstream dependencies are not being uh burdened and we should also have compensation uh operations in place for s74 00:10:17.040 --> 00:10:23.440 uh operations that uh that that can have side effects. s75 00:10:25.279 --> 00:10:43.279 Uh so uh a lot of uh teams when building AI agents think of AI agent context as just a AI cont the context that uh the AI agent has as s76 00:10:43.279 --> 00:10:45.519 uh as just a context. s77 00:10:45.519 --> 00:10:59.920 However, when that context can influence an action, it's a state and that state can become stale that can conflict with the authoritative data or corrupt future actions that the agent might perform. s78 00:10:59.920 --> 00:11:05.360 I like to classify it into two different types of uh memory that the agent has. s79 00:11:05.360 --> 00:11:16.079 First is the short-term memory which is the jet thread uh that the agent has uh the which is tied to a single execution thread and the second is the long-term memory. s80 00:11:16.079 --> 00:11:25.279 It can be project files uh system prompts uh databases that it interacts with the cache layer and so on. s81 00:11:25.279 --> 00:11:42.160 It is important to uh to to decide what will be the source of truth when these uh different data sources have conflicting information and we should ideally treat memory as a cache s82 00:11:42.160 --> 00:11:48.560 which uh can be invalidated which can have provenence attached to it. s83 00:11:48.560 --> 00:12:03.040 So for instance whenever a data store or a database is updated or the source of truth is updated we in we invalidate the context or the memory that the agent has to ensure that it is not making actions based on the s84 00:12:03.040 --> 00:12:06.399 uh incorrect or stale data. s85 00:12:08.160 --> 00:12:22.320 So usually these agents perform multi-step actions and uh the agent can succeed on uh on uh on the first couple of steps and then it fail. s86 00:12:22.320 --> 00:12:31.200 Uh it is important to reverse the entire transaction that was performed and these can uh then can cross system boundaries. s87 00:12:31.200 --> 00:12:41.519 So for instance, an agent can update an internal ticket uh send an email to a customer and fail to update the CRM. s88 00:12:41.519 --> 00:12:51.200 We need to figure out what is the uh correct compensation operation when it when it hits that failure. s89 00:12:51.279 --> 00:13:02.320 So for instance uh as I mentioned earlier that uh it improperly uh it improperly sends an incorrect email to the customer. s90 00:13:02.320 --> 00:13:17.440 It is important that the compensation operation is defined for the AI agent to ensure that it is sending an uh uh an apology email to the customer or any or or any email or an email that is correcting that mistake. s91 00:13:19.040 --> 00:13:31.120 So uh a the AI agent basically runs in a loop and uh whenever uh like it can it can do multiple calls. s92 00:13:31.120 --> 00:13:39.120 It can it can have a retry uh retry loop that it can run based uh whenever it fails. s93 00:13:39.120 --> 00:13:53.120 So it is important to have uh circuit breakers whenever it is making making external calls uh to ensure that the uh that the that it is not uh burdening the downstream s94 00:13:53.120 --> 00:13:54.399 system. s95 00:13:54.399 --> 00:14:03.600 Uh for instance if a downstream is unhealthy there should be system break uh circuit breakers in place that prevents AI agents to call call that dependency. s96 00:14:03.600 --> 00:14:14.000 Moreover, it also prevents cascading failures when for instance the downstream dependency is uh unhealthy or uh is saturated. s97 00:14:14.000 --> 00:14:17.440 It is also important to assign rate limits and budgets. s98 00:14:17.440 --> 00:14:27.600 Uh an agent can uh go over uh can run your cost uh if it's not assigned proper budgets and rate limits. s99 00:14:27.600 --> 00:14:35.199 it will uh keep retrying and try try to uh try to solve the problem that if it if it's facing. s100 00:14:35.199 --> 00:14:46.720 So it is important that it is uh that we have uh set up max turns, max parallelism, max spend uh to ensure that the model is not uh uh not s101 00:14:46.720 --> 00:14:51.760 uh crossing the uh the budget boundary that we have set. s102 00:14:52.240 --> 00:15:09.040 Moreover, uh ideally uh usually whenever we are building AI agents, uh we usually try to give all the permissions that it can have to ensure that it has all that it can perform perform the task that we have. s103 00:15:09.040 --> 00:15:24.079 That's the that's the uh first uh uh thing that we have that that's the first step that we take usually that to give the AI agents all the uh cred all the uh privileges s104 00:15:24.079 --> 00:15:32.720 to perform any actions like for instance if it's interacting with the database we just give it all uh the readr access to the entire table. s105 00:15:32.720 --> 00:15:38.639 However, uh it is important to give scoped credentials to it. s106 00:15:38.639 --> 00:15:45.040 There should be separate read and write permissions and there should be allow list for the tools that it can call. s107 00:15:45.040 --> 00:15:51.440 A harmless model can become dangerous when it can perform unsafe operations. s108 00:15:51.440 --> 00:15:57.600 Moreover, uh a human approval shouldn't be tied uh to a blanket approval. s109 00:15:57.600 --> 00:16:04.560 It should be tied to uh action, timestamp, actor and expiration. s110 00:16:04.560 --> 00:16:17.600 So for instance, if a user has given uh an approval to approve a $30 refund, it shouldn't turn into a subsequent approval for $300 refund. s111 00:16:17.600 --> 00:16:30.480 It is important that whenever an approval is given, it should be tied to the particular parameters that it was uh asked for. s112 00:16:31.759 --> 00:16:42.959 So, uh observability is an important requirement when building AI agents because uh and logs are not enough. s113 00:16:42.959 --> 00:16:51.360 Teams need to reconstruct when an agent failed, what happened, what information was was it reacting to and why it failed. s114 00:16:51.360 --> 00:16:59.519 And logs alone are not enough to uh for an agent to uh for teams to determine that. s115 00:16:59.519 --> 00:17:09.760 It is important to trace the model that was called, the prompt that was uh that was uh given to it and uh also the tool calls that were made s116 00:17:09.760 --> 00:17:29.839 uh the request uh that was made, the response from the tool, the errors that it got, the retrieved context, what the agent was uh was the the retrieved information that the agent was reacting to, the rights that it made, and the approvals that it got and so on. s117 00:17:31.360 --> 00:17:40.160 So uh I would like to uh end with uh the idea that yes model capability matters. s118 00:17:40.160 --> 00:17:48.480 Having good models uh improves the uh likelihood of it making uh correct operations. s119 00:17:48.480 --> 00:17:50.799 Smarter models reduce mistakes. s120 00:17:50.799 --> 00:17:55.760 It uh it uh improves the capability that the model has. s121 00:17:55.760 --> 00:18:02.720 However, it cannot eliminate network failures, stale data or adversarial input. s122 00:18:02.720 --> 00:18:14.480 It is important when building this architecture, we also reason about can we bound, observe and recover from actions performed by the AI agent. s123 00:18:14.480 --> 00:18:29.840 It is important to have tool contracts in place to ensure that uh it is only allowed to make uh operations that it is uh given that it is provided the contract and the contracts are clearly establishing the request and response s124 00:18:29.840 --> 00:18:38.799 uh response uh response types uh the schema and all these tools have item potency baked into it. s125 00:18:38.799 --> 00:18:46.400 so that uh when repeated requests are sent in uh it is not causing unsafe operations to be retried. s126 00:18:46.400 --> 00:18:53.600 Moreover, there should be source of truth decisions made uh when there are conflict conflicting uh memory states. s127 00:18:53.600 --> 00:19:08.720 It is important uh for the agent to realize this is the source of data that it should rely on and we should have re retry policies uh like rate limits set in to ensure that the agent is not uh retrying s128 00:19:08.720 --> 00:19:11.200 uh ext uh aggressively. s129 00:19:11.200 --> 00:19:15.600 Moreover, uh permissions should be set up. s130 00:19:15.600 --> 00:19:18.960 There should be traces and recovery paths. s131 00:19:18.960 --> 00:19:28.400 So when building AI agents, we should also ask what the system lets it do when it is wrong. s132 00:19:28.400 --> 00:19:30.559 Thank you. s133 00:19:46.218 --> 00:19:48.218 [music]