WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=GJX19pNhmSw NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/GJX19pNhmSw.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:13.519 --> 00:00:14.000 Okay. s3 00:00:14.000 --> 00:00:15.759 Um, good morning everyone. s4 00:00:15.759 --> 00:00:20.480 So, um, I'm Tisha and I have Sushim with me as my co-presenter. s5 00:00:20.480 --> 00:00:20.800 All right. s6 00:00:20.800 --> 00:00:26.080 So, we'll be talking about the most expensive question in AI today. s7 00:00:26.080 --> 00:00:40.559 I think a lot of you would have come across the scenario that um you know when you opened an AI bill like through your agent workflows um you couldn't actually trace back where that bill was actually coming from right s8 00:00:40.559 --> 00:00:53.920 and um and I don't think that's a problem right now because right now the industry is valuing you know um token maxing that is like spending the most amount of tokens for exploration s9 00:00:53.920 --> 00:01:05.280 for all of those purposes And um people are proud to call themselves token billionaires and um I think that's all right but this talk is you know the shift from token maxing s10 00:01:05.280 --> 00:01:24.159 to value maxing you know how do we get there and um we'll talk about it from this question um who spent all the tokens and um if anyone spent all the tokens there has to be value associated with this right and that s11 00:01:24.159 --> 00:01:28.400 is um the talk about. s12 00:01:30.400 --> 00:01:30.960 All right. s13 00:01:30.960 --> 00:01:47.600 Now in order to minimize the gap you know from token maxing to value maxing we'll kind of see we'll observe the patterns which the like the existing um u like the past software evolution eras had s14 00:01:47.600 --> 00:02:00.799 like for instance when we talk about the SAS era the interface was UI and the control was in the form of usage caps right like or the seat limits or tier based policies s15 00:02:00.799 --> 00:02:06.159 Now when we moved on to the cloud era, the control surface again changed. s16 00:02:06.159 --> 00:02:14.640 The model became pay as you go and the control moved like in the form of autoprovisioning and you know autoscaling policies. s17 00:02:14.640 --> 00:02:29.520 Now we are in the agentic era right and um now how the cost is calculated here is in the form of model calls right like u how like the s18 00:02:29.520 --> 00:02:43.120 code calls your model but what we've observed is that there isn't a proper control plane in place for that like we do have control plane in place for you in place as model gateways where s19 00:02:43.120 --> 00:02:58.800 they're um are hard caps or there is model routing to downgrade the model but the part like where the code you know calls the model that um is what we'll be talking about s20 00:02:58.800 --> 00:03:14.239 uh today and um we also you know see um like in the last year we've seen a lot of unbounded consumption happening like um if you've read the news. s21 00:03:14.239 --> 00:03:31.599 There was news about the like the uh AI budget for Uber getting exhausted within 4 months and um there were companies who like who ran into you know like hundreds of millions of dollars within just months or days and like there were a lot of s22 00:03:31.599 --> 00:03:46.640 um like other news in place as well where like these runaway loops um led to a very like massive increase in the cost and there wasn't proper mechanisms to control it. s23 00:03:47.519 --> 00:03:54.879 Um so when we see all of this the first thing that comes to our mind is is there a tool or is there a product to save us? s24 00:03:54.879 --> 00:04:07.840 But uh we'll instead talk about the first principles of how you know we can design a system which is actually true enough to solve the problem from the very root. s25 00:04:07.840 --> 00:04:12.319 So for that let's um like dive onto the principles. s26 00:04:12.319 --> 00:04:16.400 First of all, let's talk about token being the unit of cost. s27 00:04:16.400 --> 00:04:16.720 Right? s28 00:04:16.720 --> 00:04:19.519 We are charged in terms of token. s29 00:04:19.519 --> 00:04:24.479 So the now we have to see value also in terms of token. s30 00:04:24.479 --> 00:04:25.680 Right? s31 00:04:25.680 --> 00:04:32.960 Next um we all know that cost is created at the LLM like the model call boundary. s32 00:04:32.960 --> 00:04:50.639 Um so that is what we'll have to track and if we don't have proper attribution like if we don't know what agent want run made that particular call we we can't you know control it right we we just know the like the broad s33 00:04:50.639 --> 00:04:55.360 uh picture of what went wrong but we don't we can't you know trace it back or narrow it down. s34 00:04:55.360 --> 00:05:13.120 So that is why attribution is a very important element to have and um like once you know which particular run or which particular agent is actually you know attributing to the cost you should have proper policies in place to actually stop it. s35 00:05:13.120 --> 00:05:29.840 Like um let's take example that um if you have a you know um a loop which is you know running um very excessively and which is not required or you know if your context is growing very out of range. s36 00:05:29.840 --> 00:05:42.880 you should have in place policies which can um like solve that particular thing there and there instead of halting that and if um and as the last resort only a like a halting or a s37 00:05:42.880 --> 00:05:45.039 should happen from a budget cap. s38 00:05:45.039 --> 00:05:46.720 So these are the first principles. s39 00:05:46.720 --> 00:05:56.000 Now let's see how we can you know define an ideal uh platform on top of that from these principles which we talked about. s40 00:05:56.000 --> 00:05:56.479 All right. s41 00:05:56.479 --> 00:06:08.960 Uh so one thing which is very important that which matters here is that u when we talk about um like the um existing frameworks for token ops or for token management s42 00:06:08.960 --> 00:06:16.880 most of them are at the um like u basically monitor the model uh request. s43 00:06:16.880 --> 00:06:27.759 they like they are like model gateways which will u you know u basically um do like model routing or hard budget capping. s44 00:06:27.759 --> 00:06:35.759 But what we need right now is something which you know um like monitors you at the run instead. s45 00:06:35.759 --> 00:06:48.560 Like um if you see we need something uh which can control the loop between like the agent call between the tool um and the agent. s46 00:06:48.560 --> 00:07:02.160 something you know which can um uh see or control the the spawning of multiple sub aents happening from a one main agent or um like something which can control the growing of context. s47 00:07:02.160 --> 00:07:06.639 So like that is the need of the right and that is what we need. s48 00:07:06.639 --> 00:07:20.720 So for all of this um we like uh kind of are proposing a platform which first of all um has a cumulative budget across like the uh like the s49 00:07:20.720 --> 00:07:38.080 attribution runs which happened and then where enforcement actually happens in call path rather than um you know a separate thing like for example if something goes wrong if your like if your context is just growing heavily. s50 00:07:38.080 --> 00:07:47.120 Then like in place compaction should happen or like in place caching or something like that should happen. s51 00:07:47.120 --> 00:07:59.360 And um after that if like after basically exhausting the list of all in place policies only like uh the budget cap should happen at the very last. s52 00:07:59.360 --> 00:08:02.560 Um so that is something which we are proposing. s53 00:08:02.560 --> 00:08:13.199 But um if you look at the landscape today, if you see the uh like the uh tools like um this light LLM, port key, cloudflare, all of those they happen s54 00:08:13.199 --> 00:08:30.879 at again the request level right um like if you see like halting is there, routing is there for some of those but all of this again is at a request and you can't control the cost at the uh request layer s55 00:08:30.879 --> 00:08:34.080 uh at the model layer, Right. s56 00:08:34.080 --> 00:08:47.120 So this is the missing piece which is you know the u basically navigating it at the um you know the model the agent run layer. s57 00:08:47.120 --> 00:08:58.880 So for that we have token ops which is uh you know a runaway token governance for AI agents and u this is the uh architecture for that. s58 00:08:58.880 --> 00:09:07.920 So first of all one thing I would want to highlight is the like the intentional design decision we took here was an out ofbound plane. s59 00:09:07.920 --> 00:09:11.519 So it doesn't interfere with your code at all. s60 00:09:11.519 --> 00:09:16.959 Um so if you see here that out of the bandound plane has three modules which I'll be talking about. s61 00:09:16.959 --> 00:09:18.560 The first one being instrumentation. s62 00:09:18.560 --> 00:09:31.680 It is a common observability layer where you know you'll u like uh have u like the basic telemetry the open telemetry the cost in microns and um like the s63 00:09:31.680 --> 00:09:52.800 um like enrichment layer and basically um the uh attribution like what caused that uh like particular run and then there is um obviously accounting where you'll basically accumulate it in a kind of a ledger s64 00:09:52.800 --> 00:09:55.440 like the total runs which are happening. s65 00:09:55.440 --> 00:10:01.440 And finally we have this enforced layer which has uh two main purposes. s66 00:10:01.440 --> 00:10:13.360 one is steering it um through the policies which we've defined which I think will cover later and um then we have halt in place as the you know final s67 00:10:13.360 --> 00:10:26.800 um like u final thing if um you know your budget is getting exhausted so yeah that is there now when we again look at the landscape this kind of will s68 00:10:26.800 --> 00:10:46.240 solve a lot of problems um which kind of happened uh when we like look at the previous um tools or products there because uh it is at happening at run and it is you know uh helping you solve the problem from the very root by steering s69 00:10:46.240 --> 00:10:48.320 it in place. s70 00:10:48.320 --> 00:10:49.040 All right. s71 00:10:49.040 --> 00:10:53.920 So uh with this I would like to hand it over to Sashim for the demo. s72 00:10:53.920 --> 00:10:55.920 Yeah. s73 00:10:57.120 --> 00:10:57.760 Oh yeah. s74 00:10:57.760 --> 00:11:00.720 Now I think I should be able to everyone in the back can hear me. s75 00:11:00.720 --> 00:11:01.920 All right, perfect. s76 00:11:01.920 --> 00:11:06.000 So yeah, we have established the principles behind token ops till now. s77 00:11:06.000 --> 00:11:06.720 Right. s78 00:11:06.720 --> 00:11:13.360 Now let's shift gears, talk about the design part of it and uh maybe get into the code and the eventual demo. s79 00:11:13.360 --> 00:11:13.920 Right? s80 00:11:13.920 --> 00:11:20.079 So what I have behind me on the screen is the like bird's eye view of what token ops looks like today. s81 00:11:20.079 --> 00:11:21.760 It's it's three layers. s82 00:11:21.760 --> 00:11:23.440 We'll go left to right and top to bottom. s83 00:11:23.440 --> 00:11:32.160 So on the left most you have your own agent runtime which you're trying to instrument and kind of manage the cost for right the middle layer is what we're calling the bridge that s84 00:11:32.160 --> 00:11:41.760 basically shuffles data between your agent and the control plane and the control plane is where the mind of the system lies right so let's talk about the bridge layer very briefly s85 00:11:41.760 --> 00:11:57.680 if we uh go from top to bottom you have the attribution on top so what we're trying to do here is every agent run that you do it's attributed to some user dimensions so the idea is everything that you do every run of the agent is accounted to some usability or some usage. s86 00:11:57.680 --> 00:11:58.800 This comes in handy later. s87 00:11:58.800 --> 00:11:59.839 We'll talk about it. s88 00:11:59.839 --> 00:12:06.320 Uh the second part which is the boundary annotation that you see this is pretty much the heart and soul of this middle layer. s89 00:12:06.320 --> 00:12:09.839 So the idea behind the boundary annotation is that you take any method. s90 00:12:09.839 --> 00:12:11.360 It doesn't matter what framework you're using. s91 00:12:11.360 --> 00:12:14.320 You might be using uh let's say lang chain lang whatever. s92 00:12:14.320 --> 00:12:17.040 If you have a method you can annotate it with boundary. s93 00:12:17.040 --> 00:12:19.920 What this annotation is going to do is it's going to do two things. s94 00:12:19.920 --> 00:12:27.120 First it's going to track the input and the output and it's going to flight that up to the control layer and record it there as a ledger entry. s95 00:12:27.120 --> 00:12:31.680 Now this will be annotated with the further agent run ID and the other attributes and so on. s96 00:12:31.680 --> 00:12:38.240 The second thing the boundary annotation does is it acts as a channel through which the control plane can push actions down to the agent. s97 00:12:38.240 --> 00:12:40.000 This is where the entire intelligence lies. s98 00:12:40.000 --> 00:12:42.959 So we do not have a single directional highway. s99 00:12:42.959 --> 00:12:51.760 We want the control plane to be able to tweak the behavior of the agent on the fly to ensure that we are able to squeeze in more runs inside our budget cap. s100 00:12:51.760 --> 00:12:54.800 Right now let's say the control plane pushes down an action. s101 00:12:54.800 --> 00:12:56.000 Let's take a small example. s102 00:12:56.000 --> 00:13:03.120 Let's say you have a rag retrieval tool which is generating like 20 chunks every retrieval for every call and that's eating up eating up your budget. s103 00:13:03.120 --> 00:13:07.519 And let's say the LLM is not even using the chunks that are after five because they are just not relevant, right? s104 00:13:07.519 --> 00:13:09.040 They're sorted by relevance. s105 00:13:09.040 --> 00:13:14.160 So let's say the control plane observes this and it wants to limit the output to just five chunks. s106 00:13:14.160 --> 00:13:20.399 So it can push down an action but that action has to be received by boundary and then has to be executed by something. s107 00:13:20.399 --> 00:13:22.880 That is where the third node, the governor node comes in. s108 00:13:22.880 --> 00:13:33.600 The governor knows what actions are allowed on your agent by you as a developer and it receives those actions from the control plane and knows how to apply it in a non-destructive way. s109 00:13:33.600 --> 00:13:35.120 So that's the first three. s110 00:13:35.120 --> 00:13:39.279 The fourth one wrap uh the wrap complete is essentially just a helper method. s111 00:13:39.279 --> 00:13:45.120 So as we know most of the agent providers or the model providers they provide objects rather than methods for their LMS right. s112 00:13:45.120 --> 00:13:49.279 So wrap complete is just another way of applying boundary on objects rather than methods. s113 00:13:49.279 --> 00:13:52.000 Let's shift right to the control plane. s114 00:13:52.000 --> 00:13:54.320 On the control plane the first layer is the segment. s115 00:13:54.320 --> 00:13:58.800 Now this is where the attribution that we talked about earlier comes into picture. s116 00:13:58.800 --> 00:14:01.920 So any dimensions that you float from the attribution layer. s117 00:14:01.920 --> 00:14:14.480 Let's say you have a preview agent that you share with everyone in this room and your agent is floating a dimension saying that cohort is AIE 2026 right so you can create a segment which is a cohort of users s118 00:14:14.480 --> 00:14:31.279 which is based on this tag like dimension being AI 2026 right and you can apply your budgets at this cohort level so you don't necessarily have to restrict everything at an agent level or a run level you can do you can do rollups you can do fine grain or coarse grain control right so that's the segmentation part s119 00:14:31.279 --> 00:14:36.240 of Ledger as I mentioned is just one agent run all the traces in one place. s120 00:14:36.240 --> 00:14:37.279 Then you have budgets. s121 00:14:37.279 --> 00:14:43.680 Budgets are basically just the static thresholds that work across a time window against a particular segment or an agent run. s122 00:14:43.680 --> 00:14:45.040 And then you have actions. s123 00:14:45.040 --> 00:14:47.760 So on the actions part we have broadly two flavors. s124 00:14:47.760 --> 00:14:52.480 First is the halt type actions which basically just kill your agent if it exceeds a budget. s125 00:14:52.480 --> 00:14:56.399 The second part where we are adding value is the steer type actions. s126 00:14:56.399 --> 00:14:58.240 So here we do not kill the agent. s127 00:14:58.240 --> 00:15:07.040 Instead we try to steer the behavior of the agent or the components of the agent to try and fit that particular run within the alerted budget. s128 00:15:07.040 --> 00:15:07.519 Right? s129 00:15:07.519 --> 00:15:10.079 And then the policies layer is where it all comes together. s130 00:15:10.079 --> 00:15:17.600 You basically uh group the budgets the actions and then set your policies against certain segments or agent runs and that is where it executes. s131 00:15:17.600 --> 00:15:18.320 Right? s132 00:15:18.320 --> 00:15:23.440 So moving on uh what changes in your code that is the boundary annotation that we just talked about. s133 00:15:23.440 --> 00:15:25.760 As Disha mentioned earlier this is all out of band. s134 00:15:25.760 --> 00:15:27.120 So you do not have to change your code. s135 00:15:27.120 --> 00:15:29.920 You just have to apply the annotation on the methods that you have. s136 00:15:29.920 --> 00:15:33.839 This boundary annotation will take care of floating all the information up to the control plane. s137 00:15:33.839 --> 00:15:36.000 And uh the control plane lies in your own tenant. s138 00:15:36.000 --> 00:15:39.120 So you do not need to worry about any data leaks or anything. s139 00:15:39.120 --> 00:15:43.680 Then if I talk about the governor, so for the governor, you just have to create an instance. s140 00:15:43.680 --> 00:15:45.519 You just have to pass it your own configs. s141 00:15:45.519 --> 00:15:50.800 These configs will basically declare what sort of actions are allowed for those agents, right? s142 00:15:50.800 --> 00:15:56.720 so that your control plane cannot just willingly do any random things on your on your agents. s143 00:15:56.720 --> 00:16:02.320 So before we move on to the demo, I'll just briefly touch upon the uh test that we're going to use today. s144 00:16:02.320 --> 00:16:04.480 So it's a simple two agent workflow. s145 00:16:04.480 --> 00:16:07.440 We have a research agent which has access to a search tool. s146 00:16:07.440 --> 00:16:08.800 Uh you give it a question. s147 00:16:08.800 --> 00:16:12.720 It's allowed to look up on the web as many times as it wants. s148 00:16:12.720 --> 00:16:19.920 And once it knows that it has all the data, it passes the findings on to the second agent which is a summarizer which creates creates a research report. s149 00:16:19.920 --> 00:16:20.560 Right? s150 00:16:20.560 --> 00:16:24.240 So with that out of the way, let's just quickly walk over to the demo. s151 00:16:24.240 --> 00:16:27.279 So for the demo, we have three different scenarios that we're going to talk about. s152 00:16:27.279 --> 00:16:31.680 For the first one, we're going to run the token ops in what we call preview mode. s153 00:16:31.680 --> 00:16:36.880 So in preview mode, what happens is that all the policies run as is, but the enforcement doesn't happen. s154 00:16:36.880 --> 00:16:46.160 So if you see we ran a particular run over here which completed but we did not see any sort of failures there. s155 00:16:46.160 --> 00:16:52.480 The policies executed but the actions that were associated with those policies were not allowed to be executed. s156 00:16:52.480 --> 00:16:57.199 So we're just going to load the dashboard screen here. s157 00:16:57.199 --> 00:16:57.519 Yeah. s158 00:16:57.519 --> 00:17:00.000 So this is the governance output. s159 00:17:00.000 --> 00:17:00.959 Governance is off. s160 00:17:00.959 --> 00:17:02.240 The run completed. s161 00:17:02.240 --> 00:17:04.559 But in the dashboard you can see the policies have executed. s162 00:17:04.559 --> 00:17:07.520 So you can see the cost budget, the cost guard and so on. s163 00:17:07.520 --> 00:17:08.000 Right? s164 00:17:08.000 --> 00:17:09.439 So this was the first scenario. s165 00:17:09.439 --> 00:17:12.640 For the second scenario, what we're going to do is we're going to turn on the governance. s166 00:17:12.640 --> 00:17:16.240 Now while that is happening, I just want to touch upon why this is important. s167 00:17:16.240 --> 00:17:24.799 So if you want to like include this product into your production agents, you want to have a safe environment or a safe way to firstly put it in your production environment, s168 00:17:24.799 --> 00:17:31.039 test the guardrails, tweak the guardrail, see what's the policies are doing and then finalize the thresholds. s169 00:17:31.039 --> 00:17:31.440 Right? s170 00:17:31.440 --> 00:17:39.760 So this is the second one where we have now enforced the governance and you can see in the dashboard that the pre-all cost cap has exceeded. s171 00:17:39.760 --> 00:17:45.679 So you had a budget allotted for this run but the agent exceeded the budget and it was killed immediately. s172 00:17:45.679 --> 00:17:48.720 So that's the simple circuit breaker sort of a methodology. s173 00:17:48.720 --> 00:17:50.640 So this is the halt behavior. s174 00:17:50.640 --> 00:17:58.640 And now let's see the steer behavior which is the which is where we are trying to add value to this entire cost management scenario. s175 00:17:58.640 --> 00:18:01.440 So this time we're going to run the third the second prompt. s176 00:18:01.440 --> 00:18:07.760 The budget allotted for this one is slightly higher but it's still not high enough for the agent to complete in time. s177 00:18:07.760 --> 00:18:12.160 So what instead happens is there is something called cost guard which kicks in. s178 00:18:12.160 --> 00:18:14.720 This cost guard it takes into account two things. s179 00:18:14.720 --> 00:18:17.919 First how much of your allotted budget have you consumed? s180 00:18:17.919 --> 00:18:20.412 Second what is the velocity at which you're consuming tokens. s181 00:18:20.412 --> 00:18:20.640 [music] s182 00:18:20.640 --> 00:18:30.720 Now based on these two things if it predicts that you're going to run out of your tokens or your allotted budget by the end of the run it's going to inject something into your system instructions s183 00:18:30.720 --> 00:18:40.880 that something could be as simple as hey you're running out of budget so make sure that the LM outputs are more succinct or more summarized right so that is the s184 00:18:40.880 --> 00:18:52.480 way we are doing the steering now the this was a very simple test bench to show you like how this works on a like working code we have also benchmarked it on a couple of open source repos. s185 00:18:52.480 --> 00:18:55.919 So we have benchmarked it on browser use as well as metagp. s186 00:18:55.919 --> 00:19:01.360 Uh we ran it across multiple iterations across stress tests across simple scenarios hard scenarios and everything. s187 00:19:01.360 --> 00:19:08.880 And the results we see are the average spend goes down by almost 78% with token ops enabled with the full policy suit that we have today. s188 00:19:08.880 --> 00:19:16.720 On the completion part when we compare it with throttling just simple throttling your simple throttling is going to kill your agent runs no matter what. s189 00:19:16.720 --> 00:19:17.200 Right? s190 00:19:17.200 --> 00:19:24.799 So with the reduced average spend what you get is you get an uplift in that completion percentage from 67% to roughly 96%. s191 00:19:24.799 --> 00:19:27.520 So that is the value ad that token ops is doing here. s192 00:19:27.520 --> 00:19:30.880 Now this is the policy catalog that we run this benchmark against. s193 00:19:30.880 --> 00:19:32.400 This is what we support today. s194 00:19:32.400 --> 00:19:38.400 We kind of researched what are the different failure modes that are there today out in the wild and tried to cover most of them here. s195 00:19:38.400 --> 00:19:47.600 So you have things across spend management, you have things across context management like context compaction, tool output reduction, you have things across loop detection and progress detection and stuff like that. s196 00:19:47.600 --> 00:19:49.919 So this is the entire set of policies that we support. s197 00:19:49.919 --> 00:19:51.760 And at the bottom you can see the actions. s198 00:19:51.760 --> 00:19:53.840 So as I mentioned earlier, we have two flavors. s199 00:19:53.840 --> 00:19:58.320 You have the uh the halt type actions and then the steer type actions. s200 00:19:58.320 --> 00:20:01.600 So for the steer we can do allow, mutate, inject and so on. s201 00:20:01.600 --> 00:20:03.840 And for the halt, it can be a simple kill. s202 00:20:03.840 --> 00:20:07.200 But this is not the end state that we envision for this. s203 00:20:07.200 --> 00:20:12.559 The end state is we have a lot of data right we have a ledger that is continuously being updated. s204 00:20:12.559 --> 00:20:29.440 So what we want to try is we want to try a self-learning module within the token ops plane within the control plane which can look at this ledger and ask this question hey why or what is the failure mode that I'm still not able to catch and then based on that it can do two things one is s205 00:20:29.440 --> 00:20:45.919 it can enhance it can generate new policies on the fly based on the missing or the still uh runaway costs or it can refine the existing parameters for the existing policies that are there so that the runaway costs are managed more effectively in the future. s206 00:20:45.919 --> 00:20:49.360 So with that I think uh that is all we have for you guys today. s207 00:20:49.360 --> 00:20:53.200 Thank you so much for your time and you can scan this QR code that's the public wiki. s208 00:20:53.200 --> 00:20:55.520 We are updating it almost regularly. s209 00:20:55.520 --> 00:21:03.360 So you can scan this and stay up to date and uh Tisha and I are around so if you guys have any questions or if you want to discuss more about it just let us know. s210 00:21:03.360 --> 00:21:03.919 That's it. s211 00:21:03.919 --> 00:21:05.474 Thank you. s212 00:21:05.474 --> 00:21:07.474 [applause]