WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=dSg0pu8d6qg NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/dSg0pu8d6qg.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.719 --> 00:00:14.880 All right, let's get started. s3 00:00:14.880 --> 00:00:18.480 Apologies for the delay, but I'm really excited to be here. s4 00:00:18.480 --> 00:00:23.760 I'm Mingshan, VP of engineering focused on AI at Ironclad. s5 00:00:23.760 --> 00:00:29.679 And today I'll be telling you about something that's probably on top of many of your mind. s6 00:00:29.679 --> 00:00:34.480 uh how to control and optimize for your AI token spend. s7 00:00:34.480 --> 00:00:40.160 Can I get a get a quick show of hands that this is a relevant topic? s8 00:00:40.160 --> 00:00:41.760 Okay, awesome. s9 00:00:41.760 --> 00:00:44.320 I appreciate that. s10 00:00:45.920 --> 00:00:50.800 So, we have all heard a few sensational stories from the media. s11 00:00:50.800 --> 00:01:00.879 There's an interesting Amazon story where an employee just created kind of a voluntary dashboard and everyone start tracking their own AI token usage. s12 00:01:00.879 --> 00:01:14.479 I'm not sure there's explicit encouragement from the leadership, but the effect is you know engineers some of the engineers started competing with each other in maximizing their token usage and get to the top of the so-called leaderboard. s13 00:01:14.479 --> 00:01:25.759 There's a similar story from Meta and then another even more sensational story about some companies spending $500 million on cloud oops within a month. s14 00:01:25.759 --> 00:01:32.479 So while these may not be happening in your companies today, the threats, the risks are real. s15 00:01:32.479 --> 00:01:35.119 How do we think about the policies? s16 00:01:35.119 --> 00:01:36.720 How do we measure the cost? s17 00:01:36.720 --> 00:01:40.000 And how do we control and optimize for it? s18 00:01:40.000 --> 00:01:54.799 So one initial learning I want to share is it is really important to have dashboard that track every team every individual's token usage and cost but that should not be positioned as a leaderboard. s19 00:01:54.799 --> 00:02:00.000 We think of the the usage dashboard more as a smoke detector. s20 00:02:00.000 --> 00:02:08.080 If there are local pockets of teams or individuals that don't use much AI token that might be a signal worth investigating. s21 00:02:08.080 --> 00:02:17.120 But beyond that certainly we don't want to create even indirect incentive to maximize the token usage itself. s22 00:02:18.640 --> 00:02:21.120 So how do we think about it then? s23 00:02:21.120 --> 00:02:31.680 First I want to make sure that we position this talk for those of you whose teams have already gone through the hump of getting AI adopted. s24 00:02:31.680 --> 00:02:47.120 If you're still in the initial process of provisioning easy access to your engineers or encouraging the teams and individuals to adopt, then you may not be ready to implement some of the ideas for controlling s25 00:02:47.120 --> 00:02:49.200 and optimizing for cost. s26 00:02:49.200 --> 00:02:50.000 But that's okay. s27 00:02:50.000 --> 00:02:51.920 This could still be a good discussion. s28 00:02:51.920 --> 00:02:56.160 And frankly, we just got over that hump over the last couple quarters. s29 00:02:56.160 --> 00:03:01.840 So this is a very topical subject that every engineering leader I believe is navigating. s30 00:03:01.840 --> 00:03:07.120 So I would love to start that dialogue with you all today to explore the best practices. s31 00:03:07.120 --> 00:03:16.239 Can I get a quick show of hand for those of you whose teams have gone over the initial adoption phase now you are starting to seriously worry about the cost. s32 00:03:16.239 --> 00:03:18.879 Okay, I see roughly half of the hands raised. s33 00:03:18.879 --> 00:03:19.680 Thank you. s34 00:03:19.680 --> 00:03:32.560 So let's talk about then how we can control and how we can optimize what we call the trusted throughput as a kind of a proxy metric as a way to measure your ROI. s35 00:03:32.560 --> 00:03:42.959 But before that, just for those of you who are in the process of still increasing adoption, one lesson we learned is to after the kind of the top down leadership push s36 00:03:42.959 --> 00:03:52.480 is to sit down with the individual teams and uh the individuals who may be resistant or struggling with adoption, understand where they came from. s37 00:03:52.480 --> 00:04:03.760 For example, there are some legitimate concerns that I heard, you know, people say, "Hey, I used to really take pride and joy in handcrafting the code and now a lot of the joy and the pride s38 00:04:03.760 --> 00:04:08.879 got taken away and replaced with me reviewing AI slop code, right? s39 00:04:08.879 --> 00:04:18.160 So that doesn't sound like a very satisfying professional activity and that's where we need to kind of dig down and understand what are still the kind of the high impact s40 00:04:18.160 --> 00:04:26.880 and uh engineering tasks technical work that we can help our engineers continue to grow themselves in the era of the AI. s41 00:04:28.479 --> 00:04:40.479 So I wanted to share with you a bit more about what we at ironclad does and there's an interesting connection actually within how we think about optimizing for engineering AI token usage. s42 00:04:40.479 --> 00:04:45.600 So, ironclad is a legal contracting AI companies AI company. s43 00:04:45.600 --> 00:04:59.840 We build AI features and native AI products to help lawyers, procurement and other business users move forward new contracts, move them forward faster with controlled risk. s44 00:04:59.840 --> 00:05:06.000 What that means is building trust is the number one priority with our AI product features and products. s45 00:05:06.000 --> 00:05:14.400 And for the prior speak uh speaker speaker, she did a wonderful job telling you about the importance of trust and how to build it in their domain. s46 00:05:14.400 --> 00:05:25.600 In our ironclad product domain, it often means lawyers especially, but other persona as well taking the time to kind of test the water and see if they can trust the AI output. s47 00:05:25.600 --> 00:05:36.479 For example, they may feed our conversational search a set of contracts they are firmly familiar with and they run a search and see if the output is towards the expectation. s48 00:05:36.479 --> 00:05:49.919 If so, they may expand on searching for things they don't know about or apply other workflows using AI to solve other things like redlinining the contract um and you know finding anomalies and so on. s49 00:05:49.919 --> 00:06:05.759 And so similarly using AI and making sure AI is delivering high engineering value also involves a you know a sequence of steps in gaining trust from the internal engineers the leadership as well with as uh with our customers. s50 00:06:05.759 --> 00:06:14.960 So this is the focus of our talk today and this probably will not come as a surprise here. s51 00:06:14.960 --> 00:06:21.280 The goal is not to minimizing or not even necessarily to reduce token spend. s52 00:06:21.280 --> 00:06:24.560 So here we kind of use the word it's not about austerity. s53 00:06:24.560 --> 00:06:29.280 It's about further improving the ROI of the token spend. s54 00:06:29.280 --> 00:06:30.880 So how do we do that? s55 00:06:30.880 --> 00:06:35.919 Here we propose um a concept we call trusted throughput. s56 00:06:35.919 --> 00:06:49.039 So the trusted throughput comes from having the code reviewed and validated internally and ultimately validated in customer uh in customer deployments. s57 00:06:50.639 --> 00:07:01.039 So how do we go and how do we think about controlling the cost and uh measuring and in turn optimizing the ROI? s58 00:07:01.039 --> 00:07:08.400 The first step is I'm pretty confident that all of you your teams who have been adopting AI have been measuring the cost. s59 00:07:08.400 --> 00:07:16.800 If you're using a single tool like claw code or codeex then you tend to get very rich analytics from the vendor's dashboard already. s60 00:07:16.800 --> 00:07:27.759 If you're like us who use a combination of these different coding tools then we basically use AI to build simple dashboards and pipelines to extract such vendor data. s61 00:07:27.759 --> 00:07:29.840 So we can kind of crossorrelate them. s62 00:07:29.840 --> 00:07:39.360 Then we can break it down, aggregate and then break down by per team, per individual, what is their cost usage across all of these uh tools. s63 00:07:39.360 --> 00:07:42.400 So that's the first step for measuring cost. s64 00:07:42.400 --> 00:07:52.160 Now one pitfall I have seen and we wanted to caution everybody is to then jump from measuring cost to start reducing or minimizing the cost, right? s65 00:07:52.160 --> 00:07:53.199 Cutting cost. s66 00:07:53.199 --> 00:07:55.199 We think that is premature. s67 00:07:55.199 --> 00:08:00.319 Instead, the other important side of the equation for ROI is to measure value. s68 00:08:00.319 --> 00:08:03.759 How much value are we getting from burning the tokens? s69 00:08:03.759 --> 00:08:12.720 Once we can measure the cost and value side, we understand ROI and then to improve ROI, we want to find and then fix the bottlenecks. s70 00:08:12.720 --> 00:08:23.680 In the next couple slides, I'm going to introduce two new bottlenecks we identify in this whole new software development life cycle where code generation now becomes abundant thanks to AI. s71 00:08:23.680 --> 00:08:31.039 But the pressure is now getting pushed down to code review and continuous integration CICD the merging the code. s72 00:08:31.039 --> 00:08:42.880 So we'll talk about that and finally we'll put together these ideas into a pra pragmatic framework of how we think about optimizing the ROI and thus the leverage in using AI. s73 00:08:45.279 --> 00:08:45.920 Okay. s74 00:08:45.920 --> 00:08:51.920 So this is kind of just a slide in building or using the vendor dashboard to measure the cost. s75 00:08:51.920 --> 00:09:04.080 And again we want to caution that here the main goal for regularly reviewing the dashboard is to see a if there's still adoption gap within individual pockets of teams or the individual engineers s76 00:09:04.080 --> 00:09:16.560 and b if there are any sudden surprises in kind of the usage burst and if so understand what's been happening if they're legitimate and then also compare teams contextually. s77 00:09:16.560 --> 00:09:17.600 So this is important. s78 00:09:17.600 --> 00:09:26.959 We don't control just the AI usage per se because for example a platform infrastructure team the way they use AI and the way they get value may be different from the UI team. s79 00:09:26.959 --> 00:09:30.240 So we need to take the context into consideration. s80 00:09:30.240 --> 00:09:34.160 All of such review analysis is to help us extract learnings. s81 00:09:34.160 --> 00:09:39.839 So there's a self-learning loop that we can then feed back into institutional best practices. s82 00:09:39.839 --> 00:09:44.640 What we don't want to use the dashboards are to kind of stack rank people, right? s83 00:09:44.640 --> 00:09:49.200 making it a a leaderboard and somehow reward maximization. s84 00:09:49.200 --> 00:09:56.080 There's an interesting analogy I want to draw with uh a traditional edge productivity metric called lines of code. s85 00:09:56.080 --> 00:10:06.800 So I believe all of you will be tracking that metric but it wouldn't be wise to use that metric as the key goal to measure engine velocity because if we want s86 00:10:06.800 --> 00:10:12.160 productive and high quality engine work one can argue that removing code is even better. s87 00:10:12.160 --> 00:10:17.760 So, LOC line of code is an important metric but not something we want to directly optimize for. s88 00:10:17.760 --> 00:10:21.519 Same thing for the token usage and spend. s89 00:10:22.320 --> 00:10:26.720 So, that that gets us to the notion of trusted throughput. s90 00:10:26.720 --> 00:10:28.000 How do we think about that? s91 00:10:28.000 --> 00:10:29.680 How do we define that? s92 00:10:29.680 --> 00:10:33.040 First, I want to kind of share the quantified uh side of the things. s93 00:10:33.040 --> 00:10:37.440 What are the metrics that kind of we have been involving in defining and tracking. s94 00:10:37.440 --> 00:10:44.720 So we talked about line of code is clearly not a good way to measure if AI is you know generating a lot of value. s95 00:10:44.720 --> 00:10:50.880 So the next evolution can be let's count the number of open PRs pull requests. s96 00:10:50.880 --> 00:10:55.440 The intuition being engineers are using AI to generate a lot more code. s97 00:10:55.440 --> 00:10:57.519 So let's measure the open PR. s98 00:10:57.519 --> 00:11:03.200 So clearly we see a big kind of inflection in the open PR count. s99 00:11:03.200 --> 00:11:16.640 But eventually as we as I assume everyone would agree over the time even though people may do oneoff you know R&D work to try out things without lending them but eventually we're all measured by the code we ship. s100 00:11:16.640 --> 00:11:22.800 So therefore we evolved from tracking the open PR count to tracking the merge PR count. s101 00:11:22.800 --> 00:11:25.040 So that's an improvement. s102 00:11:25.040 --> 00:11:29.279 But the next question is not every merged PR is equal. s103 00:11:29.279 --> 00:11:38.160 There can be a PO with only 10 lines of code that takes forever that finds and fix a concurrency bug or there can be a thousand line kind of boilerplate s104 00:11:38.160 --> 00:11:46.320 code that just takes a lot of time to then kind of generate and review but otherwise it's not necessary adding as much business value. s105 00:11:46.320 --> 00:11:53.440 So as such we then started kind of tagging each merged PR with some sort of complexity score. s106 00:11:53.440 --> 00:11:55.920 There's no traditional definition of what that means. s107 00:11:55.920 --> 00:11:57.519 We looked at the literature a bit. s108 00:11:57.519 --> 00:12:09.519 So we just took a pragmatic approach of giving AI a well-crafted prompt and then we feed the PR into basically one or two M and say score the complexity based on t-shirt size. s109 00:12:09.519 --> 00:12:18.079 So I the idea being if you use AI to generate a more complex PR we consider that as being more valuable basically that's how we kind of add a weightage s110 00:12:18.079 --> 00:12:31.279 to each merged PR but that's not the end of the journey that's still something we're going to evolve keep evolving and I would love to discuss with everyone on kind of how we end up creating defining a set of metrics that kind of approximate s111 00:12:31.279 --> 00:12:34.000 the value AI is generating. s112 00:12:34.000 --> 00:12:36.639 Now let's look at the qualitative view. s113 00:12:36.639 --> 00:12:47.600 What we think about the way we would define trusted throughput is a high quality output that's interested by both internal engineering and leadership and external customers. s114 00:12:47.600 --> 00:12:50.639 We think they come from three buckets. s115 00:12:50.639 --> 00:13:00.880 The first bucket is all of the objective metrics that we run with checking the test coverage whether uh all of the predefined security checks are passing. s116 00:13:00.880 --> 00:13:06.560 Do we go through the regular canarying practice as we roll out features safely and so on. s117 00:13:06.560 --> 00:13:13.440 In addition, we complement the subjective objective metrics with our subjective human judgment. s118 00:13:13.440 --> 00:13:21.839 So that's where the code review, the design review come in to look at the code quality, clarity, maintenance, architecture fit and so on. s119 00:13:21.839 --> 00:13:35.440 And then finally we want to make sure through all of these internal objective and subjective check when the rubber meets the road how customer perceive the changes are there production fire that lead to ro rollbacks s120 00:13:35.440 --> 00:13:41.680 do customers complain have tickets that talk about usability uh friction uh bugs and so on. s121 00:13:41.680 --> 00:13:48.639 So these are the three buckets that together form what we think is trusted throughput from engineering. s122 00:13:52.160 --> 00:13:52.639 Okay. s123 00:13:52.639 --> 00:14:06.800 So now let's talk about from a software deploy deployment life cycle perspective where we observe the new bottlenecks are as I mentioned earlier AI code generation is making PR creation s124 00:14:06.800 --> 00:14:08.240 abundant. s125 00:14:08.240 --> 00:14:17.519 So now the the bottleneck from kind of the whole life cycle perspective gets shifted onto re review and they're subsequently merging the PR. s126 00:14:17.519 --> 00:14:19.920 Does that resonate? s127 00:14:20.240 --> 00:14:22.959 I see some heads nodding. s128 00:14:22.959 --> 00:14:31.680 So this is where we spend time on figuring out how we can further improve the review process as well as the continuous integration the CI process. s129 00:14:31.680 --> 00:14:35.760 So we will dive into these two topics in the next couple slides here. s130 00:14:35.760 --> 00:14:51.760 I just want to say a potential anti-attern anti-solution is that hey if the CI infrastructure gets overloaded then a workar around by engineers to stop splitting PR just start submitting large PR for review and submission s131 00:14:51.760 --> 00:14:59.760 because if it takes an hour to run all of your regression test and submit it I don't want to break my PR into 10 right which might take 10 hours s132 00:14:59.760 --> 00:15:13.920 however this in our view can be pretty risky because it makes the human review overhead higher it also reduce the quality of the review because the human attention can be spread thin so that is an anti-attern s133 00:15:13.920 --> 00:15:30.160 I wanted to caution so for code review the key principle we use is to make sure we onboard AI tooling as the first level of defense they don't replace human reviewers but we want to offload human reviewers s134 00:15:30.160 --> 00:15:39.360 as much as possible let the AI review take care of simpler things like coding style issues or if there's a missing test coverage. s135 00:15:39.360 --> 00:15:46.320 So, make sure the author gets through all of them before then the review gets routed to a human reviewer. s136 00:15:46.320 --> 00:15:57.279 And this way our human engineers can focus on applying their deep judgment on aspects that are somewhat subjective like if the code is good, if the architecture is sound, s137 00:15:57.279 --> 00:16:04.000 if the code uh uh passes kind of the security uh the security design and so on. s138 00:16:04.000 --> 00:16:09.680 so that in the end our engineering team can take the final accountability. s139 00:16:11.600 --> 00:16:13.440 Now let's look at CI. s140 00:16:13.440 --> 00:16:27.920 So I assume all of you deploy some form of CI uh CI/CD and what we're seeing is thanks to AI now making it much easier to generate code as splitting code into smaller but more PRs s141 00:16:27.920 --> 00:16:41.519 it puts a lot more pressure on the CI and this is something that uh if we don't address uh at a company level individual engineers can be struggling because that means they have to waste their human time babysitting s142 00:16:41.519 --> 00:16:43.360 the PR to get merged. s143 00:16:43.360 --> 00:16:55.600 If they run into flaky test then they have to manually they hit rerun it's very frustrating or they can recruit an AI agent to babysit and kind of do a loop but that in turn waste AI token as well. s144 00:16:55.600 --> 00:17:04.799 So these are not these are just workarounds not perfect solution and also tend to make engineers feel a little bit lower morale a little bit more frustrated. s145 00:17:04.799 --> 00:17:19.679 So what we what we are doing is kind of we put more uh developer experience uh platform kind of engineering to invest into reducing removing the flaky test improving the CI infrastructure s146 00:17:19.679 --> 00:17:32.000 and the key thing here is to also define and measure the right metrics for example uh the work clock time between when a peer is ready to submit till when it's submitted s147 00:17:32.000 --> 00:17:35.840 right if a typical CR uh run takes an hour. s148 00:17:35.840 --> 00:17:38.559 Does the typical PR submission take two or three hours? s149 00:17:38.559 --> 00:17:45.440 In which case, that's a red flag and also the number of times a PR needs to get retrieded for passing through the test. s150 00:17:45.440 --> 00:17:57.039 So, these are the key metrics that we are using to measure our developer experiences and the relevant team who is focused on improving uh these uh the developer experience. s151 00:17:58.480 --> 00:18:10.000 So with all of the analysis and ideas here we uh want to share kind of the a pragmatic framework of how we can then measure and optimize token usage. s152 00:18:10.000 --> 00:18:11.760 It has three aspects. s153 00:18:11.760 --> 00:18:25.360 The first one is set the right set of guards across setting the budget and quota tracking usage defining anomalies so that no users uh leaders can get notified if something feels wrong. s154 00:18:25.360 --> 00:18:35.840 This is complementaryary to still regular human review which can catch other interesting patterns or learnings and feedback into the institutional knowledge base. s155 00:18:35.840 --> 00:18:50.640 Let me just couple that with the third item here which is the learning loop we talk about as our leadership work with individuals to define these guard rails review the metrics and then refine that's how we kind of close the learning loop. s156 00:18:50.640 --> 00:19:05.200 In addition to that, we want to work with our teams, individual engineers to continue to search for and if needed innovate on the best practices of how to use AI, how to use AI to build products and also use it internally. s157 00:19:05.200 --> 00:19:13.039 For example, some engineers may be writing an agentic loop as part of the harness when they use cloud code. s158 00:19:13.039 --> 00:19:25.200 after they generated initial PR they go and loop around and say try and pass the set of tests and then if some tests don't pass just auto fix the test or the code and retry. s159 00:19:25.200 --> 00:19:35.520 One thing to watch out for is to put a limit on the number of loop steps to make sure if things go out of control we don't waste too many tokens on that. s160 00:19:35.520 --> 00:19:37.679 Another example is prompt caching. s161 00:19:37.679 --> 00:19:54.559 This is becoming increasingly more prevalent by the commercial uh model vendors where what they advise is if you send a prompt with the same prefix they could optimize how they process the prefix of the prompt. s162 00:19:54.559 --> 00:20:01.679 What that means then as a user to those is that we want to encourage our users to structure their prompt that way. s163 00:20:01.679 --> 00:20:12.240 For example, if your prompt consists of a system prompt followed by a user prompt, you want to put the system prompt that's fixed at the top and the varying content at the bottom. s164 00:20:12.240 --> 00:20:13.840 Context pruning is also important. s165 00:20:13.840 --> 00:20:19.120 We want to kind of drill it into each individual users kind of new kind of muscle memory. s166 00:20:19.120 --> 00:20:29.360 So they are aware that as they build out the context through a longer chat session, they would be mindful of summarizing the context and make sure that the token usage is efficient that way. s167 00:20:29.360 --> 00:20:34.960 There are increasingly more tools like claw code that will automatically manage and compact the context for you. s168 00:20:34.960 --> 00:20:40.320 And so this increases the token usage efficiency but also increase the quality of AI output. s169 00:20:40.320 --> 00:20:44.080 There are other ideas we're exploring as well. s170 00:20:48.559 --> 00:20:52.240 So I know we're at time so this is towards the end of the talk. s171 00:20:52.240 --> 00:20:55.440 There is sometimes we also face build versus by decision. s172 00:20:55.440 --> 00:21:02.799 The principle is simple for things that are non- differentiating like IDE CI infrastructure we want to buy. s173 00:21:02.799 --> 00:21:13.440 But then for things that are specific to our context like how we would generate high quality PR for small bug fixes versus building a new UI feature for refactoring and so on. s174 00:21:13.440 --> 00:21:18.559 We have our internal playbook which is a set of well-crafted AI prompts. s175 00:21:18.559 --> 00:21:20.960 So we save that and share across our team. s176 00:21:20.960 --> 00:21:22.880 So that gets reused and enhanced. s177 00:21:22.880 --> 00:21:25.440 So that's something we must build internally. s178 00:21:25.440 --> 00:21:31.600 When it comes to case to case though, sometimes it's still a bit ambiguous like we're trying to build what we call builder agent. s179 00:21:31.600 --> 00:21:36.400 That's like a cloud-based code generation that wrap the cloud codec and so on. s180 00:21:36.400 --> 00:21:39.760 While we know there are also other vendors out there that we're still exploring. s181 00:21:39.760 --> 00:21:43.200 So we love to exchange thoughts on that. s182 00:21:43.520 --> 00:21:48.799 So then to summarize here are a couple key lessons as we went through the last couple quarters of journey. s183 00:21:48.799 --> 00:21:52.880 I wanted to share so that hopefully you could kind of accelerate your process there. s184 00:21:52.880 --> 00:22:00.480 If I were to summarize these three things I would it's about learning planning ahead and learn from other people's stories mistakes. s185 00:22:00.480 --> 00:22:12.880 So what that means is think about build respences by early on as you are encouraging more code gen think about how that impact your code review and CI and how you can address these new bottlenecks. s186 00:22:12.880 --> 00:22:24.080 And finally, continue to define and instrument your system to get the right metrics to measure the health of your CI system and the whole developer experience in general. s187 00:22:24.080 --> 00:22:25.600 So that's it for the talk. s188 00:22:25.600 --> 00:22:34.000 We believe that this is the golden era of AI where maximizing token ROI is the key for every team success. s189 00:22:34.000 --> 00:22:36.559 And with that, I just want to end with saying we are hiring. s190 00:22:36.559 --> 00:22:45.120 I know this is engineering leadership crowd but if you know of someone who is interested in building cutting edge legal contracting AI we would love to talk. s191 00:22:45.120 --> 00:22:47.280 Thank you.