WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=aGIUg8zKI8A NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/aGIUg8zKI8A.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.880 --> 00:00:14.240 My name is Nayan. s3 00:00:14.240 --> 00:00:21.160 I'm the head of AI engineering at Optiver, and today I'm going to talk to you a little bit about how we scale AI with P&L on the line. s4 00:00:21.160 --> 00:00:22.920 Like, what does that mean though, right? s5 00:00:22.920 --> 00:00:27.280 So, first I think to contextualize, uh let me tell you a little bit about what we do at Optiver. s6 00:00:27.280 --> 00:00:28.960 So, we're a market maker. s7 00:00:28.960 --> 00:00:31.520 Uh what that means is that we facilitate trading. s8 00:00:31.520 --> 00:00:32.360 Uh we're a trading firm. s9 00:00:32.360 --> 00:00:33.840 We trade with our own money. s10 00:00:33.840 --> 00:00:36.120 Um and we don't have clients. s11 00:00:36.120 --> 00:00:42.760 Uh basically, you know, we in in a tech sense, we take a lot of data, you know, we build models, and then we make decisions with them. s12 00:00:42.760 --> 00:00:52.000 And these decisions need to work in a very high-performance manner, you know, we make 10 million plus trades a day, pricing million plus instruments, and petabytes of data processed daily. s13 00:00:52.000 --> 00:00:53.840 And also, we're a pretty global firm. s14 00:00:53.840 --> 00:00:56.160 We have 10 plus locations across the world. s15 00:00:56.160 --> 00:01:00.260 Um and that comes with, you know, certain implications that we'll get into for a second. s16 00:01:00.260 --> 00:01:00.920 [clears throat] s17 00:01:00.920 --> 00:01:08.040 So, typically, if you look at a traditional tech company, you can think about how they think about, you know, how they measure how much money they make. s18 00:01:08.040 --> 00:01:16.240 Uh it's something along the lines of you implement some change, you roll it out to users, and then users use it for days and weeks, etc., and you collect some engagement metrics. s19 00:01:16.240 --> 00:01:25.480 Uh and then over, you know, months and quarters, maybe years, you look at like revenue, long-time value, etc. I used to be at Apple, um so I know how this works. s20 00:01:25.480 --> 00:01:29.760 Um but um at Optiver, we implement something. s21 00:01:29.760 --> 00:01:34.920 Uh we roll out that change to our traders and our our quants, and literally within hours you can see the P&L impact. s22 00:01:34.920 --> 00:01:41.320 In fact, you know, at the top of every day, we basically zero out the balance sheet, and then look by the end of the day how much money we've made. s23 00:01:41.320 --> 00:01:48.731 So, this has certain implications around how, you know, we make decisions around how we want to scale AI and how do we want to build things. s24 00:01:48.731 --> 00:01:49.080 [clears throat] s25 00:01:49.080 --> 00:01:51.680 First of all, the feedback loop is much more immediate, right? s26 00:01:51.680 --> 00:01:55.840 Like, when you make a decision, by the end of the day you'll start seeing the impact of that decision. s27 00:01:55.840 --> 00:02:04.840 Of course, like, over long term, there are instruments that we trade that are, you know, longer horizon and there's more value to discover over long-term, but immediately you'll start seeing the impact of something. s28 00:02:04.840 --> 00:02:11.560 But, the other side of that coin is that if you miss an opportunity, then that means that opportunity loss is potentially pretty costly. s29 00:02:11.560 --> 00:02:20.160 Especially if, you know, there's like a special [clears throat] thing going on in the market, let's say that there's a SpaceX AI IPO or something like that and you miss out on that, then that's probably, you know, pretty costly. s30 00:02:20.160 --> 00:02:22.920 Which means that we need to make decisions pretty quickly. s31 00:02:22.920 --> 00:02:26.040 Uh we need to be able to measure them really effectively and really efficiently. s32 00:02:26.040 --> 00:02:28.160 And we got to have a bias for action. s33 00:02:28.160 --> 00:02:35.320 Um that means like the ceiling of the value and importance of a decision decision you can make is really, really high. s34 00:02:35.320 --> 00:02:46.840 Um so, in the in the case of like a service, like a user-facing service business, like Spotify or whatever, like fundamentally, they're competing for a finite amount of attention of the customers. s35 00:02:46.840 --> 00:02:49.600 Versus with your trading, the more you trade, the more money you make, right? s36 00:02:49.600 --> 00:02:53.560 So, you know, the economics are a little bit more different. s37 00:02:53.560 --> 00:03:03.680 So, to accomplish this and to be able to scale and operate at this way, we basically needed to build our own AI platform around a few core principles. s38 00:03:03.680 --> 00:03:04.400 Move fast. s39 00:03:04.400 --> 00:03:05.680 I mean, we talked about that already. s40 00:03:05.680 --> 00:03:09.040 But, of course, part of moving fast is that being able to be modular. s41 00:03:09.040 --> 00:03:14.120 Because, you know, in the AI world, every day there's a new model, there's a new product, there's a new paradigm. s42 00:03:14.120 --> 00:03:19.200 We're doing loops, now we're doing swarms, now we're doing graphs, etc. And we need to be able to account for that. s43 00:03:19.200 --> 00:03:22.160 And if that paradigm is valuable, like measure it. s44 00:03:22.160 --> 00:03:23.440 And if it is, bring it in. s45 00:03:23.440 --> 00:03:27.160 Because the moment you bring it in, you're going to start, you know, getting more value. s46 00:03:27.160 --> 00:03:30.200 And of course, to be able to understand that, we need to build to measure at every layer. s47 00:03:30.200 --> 00:03:40.920 You know, of course, like trading means lots of backtesting, etc. But also, you know, we need to when a new model comes out, yes, there's of course external benchmarks as to how this model is performing, s48 00:03:40.920 --> 00:03:44.200 but we also need to be able to understand how in our internal benchmarks these things perform. s49 00:03:44.200 --> 00:03:49.800 Because the data distribution of the benchmarks that we have is different from what they have available externally. s50 00:03:49.800 --> 00:03:53.160 Now, the thing is, traders are pretty smart people. s51 00:03:53.160 --> 00:03:54.880 They're pretty high agency people. s52 00:03:54.880 --> 00:03:57.400 So, we want our platform to be very self-service. s53 00:03:57.400 --> 00:04:01.160 But, of course, anytime you provide something purely self-service, it can be really chaotic. s54 00:04:01.160 --> 00:04:03.960 So, we also need to be able to provide good defaults. s55 00:04:03.960 --> 00:04:10.160 Uh and one way we kind of have this um way of working with them is also being able to forward deploy. s56 00:04:10.160 --> 00:04:18.680 So, we have a fairly decently sized AI engineering team and we take our people, we have them sit down with the traders, work with them, understand how they trade, s57 00:04:18.680 --> 00:04:24.080 teach them how AI can add more value to them, learn and bring it back to the platform. s58 00:04:24.080 --> 00:04:31.560 Now, of course, you've all heard the stories of like opening eye hugging face, etc. So, of course, as we're doing more and more with AI, we also need to make sure that we're doing in a safe manner. s59 00:04:31.560 --> 00:04:38.960 As you can imagine, there's potentially pretty negative impacts if the agents kind of go haywire and do stuff with our stack. s60 00:04:38.960 --> 00:04:42.360 Um but, I do want to underscore that everybody in the company is using AI. s61 00:04:42.360 --> 00:04:49.960 Like, everybody's using it a little bit differently, but fundamentally we're very very um AI native, I guess is the term that we're using now. s62 00:04:49.960 --> 00:04:57.280 Um And, of course, because we're a highly regulated industry and we value our IP deeply, everything that we do is on prem. s63 00:04:57.280 --> 00:05:05.320 So, we have our own on prem GPUs, our own prem infrastructure, etc. The only thing that we phone home outside of our service really is for like third-party model inference. s64 00:05:05.320 --> 00:05:11.080 And, of course, in those cases, we have like things like zero data retention, etc. agreements with the companies. s65 00:05:11.080 --> 00:05:18.440 So, here's kind of like a very, you know, shock and awe, like a lot of information on the slide type of view of our AI platform. s66 00:05:18.440 --> 00:05:25.000 I don't expect you to, you know, digest all of this, but, you know, if you want to take a photo and talk about me afterwards, be my guest. s67 00:05:25.000 --> 00:05:26.760 I'll flash this slide again at the end. s68 00:05:26.760 --> 00:05:32.680 But, broadly, the things that I want to highlight is that um we all have to be accountable for every layer of the stack. s69 00:05:32.680 --> 00:05:35.520 At the bottom, you know, we have models from third-party providers. s70 00:05:35.520 --> 00:05:39.360 We also self-host our own models and the open-source models. s71 00:05:39.360 --> 00:05:43.840 We have an inference gateway that provides things like routing between these different models. s72 00:05:43.840 --> 00:05:47.480 Um we can uh, measure how much we're spending on each of these models. s73 00:05:47.480 --> 00:05:51.000 We can assign budgets, etc. I'll go to that in in a second. s74 00:05:51.000 --> 00:05:52.280 And of course this gives us resilience. s75 00:05:52.280 --> 00:06:00.120 Like if one provider goes down, we can always swap out to a different provider, etc. Because again, any time that we're down, you know, that's basically we're not making money. s76 00:06:00.120 --> 00:06:09.080 Um, and then we have our own agent platform where uh, agents have uh, registered uh, capabilities like skills, plugins, tools. s77 00:06:09.080 --> 00:06:13.880 Each agent in each team with the using an agent can bring in their knowledge and memory systems. s78 00:06:13.880 --> 00:06:23.919 We have an agent gateway which is basically like a single ledger of all of the agents in the company being able to come in together, communicate, orchestrate humans and agents with similar identity management, s79 00:06:23.919 --> 00:06:29.760 uh, etc. So this kind of goes into things like the agent swarm, etc. concepts that you also may have heard about. s80 00:06:29.760 --> 00:06:42.960 And finally, what's really important to me, I'm a product guy as well, is that um, we have the right surfaces for our users to engage with the agents because you can build a great platform, but if nobody wants to, likes to, or knows how to use it, then it doesn't really matter. s81 00:06:42.960 --> 00:06:52.120 So we've built our own kind of um, web interface for people to access like their sessions, etc. Uh, we have this thing called our agent workbench which is basically persona management. s82 00:06:52.120 --> 00:06:55.960 Teams can kind of build their own profile of how they use the agents and share. s83 00:06:55.960 --> 00:06:59.880 Uh, we have a personal assistant that everybody in the company uh, has access to. s84 00:06:59.880 --> 00:07:01.240 Um, yeah. s85 00:07:01.240 --> 00:07:06.720 So again, I'll get back to this later, uh, but if you have questions we can talk about it afterwards. s86 00:07:06.840 --> 00:07:07.640 Right. s87 00:07:07.640 --> 00:07:09.560 So how do we move on these principles? s88 00:07:09.560 --> 00:07:18.960 Uh, one principle that we have is that, you know, if a new frontier model is made available publicly, uh, we want to be able to leverage it in production flows the same day because there's immense value in that if it is valuable. s89 00:07:18.960 --> 00:07:20.400 But is it valuable, right? s90 00:07:20.400 --> 00:07:23.200 We want to be able to benchmark it on golden sets. s91 00:07:23.200 --> 00:07:26.560 So that means that we need to have a mature evaluation framework. s92 00:07:26.560 --> 00:07:31.600 Um, and then we partner with key customer trading teams where we collect their benchmarks. s93 00:07:31.600 --> 00:07:42.360 We onboard onto them and whenever we get a new model, we run the model on these new benchmarks, immediately know the metrics, and can share with the folks in the company to see, "Hey, is this really valuable for our use cases?" s94 00:07:42.360 --> 00:07:51.520 But additionally, you know, anytime there's a different model, usually that means that we have to make the model available in our gateway, we have the right APIs, interfaces, we have the cost management, etc. set up correctly. s95 00:07:51.520 --> 00:07:59.240 And often times new models mean like the harnesses need to be updated, etc. and we also manage the update and delivery of that as well. s96 00:07:59.240 --> 00:08:03.120 And then of course, yes, we launch it, but then that also comes with the expectations. s97 00:08:03.120 --> 00:08:05.880 We're a global company, so we need to have round-the-clock support. s98 00:08:05.880 --> 00:08:11.320 So our team, you know, has in different regions support rotations to make sure that the uptime is high. s99 00:08:11.320 --> 00:08:15.240 I think some people here in the audience actually are in support today, but anyway. s100 00:08:15.240 --> 00:08:26.520 Um and then of course, like we run these benchmarks, we also need to be able to have um the people in the company understand like, okay, we looked into this model, we benchmarked it, here's how well it works, here's how you use it, here's our recommendation, s101 00:08:26.520 --> 00:08:30.080 and kind of keep that communication line open. s102 00:08:30.520 --> 00:08:37.840 Um another thing is that because we have optionality between different models, uh if any model's adding value, we want to integrate it. s103 00:08:37.840 --> 00:08:49.560 Uh and we need to be able to insulate against model availability in like different regions, uh or different, you know, cloud providers, etc. Uh and also we have a requirement that some of our workloads run in a completely DMZ s104 00:08:49.560 --> 00:08:50.960 on-prem zone. s105 00:08:50.960 --> 00:08:56.200 Uh so we also there's cost reasons for that, there is um reasons of isolation for that. s106 00:08:56.200 --> 00:09:06.600 So for those reasons, we want to be able to be agnostic to the model and harness, so which means that we have the third-party providers, uh we have on-prem models, and all of it kind of goes through our inference gateway. s107 00:09:06.600 --> 00:09:14.560 We log everything, uh every interaction going with the models is logged, and we also automatically move them to our evaluation system. s108 00:09:14.560 --> 00:09:25.560 Uh we have a bunch of different harnesses that we support, and on top of that we have a meta harness layer that basically lets teams configure their experience on top of that and have some portability between these different harnesses. s109 00:09:25.560 --> 00:09:26.581 Um s110 00:09:26.581 --> 00:09:27.040 [clears throat] s111 00:09:27.040 --> 00:09:32.880 so teams can define their own harness configs in this meta harness layer, and that plugs into the team's knowledge base. s112 00:09:32.880 --> 00:09:39.600 Automatically knowledge is harvested over time, we look at the traces and then derive ways to uh get additional knowledge as well. s113 00:09:39.600 --> 00:09:50.440 Uh and then through this meta harness, you can either deploy manage the agents on our platform or you get access to them through the web portal IDE UI that I described earlier. s114 00:09:50.440 --> 00:09:53.800 I see some people taking photos, so I'll give it a second. s115 00:09:53.880 --> 00:09:54.746 All right. s116 00:09:54.746 --> 00:09:55.240 [clears throat] s117 00:09:55.240 --> 00:09:59.080 So, um this type of a chart may be familiar to a bunch of you. s118 00:09:59.080 --> 00:10:04.760 Uh this is kind of a roughly illustrated version of how much we're spending on AI, no numbers on it. s119 00:10:04.760 --> 00:10:05.840 Um but we are doing more with it. s120 00:10:05.840 --> 00:10:09.520 And this is, you know, in December you see like the Claude code moment and things go up. s121 00:10:09.520 --> 00:10:14.040 By the way, this inflection point here is like when I joined the company, so I don't know if I have anything to do with that. s122 00:10:14.040 --> 00:10:20.960 Um and then um then we have here, you know, like the big frontier model releases of the summer. s123 00:10:20.960 --> 00:10:31.400 As you can see, like spend really shot up there, but then we kind of got it back under control because we built some optimizations into the process that helped us really get this thing uh in check. s124 00:10:31.400 --> 00:10:36.360 So, um the thing is the more we spend, the more value we can get. s125 00:10:36.360 --> 00:10:39.720 We are pretty confident that we can, you know, keep getting returns. s126 00:10:39.720 --> 00:10:42.800 We want to make sure that we're spending wisely, right? s127 00:10:42.800 --> 00:10:53.839 So, one thing that we discovered was that um we have a bunch of MCPs, we have a bunch of knowledge bases and information, etc. Um and on a fresh session, we were seeing that, you know, sometimes people have like 200k tokens s128 00:10:53.839 --> 00:10:57.160 worth of context like by default minimum, sometimes even more. s129 00:10:57.160 --> 00:10:58.280 That's That's ridiculous, right? s130 00:10:58.280 --> 00:11:00.839 Because everybody was kind of dumping a bunch of stuff in there. s131 00:11:00.839 --> 00:11:04.040 This is the kind of problem with like doing things completely self-service, right? s132 00:11:04.040 --> 00:11:05.920 We need to have skin in the game. s133 00:11:05.920 --> 00:11:08.160 So, we implemented a bunch of different changes. s134 00:11:08.160 --> 00:11:10.600 Uh we brought in MCP tool search. s135 00:11:10.600 --> 00:11:19.400 Uh we gave the agents the option to um rather than calling MCPs, have their own CLI that's wrapping around a bunch of the MCPs, which is like much more efficient. s136 00:11:19.400 --> 00:11:25.240 Um we looked at the caching characteristics of like when caches are getting invalidated, how long do we need to set these parameters? s137 00:11:25.240 --> 00:11:26.360 Do we need to set them differently? s138 00:11:26.360 --> 00:11:28.120 Do we set them dynamically? s139 00:11:28.120 --> 00:11:34.080 Um and then we work with the individual teams to kind of come up with further optimizations to their harnesses. s140 00:11:34.080 --> 00:11:47.400 You know, we do we put some things into knowledge base rather than this like massive system prompt etc. And the good thing about these improvements is that um these things all scale with token cost, right? s141 00:11:47.400 --> 00:11:51.480 So basically if the tokens are more expensive, you're saving more on this. s142 00:11:51.480 --> 00:11:57.600 But the point here is that um yes, you can dump models out there, make them available to people. s143 00:11:57.600 --> 00:12:09.920 And I'm not even going into the work that we're doing on things like, you know, model routing or sub-agent optimizations etc. Sheer single session stuff, there's almost always value left on the table. s144 00:12:09.920 --> 00:12:14.960 And we're, you know, ruthlessly optimizing towards that. s145 00:12:15.440 --> 00:12:17.240 Yeah, there we go. s146 00:12:17.240 --> 00:12:26.560 And we shipped optimizations at this point in time, but of course there's always like roll out etc. Our tech stack is a little bit interesting when it comes to rolling out across the different global regions. s147 00:12:26.560 --> 00:12:39.160 But then also other things that really helped us spend less was that um turns out that if you tell people like how much they're spending and like give them a little bit more visibility, a little bit more alerting, and give them the agency to self-manage, they actually do a pretty good job, you know. s148 00:12:39.160 --> 00:12:43.680 We built a system to basically have teams kind of set their own alerting and limits. s149 00:12:43.680 --> 00:12:48.760 And we've seen like many teams, you know, basically set reasonable limits for themselves. s150 00:12:48.760 --> 00:12:52.280 Another option is, you know, exploring open-source models. s151 00:12:52.280 --> 00:12:58.680 Uh you know, there are a bunch of open-source models that are on token cost pretty competitive with the frontier models out there. s152 00:12:58.680 --> 00:13:04.360 It's not necessarily right for every use case, but we found opportunities where they're a good value replacement. s153 00:13:04.360 --> 00:13:07.839 And we've been basically making that available internally. s154 00:13:07.839 --> 00:13:11.560 Uh and we've seen more and more people leverage those. s155 00:13:11.560 --> 00:13:22.120 Then of course, like there's things like model routing etc. There's we've been exploring different approaches to dynamically send the prompt to like a cheaper model etc. Or sometimes have a way where s156 00:13:22.120 --> 00:13:33.080 we have the like plan generated by like a more frontier model and the implementation be broken down by smaller models, etc. Um and finally, you know, I think knowledge bases are underrated, s157 00:13:33.080 --> 00:13:44.520 uh which yeah, I mean they're very important, but they also help get spend in control if you do your knowledge base effectively because then you're retrieving less context fundamentally if you just retrieve the right amount of context. s158 00:13:44.520 --> 00:13:53.920 Um so, long story short is that we're still investing I mean using a lot of tokens, but this is just about getting more ROI out of the same tokens. s159 00:13:54.880 --> 00:13:58.920 And finally, I kind of want to talk about uh meeting the users where they're at. s160 00:13:58.920 --> 00:14:04.520 Um so, a few months ago we were in this place where like a bunch of people are using different harnesses. s161 00:14:04.520 --> 00:14:12.840 They have all these weird bespoke setups to kind of team up into their dev boxes and some people were building, you know, hacky web UIs, etc. etc. Whatever. s162 00:14:12.840 --> 00:14:16.280 So, basically what we said was like, "Hey, you know, there's a bunch of people doing these types of things. s163 00:14:16.280 --> 00:14:19.640 Why don't we just build a nice version of this and then make it available to everyone in the company?" s164 00:14:19.640 --> 00:14:21.320 So, that's exactly what we did. s165 00:14:21.320 --> 00:14:25.520 In our web portal, we built this basically system where you can manage your sessions. s166 00:14:25.520 --> 00:14:26.800 You can have chat windows. s167 00:14:26.800 --> 00:14:29.480 You can have like artifacts in there. s168 00:14:29.480 --> 00:14:38.920 Um there are similar products out there offered by, you know, you know, some of the different uh harness companies, but we feel that this worked very well with our infrastructure s169 00:14:38.920 --> 00:14:42.760 um and we can kind of iterate it at the rate that we would like to. s170 00:14:42.760 --> 00:14:51.080 Um so, basically this was actually uh also at a at a place where different traders in the company are now kind of contributing code. s171 00:14:51.080 --> 00:15:01.280 They find something that's valuable to their workflow and then they did they just send up a PR and we just update it and then now we have things like, you know, collaborative document editing and things like that. s172 00:15:01.280 --> 00:15:08.040 And in fact, we're kind of getting this to a place where uh we're doing this, you know, agent multiplayer thing where now I have my sessions in there. s173 00:15:08.040 --> 00:15:19.160 We can create rooms with like my agents, your agents, like other deployed agents and we can kind of have all of that in one place and this really helps uh people be more productive and s174 00:15:19.160 --> 00:15:23.440 uh you know, we have now a Slack client, etc. So, you can also do it on the go and whatnot. s175 00:15:23.440 --> 00:15:27.520 So, basically, um our users are very high agency. s176 00:15:27.520 --> 00:15:30.040 Uh they will go and build something if we don't build it for them. s177 00:15:30.040 --> 00:15:33.360 So, we just built a better version of it. s178 00:15:34.200 --> 00:15:34.560 Yeah. s179 00:15:34.560 --> 00:15:41.560 So, uh basically, this is um the overall overarching philosophy of how we did this. s180 00:15:41.560 --> 00:15:46.560 Uh we have a platform, very modular, uh and um we continue to iterate on it. s181 00:15:46.560 --> 00:15:50.120 But, so far, we're having a pretty good time. s182 00:15:50.120 --> 00:15:52.160 Um that's the end of the talk. s183 00:15:52.160 --> 00:15:55.520 Uh but, if you have questions about anything on this or any other area, we can talk. s184 00:15:55.520 --> 00:15:59.000 And by the way, we have a networking event uh this evening. s185 00:15:59.000 --> 00:16:00.560 Uh please come to our booth and register. s186 00:16:00.560 --> 00:16:01.120 It's going to be fun. s187 00:16:01.120 --> 00:16:03.520 We have cool prizes. s188 00:16:04.171 --> 00:16:06.171 [applause] s189 00:16:08.756 --> 00:16:10.756 [applause] s190 00:16:11.920 --> 00:16:13.920 Questions? s191 00:16:19.200 --> 00:16:19.520 Yeah. s192 00:16:19.520 --> 00:16:23.800 So, we've um definitely been exploring a bunch of different options. s193 00:16:23.800 --> 00:16:28.120 Um personally, I'm not happy with any of the options out there. s194 00:16:28.120 --> 00:16:36.800 There's tradeoffs, but I mean, if you ask like some of the Frontier Labs what you do for knowledge base, they're like just huge markdown files, hierarchical markdown, etc. Um s195 00:16:36.800 --> 00:16:44.440 there's places where that works, but um there is um vector search has its own benefits and disadvantages. s196 00:16:44.440 --> 00:16:49.120 We have a hybrid approach that kind of uh allows us to explore these different options. s197 00:16:49.120 --> 00:16:57.720 Personally, I want us to spend more time around like doing things like perhaps some graph databases, etc. that are going to be uh more valuable. s198 00:16:57.720 --> 00:17:03.040 I mean, every day on Twitter, I see something like new way of like handling memory and knowledge. s199 00:17:03.040 --> 00:17:05.720 Um that's a problem that we want to spend more time on. s200 00:17:05.720 --> 00:17:08.360 I'm not happy with our solution. s201 00:17:10.800 --> 00:17:12.880 Go ahead. s202 00:17:15.000 --> 00:17:17.720 There's hand built or s203 00:17:20.920 --> 00:17:22.000 Yeah, good question. s204 00:17:22.000 --> 00:17:25.600 Um, value gained from using hand built solutions versus Codex. s205 00:17:25.600 --> 00:17:27.199 So, there's a few things. s206 00:17:27.199 --> 00:17:37.240 Um, I mean, one kind of boring answer is that because everything is kind of in our own infrastructure, a lot of these tools don't work in the way that you would want them to. s207 00:17:37.240 --> 00:17:44.280 Um, un- until recently, Codex didn't really have the option to kind of run managed agents, etc. or some of these things on your own infrastructure. s208 00:17:44.280 --> 00:17:47.240 Now, they've started offering them and we've been working with them on it, actually. s209 00:17:47.240 --> 00:17:51.680 But, um that was one reason why we started this way. s210 00:17:51.680 --> 00:18:03.120 Secondly, I think the um most Many of the places that offer these types of solutions also try to kind of lock you into either their model or some rest of their solution. s211 00:18:03.120 --> 00:18:14.520 But, the moment we kind of go down that path, then we end up in a place where if our infrastructure is overly invested in one particular provider, we move at the speed that they let us move. s212 00:18:14.520 --> 00:18:20.200 Uh, we've actually encountered with um that was one of the uh inference gateway providers that we used to use. s213 00:18:20.200 --> 00:18:28.120 Um, and basically, we would encounter scenarios where something comes out, we need it, and they're like, "Okay, uh let's talk to our GTM team. s214 00:18:28.120 --> 00:18:29.240 Let's put you on the road map." s215 00:18:29.240 --> 00:18:33.240 Well, it's like and then by that point, it's like, "Okay, no, we need to actually move faster." s216 00:18:33.240 --> 00:18:35.760 Um, so that's also one of the reasons. s217 00:18:35.760 --> 00:18:43.600 Uh, and finally, I think the in some of these areas, we've been um consistently ahead of the curve. s218 00:18:43.600 --> 00:18:54.880 Like, uh talking to some of the frontier labs, um the the way that we've been doing managed agent capabilities, etc. like, they're asking us to tell them how we did it. s219 00:18:54.880 --> 00:19:01.840 Not to be bragging too much, but I think the um we feel pretty confident in our engineering and kind of our road map. s220 00:19:01.840 --> 00:19:03.640 So, that's also uh been well. s221 00:19:03.640 --> 00:19:11.680 I don't think this approach works well for everybody, but I think we have a unique um coalition of circumstances that allow for us to do that. s222 00:19:11.880 --> 00:19:12.760 Good. s223 00:19:12.760 --> 00:19:25.400 Can you talk a bit about your Um yeah. s224 00:19:25.400 --> 00:19:29.400 So, just to make sure I understand your question correctly. s225 00:19:29.400 --> 00:19:34.320 What are some of the best practices of how we integrate the rest of this into eval and memory? s226 00:19:36.400 --> 00:19:37.760 Right. s227 00:19:37.760 --> 00:19:47.960 Um so, I think the we started at a place where, you know, we log everything, like I said, every tool call, input, output, etc. Just hotel logging. s228 00:19:47.960 --> 00:19:52.600 And that goes into our evaluation system. s229 00:19:52.600 --> 00:20:07.280 Um the challenge that we found with that, and we it's kind of an open problem, I would say, is that um when you're kind of trying to evaluate like a tool call, I mean, we have like, for example, a support bot agent that we run in Slack. s230 00:20:07.280 --> 00:20:08.760 And we want to evaluate that agent. s231 00:20:08.760 --> 00:20:11.040 Let's say, what if we ran this agent on a different model? s232 00:20:11.040 --> 00:20:16.520 Well, the support bot is leveraging a bunch of tools that depend on the state of the world that were different at the time when you ran it. s233 00:20:16.520 --> 00:20:18.680 And that's the pretty like typical problem. s234 00:20:18.680 --> 00:20:24.960 Um we've been working on basically things like, can we snapshot world states with these tools? s235 00:20:24.960 --> 00:20:29.280 And depending on the system that it ingresses with, it's, you know, there's different answers to that question. s236 00:20:29.280 --> 00:20:32.480 If you're looking at some like trading back test database, easy. s237 00:20:32.480 --> 00:20:33.680 Like that you can always point in time. s238 00:20:33.680 --> 00:20:35.200 That's what we do. s239 00:20:35.200 --> 00:20:41.920 But if you're looking at some other system, like if you're looking at Slack, for example, what was the state of Slack when this was originally evaluated? s240 00:20:41.920 --> 00:20:46.840 Maybe there's some ways to kind of replay Slack from a particular point in time, but other systems, not really. s241 00:20:46.840 --> 00:20:52.880 So, we kind of try to assign the confidence value as to how well we can point in time this particular tool invocation. s242 00:20:52.880 --> 00:20:55.280 And we still log it as such. s243 00:20:55.280 --> 00:21:00.880 And when we're rerunning the eval, we will run the tool with kind of saying, okay, I want to kind of the snapshot. s244 00:21:00.880 --> 00:21:04.280 We have like a proxy around the MCPs. s245 00:21:04.280 --> 00:21:05.960 But, you know, it's imperfect. s246 00:21:05.960 --> 00:21:11.920 And I don't know if there's a perfect solution other than just record, you know, everything. s247 00:21:15.400 --> 00:21:17.840 Yeah, I maybe I talk time for one more? s248 00:21:17.840 --> 00:21:19.840 Good. s249 00:21:31.400 --> 00:21:33.400 Yeah. s250 00:21:36.120 --> 00:21:39.880 That's a very classic problem that we've actually run into several times. s251 00:21:39.880 --> 00:21:46.000 You know, we had this one team that built an agent that did like coding and like analysis with them. s252 00:21:46.000 --> 00:21:53.160 It was kind of scoped to their repo and it was like, well, what if we also had worked with this team and have their thing also come to us and immediately kind of explodes in complexity. s253 00:21:53.160 --> 00:21:58.400 Now it's checking out multiple repos that have different off, etc. like typical problem. s254 00:21:58.400 --> 00:22:03.080 So, initially our solution was explore, okay, what if we like literally just did that? s255 00:22:03.080 --> 00:22:12.360 What if we just made the new agent that just had a union of the functionality, but often times that ends up, you know, a lot of tech debt, works very poorly, etc. s256 00:22:12.360 --> 00:22:23.480 I think our current answer to that type of paradigm is obviously every agent is version, etc. So, you can if it's small increments, you can push a new version to the agent, make make new capabilities, s257 00:22:23.480 --> 00:22:34.120 but going forward we would more encourage people to basically leverage our agent gateway where rather than one agent having like a 10 different things that it does, scope your agents to like a task that they're good at. s258 00:22:34.120 --> 00:22:48.480 You can easily measure them at, you can easily manage the life cycle of it and then there'll be another agent that it can talk to easily as if it was, you know, kind of operating on itself because we kind of have this construct of agents being able to go through one ledger, spin up each other, etc. s259 00:22:48.480 --> 00:22:56.320 And that will help us scale these types of functionality more effectively rather than just bloating one agent to do everything. s260 00:22:56.320 --> 00:23:17.080 It becomes kind of like a multi-agent orchestration problem, but you know, that's I think keeping things more measurable is really important because if you really bloat the agent then you can end up in scenarios where like, you know, some of the functionality gets better, some gets worse, and you can't really, you know, tweak it easily.