How a Trading Firm Ships New AI Models the Same Day - Noyan Tokgozoglu, Optiver

AI Engineer · 23 min · 260 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12My name is Nayan.
00:14I'm the head of AI engineering at Optiver, and today I'm going to talk to you a little bit about how we scale AI with P&L on the line.
00:21Like, what does that mean though, right?
00:22So, first I think to contextualize, uh let me tell you a little bit about what we do at Optiver.
00:27So, we're a market maker.
00:28Uh what that means is that we facilitate trading.
00:31Uh we're a trading firm.
00:32We trade with our own money.
00:33Um and we don't have clients.
00:36Uh basically, you know, we in in a tech sense, we take a lot of data, you know, we build models, and then we make decisions with them.
00:42And these decisions need to work in a very high-performance manner, you know, we make 10 million plus trades a day, pricing million plus instruments, and petabytes of data processed daily.
00:52And also, we're a pretty global firm.
00:53We have 10 plus locations across the world.
00:56Um and that comes with, you know, certain implications that we'll get into for a second.
01:00[clears throat]
01:00So, typically, if you look at a traditional tech company, you can think about how they think about, you know, how they measure how much money they make.
01:08Uh it's something along the lines of you implement some change, you roll it out to users, and then users use it for days and weeks, etc., and you collect some engagement metrics.
01:16Uh and then over, you know, months and quarters, maybe years, you look at like revenue, long-time value, etc. I used to be at Apple, um so I know how this works.
01:25Um but um at Optiver, we implement something.
01:29Uh we roll out that change to our traders and our our quants, and literally within hours you can see the P&L impact.
01:34In fact, you know, at the top of every day, we basically zero out the balance sheet, and then look by the end of the day how much money we've made.
01:41So, this has certain implications around how, you know, we make decisions around how we want to scale AI and how do we want to build things.
01:48[clears throat]
01:49First of all, the feedback loop is much more immediate, right?
01:51Like, when you make a decision, by the end of the day you'll start seeing the impact of that decision.
01:55Of course, like, over long term, there are instruments that we trade that are, you know, longer horizon and there's more value to discover over long-term, but immediately you'll start seeing the impact of something.
02:04But, the other side of that coin is that if you miss an opportunity, then that means that opportunity loss is potentially pretty costly.
02:11Especially if, you know, there's like a special [clears throat] thing going on in the market, let's say that there's a SpaceX AI IPO or something like that and you miss out on that, then that's probably, you know, pretty costly.
02:20Which means that we need to make decisions pretty quickly.
02:22Uh we need to be able to measure them really effectively and really efficiently.
02:26And we got to have a bias for action.
02:28Um that means like the ceiling of the value and importance of a decision decision you can make is really, really high.
02:35Um so, in the in the case of like a service, like a user-facing service business, like Spotify or whatever, like fundamentally, they're competing for a finite amount of attention of the customers.
02:46Versus with your trading, the more you trade, the more money you make, right?
02:49So, you know, the economics are a little bit more different.
02:53So, to accomplish this and to be able to scale and operate at this way, we basically needed to build our own AI platform around a few core principles.
03:03Move fast.
03:04I mean, we talked about that already.
03:05But, of course, part of moving fast is that being able to be modular.
03:09Because, you know, in the AI world, every day there's a new model, there's a new product, there's a new paradigm.
03:14We're doing loops, now we're doing swarms, now we're doing graphs, etc. And we need to be able to account for that.
03:19And if that paradigm is valuable, like measure it.
03:22And if it is, bring it in.
03:23Because the moment you bring it in, you're going to start, you know, getting more value.
03:27And of course, to be able to understand that, we need to build to measure at every layer.
03:30You know, of course, like trading means lots of backtesting, etc. But also, you know, we need to when a new model comes out, yes, there's of course external benchmarks as to how this model is performing,
03:40but we also need to be able to understand how in our internal benchmarks these things perform.
03:44Because the data distribution of the benchmarks that we have is different from what they have available externally.
03:49Now, the thing is, traders are pretty smart people.
03:53They're pretty high agency people.
03:54So, we want our platform to be very self-service.
03:57But, of course, anytime you provide something purely self-service, it can be really chaotic.
04:01So, we also need to be able to provide good defaults.
04:03Uh and one way we kind of have this um way of working with them is also being able to forward deploy.
04:10So, we have a fairly decently sized AI engineering team and we take our people, we have them sit down with the traders, work with them, understand how they trade,
04:18teach them how AI can add more value to them, learn and bring it back to the platform.
04:24Now, of course, you've all heard the stories of like opening eye hugging face, etc. So, of course, as we're doing more and more with AI, we also need to make sure that we're doing in a safe manner.
04:31As you can imagine, there's potentially pretty negative impacts if the agents kind of go haywire and do stuff with our stack.
04:38Um but, I do want to underscore that everybody in the company is using AI.
04:42Like, everybody's using it a little bit differently, but fundamentally we're very very um AI native, I guess is the term that we're using now.
04:49Um And, of course, because we're a highly regulated industry and we value our IP deeply, everything that we do is on prem.
04:57So, we have our own on prem GPUs, our own prem infrastructure, etc. The only thing that we phone home outside of our service really is for like third-party model inference.
05:05And, of course, in those cases, we have like things like zero data retention, etc. agreements with the companies.
05:11So, here's kind of like a very, you know, shock and awe, like a lot of information on the slide type of view of our AI platform.
05:18I don't expect you to, you know, digest all of this, but, you know, if you want to take a photo and talk about me afterwards, be my guest.
05:25I'll flash this slide again at the end.
05:26But, broadly, the things that I want to highlight is that um we all have to be accountable for every layer of the stack.
05:32At the bottom, you know, we have models from third-party providers.
05:35We also self-host our own models and the open-source models.
05:39We have an inference gateway that provides things like routing between these different models.
05:43Um we can uh, measure how much we're spending on each of these models.
05:47We can assign budgets, etc. I'll go to that in in a second.
05:51And of course this gives us resilience.
05:52Like if one provider goes down, we can always swap out to a different provider, etc. Because again, any time that we're down, you know, that's basically we're not making money.
06:00Um, and then we have our own agent platform where uh, agents have uh, registered uh, capabilities like skills, plugins, tools.
06:09Each agent in each team with the using an agent can bring in their knowledge and memory systems.
06:13We have an agent gateway which is basically like a single ledger of all of the agents in the company being able to come in together, communicate, orchestrate humans and agents with similar identity management,
06:23uh, etc. So this kind of goes into things like the agent swarm, etc. concepts that you also may have heard about.
06:29And finally, what's really important to me, I'm a product guy as well, is that um, we have the right surfaces for our users to engage with the agents because you can build a great platform, but if nobody wants to, likes to, or knows how to use it, then it doesn't really matter.
06:42So we've built our own kind of um, web interface for people to access like their sessions, etc. Uh, we have this thing called our agent workbench which is basically persona management.
06:52Teams can kind of build their own profile of how they use the agents and share.
06:55Uh, we have a personal assistant that everybody in the company uh, has access to.
06:59Um, yeah.
07:01So again, I'll get back to this later, uh, but if you have questions we can talk about it afterwards.
07:06Right.
07:07So how do we move on these principles?
07:09Uh, one principle that we have is that, you know, if a new frontier model is made available publicly, uh, we want to be able to leverage it in production flows the same day because there's immense value in that if it is valuable.
07:18But is it valuable, right?
07:20We want to be able to benchmark it on golden sets.
07:23So that means that we need to have a mature evaluation framework.
07:26Um, and then we partner with key customer trading teams where we collect their benchmarks.
07:31We onboard onto them and whenever we get a new model, we run the model on these new benchmarks, immediately know the metrics, and can share with the folks in the company to see, "Hey, is this really valuable for our use cases?"
07:42But additionally, you know, anytime there's a different model, usually that means that we have to make the model available in our gateway, we have the right APIs, interfaces, we have the cost management, etc. set up correctly.
07:51And often times new models mean like the harnesses need to be updated, etc. and we also manage the update and delivery of that as well.
07:59And then of course, yes, we launch it, but then that also comes with the expectations.
08:03We're a global company, so we need to have round-the-clock support.
08:05So our team, you know, has in different regions support rotations to make sure that the uptime is high.
08:11I think some people here in the audience actually are in support today, but anyway.
08:15Um and then of course, like we run these benchmarks, we also need to be able to have um the people in the company understand like, okay, we looked into this model, we benchmarked it, here's how well it works, here's how you use it, here's our recommendation,
08:26and kind of keep that communication line open.
08:30Um another thing is that because we have optionality between different models, uh if any model's adding value, we want to integrate it.
08:37Uh and we need to be able to insulate against model availability in like different regions, uh or different, you know, cloud providers, etc. Uh and also we have a requirement that some of our workloads run in a completely DMZ
08:49on-prem zone.
08:50Uh so we also there's cost reasons for that, there is um reasons of isolation for that.
08:56So for those reasons, we want to be able to be agnostic to the model and harness, so which means that we have the third-party providers, uh we have on-prem models, and all of it kind of goes through our inference gateway.
09:06We log everything, uh every interaction going with the models is logged, and we also automatically move them to our evaluation system.
09:14Uh we have a bunch of different harnesses that we support, and on top of that we have a meta harness layer that basically lets teams configure their experience on top of that and have some portability between these different harnesses.
09:25Um
09:26[clears throat]
09:27so teams can define their own harness configs in this meta harness layer, and that plugs into the team's knowledge base.
09:32Automatically knowledge is harvested over time, we look at the traces and then derive ways to uh get additional knowledge as well.
09:39Uh and then through this meta harness, you can either deploy manage the agents on our platform or you get access to them through the web portal IDE UI that I described earlier.
09:50I see some people taking photos, so I'll give it a second.
09:53All right.
09:54[clears throat]
09:55So, um this type of a chart may be familiar to a bunch of you.
09:59Uh this is kind of a roughly illustrated version of how much we're spending on AI, no numbers on it.
10:04Um but we are doing more with it.
10:05And this is, you know, in December you see like the Claude code moment and things go up.
10:09By the way, this inflection point here is like when I joined the company, so I don't know if I have anything to do with that.
10:14Um and then um then we have here, you know, like the big frontier model releases of the summer.
10:20As you can see, like spend really shot up there, but then we kind of got it back under control because we built some optimizations into the process that helped us really get this thing uh in check.
10:31So, um the thing is the more we spend, the more value we can get.
10:36We are pretty confident that we can, you know, keep getting returns.
10:39We want to make sure that we're spending wisely, right?
10:42So, one thing that we discovered was that um we have a bunch of MCPs, we have a bunch of knowledge bases and information, etc. Um and on a fresh session, we were seeing that, you know, sometimes people have like 200k tokens
10:53worth of context like by default minimum, sometimes even more.
10:57That's That's ridiculous, right?
10:58Because everybody was kind of dumping a bunch of stuff in there.
11:00This is the kind of problem with like doing things completely self-service, right?
11:04We need to have skin in the game.
11:05So, we implemented a bunch of different changes.
11:08Uh we brought in MCP tool search.
11:10Uh we gave the agents the option to um rather than calling MCPs, have their own CLI that's wrapping around a bunch of the MCPs, which is like much more efficient.
11:19Um we looked at the caching characteristics of like when caches are getting invalidated, how long do we need to set these parameters?
11:25Do we need to set them differently?
11:26Do we set them dynamically?
11:28Um and then we work with the individual teams to kind of come up with further optimizations to their harnesses.
11:34You know, we do we put some things into knowledge base rather than this like massive system prompt etc. And the good thing about these improvements is that um these things all scale with token cost, right?
11:47So basically if the tokens are more expensive, you're saving more on this.
11:51But the point here is that um yes, you can dump models out there, make them available to people.
11:57And I'm not even going into the work that we're doing on things like, you know, model routing or sub-agent optimizations etc. Sheer single session stuff, there's almost always value left on the table.
12:09And we're, you know, ruthlessly optimizing towards that.
12:15Yeah, there we go.
12:17And we shipped optimizations at this point in time, but of course there's always like roll out etc. Our tech stack is a little bit interesting when it comes to rolling out across the different global regions.
12:26But then also other things that really helped us spend less was that um turns out that if you tell people like how much they're spending and like give them a little bit more visibility, a little bit more alerting, and give them the agency to self-manage, they actually do a pretty good job, you know.
12:39We built a system to basically have teams kind of set their own alerting and limits.
12:43And we've seen like many teams, you know, basically set reasonable limits for themselves.
12:48Another option is, you know, exploring open-source models.
12:52Uh you know, there are a bunch of open-source models that are on token cost pretty competitive with the frontier models out there.
12:58It's not necessarily right for every use case, but we found opportunities where they're a good value replacement.
13:04And we've been basically making that available internally.
13:07Uh and we've seen more and more people leverage those.
13:11Then of course, like there's things like model routing etc. There's we've been exploring different approaches to dynamically send the prompt to like a cheaper model etc. Or sometimes have a way where
13:22we have the like plan generated by like a more frontier model and the implementation be broken down by smaller models, etc. Um and finally, you know, I think knowledge bases are underrated,
13:33uh which yeah, I mean they're very important, but they also help get spend in control if you do your knowledge base effectively because then you're retrieving less context fundamentally if you just retrieve the right amount of context.
13:44Um so, long story short is that we're still investing I mean using a lot of tokens, but this is just about getting more ROI out of the same tokens.
13:54And finally, I kind of want to talk about uh meeting the users where they're at.
13:58Um so, a few months ago we were in this place where like a bunch of people are using different harnesses.
14:04They have all these weird bespoke setups to kind of team up into their dev boxes and some people were building, you know, hacky web UIs, etc. etc. Whatever.
14:12So, basically what we said was like, "Hey, you know, there's a bunch of people doing these types of things.
14:16Why don't we just build a nice version of this and then make it available to everyone in the company?"
14:19So, that's exactly what we did.
14:21In our web portal, we built this basically system where you can manage your sessions.
14:25You can have chat windows.
14:26You can have like artifacts in there.
14:29Um there are similar products out there offered by, you know, you know, some of the different uh harness companies, but we feel that this worked very well with our infrastructure
14:38um and we can kind of iterate it at the rate that we would like to.
14:42Um so, basically this was actually uh also at a at a place where different traders in the company are now kind of contributing code.
14:51They find something that's valuable to their workflow and then they did they just send up a PR and we just update it and then now we have things like, you know, collaborative document editing and things like that.
15:01And in fact, we're kind of getting this to a place where uh we're doing this, you know, agent multiplayer thing where now I have my sessions in there.
15:08We can create rooms with like my agents, your agents, like other deployed agents and we can kind of have all of that in one place and this really helps uh people be more productive and
15:19uh you know, we have now a Slack client, etc. So, you can also do it on the go and whatnot.
15:23So, basically, um our users are very high agency.
15:27Uh they will go and build something if we don't build it for them.
15:30So, we just built a better version of it.
15:34Yeah.
15:34So, uh basically, this is um the overall overarching philosophy of how we did this.
15:41Uh we have a platform, very modular, uh and um we continue to iterate on it.
15:46But, so far, we're having a pretty good time.
15:50Um that's the end of the talk.
15:52Uh but, if you have questions about anything on this or any other area, we can talk.
15:55And by the way, we have a networking event uh this evening.
15:59Uh please come to our booth and register.
16:00It's going to be fun.
16:01We have cool prizes.
16:04[applause]
16:08[applause]
16:11Questions?
16:19Yeah.
16:19So, we've um definitely been exploring a bunch of different options.
16:23Um personally, I'm not happy with any of the options out there.
16:28There's tradeoffs, but I mean, if you ask like some of the Frontier Labs what you do for knowledge base, they're like just huge markdown files, hierarchical markdown, etc. Um
16:36there's places where that works, but um there is um vector search has its own benefits and disadvantages.
16:44We have a hybrid approach that kind of uh allows us to explore these different options.
16:49Personally, I want us to spend more time around like doing things like perhaps some graph databases, etc. that are going to be uh more valuable.
16:57I mean, every day on Twitter, I see something like new way of like handling memory and knowledge.
17:03Um that's a problem that we want to spend more time on.
17:05I'm not happy with our solution.
17:10Go ahead.
17:15There's hand built or
17:20Yeah, good question.
17:22Um, value gained from using hand built solutions versus Codex.
17:25So, there's a few things.
17:27Um, I mean, one kind of boring answer is that because everything is kind of in our own infrastructure, a lot of these tools don't work in the way that you would want them to.
17:37Um, un- until recently, Codex didn't really have the option to kind of run managed agents, etc. or some of these things on your own infrastructure.
17:44Now, they've started offering them and we've been working with them on it, actually.
17:47But, um that was one reason why we started this way.
17:51Secondly, I think the um most Many of the places that offer these types of solutions also try to kind of lock you into either their model or some rest of their solution.
18:03But, the moment we kind of go down that path, then we end up in a place where if our infrastructure is overly invested in one particular provider, we move at the speed that they let us move.
18:14Uh, we've actually encountered with um that was one of the uh inference gateway providers that we used to use.
18:20Um, and basically, we would encounter scenarios where something comes out, we need it, and they're like, "Okay, uh let's talk to our GTM team.
18:28Let's put you on the road map."
18:29Well, it's like and then by that point, it's like, "Okay, no, we need to actually move faster."
18:33Um, so that's also one of the reasons.
18:35Uh, and finally, I think the in some of these areas, we've been um consistently ahead of the curve.
18:43Like, uh talking to some of the frontier labs, um the the way that we've been doing managed agent capabilities, etc. like, they're asking us to tell them how we did it.
18:54Not to be bragging too much, but I think the um we feel pretty confident in our engineering and kind of our road map.
19:01So, that's also uh been well.
19:03I don't think this approach works well for everybody, but I think we have a unique um coalition of circumstances that allow for us to do that.
19:11Good.
19:12Can you talk a bit about your Um yeah.
19:25So, just to make sure I understand your question correctly.
19:29What are some of the best practices of how we integrate the rest of this into eval and memory?
19:36Right.
19:37Um so, I think the we started at a place where, you know, we log everything, like I said, every tool call, input, output, etc. Just hotel logging.
19:47And that goes into our evaluation system.
19:52Um the challenge that we found with that, and we it's kind of an open problem, I would say, is that um when you're kind of trying to evaluate like a tool call, I mean, we have like, for example, a support bot agent that we run in Slack.
20:07And we want to evaluate that agent.
20:08Let's say, what if we ran this agent on a different model?
20:11Well, the support bot is leveraging a bunch of tools that depend on the state of the world that were different at the time when you ran it.
20:16And that's the pretty like typical problem.
20:18Um we've been working on basically things like, can we snapshot world states with these tools?
20:24And depending on the system that it ingresses with, it's, you know, there's different answers to that question.
20:29If you're looking at some like trading back test database, easy.
20:32Like that you can always point in time.
20:33That's what we do.
20:35But if you're looking at some other system, like if you're looking at Slack, for example, what was the state of Slack when this was originally evaluated?
20:41Maybe there's some ways to kind of replay Slack from a particular point in time, but other systems, not really.
20:46So, we kind of try to assign the confidence value as to how well we can point in time this particular tool invocation.
20:52And we still log it as such.
20:55And when we're rerunning the eval, we will run the tool with kind of saying, okay, I want to kind of the snapshot.
21:00We have like a proxy around the MCPs.
21:04But, you know, it's imperfect.
21:05And I don't know if there's a perfect solution other than just record, you know, everything.
21:15Yeah, I maybe I talk time for one more?
21:17Good.
21:31Yeah.
21:36That's a very classic problem that we've actually run into several times.
21:39You know, we had this one team that built an agent that did like coding and like analysis with them.
21:46It was kind of scoped to their repo and it was like, well, what if we also had worked with this team and have their thing also come to us and immediately kind of explodes in complexity.
21:53Now it's checking out multiple repos that have different off, etc. like typical problem.
21:58So, initially our solution was explore, okay, what if we like literally just did that?
22:03What if we just made the new agent that just had a union of the functionality, but often times that ends up, you know, a lot of tech debt, works very poorly, etc.
22:12I think our current answer to that type of paradigm is obviously every agent is version, etc. So, you can if it's small increments, you can push a new version to the agent, make make new capabilities,
22:23but going forward we would more encourage people to basically leverage our agent gateway where rather than one agent having like a 10 different things that it does, scope your agents to like a task that they're good at.
22:34You can easily measure them at, you can easily manage the life cycle of it and then there'll be another agent that it can talk to easily as if it was, you know, kind of operating on itself because we kind of have this construct of agents being able to go through one ledger, spin up each other, etc.
22:48And that will help us scale these types of functionality more effectively rather than just bloating one agent to do everything.
22:56It becomes kind of like a multi-agent orchestration problem, but you know, that's I think keeping things more measurable is really important because if you really bloat the agent then you can end up in scenarios where like, you know, some of the functionality gets better, some gets worse, and you can't really, you know, tweak it easily.