How Revolut Cut AI Coding Costs 90% With Self-Hosted Open Models - David Abitbol, Revolut
https://www.youtube.com/watch?v=Ok6QdZHCcI8
AI Engineer · 11 min · 147 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:12] My name is David.
[00:14] I lead AI operations at a company called Revolut.
[00:19] We have 50 minutes today together.
[00:22] We'll talk about high-performance engineering and self-hosting open-source model on dedicated GPU.
[00:28] How's that sound?
[00:31] Cool.
[00:32] So, it's going to be a technical one, so bear with me.
[00:36] Um so, maybe to start off because I know this is the US and you may not know Revolut.
[00:41] Uh quick words about our company.
[00:44] We're a digital bank in Europe.
[00:46] We have 80 million customers.
[00:48] We're live in 45 markets.
[00:52] With the most downloaded financial app in Europe for the last year.
[00:56] And as of July, we're Europe's largest private company.
[01:01] Um So, we're doing okay.
[01:05] The the emphasis of Revolut is, you know, shipping a lot of product and so there's a high focus on engineering culture and uh excellence on that domain.
[01:16] And so, we'll talk about that.
[01:18] So, let's talk about some AI.
[01:22] We have a chart here.
[01:23] I going to see two lines.
[01:25] This is a sample of 2,000 engineers performance over the last year.
[01:31] So, the green line is going to be lead time.
[01:33] So, it's going to be when engineer picks up a feature until a feature is merged and ready to be deployed.
[01:41] The blue line is going to be coding time.
[01:43] So, the green line is in days.
[01:44] The blue line is in hours.
[01:48] Just to reset the timeline in case people forget because AI moves really fast.
[01:53] Cloud code came out around February 2025, right?
[01:57] And that's the Cloud Code Sonnet 3.7, the one would delete your prod database, right?
[02:02] Not the safe Cloud and Codex that we have today.
[02:05] So, bear in mind when we talk about adoption.
[02:09] So, what you see here on this chart is January 25th, first introduced AI in the company, right?
[02:15] Gemini.
[02:16] First time people were able to use private data within AI.
[02:21] Around February, we start playing around with Cloud Code.
[02:24] And then January is when we first launched Cloud Code at scale.
[02:29] What you see is Gemini had 10% improvement in lead time, 20% improvement in coding time, so that's people no longer needed to look Stack Overflow and read documentation.
[02:40] But then Cloud Code not so much impact at the time.
[02:45] So, when we launched it, we were highly skeptical about the cost of AI.
[02:49] We were seeing engineers spend a thousand a month, ten thousand a month with not so much ROI.
[02:55] And so, when we launched it, we launched it with strict budget limits.
[03:00] And so, we're looking at the next thing, right?
[03:01] So, how can we provide unlimited tokens to engineers so they can use AI guilt-free for a fraction of the cost.
[03:08] And so, we looked into open source.
[03:11] So, around April 26th is when we started self-hosting open source model.
[03:16] We're a big fan of GLM for coding.
[03:18] So, around that time what you see, April 26th, GLM 51 we see 60% improvement in coding time, 70% improvement in lead time.
[03:30] So, dramatic impact.
[03:33] What do we learn from this?
[03:34] Well, there's more impact on the lead time than on the coding time, so what we find is the bottleneck for engineering is not necessarily coding.
[03:42] You can save a lot of time not having to dig through your inbox or messing with Jira and Confluence.
[03:47] And so, when you have unlimited token, you're free to do that.
[03:51] You no longer have to save your tokens for end of the quarter or the most difficult task.
[03:59] So, where do we stand?
[04:00] Well, open source is now a priority at Revolut and yet the economics are favorable.
[04:06] The billing is flat, we don't have any bill shock at the end of the month.
[04:10] And the French here is the exception.
[04:12] So, obviously, we keep using CodeX code code.
[04:15] The latest models are fantastic.
[04:17] Uh Jimmy and I are going very well as very good as well.
[04:20] But, um we keep them as exception.
[04:24] So, how do we get there?
[04:28] Some metrics that matter when you're self-hosting model.
[04:31] So, to serve 2,000 engineers, we roughly need 56 B200.
[04:37] And that's to uh host GLM-53.
[04:39] GLM is a big model, right?
[04:41] 800 billion parameters.
[04:44] What you want to aim for is sub-3 second time to first token.
[04:50] This is empirical evidence.
[04:52] As soon as you exceed 5 seconds, my Slack inbox starts to blow up and so we know that's the limit.
[04:59] In terms of throughput, it doesn't really matter as much, right?
[05:01] So, we serve in terms of interactivity, 200 tokens per second.
[05:05] People are fine, you don't need to go higher.
[05:08] Um the way people work has changed, we no longer look at the stream of thinking tokens, people run tasks in parallel and check in when the task is done.
[05:17] But, if you want to have high throughputs, when you're an engineer at Revolut, you just connect after 5:00 p.m. when no one is on GPU and you can get 800 TPS and then,
[05:25] you know, it's a dream.
[05:28] In terms of cost, in terms of impact, uh self-hosting GLM, our blended cost of token is 30 cents per million.
[05:35] And so, our invoice is 1/10 of what we were spending on Anthropic and um OpenAI.
[05:45] So, there's a lot of tuning that goes into running open source models.
[05:51] If you just get GPUs and upload weights on a GPU and you start doing some inference, you're going to get terrible performance.
[06:01] It's because your workload is going to be very dependent um on the parameters that you set.
[06:07] So, if you look all the way to the left of that chart, the blue line is going to be demand, so that's request per second.
[06:12] That's going to be what's going to be the bottleneck for your latency for your GPUs.
[06:17] And the red line is what we're optimizing for, that's the time to first token.
[06:22] So, when we started back in April, you could see request per second around three, which is high but not so high.
[06:30] And time to first token over 10 second, which is very slow.
[06:35] All of the data that you see is the same number of GPU.
[06:38] We didn't increase GPU count, we're still at 56.
[06:42] And by the end, you see in August request per second, we're about over seven, and time to first token we're around 1.6. So, we've had 88% improvement in performance for
[06:52] 76% increased demand.
[06:55] [snorts]
[06:56] So, the learning from this is there's a lot of parameters that you can improve on, and you really need to really scaling if you're looking to serve for some model.
[07:05] So, let's talk parameters.
[07:08] There's no secret recipe, but the things to consider are the following.
[07:14] First one, long context is a scam.
[07:17] You really need to be mindful that even though you can serve 1 million token, 256 256 is really what you need.
[07:26] Quantization, if you're running Blackwell generation GPU, you don't want to run anything higher than FP4.
[07:32] And VFP4 is best.
[07:34] Roughly going from FP4 to FP8 is a 40% saving in GPU footprint for zero loss in uh, precision.
[07:45] Other parameters to consider, tensor parallelism, even though GLM fits on uh, TP4, you don't have enough KV cache, and so maybe a TP8 works better depending on how large your workload is.
[07:57] And then the more esoteric, check your cache hit.
[08:00] Coding test needs to be above 90% uh, cache hit.
[08:04] And you can look into optimize spec decoding, uh, sequential pre-fill, PGD zag, all of these.
[08:13] Cool.
[08:14] So, now let's talk about production setup.
[08:17] Gnome obviously is a text-only model, which is a limitation for some people.
[08:23] The trade-off I would say we would make every time is get the smarter model and figure out how to do image recognition another way.
[08:30] You can easily have a local router.
[08:32] Open Codex is one where you would route any image recognition to Gemini or Luna and keep Gemini for the main task.
[08:42] If you really need some vision model, Gnome 5G flash works really well.
[08:48] We have our UI designer making great use of it.
[08:53] Turnaround for UI product has gone a really, really long way.
[09:00] And then the last one is going to be about the harness.
[09:04] So, all of these endpoints are going to be OpenAI compatible API with response API and so you can use them inside Open Code Codex Codex app as well.
[09:15] Works really well.
[09:17] And so, that's what we we recommend.
[09:23] And then last message, this one is for the CIO in the room.
[09:29] There's been a lot of stigma in using Chinese open source model.
[09:35] So, the thing to consider is you're not using the inference dedicated endpoint that goes to China.
[09:40] You're just getting the weights and putting them on a US-based service.
[09:45] So, for data sharing side, this is totally fine.
[09:50] Revolut in particular, because we're a EU regulated bank, we fall under the most strict regime in terms of compliance.
[09:59] And so, what we find that works is all of these coding tasks are not considered autonomous agents.
[10:07] They're always with a human in the loop.
[10:09] And so, how do you give uh reassurance that you have a human in the loop?
[10:14] You start by creating a sandbox that guarantees that.
[10:17] And the sandbox means now, if you've used Codex code, it has all of these features of approve for me.
[10:23] You just have to set the rules for the operations that you don't want to allow.
[10:27] So, you can allow grab but not Gradle, for instance.
[10:30] And that's going to give confidence uh to your regulator that this is not an autonomous agent that needs different uh supervision.
[10:39] The sandbox also protects you from uh rogue use, right?
[10:42] You may have bad actor inside your company, and you want to shield uh for that.
[10:46] You'd be surprised what you find in a large company.
[10:50] And then, the last one is prompt injection.
[10:52] Not not so much a thing uh now given how smart models are, but always a thing.
[10:58] In terms of data privacy, we spoke about it.
[11:00] So, as long as the data sits in one of the countries that are in the data equivalency framework, so US, EU, UK, Israel, you're fine.
[11:09] You can get your data to be used there.
[11:11] All of your inference provider or GPT provider are going to have zero data retention by default.
[11:19] And then, the last thing you need is a very strong uh open telemetry so that if you ever have an incident, um you need to be able to go deep
[11:29] uh and look at the logs.
[11:31] Also gives you some uh yeah, analytics.
[11:37] All right.
[11:37] Thanks, everyone.
[11:38] That was all for my talk.
[11:40] [applause]