How Revolut Cut AI Coding Costs 90% With Self-Hosted Open Models - David Abitbol, Revolut

AI Engineer · 11 min · 147 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:01[music]
  2. 00:12My name is David.
  3. 00:14I lead AI operations at a company called Revolut.
  4. 00:19We have 50 minutes today together.
  5. 00:22We'll talk about high-performance engineering and self-hosting open-source model on dedicated GPU.
  6. 00:28How's that sound?
  7. 00:31Cool.
  8. 00:32So, it's going to be a technical one, so bear with me.
  9. 00:36Um so, maybe to start off because I know this is the US and you may not know Revolut.
  10. 00:41Uh quick words about our company.
  11. 00:44We're a digital bank in Europe.
  12. 00:46We have 80 million customers.
  13. 00:48We're live in 45 markets.
  14. 00:52With the most downloaded financial app in Europe for the last year.
  15. 00:56And as of July, we're Europe's largest private company.
  16. 01:01Um So, we're doing okay.
  17. 01:05The the emphasis of Revolut is, you know, shipping a lot of product and so there's a high focus on engineering culture and uh excellence on that domain.
  18. 01:16And so, we'll talk about that.
  19. 01:18So, let's talk about some AI.
  20. 01:22We have a chart here.
  21. 01:23I going to see two lines.
  22. 01:25This is a sample of 2,000 engineers performance over the last year.
  23. 01:31So, the green line is going to be lead time.
  24. 01:33So, it's going to be when engineer picks up a feature until a feature is merged and ready to be deployed.
  25. 01:41The blue line is going to be coding time.
  26. 01:43So, the green line is in days.
  27. 01:44The blue line is in hours.
  28. 01:48Just to reset the timeline in case people forget because AI moves really fast.
  29. 01:53Cloud code came out around February 2025, right?
  30. 01:57And that's the Cloud Code Sonnet 3.7, the one would delete your prod database, right?
  31. 02:02Not the safe Cloud and Codex that we have today.
  32. 02:05So, bear in mind when we talk about adoption.
  33. 02:09So, what you see here on this chart is January 25th, first introduced AI in the company, right?
  34. 02:15Gemini.
  35. 02:16First time people were able to use private data within AI.
  36. 02:21Around February, we start playing around with Cloud Code.
  37. 02:24And then January is when we first launched Cloud Code at scale.
  38. 02:29What you see is Gemini had 10% improvement in lead time, 20% improvement in coding time, so that's people no longer needed to look Stack Overflow and read documentation.
  39. 02:40But then Cloud Code not so much impact at the time.
  40. 02:45So, when we launched it, we were highly skeptical about the cost of AI.
  41. 02:49We were seeing engineers spend a thousand a month, ten thousand a month with not so much ROI.
  42. 02:55And so, when we launched it, we launched it with strict budget limits.
  43. 03:00And so, we're looking at the next thing, right?
  44. 03:01So, how can we provide unlimited tokens to engineers so they can use AI guilt-free for a fraction of the cost.
  45. 03:08And so, we looked into open source.
  46. 03:11So, around April 26th is when we started self-hosting open source model.
  47. 03:16We're a big fan of GLM for coding.
  48. 03:18So, around that time what you see, April 26th, GLM 51 we see 60% improvement in coding time, 70% improvement in lead time.
  49. 03:30So, dramatic impact.
  50. 03:33What do we learn from this?
  51. 03:34Well, there's more impact on the lead time than on the coding time, so what we find is the bottleneck for engineering is not necessarily coding.
  52. 03:42You can save a lot of time not having to dig through your inbox or messing with Jira and Confluence.
  53. 03:47And so, when you have unlimited token, you're free to do that.
  54. 03:51You no longer have to save your tokens for end of the quarter or the most difficult task.
  55. 03:59So, where do we stand?
  56. 04:00Well, open source is now a priority at Revolut and yet the economics are favorable.
  57. 04:06The billing is flat, we don't have any bill shock at the end of the month.
  58. 04:10And the French here is the exception.
  59. 04:12So, obviously, we keep using CodeX code code.
  60. 04:15The latest models are fantastic.
  61. 04:17Uh Jimmy and I are going very well as very good as well.
  62. 04:20But, um we keep them as exception.
  63. 04:24So, how do we get there?
  64. 04:28Some metrics that matter when you're self-hosting model.
  65. 04:31So, to serve 2,000 engineers, we roughly need 56 B200.
  66. 04:37And that's to uh host GLM-53.
  67. 04:39GLM is a big model, right?
  68. 04:41800 billion parameters.
  69. 04:44What you want to aim for is sub-3 second time to first token.
  70. 04:50This is empirical evidence.
  71. 04:52As soon as you exceed 5 seconds, my Slack inbox starts to blow up and so we know that's the limit.
  72. 04:59In terms of throughput, it doesn't really matter as much, right?
  73. 05:01So, we serve in terms of interactivity, 200 tokens per second.
  74. 05:05People are fine, you don't need to go higher.
  75. 05:08Um the way people work has changed, we no longer look at the stream of thinking tokens, people run tasks in parallel and check in when the task is done.
  76. 05:17But, if you want to have high throughputs, when you're an engineer at Revolut, you just connect after 5:00 p.m. when no one is on GPU and you can get 800 TPS and then,
  77. 05:25you know, it's a dream.
  78. 05:28In terms of cost, in terms of impact, uh self-hosting GLM, our blended cost of token is 30 cents per million.
  79. 05:35And so, our invoice is 1/10 of what we were spending on Anthropic and um OpenAI.
  80. 05:45So, there's a lot of tuning that goes into running open source models.
  81. 05:51If you just get GPUs and upload weights on a GPU and you start doing some inference, you're going to get terrible performance.
  82. 06:01It's because your workload is going to be very dependent um on the parameters that you set.
  83. 06:07So, if you look all the way to the left of that chart, the blue line is going to be demand, so that's request per second.
  84. 06:12That's going to be what's going to be the bottleneck for your latency for your GPUs.
  85. 06:17And the red line is what we're optimizing for, that's the time to first token.
  86. 06:22So, when we started back in April, you could see request per second around three, which is high but not so high.
  87. 06:30And time to first token over 10 second, which is very slow.
  88. 06:35All of the data that you see is the same number of GPU.
  89. 06:38We didn't increase GPU count, we're still at 56.
  90. 06:42And by the end, you see in August request per second, we're about over seven, and time to first token we're around 1.6. So, we've had 88% improvement in performance for
  91. 06:5276% increased demand.
  92. 06:55[snorts]
  93. 06:56So, the learning from this is there's a lot of parameters that you can improve on, and you really need to really scaling if you're looking to serve for some model.
  94. 07:05So, let's talk parameters.
  95. 07:08There's no secret recipe, but the things to consider are the following.
  96. 07:14First one, long context is a scam.
  97. 07:17You really need to be mindful that even though you can serve 1 million token, 256 256 is really what you need.
  98. 07:26Quantization, if you're running Blackwell generation GPU, you don't want to run anything higher than FP4.
  99. 07:32And VFP4 is best.
  100. 07:34Roughly going from FP4 to FP8 is a 40% saving in GPU footprint for zero loss in uh, precision.
  101. 07:45Other parameters to consider, tensor parallelism, even though GLM fits on uh, TP4, you don't have enough KV cache, and so maybe a TP8 works better depending on how large your workload is.
  102. 07:57And then the more esoteric, check your cache hit.
  103. 08:00Coding test needs to be above 90% uh, cache hit.
  104. 08:04And you can look into optimize spec decoding, uh, sequential pre-fill, PGD zag, all of these.
  105. 08:13Cool.
  106. 08:14So, now let's talk about production setup.
  107. 08:17Gnome obviously is a text-only model, which is a limitation for some people.
  108. 08:23The trade-off I would say we would make every time is get the smarter model and figure out how to do image recognition another way.
  109. 08:30You can easily have a local router.
  110. 08:32Open Codex is one where you would route any image recognition to Gemini or Luna and keep Gemini for the main task.
  111. 08:42If you really need some vision model, Gnome 5G flash works really well.
  112. 08:48We have our UI designer making great use of it.
  113. 08:53Turnaround for UI product has gone a really, really long way.
  114. 09:00And then the last one is going to be about the harness.
  115. 09:04So, all of these endpoints are going to be OpenAI compatible API with response API and so you can use them inside Open Code Codex Codex app as well.
  116. 09:15Works really well.
  117. 09:17And so, that's what we we recommend.
  118. 09:23And then last message, this one is for the CIO in the room.
  119. 09:29There's been a lot of stigma in using Chinese open source model.
  120. 09:35So, the thing to consider is you're not using the inference dedicated endpoint that goes to China.
  121. 09:40You're just getting the weights and putting them on a US-based service.
  122. 09:45So, for data sharing side, this is totally fine.
  123. 09:50Revolut in particular, because we're a EU regulated bank, we fall under the most strict regime in terms of compliance.
  124. 09:59And so, what we find that works is all of these coding tasks are not considered autonomous agents.
  125. 10:07They're always with a human in the loop.
  126. 10:09And so, how do you give uh reassurance that you have a human in the loop?
  127. 10:14You start by creating a sandbox that guarantees that.
  128. 10:17And the sandbox means now, if you've used Codex code, it has all of these features of approve for me.
  129. 10:23You just have to set the rules for the operations that you don't want to allow.
  130. 10:27So, you can allow grab but not Gradle, for instance.
  131. 10:30And that's going to give confidence uh to your regulator that this is not an autonomous agent that needs different uh supervision.
  132. 10:39The sandbox also protects you from uh rogue use, right?
  133. 10:42You may have bad actor inside your company, and you want to shield uh for that.
  134. 10:46You'd be surprised what you find in a large company.
  135. 10:50And then, the last one is prompt injection.
  136. 10:52Not not so much a thing uh now given how smart models are, but always a thing.
  137. 10:58In terms of data privacy, we spoke about it.
  138. 11:00So, as long as the data sits in one of the countries that are in the data equivalency framework, so US, EU, UK, Israel, you're fine.
  139. 11:09You can get your data to be used there.
  140. 11:11All of your inference provider or GPT provider are going to have zero data retention by default.
  141. 11:19And then, the last thing you need is a very strong uh open telemetry so that if you ever have an incident, um you need to be able to go deep
  142. 11:29uh and look at the logs.
  143. 11:31Also gives you some uh yeah, analytics.
  144. 11:37All right.
  145. 11:37Thanks, everyone.
  146. 11:38That was all for my talk.
  147. 11:40[applause]