WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Ok6QdZHCcI8

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Ok6QdZHCcI8.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.720 --> 00:00:14.880
My name is David.

s3
00:00:14.880 --> 00:00:19.280
I lead AI operations at a company called Revolut.

s4
00:00:19.520 --> 00:00:22.560
We have 50 minutes today together.

s5
00:00:22.560 --> 00:00:28.760
We'll talk about high-performance engineering and self-hosting open-source model on dedicated GPU.

s6
00:00:28.760 --> 00:00:31.080
How's that sound?

s7
00:00:31.760 --> 00:00:32.439
Cool.

s8
00:00:32.439 --> 00:00:36.040
So, it's going to be a technical one, so bear with me.

s9
00:00:36.040 --> 00:00:41.560
Um so, maybe to start off because I know this is the US and you may not know Revolut.

s10
00:00:41.560 --> 00:00:44.000
Uh quick words about our company.

s11
00:00:44.000 --> 00:00:46.440
We're a digital bank in Europe.

s12
00:00:46.440 --> 00:00:48.600
We have 80 million customers.

s13
00:00:48.600 --> 00:00:51.840
We're live in 45 markets.

s14
00:00:52.000 --> 00:00:56.400
With the most downloaded financial app in Europe for the last year.

s15
00:00:56.400 --> 00:01:01.040
And as of July, we're Europe's largest private company.

s16
00:01:01.040 --> 00:01:05.000
Um So, we're doing okay.

s17
00:01:05.000 --> 00:01:16.120
The the emphasis of Revolut is, you know, shipping a lot of product and so there's a high focus on engineering culture and uh excellence on that domain.

s18
00:01:16.280 --> 00:01:18.720
And so, we'll talk about that.

s19
00:01:18.720 --> 00:01:22.400
So, let's talk about some AI.

s20
00:01:22.800 --> 00:01:23.920
We have a chart here.

s21
00:01:23.920 --> 00:01:25.320
I going to see two lines.

s22
00:01:25.320 --> 00:01:31.200
This is a sample of 2,000 engineers performance over the last year.

s23
00:01:31.320 --> 00:01:33.600
So, the green line is going to be lead time.

s24
00:01:33.600 --> 00:01:41.360
So, it's going to be when engineer picks up a feature until a feature is merged and ready to be deployed.

s25
00:01:41.360 --> 00:01:43.080
The blue line is going to be coding time.

s26
00:01:43.080 --> 00:01:44.520
So, the green line is in days.

s27
00:01:44.520 --> 00:01:47.840
The blue line is in hours.

s28
00:01:48.240 --> 00:01:53.240
Just to reset the timeline in case people forget because AI moves really fast.

s29
00:01:53.240 --> 00:01:57.080
Cloud code came out around February 2025, right?

s30
00:01:57.080 --> 00:02:02.252
And that's the Cloud Code Sonnet 3.7, the one would delete your prod database, right?

s31
00:02:02.252 --> 00:02:05.572
Not the safe Cloud and Codex that we have today.

s32
00:02:05.572 --> 00:02:08.692
So, bear in mind when we talk about adoption.

s33
00:02:09.292 --> 00:02:15.052
So, what you see here on this chart is January 25th, first introduced AI in the company, right?

s34
00:02:15.052 --> 00:02:16.252
Gemini.

s35
00:02:16.252 --> 00:02:21.172
First time people were able to use private data within AI.

s36
00:02:21.452 --> 00:02:24.132
Around February, we start playing around with Cloud Code.

s37
00:02:24.132 --> 00:02:29.372
And then January is when we first launched Cloud Code at scale.

s38
00:02:29.412 --> 00:02:40.772
What you see is Gemini had 10% improvement in lead time, 20% improvement in coding time, so that's people no longer needed to look Stack Overflow and read documentation.

s39
00:02:40.892 --> 00:02:45.092
But then Cloud Code not so much impact at the time.

s40
00:02:45.092 --> 00:02:49.532
So, when we launched it, we were highly skeptical about the cost of AI.

s41
00:02:49.532 --> 00:02:55.172
We were seeing engineers spend a thousand a month, ten thousand a month with not so much ROI.

s42
00:02:55.172 --> 00:03:00.372
And so, when we launched it, we launched it with strict budget limits.

s43
00:03:00.372 --> 00:03:01.612
And so, we're looking at the next thing, right?

s44
00:03:01.612 --> 00:03:08.372
So, how can we provide unlimited tokens to engineers so they can use AI guilt-free for a fraction of the cost.

s45
00:03:08.372 --> 00:03:11.732
And so, we looked into open source.

s46
00:03:11.732 --> 00:03:16.172
So, around April 26th is when we started self-hosting open source model.

s47
00:03:16.172 --> 00:03:18.652
We're a big fan of GLM for coding.

s48
00:03:18.652 --> 00:03:30.452
So, around that time what you see, April 26th, GLM 51 we see 60% improvement in coding time, 70% improvement in lead time.

s49
00:03:30.452 --> 00:03:33.412
So, dramatic impact.

s50
00:03:33.412 --> 00:03:34.252
What do we learn from this?

s51
00:03:34.252 --> 00:03:42.532
Well, there's more impact on the lead time than on the coding time, so what we find is the bottleneck for engineering is not necessarily coding.

s52
00:03:42.532 --> 00:03:47.812
You can save a lot of time not having to dig through your inbox or messing with Jira and Confluence.

s53
00:03:47.812 --> 00:03:51.932
And so, when you have unlimited token, you're free to do that.

s54
00:03:51.932 --> 00:03:58.732
You no longer have to save your tokens for end of the quarter or the most difficult task.

s55
00:03:59.412 --> 00:04:00.932
So, where do we stand?

s56
00:04:00.932 --> 00:04:06.534
Well, open source is now a priority at Revolut and yet the economics are favorable.

s57
00:04:06.534 --> 00:04:10.694
The billing is flat, we don't have any bill shock at the end of the month.

s58
00:04:10.694 --> 00:04:12.734
And the French here is the exception.

s59
00:04:12.734 --> 00:04:15.614
So, obviously, we keep using CodeX code code.

s60
00:04:15.614 --> 00:04:17.174
The latest models are fantastic.

s61
00:04:17.174 --> 00:04:20.214
Uh Jimmy and I are going very well as very good as well.

s62
00:04:20.214 --> 00:04:24.214
But, um we keep them as exception.

s63
00:04:24.894 --> 00:04:27.414
So, how do we get there?

s64
00:04:28.054 --> 00:04:31.854
Some metrics that matter when you're self-hosting model.

s65
00:04:31.854 --> 00:04:37.094
So, to serve 2,000 engineers, we roughly need 56 B200.

s66
00:04:37.094 --> 00:04:39.014
And that's to uh host GLM-53.

s67
00:04:39.014 --> 00:04:41.534
GLM is a big model, right?

s68
00:04:41.534 --> 00:04:44.014
800 billion parameters.

s69
00:04:44.534 --> 00:04:50.294
What you want to aim for is sub-3 second time to first token.

s70
00:04:50.294 --> 00:04:52.134
This is empirical evidence.

s71
00:04:52.134 --> 00:04:58.894
As soon as you exceed 5 seconds, my Slack inbox starts to blow up and so we know that's the limit.

s72
00:04:59.374 --> 00:05:01.374
In terms of throughput, it doesn't really matter as much, right?

s73
00:05:01.374 --> 00:05:05.974
So, we serve in terms of interactivity, 200 tokens per second.

s74
00:05:05.974 --> 00:05:08.214
People are fine, you don't need to go higher.

s75
00:05:08.214 --> 00:05:17.654
Um the way people work has changed, we no longer look at the stream of thinking tokens, people run tasks in parallel and check in when the task is done.

s76
00:05:17.654 --> 00:05:25.654
But, if you want to have high throughputs, when you're an engineer at Revolut, you just connect after 5:00 p.m. when no one is on GPU and you can get 800 TPS and then,

s77
00:05:25.654 --> 00:05:28.294
you know, it's a dream.

s78
00:05:28.894 --> 00:05:35.294
In terms of cost, in terms of impact, uh self-hosting GLM, our blended cost of token is 30 cents per million.

s79
00:05:35.294 --> 00:05:43.974
And so, our invoice is 1/10 of what we were spending on Anthropic and um OpenAI.

s80
00:05:45.974 --> 00:05:51.454
So, there's a lot of tuning that goes into running open source models.

s81
00:05:51.494 --> 00:06:01.094
If you just get GPUs and upload weights on a GPU and you start doing some inference, you're going to get terrible performance.

s82
00:06:01.094 --> 00:06:07.255
It's because your workload is going to be very dependent um on the parameters that you set.

s83
00:06:07.255 --> 00:06:12.895
So, if you look all the way to the left of that chart, the blue line is going to be demand, so that's request per second.

s84
00:06:12.895 --> 00:06:17.415
That's going to be what's going to be the bottleneck for your latency for your GPUs.

s85
00:06:17.495 --> 00:06:22.135
And the red line is what we're optimizing for, that's the time to first token.

s86
00:06:22.175 --> 00:06:30.255
So, when we started back in April, you could see request per second around three, which is high but not so high.

s87
00:06:30.255 --> 00:06:35.295
And time to first token over 10 second, which is very slow.

s88
00:06:35.335 --> 00:06:38.415
All of the data that you see is the same number of GPU.

s89
00:06:38.415 --> 00:06:42.055
We didn't increase GPU count, we're still at 56.

s90
00:06:42.055 --> 00:06:52.695
And by the end, you see in August request per second, we're about over seven, and time to first token we're around 1.6. So, we've had 88% improvement in performance for

s91
00:06:52.695 --> 00:06:55.929
76% increased demand.

s92
00:06:55.929 --> 00:06:56.135
[snorts]

s93
00:06:56.135 --> 00:07:05.735
So, the learning from this is there's a lot of parameters that you can improve on, and you really need to really scaling if you're looking to serve for some model.

s94
00:07:05.815 --> 00:07:08.535
So, let's talk parameters.

s95
00:07:08.695 --> 00:07:14.015
There's no secret recipe, but the things to consider are the following.

s96
00:07:14.535 --> 00:07:17.855
First one, long context is a scam.

s97
00:07:17.855 --> 00:07:25.855
You really need to be mindful that even though you can serve 1 million token, 256 256 is really what you need.

s98
00:07:26.655 --> 00:07:32.895
Quantization, if you're running Blackwell generation GPU, you don't want to run anything higher than FP4.

s99
00:07:32.895 --> 00:07:34.695
And VFP4 is best.

s100
00:07:34.695 --> 00:07:44.495
Roughly going from FP4 to FP8 is a 40% saving in GPU footprint for zero loss in uh, precision.

s101
00:07:45.255 --> 00:07:56.975
Other parameters to consider, tensor parallelism, even though GLM fits on uh, TP4, you don't have enough KV cache, and so maybe a TP8 works better depending on how large your workload is.

s102
00:07:57.095 --> 00:08:00.255
And then the more esoteric, check your cache hit.

s103
00:08:00.255 --> 00:08:04.855
Coding test needs to be above 90% uh, cache hit.

s104
00:08:04.935 --> 00:08:12.577
And you can look into optimize spec decoding, uh, sequential pre-fill, PGD zag, all of these.

s105
00:08:13.977 --> 00:08:14.297
Cool.

s106
00:08:14.297 --> 00:08:17.977
So, now let's talk about production setup.

s107
00:08:17.977 --> 00:08:23.297
Gnome obviously is a text-only model, which is a limitation for some people.

s108
00:08:23.297 --> 00:08:30.537
The trade-off I would say we would make every time is get the smarter model and figure out how to do image recognition another way.

s109
00:08:30.537 --> 00:08:32.457
You can easily have a local router.

s110
00:08:32.457 --> 00:08:42.377
Open Codex is one where you would route any image recognition to Gemini or Luna and keep Gemini for the main task.

s111
00:08:42.657 --> 00:08:48.297
If you really need some vision model, Gnome 5G flash works really well.

s112
00:08:48.297 --> 00:08:53.817
We have our UI designer making great use of it.

s113
00:08:53.817 --> 00:08:59.777
Turnaround for UI product has gone a really, really long way.

s114
00:09:00.497 --> 00:09:04.857
And then the last one is going to be about the harness.

s115
00:09:04.897 --> 00:09:15.977
So, all of these endpoints are going to be OpenAI compatible API with response API and so you can use them inside Open Code Codex Codex app as well.

s116
00:09:15.977 --> 00:09:17.097
Works really well.

s117
00:09:17.097 --> 00:09:20.377
And so, that's what we we recommend.

s118
00:09:23.977 --> 00:09:29.577
And then last message, this one is for the CIO in the room.

s119
00:09:29.577 --> 00:09:35.057
There's been a lot of stigma in using Chinese open source model.

s120
00:09:35.057 --> 00:09:40.457
So, the thing to consider is you're not using the inference dedicated endpoint that goes to China.

s121
00:09:40.457 --> 00:09:45.577
You're just getting the weights and putting them on a US-based service.

s122
00:09:45.577 --> 00:09:50.457
So, for data sharing side, this is totally fine.

s123
00:09:50.937 --> 00:09:59.857
Revolut in particular, because we're a EU regulated bank, we fall under the most strict regime in terms of compliance.

s124
00:09:59.857 --> 00:10:07.257
And so, what we find that works is all of these coding tasks are not considered autonomous agents.

s125
00:10:07.257 --> 00:10:09.577
They're always with a human in the loop.

s126
00:10:09.577 --> 00:10:14.018
And so, how do you give uh reassurance that you have a human in the loop?

s127
00:10:14.098 --> 00:10:17.418
You start by creating a sandbox that guarantees that.

s128
00:10:17.418 --> 00:10:23.578
And the sandbox means now, if you've used Codex code, it has all of these features of approve for me.

s129
00:10:23.578 --> 00:10:27.218
You just have to set the rules for the operations that you don't want to allow.

s130
00:10:27.218 --> 00:10:30.738
So, you can allow grab but not Gradle, for instance.

s131
00:10:30.738 --> 00:10:38.818
And that's going to give confidence uh to your regulator that this is not an autonomous agent that needs different uh supervision.

s132
00:10:39.418 --> 00:10:42.298
The sandbox also protects you from uh rogue use, right?

s133
00:10:42.298 --> 00:10:46.658
You may have bad actor inside your company, and you want to shield uh for that.

s134
00:10:46.658 --> 00:10:50.138
You'd be surprised what you find in a large company.

s135
00:10:50.378 --> 00:10:52.298
And then, the last one is prompt injection.

s136
00:10:52.298 --> 00:10:58.778
Not not so much a thing uh now given how smart models are, but always a thing.

s137
00:10:58.858 --> 00:11:00.498
In terms of data privacy, we spoke about it.

s138
00:11:00.498 --> 00:11:09.498
So, as long as the data sits in one of the countries that are in the data equivalency framework, so US, EU, UK, Israel, you're fine.

s139
00:11:09.498 --> 00:11:11.618
You can get your data to be used there.

s140
00:11:11.618 --> 00:11:18.978
All of your inference provider or GPT provider are going to have zero data retention by default.

s141
00:11:19.738 --> 00:11:29.018
And then, the last thing you need is a very strong uh open telemetry so that if you ever have an incident, um you need to be able to go deep

s142
00:11:29.018 --> 00:11:31.378
uh and look at the logs.

s143
00:11:31.378 --> 00:11:35.138
Also gives you some uh yeah, analytics.

s144
00:11:37.458 --> 00:11:37.898
All right.

s145
00:11:37.898 --> 00:11:38.538
Thanks, everyone.

s146
00:11:38.538 --> 00:11:40.858
That was all for my talk.

s147
00:11:40.858 --> 00:11:42.858
[applause]