WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=UB6hoWyh3B0

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/UB6hoWyh3B0.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.600 --> 00:00:20.720
And I'm going to show you the work we've been doing lastly on building a our first foundation model Wells Fargo foundation model.

s3
00:00:20.720 --> 00:00:26.440
So I explain why we did that, how we did it and what we accomplished in doing so.

s4
00:00:26.440 --> 00:00:31.440
So so first foundation model it doesn't necessarily to be language based.

s5
00:00:31.440 --> 00:00:39.280
I mean we all playing with Claude and chat GPT and others they all language based foundation models.

s6
00:00:39.280 --> 00:00:48.840
In financial services we also have language of course from a structure and structure representation data but we have also have a lot of structure representation.

s7
00:00:48.840 --> 00:00:50.960
A lot of this data is tabular data.

s8
00:00:50.960 --> 00:00:53.800
We have a lot of information from our customers.

s9
00:00:53.800 --> 00:01:07.720
We have information in investment banking, consumer banking and all of these data is data that we have in in Wells Fargo and all fun big financial organizations have to and we want to take benefit

s10
00:01:07.720 --> 00:01:17.160
of all of these huge amount of data that we have and it's not like you just take it and put it in a in a language model and get the best of it.

s11
00:01:17.160 --> 00:01:23.080
Of course you can get a lot of benefit out of it but it's not necessarily the best way to do it.

s12
00:01:23.080 --> 00:01:33.320
So so what we decided to do is to look at the data we have all the transactions all the customer transactions that they're doing with their debit card, credit card, digital check

s13
00:01:33.320 --> 00:01:44.680
all of those coming from many different channels and see if we can build a foundation model that we can use and fine tune for downstream task like credit scoring

s14
00:01:44.680 --> 00:01:48.720
or that you can use it for fraud detection, anti-money laundering, do marketing.

s15
00:01:48.720 --> 00:01:55.360
So how we can build one backbone that you can use and streamline across downstream task.

s16
00:01:55.360 --> 00:02:09.039
So now you're moving from many models very solid across different organizations in a really one backbone and every organization can just reuse because you learn the representations of customers transactions

s17
00:02:09.039 --> 00:02:10.759
at at scale.

s18
00:02:10.759 --> 00:02:11.560
So, that's the intent.

s19
00:02:11.560 --> 00:02:14.920
And just for that, you may say, "Hey, that's tabular data.

s20
00:02:14.920 --> 00:02:18.360
Why do you just don't use XGBoost and the usual suspect?"

s21
00:02:18.360 --> 00:02:19.000
We did that.

s22
00:02:19.000 --> 00:02:21.080
This is the model we had in production.

s23
00:02:21.080 --> 00:02:32.640
And but the model I'm going to show you is something that get that beats XGBoost even from a latency perspective, but also from a from a fraud detections, we get much better false

s24
00:02:32.640 --> 00:02:37.320
uh um reduction of false positive and better detection rate.

s25
00:02:37.320 --> 00:02:44.320
So, it was a really good lesson learned in term of having better model using foundation model.

s26
00:02:44.320 --> 00:02:47.440
And to foundation model, I've been transformer blocks.

s27
00:02:47.440 --> 00:02:49.320
And that that's that's what we built.

s28
00:02:49.320 --> 00:03:02.239
So, uh so, we have as I mentioned billion of transactions uh on a monthly basis um our customers interacting with their debit, credit card, and they do check, and they do different sort of transactions.

s29
00:03:02.239 --> 00:03:10.440
And the the question is really how we can use these informations at at scale and not for one and every single downstream task.

s30
00:03:10.440 --> 00:03:22.440
So, I'm going to cover a bit of the data, the representations, the architecture that we use, uh how that what the result we get, and the lesson we learned in building this.

s31
00:03:22.440 --> 00:03:22.720
All right.

s32
00:03:22.720 --> 00:03:34.920
So, I mean, left-hand side, language foundation model, they're processing language tokens, and then the the goal is to learn a representation that's going to predict the next token.

s33
00:03:34.920 --> 00:03:46.680
And then you have uh you have supervised fine-tuning, and then you do our LCHEF, LRVR, and all of the great uh post-training process to get uh better at the task that you want to, right?

s34
00:03:46.680 --> 00:03:47.640
And that's for languages.

s35
00:03:47.640 --> 00:03:48.840
That's great.

s36
00:03:48.840 --> 00:03:54.920
Uh but what if your data is uh very much more specialized in term of representation.

s37
00:03:54.920 --> 00:03:58.400
So, that's where we build this transaction foundation model.

s38
00:03:58.400 --> 00:04:04.360
So, instead of having languages as the base, the base is now transactions.

s39
00:04:04.360 --> 00:04:06.320
Transactions from our customers.

s40
00:04:06.320 --> 00:04:16.000
And you still try to predict the next thing and here the next thing is the next transaction or it's the next token in the transaction using all of the channel that you have.

s41
00:04:16.000 --> 00:04:17.560
But that's not the end goal.

s42
00:04:17.560 --> 00:04:23.160
I mean, the similarly to language model where you try to predict the next token, that's not the end of the story, right?

s43
00:04:23.160 --> 00:04:31.480
You learn a representation and then you use that representation and then you you you fine tune it and you post train it for any other task, right?

s44
00:04:31.480 --> 00:04:41.240
So, it could be classification task for languages, but in our context for transaction foundation model, it could be used for credit line increase, could be used for doing fraud detections

s45
00:04:41.240 --> 00:04:45.160
and flagging bad actors from an anti-money laundering perspective.

s46
00:04:45.160 --> 00:04:52.680
And the goal is to learn this relationship between transactions over time and transactions across customers.

s47
00:04:52.680 --> 00:04:53.640
That's really the intent.

s48
00:04:53.640 --> 00:04:58.280
So, the intent number one is you do pre-train from scratch.

s49
00:04:58.280 --> 00:05:01.160
I'll say it again because people sometimes confuse.

s50
00:05:01.160 --> 00:05:08.919
We pre-train transformer blocks at scale from scratch using our transaction data.

s51
00:05:08.919 --> 00:05:12.120
So, the weights is zero weights that we use from open weights.

s52
00:05:12.120 --> 00:05:17.560
We just use the architecture and then we we pre-train it from scratch using our GPU cluster.

s53
00:05:17.560 --> 00:05:21.560
And then we train it for couple of days and and weeks.

s54
00:05:21.560 --> 00:05:25.960
And and then you do the the downstream task of of post training.

s55
00:05:25.960 --> 00:05:32.040
But the goal here and you pre-train it for predicting the next token or the next transactions.

s56
00:05:32.040 --> 00:05:33.440
And you you try to to do this.

s57
00:05:33.440 --> 00:05:36.919
Well, you try to predict the next the next pattern pattern of the customer.

s58
00:05:36.919 --> 00:05:38.600
But that's not the intent.

s59
00:05:38.600 --> 00:05:41.000
The intent is to do the downstream task later on.

s60
00:05:41.000 --> 00:05:45.760
And in term of scale, I mean, we we operate at a large scale.

s61
00:05:45.760 --> 00:05:58.520
So, we have uh billions of transactions, a million of accounts, uh hundreds of transactions per clients on a monthly basis interacting with their debit card, credit card, or doing check.

s62
00:05:58.520 --> 00:06:08.800
And uh the five main dimensions we look at is really who is doing I mean, who is processing and who is asking on who is providing a service.

s63
00:06:08.800 --> 00:06:11.520
So, really the customer or the merchant.

s64
00:06:11.520 --> 00:06:15.720
Uh the what, so I mean, what what do we what do we buy?

s65
00:06:15.720 --> 00:06:17.320
What do we transact?

s66
00:06:17.320 --> 00:06:18.760
And how, so which channel?

s67
00:06:18.760 --> 00:06:19.520
Is it credit?

s68
00:06:19.520 --> 00:06:20.280
Is it debit?

s69
00:06:20.280 --> 00:06:21.560
Is it digital?

s70
00:06:21.560 --> 00:06:23.040
When is it happening?

s71
00:06:23.040 --> 00:06:24.040
And where is it happening?

s72
00:06:24.040 --> 00:06:28.200
And all of those, they basically breaking down in Android of features.

s73
00:06:28.200 --> 00:06:38.360
And uh if you look at the existing model we have in production, there's a lot of I mean, and people that been build those model in similar organizations, my

s74
00:06:38.360 --> 00:06:46.240
uh my understand that we build a lot of very hard crafted uncrafted engineering uh feature engineering effort on that.

s75
00:06:46.240 --> 00:06:48.440
And that's required a lot and a lot of time.

s76
00:06:48.440 --> 00:06:59.800
So, we step back and look at really the raw features we need, not the one that been really pre-built and post-built for doing the downstream task, but try to understand which raw features.

s77
00:06:59.800 --> 00:07:01.169
And we ended up with uh basically

s78
00:07:01.169 --> 00:07:01.640
[snorts]

s79
00:07:01.640 --> 00:07:02.880
100 features.

s80
00:07:02.880 --> 00:07:10.680
Even with at the end of the day, we had thousands of them, but we tried to find which ones are really the the raw information unit.

s81
00:07:10.680 --> 00:07:17.960
And then we let the architecture, the transformer blocks to figure out the relationship between those core features.

s82
00:07:17.960 --> 00:07:25.200
And uh as you can see, uh we do have a a yeah, a bunch of features on those five dimensions.

s83
00:07:25.200 --> 00:07:30.760
And the vocabulary is very unique to the financial services setup.

s84
00:07:30.760 --> 00:07:42.920
So, and that's the reason why I mean, we didn't really use uh existing pre-trained model uh because those the vocabulary that we use is very, very be to the work that you want to do

s85
00:07:42.920 --> 00:07:59.160
and even if you merge well all digital and checking and saving and all of those sort of transactions, even if you merge all of those, you still end up with a very limited numbers of features and characteristics.

s86
00:07:59.160 --> 00:08:03.640
So at that scale then the question is well how how you work with that?

s87
00:08:03.640 --> 00:08:06.160
So So we started everything is tabular data.

s88
00:08:06.160 --> 00:08:16.320
I mean there's no secret it's tabular data and then the goal is to understand how you going to capture the vocabulary because you do have numerical values, you have categorical values,

s89
00:08:16.320 --> 00:08:20.200
you have cat I mean you have information like zip code.

s90
00:08:20.200 --> 00:08:23.000
Do you encode each and every zip code in a token?

s91
00:08:23.000 --> 00:08:24.200
Does that make sense?

s92
00:08:24.200 --> 00:08:35.360
So there's a lot of effort need to push in understanding the vocabulary that you need and vocabulary that you want also to scale because at the training if you have a too huge vocabulary,

s93
00:08:35.360 --> 00:08:40.800
it would be kind of messy to train such such a model with a vocabulary which is far far too big.

s94
00:08:40.800 --> 00:08:45.000
So So there's a lot of effort in understanding this value to token.

s95
00:08:45.000 --> 00:08:48.080
So which information do you encode in a token?

s96
00:08:48.080 --> 00:08:51.920
And then you have the token 3D, the tokenizations and then the embedding lookup.

s97
00:08:51.920 --> 00:08:56.080
So then when we have a transaction coming up, you do have the vocabulary.

s98
00:08:56.080 --> 00:09:07.120
So it's a basically it's a lookup and aligning a transactions and all the characteristics of that transactions to to a token and an ID and then an embedding.

s99
00:09:08.000 --> 00:09:11.160
So it just to give you a number well some numbers.

s100
00:09:11.160 --> 00:09:16.160
So we have quantitative, qualitative, categorical of different kind, right?

s101
00:09:16.160 --> 00:09:20.720
So categories with only eight types, some with much bigger types.

s102
00:09:20.720 --> 00:09:23.360
You have transaction amount which is a big deal.

s103
00:09:23.360 --> 00:09:26.000
You have the delta time between transaction.

s104
00:09:26.000 --> 00:09:38.920
So you have anything related to IP address, postal code to just to give you example of uh uh variety and heterogeneity of each and every uh, characteristic could be encoded in a binary token

s105
00:09:38.920 --> 00:09:42.280
or could be coded in a much larger number of tokens.

s106
00:09:42.280 --> 00:09:50.480
And even if you just look at the bit that the zip code, I mean, you have a much larger number of zip code in the in in the US than those 2800,

s107
00:09:50.480 --> 00:10:01.040
but you you have fancier, well, you have much better way to structure how you're going to uh, regroup zip code based on distributions, for instance.

s108
00:10:01.040 --> 00:10:12.880
So, uh, architecture So, so here just um, what we So, we use the the Chinchilla um, well, mathematics behind the to define how we build it, right?

s109
00:10:12.880 --> 00:10:24.040
I mean, initially, cuz we we had no clue how it's going to work in a tabular space, even though there are existing uh, well, well, open weights and open solutions for doing

s110
00:10:24.040 --> 00:10:26.560
um, tabular data foundation models.

s111
00:10:26.560 --> 00:10:31.760
Uh, so we we we started from what they learned and what I did to what we needed to do.

s112
00:10:31.760 --> 00:10:41.400
And I'll give you an example of innovating dimensions, number of decoder layers we used, and um, all of the the recipe from a from an architecture perspective.

s113
00:10:41.400 --> 00:10:47.400
Um, so we have And as you can see, we're not trying to build a trillion parameters model.

s114
00:10:47.400 --> 00:10:57.560
We don't try to compete with any of the uh, the the leaders in the language models, because we're working really and focusing on transactions, right?

s115
00:10:57.560 --> 00:11:00.920
Transactions foundation models at billion level.

s116
00:11:00.920 --> 00:11:09.000
And one 1 billion is because that's the model that give us uh, that was the well, kind of the smallest and give us the highest boost.

s117
00:11:09.000 --> 00:11:19.360
And now we're working on a on a next version that's 11 billion, but that's good enough to to give a lot of uh, ROI uh, from a false positive reductions for

s118
00:11:19.360 --> 00:11:24.920
downstream tasks, like fraud detections and detection unit detection rate.

s119
00:11:24.920 --> 00:11:34.640
So, and the and the type of architecture So, the decoder-only architecture, uh, that's basically for predicting the next the next token or the next transactions.

s120
00:11:34.640 --> 00:11:36.360
And that's really the the pre-training.

s121
00:11:36.360 --> 00:11:42.760
So So here we initially used 20 times the number of parameters to define the number of tokens we might need.

s122
00:11:42.760 --> 00:11:45.560
We ended up actually 10 times factor factor than that.

s123
00:11:45.560 --> 00:11:51.760
So by doing a a few experimentations, we needed much more much more tokens per parameters.

s124
00:11:51.760 --> 00:11:56.960
I mean this is this kind of the initial experimentation we did.

s125
00:11:56.960 --> 00:12:10.760
So I'll pass on this because this is the the usual suspects from uh objective functions causal language modeling and mass language modeling type of uh uh optimization work.

s126
00:12:10.760 --> 00:12:15.760
And and yeah, from an architecture perspective, nothing really I mean fancy here.

s127
00:12:15.760 --> 00:12:19.960
The the the again, the most complex part was really the tokenization.

s128
00:12:19.960 --> 00:12:22.200
And then and then adapting and doing the pre-training.

s129
00:12:22.200 --> 00:12:24.080
And then how you do the fine-tuning.

s130
00:12:24.080 --> 00:12:31.480
So if I would say the the recipe or the secret sauce is really how you do the tokenization and then how you going to do the post the post training.

s131
00:12:31.480 --> 00:12:44.120
But anything from a training perspective, it's uh it it the question is just how many how many transaction you're going to put in your in your context or in your historic for for doing the training.

s132
00:12:44.120 --> 00:12:52.080
And and evaluation perplexity base accuracy accuracy dashboard on downstream tasks, that's that's the usual suspect.

s133
00:12:52.080 --> 00:13:10.400
And uh so Yeah, so use mixed precision distributed data parallel um maximize micro batch size um get faster convergence um yeah, using one rope usual and memory efficiency flash attention

s134
00:13:10.400 --> 00:13:11.960
activation checkpointing.

s135
00:13:11.960 --> 00:13:13.839
I mean here there's nothing really new.

s136
00:13:13.839 --> 00:13:22.640
It just we used the the the the the advices from the from the best and from the the one that built even much bigger models.

s137
00:13:22.640 --> 00:13:24.920
So I'll pass on this one.

s138
00:13:24.920 --> 00:13:30.520
Time is short, but I wanted to spend some time yeah on the results.

s139
00:13:30.520 --> 00:13:41.600
So um Yeah, so the when you build this foundation, that foundation is really is is of is really is on your backbone.

s140
00:13:41.600 --> 00:13:44.280
It's something you can use for downstream task.

s141
00:13:44.280 --> 00:13:48.160
And the downstream task was to do fraud classification.

s142
00:13:48.160 --> 00:13:53.000
So you have a transactions and then you try to predict if that transaction is going to be fraudulent.

s143
00:13:53.000 --> 00:14:02.880
So in the model we have in production, it's an XGBoost model and it's trained on large amount of transactions which is fraudulent or non-fraudulent.

s144
00:14:02.880 --> 00:14:15.880
And the missing piece is that was the relationship between whatever transaction you try to predict try to classify and the relationship with this transaction with all of the past transactions.

s145
00:14:15.880 --> 00:14:24.200
And that information is kind of gone when you use an XGBoost model unless you encode it in a very hard way which is which is not so easy.

s146
00:14:24.200 --> 00:14:32.560
But if you use transformer architectures, you do have this nice attentions from a from transactions to past transactions.

s147
00:14:32.560 --> 00:14:39.720
And that's basically what helped and this is what we think this is what really helped pushing the fraud detection rate much higher.

s148
00:14:39.720 --> 00:14:44.560
But at the end of the day the downstream task was to say here's the sequence of transactions.

s149
00:14:44.560 --> 00:14:46.960
I have someone just using the debit card.

s150
00:14:46.960 --> 00:14:52.440
Is that transaction fraudulent with respect of all his historical information?

s151
00:14:52.440 --> 00:14:59.560
And and that's that's exactly what you put in in at inference time when you try to do the prediction.

s152
00:14:59.560 --> 00:15:03.400
So so that's the yeah architecture.

s153
00:15:03.400 --> 00:15:05.000
So nothing really fancy here.

s154
00:15:05.000 --> 00:15:18.480
You just need to have the good engineering in place to do less than 250 milliseconds for doing the processes because the processing because the the decline need to happen within that time frame.

s155
00:15:18.480 --> 00:15:25.840
But it's it's it's we were well 10 times slower than that from a latency perspective, so it was all good.

s156
00:15:25.840 --> 00:15:28.920
And and just in terms of final result we get.

s157
00:15:28.920 --> 00:15:38.440
So, the main metrics and those metrics relate right away to to dollar to cost well to to savings for the firm.

s158
00:15:38.440 --> 00:15:41.480
So, it's detection rate and false positive rate.

s159
00:15:41.480 --> 00:15:47.880
Of course, you can detect much higher, but in many cases sometimes we get much more false positive.

s160
00:15:47.880 --> 00:15:55.600
And in this case, we managed to get more weak flag more fraudulent activities and we even reduced the false positive.

s161
00:15:55.600 --> 00:15:56.840
So, that that was a big win.

s162
00:15:56.840 --> 00:16:04.520
That was a big win because at the end of the day, if you decline a transaction which is not fraudulent, then that person is going to call.

s163
00:16:04.520 --> 00:16:07.240
There's often operation cost associated to it.

s164
00:16:07.240 --> 00:16:11.560
You lose the transaction related to that that non-fraudulent action.

s165
00:16:11.560 --> 00:16:17.680
So, it's a it's a very it's a very well, something that you you don't really want to happen.

s166
00:16:17.680 --> 00:16:24.560
So, so again, that's this that's compared to the state of the art of XGBoost model using fraud detection.

s167
00:16:24.560 --> 00:16:30.760
So, it's it's basically huge it's millions of dollar savings.

s168
00:16:30.760 --> 00:16:33.400
So, it's it's a very big boost.

s169
00:16:33.400 --> 00:16:36.400
So, just to finish up on that slide.

s170
00:16:36.400 --> 00:16:47.520
So, um So, that foundation model has been built on billions of transactions from Wells Fargo customers.

s171
00:16:47.520 --> 00:16:49.080
That's number one.

s172
00:16:49.080 --> 00:16:51.800
It's it's pre-trained from scratch.

s173
00:16:51.800 --> 00:16:54.880
So, we don't pre-train from any open weight existing model.

s174
00:16:54.880 --> 00:17:00.960
We use existing architecture, but the way they zero rate, we pre-training on couple of days.

s175
00:17:00.960 --> 00:17:04.319
I mean, we did it in a couple of weeks initially, but then that's pre-trained.

s176
00:17:04.319 --> 00:17:09.120
And then you and then when it's pre-trained, it's pretty good in predicting the next transaction.

s177
00:17:09.120 --> 00:17:12.520
And when you done with that, then you can use it for many downstream task.

s178
00:17:12.520 --> 00:17:14.319
We did it for fraud detection.

s179
00:17:14.319 --> 00:17:20.760
Now we investigating for credit line increase, investigating for anti-money laundering, investigating for marketing.

s180
00:17:20.760 --> 00:17:26.800
So, now the same foundation is used across different organizations even Wells Fargo.

s181
00:17:26.800 --> 00:17:28.760
And yeah.

s182
00:17:28.760 --> 00:17:32.520
Thank you for having me and thank you.

s183
00:17:33.907 --> 00:17:35.907
[applause]