A Bank Trained Its Own Foundation Model to Catch Fraud — Freddy Lecue, Wells Fargo

AI Engineer · 17 min · 183 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12And I'm going to show you the work we've been doing lastly on building a our first foundation model Wells Fargo foundation model.
00:20So I explain why we did that, how we did it and what we accomplished in doing so.
00:26So so first foundation model it doesn't necessarily to be language based.
00:31I mean we all playing with Claude and chat GPT and others they all language based foundation models.
00:39In financial services we also have language of course from a structure and structure representation data but we have also have a lot of structure representation.
00:48A lot of this data is tabular data.
00:50We have a lot of information from our customers.
00:53We have information in investment banking, consumer banking and all of these data is data that we have in in Wells Fargo and all fun big financial organizations have to and we want to take benefit
01:07of all of these huge amount of data that we have and it's not like you just take it and put it in a in a language model and get the best of it.
01:17Of course you can get a lot of benefit out of it but it's not necessarily the best way to do it.
01:23So so what we decided to do is to look at the data we have all the transactions all the customer transactions that they're doing with their debit card, credit card, digital check
01:33all of those coming from many different channels and see if we can build a foundation model that we can use and fine tune for downstream task like credit scoring
01:44or that you can use it for fraud detection, anti-money laundering, do marketing.
01:48So how we can build one backbone that you can use and streamline across downstream task.
01:55So now you're moving from many models very solid across different organizations in a really one backbone and every organization can just reuse because you learn the representations of customers transactions
02:09at at scale.
02:10So, that's the intent.
02:11And just for that, you may say, "Hey, that's tabular data.
02:14Why do you just don't use XGBoost and the usual suspect?"
02:18We did that.
02:19This is the model we had in production.
02:21And but the model I'm going to show you is something that get that beats XGBoost even from a latency perspective, but also from a from a fraud detections, we get much better false
02:32uh um reduction of false positive and better detection rate.
02:37So, it was a really good lesson learned in term of having better model using foundation model.
02:44And to foundation model, I've been transformer blocks.
02:47And that that's that's what we built.
02:49So, uh so, we have as I mentioned billion of transactions uh on a monthly basis um our customers interacting with their debit, credit card, and they do check, and they do different sort of transactions.
03:02And the the question is really how we can use these informations at at scale and not for one and every single downstream task.
03:10So, I'm going to cover a bit of the data, the representations, the architecture that we use, uh how that what the result we get, and the lesson we learned in building this.
03:22All right.
03:22So, I mean, left-hand side, language foundation model, they're processing language tokens, and then the the goal is to learn a representation that's going to predict the next token.
03:34And then you have uh you have supervised fine-tuning, and then you do our LCHEF, LRVR, and all of the great uh post-training process to get uh better at the task that you want to, right?
03:46And that's for languages.
03:47That's great.
03:48Uh but what if your data is uh very much more specialized in term of representation.
03:54So, that's where we build this transaction foundation model.
03:58So, instead of having languages as the base, the base is now transactions.
04:04Transactions from our customers.
04:06And you still try to predict the next thing and here the next thing is the next transaction or it's the next token in the transaction using all of the channel that you have.
04:16But that's not the end goal.
04:17I mean, the similarly to language model where you try to predict the next token, that's not the end of the story, right?
04:23You learn a representation and then you use that representation and then you you you fine tune it and you post train it for any other task, right?
04:31So, it could be classification task for languages, but in our context for transaction foundation model, it could be used for credit line increase, could be used for doing fraud detections
04:41and flagging bad actors from an anti-money laundering perspective.
04:45And the goal is to learn this relationship between transactions over time and transactions across customers.
04:52That's really the intent.
04:53So, the intent number one is you do pre-train from scratch.
04:58I'll say it again because people sometimes confuse.
05:01We pre-train transformer blocks at scale from scratch using our transaction data.
05:08So, the weights is zero weights that we use from open weights.
05:12We just use the architecture and then we we pre-train it from scratch using our GPU cluster.
05:17And then we train it for couple of days and and weeks.
05:21And and then you do the the downstream task of of post training.
05:25But the goal here and you pre-train it for predicting the next token or the next transactions.
05:32And you you try to to do this.
05:33Well, you try to predict the next the next pattern pattern of the customer.
05:36But that's not the intent.
05:38The intent is to do the downstream task later on.
05:41And in term of scale, I mean, we we operate at a large scale.
05:45So, we have uh billions of transactions, a million of accounts, uh hundreds of transactions per clients on a monthly basis interacting with their debit card, credit card, or doing check.
05:58And uh the five main dimensions we look at is really who is doing I mean, who is processing and who is asking on who is providing a service.
06:08So, really the customer or the merchant.
06:11Uh the what, so I mean, what what do we what do we buy?
06:15What do we transact?
06:17And how, so which channel?
06:18Is it credit?
06:19Is it debit?
06:20Is it digital?
06:21When is it happening?
06:23And where is it happening?
06:24And all of those, they basically breaking down in Android of features.
06:28And uh if you look at the existing model we have in production, there's a lot of I mean, and people that been build those model in similar organizations, my
06:38uh my understand that we build a lot of very hard crafted uncrafted engineering uh feature engineering effort on that.
06:46And that's required a lot and a lot of time.
06:48So, we step back and look at really the raw features we need, not the one that been really pre-built and post-built for doing the downstream task, but try to understand which raw features.
06:59And we ended up with uh basically
07:01[snorts]
07:01100 features.
07:02Even with at the end of the day, we had thousands of them, but we tried to find which ones are really the the raw information unit.
07:10And then we let the architecture, the transformer blocks to figure out the relationship between those core features.
07:17And uh as you can see, uh we do have a a yeah, a bunch of features on those five dimensions.
07:25And the vocabulary is very unique to the financial services setup.
07:30So, and that's the reason why I mean, we didn't really use uh existing pre-trained model uh because those the vocabulary that we use is very, very be to the work that you want to do
07:42and even if you merge well all digital and checking and saving and all of those sort of transactions, even if you merge all of those, you still end up with a very limited numbers of features and characteristics.
07:59So at that scale then the question is well how how you work with that?
08:03So So we started everything is tabular data.
08:06I mean there's no secret it's tabular data and then the goal is to understand how you going to capture the vocabulary because you do have numerical values, you have categorical values,
08:16you have cat I mean you have information like zip code.
08:20Do you encode each and every zip code in a token?
08:23Does that make sense?
08:24So there's a lot of effort need to push in understanding the vocabulary that you need and vocabulary that you want also to scale because at the training if you have a too huge vocabulary,
08:35it would be kind of messy to train such such a model with a vocabulary which is far far too big.
08:40So So there's a lot of effort in understanding this value to token.
08:45So which information do you encode in a token?
08:48And then you have the token 3D, the tokenizations and then the embedding lookup.
08:51So then when we have a transaction coming up, you do have the vocabulary.
08:56So it's a basically it's a lookup and aligning a transactions and all the characteristics of that transactions to to a token and an ID and then an embedding.
09:08So it just to give you a number well some numbers.
09:11So we have quantitative, qualitative, categorical of different kind, right?
09:16So categories with only eight types, some with much bigger types.
09:20You have transaction amount which is a big deal.
09:23You have the delta time between transaction.
09:26So you have anything related to IP address, postal code to just to give you example of uh uh variety and heterogeneity of each and every uh, characteristic could be encoded in a binary token
09:38or could be coded in a much larger number of tokens.
09:42And even if you just look at the bit that the zip code, I mean, you have a much larger number of zip code in the in in the US than those 2800,
09:50but you you have fancier, well, you have much better way to structure how you're going to uh, regroup zip code based on distributions, for instance.
10:01So, uh, architecture So, so here just um, what we So, we use the the Chinchilla um, well, mathematics behind the to define how we build it, right?
10:12I mean, initially, cuz we we had no clue how it's going to work in a tabular space, even though there are existing uh, well, well, open weights and open solutions for doing
10:24um, tabular data foundation models.
10:26Uh, so we we we started from what they learned and what I did to what we needed to do.
10:31And I'll give you an example of innovating dimensions, number of decoder layers we used, and um, all of the the recipe from a from an architecture perspective.
10:41Um, so we have And as you can see, we're not trying to build a trillion parameters model.
10:47We don't try to compete with any of the uh, the the leaders in the language models, because we're working really and focusing on transactions, right?
10:57Transactions foundation models at billion level.
11:00And one 1 billion is because that's the model that give us uh, that was the well, kind of the smallest and give us the highest boost.
11:09And now we're working on a on a next version that's 11 billion, but that's good enough to to give a lot of uh, ROI uh, from a false positive reductions for
11:19downstream tasks, like fraud detections and detection unit detection rate.
11:24So, and the and the type of architecture So, the decoder-only architecture, uh, that's basically for predicting the next the next token or the next transactions.
11:34And that's really the the pre-training.
11:36So So here we initially used 20 times the number of parameters to define the number of tokens we might need.
11:42We ended up actually 10 times factor factor than that.
11:45So by doing a a few experimentations, we needed much more much more tokens per parameters.
11:51I mean this is this kind of the initial experimentation we did.
11:56So I'll pass on this because this is the the usual suspects from uh objective functions causal language modeling and mass language modeling type of uh uh optimization work.
12:10And and yeah, from an architecture perspective, nothing really I mean fancy here.
12:15The the the again, the most complex part was really the tokenization.
12:19And then and then adapting and doing the pre-training.
12:22And then how you do the fine-tuning.
12:24So if I would say the the recipe or the secret sauce is really how you do the tokenization and then how you going to do the post the post training.
12:31But anything from a training perspective, it's uh it it the question is just how many how many transaction you're going to put in your in your context or in your historic for for doing the training.
12:44And and evaluation perplexity base accuracy accuracy dashboard on downstream tasks, that's that's the usual suspect.
12:52And uh so Yeah, so use mixed precision distributed data parallel um maximize micro batch size um get faster convergence um yeah, using one rope usual and memory efficiency flash attention
13:10activation checkpointing.
13:11I mean here there's nothing really new.
13:13It just we used the the the the the advices from the from the best and from the the one that built even much bigger models.
13:22So I'll pass on this one.
13:24Time is short, but I wanted to spend some time yeah on the results.
13:30So um Yeah, so the when you build this foundation, that foundation is really is is of is really is on your backbone.
13:41It's something you can use for downstream task.
13:44And the downstream task was to do fraud classification.
13:48So you have a transactions and then you try to predict if that transaction is going to be fraudulent.
13:53So in the model we have in production, it's an XGBoost model and it's trained on large amount of transactions which is fraudulent or non-fraudulent.
14:02And the missing piece is that was the relationship between whatever transaction you try to predict try to classify and the relationship with this transaction with all of the past transactions.
14:15And that information is kind of gone when you use an XGBoost model unless you encode it in a very hard way which is which is not so easy.
14:24But if you use transformer architectures, you do have this nice attentions from a from transactions to past transactions.
14:32And that's basically what helped and this is what we think this is what really helped pushing the fraud detection rate much higher.
14:39But at the end of the day the downstream task was to say here's the sequence of transactions.
14:44I have someone just using the debit card.
14:46Is that transaction fraudulent with respect of all his historical information?
14:52And and that's that's exactly what you put in in at inference time when you try to do the prediction.
14:59So so that's the yeah architecture.
15:03So nothing really fancy here.
15:05You just need to have the good engineering in place to do less than 250 milliseconds for doing the processes because the processing because the the decline need to happen within that time frame.
15:18But it's it's it's we were well 10 times slower than that from a latency perspective, so it was all good.
15:25And and just in terms of final result we get.
15:28So, the main metrics and those metrics relate right away to to dollar to cost well to to savings for the firm.
15:38So, it's detection rate and false positive rate.
15:41Of course, you can detect much higher, but in many cases sometimes we get much more false positive.
15:47And in this case, we managed to get more weak flag more fraudulent activities and we even reduced the false positive.
15:55So, that that was a big win.
15:56That was a big win because at the end of the day, if you decline a transaction which is not fraudulent, then that person is going to call.
16:04There's often operation cost associated to it.
16:07You lose the transaction related to that that non-fraudulent action.
16:11So, it's a it's a very it's a very well, something that you you don't really want to happen.
16:17So, so again, that's this that's compared to the state of the art of XGBoost model using fraud detection.
16:24So, it's it's basically huge it's millions of dollar savings.
16:30So, it's it's a very big boost.
16:33So, just to finish up on that slide.
16:36So, um So, that foundation model has been built on billions of transactions from Wells Fargo customers.
16:47That's number one.
16:49It's it's pre-trained from scratch.
16:51So, we don't pre-train from any open weight existing model.
16:54We use existing architecture, but the way they zero rate, we pre-training on couple of days.
17:00I mean, we did it in a couple of weeks initially, but then that's pre-trained.
17:04And then you and then when it's pre-trained, it's pretty good in predicting the next transaction.
17:09And when you done with that, then you can use it for many downstream task.
17:12We did it for fraud detection.
17:14Now we investigating for credit line increase, investigating for anti-money laundering, investigating for marketing.
17:20So, now the same foundation is used across different organizations even Wells Fargo.
17:26And yeah.
17:28Thank you for having me and thank you.
17:33[applause]