WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=SHvwutPSeZg

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/SHvwutPSeZg.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:04.240 --> 00:00:08.720
Hi, I'm Sam Ray, VP of AI engineering.

s2
00:00:09.040 --> 00:00:13.440
What we see here are two score, same transaction both the times.

s3
00:00:13.440 --> 00:00:25.039
On the left, a single online purchase where we look at the amount spent, who is the merchant, MCC code, device type or time of the day and with 198 features by the time we are done engineering

s4
00:00:25.039 --> 00:00:27.199
and it scores 0.04.

s5
00:00:27.199 --> 00:00:30.240
comfortably under any threshold we would alert on.

s6
00:00:30.240 --> 00:00:34.160
As far as we are concerned, this one looks clean on the right.

s7
00:00:34.160 --> 00:00:36.559
Same transaction, same feature vector.

s8
00:00:36.559 --> 00:00:38.879
Not one bite of that row is different.

s9
00:00:38.879 --> 00:00:44.879
The only thing that we change is what model is allowed to look at the neighborhood it sits in.

s10
00:00:44.879 --> 00:00:51.840
Three of the four transactions, a user, a merchant, and two other transactions were flat fraud within the last hour.

s11
00:00:51.840 --> 00:00:54.000
Now the score is 0.91.

s12
00:00:54.000 --> 00:00:57.760
So the signal was never in the row, it was in the edges.

s13
00:00:57.760 --> 00:01:00.399
Therefore, fraud isn't a property of a transaction.

s14
00:01:00.399 --> 00:01:04.479
It is a property of the neighborhood it sits in.

s15
00:01:04.479 --> 00:01:09.680
Let me describe the shape of the problem because it is two problems wearing one name.

s16
00:01:09.680 --> 00:01:14.960
Physical transaction at a merchant point of sale roughly has 0.05% natural fraud rate.

s17
00:01:14.960 --> 00:01:16.720
That is one in 2,000.

s18
00:01:16.720 --> 00:01:22.720
These are tight low noise signal and fraud largely resigns the card's own transaction velocity.

s19
00:01:22.720 --> 00:01:34.640
Whereas online transactions have 0.6% of the fraud 12 times higher with higher volume noisier and far more of it of the signal sits in the merchant and the neighbor context.

s20
00:01:34.640 --> 00:01:39.119
These two portfolios have a different base rate and a different signal geometry.

s21
00:01:39.119 --> 00:01:42.079
Therefore in practice two models not one.

s22
00:01:42.079 --> 00:01:47.759
And here is the constraint that makes this as an engineering problem not and not just the modeling one.

s23
00:01:47.759 --> 00:01:49.680
The alert budget is fixed.

s24
00:01:49.680 --> 00:01:54.000
Analysts review the top 0.5 to 1.5% of the score transaction.

s25
00:01:54.000 --> 00:01:55.200
That's it.

s26
00:01:55.200 --> 00:02:06.799
So the area under the curve isn't just the objective we care about, rather the precision at a fixed K. To be worth anything, a model has to move a real fraud into that slice

s27
00:02:06.799 --> 00:02:09.920
and do it at the same latency.

s28
00:02:09.920 --> 00:02:14.400
So the obvious question here is why not just build a better HD boost model.

s29
00:02:14.400 --> 00:02:16.640
We did physical point of sale.

s30
00:02:16.640 --> 00:02:20.480
We went from the 40 raw attributes to 87 engineered features.

s31
00:02:20.480 --> 00:02:26.959
After selection and the leakage checks, the two positive rates moves from 35.5 to 36.6.

s32
00:02:26.959 --> 00:02:29.120
And for the online transaction, same story.

s33
00:02:29.120 --> 00:02:34.480
Nearly 200 engineered features and we pick up about a point.

s34
00:02:34.480 --> 00:02:38.640
So you know two to five times the feature surface one point of DPR.

s35
00:02:38.640 --> 00:02:40.640
Now that's not a tuning failure, right?

s36
00:02:40.640 --> 00:02:47.519
That's ceiling because every feature we engineered is just a function of the same row and the safe card's own history.

s37
00:02:47.519 --> 00:02:50.800
We are projecting the information we already had.

s38
00:02:50.800 --> 00:02:53.440
Tableau features answers one question.

s39
00:02:53.440 --> 00:02:55.599
What does this transaction look like?

s40
00:02:55.599 --> 00:02:57.120
They cannot answer the second one.

s41
00:02:57.120 --> 00:03:00.640
Who is it connected to and what are they doing right now?

s42
00:03:00.640 --> 00:03:04.159
Range sharing a device velocity chains across cart.

s43
00:03:04.159 --> 00:03:07.680
A merchant went that went bad 40 minutes ago.

s44
00:03:07.680 --> 00:03:14.400
This lives in edges between the rows and a model that sees one row at a time is blind to it by design.

s45
00:03:14.400 --> 00:03:17.920
No matter how many columns we add to it.

s46
00:03:17.920 --> 00:03:21.599
Okay, so here's the metal model for the rest of the talk.

s47
00:03:21.599 --> 00:03:25.519
A graph neural network does not learn a node from its own feature.

s48
00:03:25.519 --> 00:03:27.920
It learns from its its neighborhood.

s49
00:03:27.920 --> 00:03:30.000
One layer, three operations.

s50
00:03:30.000 --> 00:03:31.599
First, gather.

s51
00:03:31.599 --> 00:03:39.200
We pull in each neighbor's hidden states across the tie edges such as transaction to user, transaction to merchant aggregate.

s52
00:03:39.200 --> 00:03:48.640
We collapse that neighbor set which is you know a different size for every node one into a fixed width vector and we do it with a permutation in function.

s53
00:03:48.640 --> 00:03:54.159
Which function we picked mattered way more than we expected and charik is going to come back to that.

s54
00:03:54.159 --> 00:03:55.360
Third update.

s55
00:03:55.360 --> 00:04:01.680
We concentrate that aggregate with the node's own state, push it through learns and then we stack it twice.

s56
00:04:01.680 --> 00:04:08.959
So every transaction knows about its neighbors and its neighbors neighbor before the classifiers ever sees it.

s57
00:04:08.959 --> 00:04:10.879
So does these results work up?

s58
00:04:10.879 --> 00:04:12.480
Yes, it does.

s59
00:04:12.480 --> 00:04:15.439
And I want to be precise about what's being compared.

s60
00:04:15.439 --> 00:04:17.600
We did not replace XG post.

s61
00:04:17.600 --> 00:04:18.479
We fed it.

s62
00:04:18.479 --> 00:04:20.400
The genet produces the embeddings.

s63
00:04:20.400 --> 00:04:24.960
these embedding scores into the same gradient boosted model and that's what can score.

s64
00:04:24.960 --> 00:04:30.479
So XG boost alone versus XG boost plus the curve is what we are comparing at.

s65
00:04:30.479 --> 00:04:45.840
So the three numbers that we would like to highlight first is 27 to 30% 33% related lift in the true positive rate over a two leg baseline using the same alert rate same evaluation window and same features available to both the models.

s66
00:04:45.840 --> 00:04:57.520
Second, approximately $65 million projected annualized incremental fraud capture across the two cardboard or third the 3 millisecond P99 in barash.

s67
00:04:57.520 --> 00:05:06.160
So this talk is just not about the numbers, it's about how we got here four decisions and order matters.

s68
00:05:06.160 --> 00:05:10.000
First and the foremost sampling fraud is 1 in 2,000.

s69
00:05:10.000 --> 00:05:12.320
So we cannot train a raw distribution.

s70
00:05:12.320 --> 00:05:17.440
We have to lift the thought rate but this is not a tablet data set where the rows are independent.

s71
00:05:17.440 --> 00:05:20.639
Every negative we drop takes its edges with it.

s72
00:05:20.639 --> 00:05:26.240
Down sample the easy way we keep the labels but lose the neighborhood that makes them worth anything.

s73
00:05:26.240 --> 00:05:32.720
So holding a soft target fraud rate without really breaking the connectivity was the key.

s74
00:05:32.720 --> 00:05:44.479
Second, graphing features building at leakage state which is harder than it sounds when your edges carry time stamps plus 32 derived features created to detect fraud.

s75
00:05:44.479 --> 00:05:48.720
The third explanability analyst won't action a number.

s76
00:05:48.720 --> 00:05:56.960
This is a regulated work where the score has to come with an evidence so or it does not ship and serving at 3 milliseconds.

s77
00:05:56.960 --> 00:05:58.080
Two thing got us here.

s78
00:05:58.080 --> 00:06:02.479
The aggregation method and the pivot from the node level prediction to edge level.

s79
00:06:02.479 --> 00:06:14.639
Then at the end one iteration that we validated after launch was creating and training the training the model at a rolling window plus an edge himed DK approach and this move the numbers again

s80
00:06:14.639 --> 00:06:26.800
with this I would like to hand it over to Shahi to get into the nuts and bolts of how did we create the graphs how did we optimize the same and what were the engineering decisions that we have taken in order to make this model more optimized.

s81
00:06:27.199 --> 00:06:30.880
Hi I am Mohammed Sarik principal AI architect.

s82
00:06:30.880 --> 00:06:34.080
This is judge a transaction by the company it gives.

s83
00:06:34.080 --> 00:06:42.000
How a graph neural network bid exib boost by over 30% on fraud capture and how we serve it in 3 milliseconds.

s84
00:06:42.000 --> 00:06:48.240
I will walk through the technical build, sampling, graph construction, serving and explanability.

s85
00:06:48.240 --> 00:06:51.120
So let's get into it.

s86
00:06:51.120 --> 00:06:55.759
First decision most teams get wrong is the data sampling.

s87
00:06:55.759 --> 00:07:14.000
We hold a soft target fraud rate of.5% for card related merchant transactions about 10 times the 0.05% natural fraud rate and 6% for online transactions again about 10 times the 0.6%

s88
00:07:14.000 --> 00:07:15.440
natural fraud rate.

s89
00:07:15.440 --> 00:07:21.919
Both far gentle than the aggressive over or under sampling a tabular pipeline would use.

s90
00:07:21.919 --> 00:07:23.280
Why?

s91
00:07:23.280 --> 00:07:28.240
Because every selected user's complete transactional history stays in the graph.

s92
00:07:28.240 --> 00:07:35.360
If you trim a genuine user's history to hit a class ratio, you break the connectivity their neighbors depend on.

s93
00:07:35.360 --> 00:07:39.120
And a fraud user's own genuine transactions stay in two.

s94
00:07:39.120 --> 00:07:43.680
As hard negatives, the most useful example for the decision boundary.

s95
00:07:43.680 --> 00:07:48.479
If capping ever triggers, only the genuine users word budget shrinks.

s96
00:07:48.479 --> 00:07:50.479
Fraud users are never capped.

s97
00:07:50.479 --> 00:07:57.840
Class imbalance gets handled downstream instead with balanced batch sampling inside the trainer.

s98
00:07:58.400 --> 00:08:00.080
Here's the receipt.

s99
00:08:00.080 --> 00:08:11.520
Average user degree at training time 11 and a half to almost 14 for card related merchant transactions matches what the model sees at inference.

s100
00:08:11.520 --> 00:08:17.520
If training time neighborhood don't match inference time neighborhood, the model learns the wrong thing.

s101
00:08:17.520 --> 00:08:25.440
It's stable across all 12 months we tested at natural rate of 0.05% and 0.6%.

s102
00:08:25.440 --> 00:08:36.880
No windage drift to correct for second decision leakage control built into the graph itself not checked for afterward.

s103
00:08:36.880 --> 00:08:40.399
Each calendar month is its own disjoint graph.

s104
00:08:40.399 --> 00:08:42.880
No edge overcrosses the boundary.

s105
00:08:42.880 --> 00:08:49.120
The transaction is the central node with types out to its user and its merchant.

s106
00:08:49.120 --> 00:08:53.920
Plus a five hop follows chain back to its own recent history.

s107
00:08:53.920 --> 00:09:02.000
Because months never connect, there's no path for information to leak from one split into another.

s108
00:09:02.560 --> 00:09:06.399
Four properties fall out of that construction.

s109
00:09:06.399 --> 00:09:07.760
No future leakage.

s110
00:09:07.760 --> 00:09:12.560
A transaction only ever aggregates neighbors with an earlier time stamp.

s111
00:09:12.560 --> 00:09:16.640
No train val test leakage because of the disjointed month.

s112
00:09:16.640 --> 00:09:17.920
No label leakage.

s113
00:09:17.920 --> 00:09:22.240
The fraud flag is the target, never a feature or an edge.

s114
00:09:22.240 --> 00:09:24.160
And no distribution leakage.

s115
00:09:24.160 --> 00:09:29.519
Every scalar and vocabulary is fit on training month only then frozen.

s116
00:09:29.519 --> 00:09:36.320
Four for four verified by construction not by post hawk audit.

s117
00:09:37.040 --> 00:09:40.240
One more decision before the architecture.

s118
00:09:40.240 --> 00:09:44.320
202 model features across three node types.

s119
00:09:44.320 --> 00:09:50.959
73 on the user, 109 on the transaction, 20 on the merchant.

s120
00:09:50.959 --> 00:09:54.720
32 of the transactional features are derived.

s121
00:09:54.720 --> 00:09:59.600
Purpose-built for fraud added in three waves.

s122
00:09:59.600 --> 00:10:02.080
10 entity history features.

s123
00:10:02.080 --> 00:10:15.279
Past transaction count, days since last activity, the mean standard deviation and max of the last five amounts, a behavioral baseline per user and merchant.

s124
00:10:15.279 --> 00:10:24.720
Eight risk aggregates, an amount jet score against that history, a recent versus past ratio, transactions per day.

s125
00:10:24.720 --> 00:10:30.720
These flag deviation from the entity's own baseline, not some global threshold.

s126
00:10:30.720 --> 00:10:36.079
and 14 true rolling window velocity and temporal features.

s127
00:10:36.079 --> 00:10:42.079
5 minute, 1 hour, and 24hour transactional counts and amounts.

s128
00:10:42.079 --> 00:10:52.480
Distinct counterparties in the last day, seconds since the last transaction, even a country change rate for impossible travel patterns.

s129
00:10:52.480 --> 00:10:53.839
Why this matters?

s130
00:10:53.839 --> 00:10:57.920
The graph tells the model who a transaction is connected to.

s131
00:10:57.920 --> 00:11:10.959
These derived features tell it how that entity is behaving differently from its own normal complimentary signals not redundant ones under the hood.

s132
00:11:10.959 --> 00:11:22.880
This is graph says architecture sample aggregate combine embed sampling a fixed recency bias set of neighbors per hop.

s133
00:11:22.880 --> 00:11:27.440
So, one enormous merchant node can't dominate every message.

s134
00:11:27.440 --> 00:11:28.720
Aggregate.

s135
00:11:28.720 --> 00:11:31.920
And here's a detailed worth pausing on.

s136
00:11:31.920 --> 00:11:33.680
We don't just average.

s137
00:11:33.680 --> 00:11:37.360
We concatenate three order invariant statistics.

s138
00:11:37.360 --> 00:11:40.640
Mean, max, and standard deviation.

s139
00:11:40.640 --> 00:11:44.480
Combine that with the nodes own vector through learn weights.

s140
00:11:44.480 --> 00:11:47.839
Embed a fixed length vector to hops.

s141
00:11:47.839 --> 00:11:49.200
It's inductive.

s142
00:11:49.200 --> 00:11:54.160
We learned that the aggregator waits not a fixed vector per node.

s143
00:11:54.160 --> 00:11:58.880
So a brand new user or merchant gets embedded on the fly.

s144
00:11:58.880 --> 00:12:00.399
No retraining.

s145
00:12:00.399 --> 00:12:02.800
Why three statistics instead of one?

s146
00:12:02.800 --> 00:12:05.360
Let me show you.

s147
00:12:05.760 --> 00:12:08.639
Picture four neighbors of a transaction.

s148
00:12:08.639 --> 00:12:10.720
Three look completely normal.

s149
00:12:10.720 --> 00:12:16.240
Scores around 0.1. The fourth is tied to a fraud ring.

s150
00:12:16.240 --> 00:12:18.240
0.92.

s151
00:12:18.720 --> 00:12:23.279
Average those four together and you get 0.32.

s152
00:12:23.279 --> 00:12:29.279
That ring linked neighbor just got diluted into the crowd.

s153
00:12:29.279 --> 00:12:33.040
Take the max instead and it comes through untouched.

s154
00:12:33.040 --> 00:12:34.880
0.92.

s155
00:12:34.880 --> 00:12:37.360
That's why we don't rely on mean alone.

s156
00:12:37.360 --> 00:12:44.160
A single dangerous neighbor among many genuine ones is exactly the pattern fraud rings create.

s157
00:12:44.160 --> 00:12:49.279
and mean pooling is precisely the operation that would hide it.

s158
00:12:49.279 --> 00:12:52.160
The standard deviation adds a third view.

s159
00:12:52.160 --> 00:12:55.839
How much the neighbor disagree with itself.

s160
00:12:55.839 --> 00:13:03.040
A uniform boring neighborhood looks nothing like one with a live outlier in it.

s161
00:13:03.040 --> 00:13:06.639
And this cost us nothing extra at serving time.

s162
00:13:06.639 --> 00:13:08.800
No additional learned parameters.

s163
00:13:08.800 --> 00:13:28.079
still a fixed size output no matter how many neighbors a node has one quick hyperparameter note 128 dimension is the sweet spot 64 dimension is close but slightly under fit

s164
00:13:28.079 --> 00:13:43.279
256 cost more GPU and gives you nothing back validation performance flattens past 128 and then starts to overfeit It's not a philosophical debate, just a sweep.

s165
00:13:43.839 --> 00:13:47.360
Now the actual numbers side by side.

s166
00:13:47.360 --> 00:13:59.519
For card related merchant transactions, best Exib Boost gets 36% true positive rate at a fixed 0.5% alert rate.

s167
00:13:59.519 --> 00:14:02.959
GNN plus Exiboost gets 48%.

s168
00:14:02.959 --> 00:14:12.000
A 33% relative lift, $5.6 6 million captured at an 83% capture rate.

s169
00:14:12.000 --> 00:14:17.839
For online transactions, 48% up to 61%.

s170
00:14:17.839 --> 00:14:25.279
27% relative lift, $35.4 million captured at 71%.

s171
00:14:25.279 --> 00:14:38.800
Online transaction is where most of that $65 million savings lives since it runs at 12 times the fraud rate of card related merchant transactions.

s172
00:14:39.519 --> 00:14:43.120
Third decision explanability.

s173
00:14:43.120 --> 00:14:49.519
We ran subgraph attribution across 200 flag cases in each channel.

s174
00:14:49.519 --> 00:15:07.519
Transaction totaction velocity edges carry 0.94 mean importance in card related transactions 0.62 in online transactions and they are the top piece of evidence in all 200 card present cases.

s175
00:15:07.519 --> 00:15:24.800
Merchant edges matter more in online transactions 0.19 versus 0.04 04 in card present where fraud writes the card's own velocity rather than the acceptance point.

s176
00:15:24.800 --> 00:15:37.600
That's the difference between the model says 91 and the model says 91 because of these three linked transaction in the last 40 minutes.

s177
00:15:38.639 --> 00:15:43.600
Why does this matter so much specifically in fraud?

s178
00:15:43.600 --> 00:15:45.040
Four reasons.

s179
00:15:45.040 --> 00:15:49.279
One, regulatory and model risk management.

s180
00:15:49.279 --> 00:15:57.040
Financial institutions operate under frameworks that require documented rational for automated decision.

s181
00:15:57.040 --> 00:15:58.959
This isn't optional.

s182
00:15:58.959 --> 00:16:07.759
Two, analyst trust and investigation speed and unexplained high score gets rubber stamped or ignored.

s183
00:16:07.759 --> 00:16:15.279
Edge level evidence turns recomputee the case from scratch into verified three linked transactions.

s184
00:16:15.279 --> 00:16:27.199
Three, model debugging attribution patterns show us when the model leans on a speurious or decaying signal before performance visibly drops in production.

s185
00:16:27.199 --> 00:16:30.000
And four, feedback loop quality.

s186
00:16:30.000 --> 00:16:41.680
When explanation match what an analyst already suspects, they trust the borderline calls enough to label them correctly, improving the next round of training data.

s187
00:16:41.680 --> 00:16:46.560
An accurate black box simply doesn't get used.

s188
00:16:47.040 --> 00:16:53.440
Now the real fight serving these solution in 3 milliseconds.

s189
00:16:53.440 --> 00:17:13.439
offline van memory bound GPU sarded layer wise graph says forward pass over the entire graph bashed by destination node so it's numerically identical to a full graph forward just bounded to a few gigabyte at a time across multiple GPUs

s190
00:17:13.439 --> 00:17:22.959
every user merchant and recent transaction gets its per layer representation cast in a low latency embedding store.

s191
00:17:22.959 --> 00:17:30.559
Each user's last five transaction sit in a ring buffer online at request time.

s192
00:17:30.559 --> 00:17:37.200
A lightweight assembly step reconstructs the neighbor states straight from that cast.

s193
00:17:37.200 --> 00:17:39.840
No grab build, no sampling.

s194
00:17:39.840 --> 00:17:46.000
Recense weighted by an exponential decay that reproduces training exactly.

s195
00:17:46.000 --> 00:17:57.919
Then the realtime scoring services runs only the new transactions own two layer forward from those cache states plus the exeost head.

s196
00:17:57.919 --> 00:18:02.960
Nothing that scales with graph size runs per request.

s197
00:18:02.960 --> 00:18:06.799
Three small tricks make it fast in practice.

s198
00:18:06.799 --> 00:18:13.440
We pin math library thread pulls to one before the heavy libraries even load.

s199
00:18:13.440 --> 00:18:17.679
That alone is the difference between 30 milliseconds and one.

s200
00:18:17.679 --> 00:18:23.520
We warm the model up at startup so the first real request isn't slow.

s201
00:18:23.520 --> 00:18:26.799
And today it's a single in-memory process.

s202
00:18:26.799 --> 00:18:34.080
The same interface maps directly into a distributed cache when we need to scale out.

s203
00:18:34.720 --> 00:18:36.960
Now there's a catch.

s204
00:18:36.960 --> 00:18:45.039
That trick requires caching a representation for every node including every transaction at production scale.

s205
00:18:45.039 --> 00:18:53.760
That's 60 GBTE and it needs expensive high memory serving with a one day embedding lag in real time.

s206
00:18:53.760 --> 00:18:59.600
True positive rate actually drops from 48 to 44%.

s207
00:18:59.600 --> 00:19:03.840
The fix predict on the edge not the node.

s208
00:19:03.840 --> 00:19:09.120
premputee embeddings for user and merchants only, never per transaction.

s209
00:19:09.120 --> 00:19:21.360
That collapses the footprint to two to three gigabytes, about 20 times smaller, cheap enough to run on the same CPU class as the Exib Boost model it sits beside.

s210
00:19:21.360 --> 00:19:29.679
Nvidia independently recommended the same approach and expects it to perform on par with the node model.

s211
00:19:29.840 --> 00:19:32.400
Here's the actual request path.

s212
00:19:32.400 --> 00:19:39.280
pre-processing, cast lookup, and the exubost head are all fractions of a millisecond.

s213
00:19:39.280 --> 00:19:47.840
The GNN's own true two layer forward is the biggest single piece at 1.61 millisecond.

s214
00:19:47.840 --> 00:19:59.520
End to end 2.3 millisecond at the median, 3.1 at P99, comfortably inside our 3 to 4 millisecond SLA.

s215
00:19:59.520 --> 00:20:08.000
If we ever need more headroom that GNN forward is the first thing we would recompile or contise.

s216
00:20:12.418 --> 00:20:14.418
[music]

s217
00:20:16.720 --> 00:20:18.160
Thank you for watching.

s218
00:20:18.160 --> 00:20:24.400
We are both happy to go deeper sampling, graph construction, explanability or serving.

s219
00:20:24.400 --> 00:20:25.840
Reach out anytime.

s220
00:20:25.840 --> 00:20:28.000
Thank you.