Judge a Transaction by the Company It Keeps — Somya Rai & MD Sharique, EXL

AI Engineer · 20 min · 220 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:04Hi, I'm Sam Ray, VP of AI engineering.
00:09What we see here are two score, same transaction both the times.
00:13On the left, a single online purchase where we look at the amount spent, who is the merchant, MCC code, device type or time of the day and with 198 features by the time we are done engineering
00:25and it scores 0.04.
00:27comfortably under any threshold we would alert on.
00:30As far as we are concerned, this one looks clean on the right.
00:34Same transaction, same feature vector.
00:36Not one bite of that row is different.
00:38The only thing that we change is what model is allowed to look at the neighborhood it sits in.
00:44Three of the four transactions, a user, a merchant, and two other transactions were flat fraud within the last hour.
00:51Now the score is 0.91.
00:54So the signal was never in the row, it was in the edges.
00:57Therefore, fraud isn't a property of a transaction.
01:00It is a property of the neighborhood it sits in.
01:04Let me describe the shape of the problem because it is two problems wearing one name.
01:09Physical transaction at a merchant point of sale roughly has 0.05% natural fraud rate.
01:14That is one in 2,000.
01:16These are tight low noise signal and fraud largely resigns the card's own transaction velocity.
01:22Whereas online transactions have 0.6% of the fraud 12 times higher with higher volume noisier and far more of it of the signal sits in the merchant and the neighbor context.
01:34These two portfolios have a different base rate and a different signal geometry.
01:39Therefore in practice two models not one.
01:42And here is the constraint that makes this as an engineering problem not and not just the modeling one.
01:47The alert budget is fixed.
01:49Analysts review the top 0.5 to 1.5% of the score transaction.
01:54That's it.
01:55So the area under the curve isn't just the objective we care about, rather the precision at a fixed K. To be worth anything, a model has to move a real fraud into that slice
02:06and do it at the same latency.
02:09So the obvious question here is why not just build a better HD boost model.
02:14We did physical point of sale.
02:16We went from the 40 raw attributes to 87 engineered features.
02:20After selection and the leakage checks, the two positive rates moves from 35.5 to 36.6.
02:26And for the online transaction, same story.
02:29Nearly 200 engineered features and we pick up about a point.
02:34So you know two to five times the feature surface one point of DPR.
02:38Now that's not a tuning failure, right?
02:40That's ceiling because every feature we engineered is just a function of the same row and the safe card's own history.
02:47We are projecting the information we already had.
02:50Tableau features answers one question.
02:53What does this transaction look like?
02:55They cannot answer the second one.
02:57Who is it connected to and what are they doing right now?
03:00Range sharing a device velocity chains across cart.
03:04A merchant went that went bad 40 minutes ago.
03:07This lives in edges between the rows and a model that sees one row at a time is blind to it by design.
03:14No matter how many columns we add to it.
03:17Okay, so here's the metal model for the rest of the talk.
03:21A graph neural network does not learn a node from its own feature.
03:25It learns from its its neighborhood.
03:27One layer, three operations.
03:30First, gather.
03:31We pull in each neighbor's hidden states across the tie edges such as transaction to user, transaction to merchant aggregate.
03:39We collapse that neighbor set which is you know a different size for every node one into a fixed width vector and we do it with a permutation in function.
03:48Which function we picked mattered way more than we expected and charik is going to come back to that.
03:54Third update.
03:55We concentrate that aggregate with the node's own state, push it through learns and then we stack it twice.
04:01So every transaction knows about its neighbors and its neighbors neighbor before the classifiers ever sees it.
04:08So does these results work up?
04:10Yes, it does.
04:12And I want to be precise about what's being compared.
04:15We did not replace XG post.
04:17We fed it.
04:18The genet produces the embeddings.
04:20these embedding scores into the same gradient boosted model and that's what can score.
04:24So XG boost alone versus XG boost plus the curve is what we are comparing at.
04:30So the three numbers that we would like to highlight first is 27 to 30% 33% related lift in the true positive rate over a two leg baseline using the same alert rate same evaluation window and same features available to both the models.
04:45Second, approximately $65 million projected annualized incremental fraud capture across the two cardboard or third the 3 millisecond P99 in barash.
04:57So this talk is just not about the numbers, it's about how we got here four decisions and order matters.
05:06First and the foremost sampling fraud is 1 in 2,000.
05:10So we cannot train a raw distribution.
05:12We have to lift the thought rate but this is not a tablet data set where the rows are independent.
05:17Every negative we drop takes its edges with it.
05:20Down sample the easy way we keep the labels but lose the neighborhood that makes them worth anything.
05:26So holding a soft target fraud rate without really breaking the connectivity was the key.
05:32Second, graphing features building at leakage state which is harder than it sounds when your edges carry time stamps plus 32 derived features created to detect fraud.
05:44The third explanability analyst won't action a number.
05:48This is a regulated work where the score has to come with an evidence so or it does not ship and serving at 3 milliseconds.
05:56Two thing got us here.
05:58The aggregation method and the pivot from the node level prediction to edge level.
06:02Then at the end one iteration that we validated after launch was creating and training the training the model at a rolling window plus an edge himed DK approach and this move the numbers again
06:14with this I would like to hand it over to Shahi to get into the nuts and bolts of how did we create the graphs how did we optimize the same and what were the engineering decisions that we have taken in order to make this model more optimized.
06:27Hi I am Mohammed Sarik principal AI architect.
06:30This is judge a transaction by the company it gives.
06:34How a graph neural network bid exib boost by over 30% on fraud capture and how we serve it in 3 milliseconds.
06:42I will walk through the technical build, sampling, graph construction, serving and explanability.
06:48So let's get into it.
06:51First decision most teams get wrong is the data sampling.
06:55We hold a soft target fraud rate of.5% for card related merchant transactions about 10 times the 0.05% natural fraud rate and 6% for online transactions again about 10 times the 0.6%
07:14natural fraud rate.
07:15Both far gentle than the aggressive over or under sampling a tabular pipeline would use.
07:21Why?
07:23Because every selected user's complete transactional history stays in the graph.
07:28If you trim a genuine user's history to hit a class ratio, you break the connectivity their neighbors depend on.
07:35And a fraud user's own genuine transactions stay in two.
07:39As hard negatives, the most useful example for the decision boundary.
07:43If capping ever triggers, only the genuine users word budget shrinks.
07:48Fraud users are never capped.
07:50Class imbalance gets handled downstream instead with balanced batch sampling inside the trainer.
07:58Here's the receipt.
08:00Average user degree at training time 11 and a half to almost 14 for card related merchant transactions matches what the model sees at inference.
08:11If training time neighborhood don't match inference time neighborhood, the model learns the wrong thing.
08:17It's stable across all 12 months we tested at natural rate of 0.05% and 0.6%.
08:25No windage drift to correct for second decision leakage control built into the graph itself not checked for afterward.
08:36Each calendar month is its own disjoint graph.
08:40No edge overcrosses the boundary.
08:42The transaction is the central node with types out to its user and its merchant.
08:49Plus a five hop follows chain back to its own recent history.
08:53Because months never connect, there's no path for information to leak from one split into another.
09:02Four properties fall out of that construction.
09:06No future leakage.
09:07A transaction only ever aggregates neighbors with an earlier time stamp.
09:12No train val test leakage because of the disjointed month.
09:16No label leakage.
09:17The fraud flag is the target, never a feature or an edge.
09:22And no distribution leakage.
09:24Every scalar and vocabulary is fit on training month only then frozen.
09:29Four for four verified by construction not by post hawk audit.
09:37One more decision before the architecture.
09:40202 model features across three node types.
09:4473 on the user, 109 on the transaction, 20 on the merchant.
09:5032 of the transactional features are derived.
09:54Purpose-built for fraud added in three waves.
09:5910 entity history features.
10:02Past transaction count, days since last activity, the mean standard deviation and max of the last five amounts, a behavioral baseline per user and merchant.
10:15Eight risk aggregates, an amount jet score against that history, a recent versus past ratio, transactions per day.
10:24These flag deviation from the entity's own baseline, not some global threshold.
10:30and 14 true rolling window velocity and temporal features.
10:365 minute, 1 hour, and 24hour transactional counts and amounts.
10:42Distinct counterparties in the last day, seconds since the last transaction, even a country change rate for impossible travel patterns.
10:52Why this matters?
10:53The graph tells the model who a transaction is connected to.
10:57These derived features tell it how that entity is behaving differently from its own normal complimentary signals not redundant ones under the hood.
11:10This is graph says architecture sample aggregate combine embed sampling a fixed recency bias set of neighbors per hop.
11:22So, one enormous merchant node can't dominate every message.
11:27Aggregate.
11:28And here's a detailed worth pausing on.
11:31We don't just average.
11:33We concatenate three order invariant statistics.
11:37Mean, max, and standard deviation.
11:40Combine that with the nodes own vector through learn weights.
11:44Embed a fixed length vector to hops.
11:47It's inductive.
11:49We learned that the aggregator waits not a fixed vector per node.
11:54So a brand new user or merchant gets embedded on the fly.
11:58No retraining.
12:00Why three statistics instead of one?
12:02Let me show you.
12:05Picture four neighbors of a transaction.
12:08Three look completely normal.
12:10Scores around 0.1. The fourth is tied to a fraud ring.
12:160.92.
12:18Average those four together and you get 0.32.
12:23That ring linked neighbor just got diluted into the crowd.
12:29Take the max instead and it comes through untouched.
12:330.92.
12:34That's why we don't rely on mean alone.
12:37A single dangerous neighbor among many genuine ones is exactly the pattern fraud rings create.
12:44and mean pooling is precisely the operation that would hide it.
12:49The standard deviation adds a third view.
12:52How much the neighbor disagree with itself.
12:55A uniform boring neighborhood looks nothing like one with a live outlier in it.
13:03And this cost us nothing extra at serving time.
13:06No additional learned parameters.
13:08still a fixed size output no matter how many neighbors a node has one quick hyperparameter note 128 dimension is the sweet spot 64 dimension is close but slightly under fit
13:28256 cost more GPU and gives you nothing back validation performance flattens past 128 and then starts to overfeit It's not a philosophical debate, just a sweep.
13:43Now the actual numbers side by side.
13:47For card related merchant transactions, best Exib Boost gets 36% true positive rate at a fixed 0.5% alert rate.
13:59GNN plus Exiboost gets 48%.
14:02A 33% relative lift, $5.6 6 million captured at an 83% capture rate.
14:12For online transactions, 48% up to 61%.
14:1727% relative lift, $35.4 million captured at 71%.
14:25Online transaction is where most of that $65 million savings lives since it runs at 12 times the fraud rate of card related merchant transactions.
14:39Third decision explanability.
14:43We ran subgraph attribution across 200 flag cases in each channel.
14:49Transaction totaction velocity edges carry 0.94 mean importance in card related transactions 0.62 in online transactions and they are the top piece of evidence in all 200 card present cases.
15:07Merchant edges matter more in online transactions 0.19 versus 0.04 04 in card present where fraud writes the card's own velocity rather than the acceptance point.
15:24That's the difference between the model says 91 and the model says 91 because of these three linked transaction in the last 40 minutes.
15:38Why does this matter so much specifically in fraud?
15:43Four reasons.
15:45One, regulatory and model risk management.
15:49Financial institutions operate under frameworks that require documented rational for automated decision.
15:57This isn't optional.
15:58Two, analyst trust and investigation speed and unexplained high score gets rubber stamped or ignored.
16:07Edge level evidence turns recomputee the case from scratch into verified three linked transactions.
16:15Three, model debugging attribution patterns show us when the model leans on a speurious or decaying signal before performance visibly drops in production.
16:27And four, feedback loop quality.
16:30When explanation match what an analyst already suspects, they trust the borderline calls enough to label them correctly, improving the next round of training data.
16:41An accurate black box simply doesn't get used.
16:47Now the real fight serving these solution in 3 milliseconds.
16:53offline van memory bound GPU sarded layer wise graph says forward pass over the entire graph bashed by destination node so it's numerically identical to a full graph forward just bounded to a few gigabyte at a time across multiple GPUs
17:13every user merchant and recent transaction gets its per layer representation cast in a low latency embedding store.
17:22Each user's last five transaction sit in a ring buffer online at request time.
17:30A lightweight assembly step reconstructs the neighbor states straight from that cast.
17:37No grab build, no sampling.
17:39Recense weighted by an exponential decay that reproduces training exactly.
17:46Then the realtime scoring services runs only the new transactions own two layer forward from those cache states plus the exeost head.
17:57Nothing that scales with graph size runs per request.
18:02Three small tricks make it fast in practice.
18:06We pin math library thread pulls to one before the heavy libraries even load.
18:13That alone is the difference between 30 milliseconds and one.
18:17We warm the model up at startup so the first real request isn't slow.
18:23And today it's a single in-memory process.
18:26The same interface maps directly into a distributed cache when we need to scale out.
18:34Now there's a catch.
18:36That trick requires caching a representation for every node including every transaction at production scale.
18:45That's 60 GBTE and it needs expensive high memory serving with a one day embedding lag in real time.
18:53True positive rate actually drops from 48 to 44%.
18:59The fix predict on the edge not the node.
19:03premputee embeddings for user and merchants only, never per transaction.
19:09That collapses the footprint to two to three gigabytes, about 20 times smaller, cheap enough to run on the same CPU class as the Exib Boost model it sits beside.
19:21Nvidia independently recommended the same approach and expects it to perform on par with the node model.
19:29Here's the actual request path.
19:32pre-processing, cast lookup, and the exubost head are all fractions of a millisecond.
19:39The GNN's own true two layer forward is the biggest single piece at 1.61 millisecond.
19:47End to end 2.3 millisecond at the median, 3.1 at P99, comfortably inside our 3 to 4 millisecond SLA.
19:59If we ever need more headroom that GNN forward is the first thing we would recompile or contise.
20:12[music]
20:16Thank you for watching.
20:18We are both happy to go deeper sampling, graph construction, explanability or serving.
20:24Reach out anytime.
20:25Thank you.