{
  "video": {
    "id": "UB6hoWyh3B0",
    "title": "A Bank Trained Its Own Foundation Model to Catch Fraud — Freddy Lecue, Wells Fargo",
    "duration": 1060,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "UB6hoWyh3B0",
    "title": "A Bank Trained Its Own Foundation Model to Catch Fraud — Freddy Lecue, Wells Fargo",
    "one_liner": "Wells Fargo pre-trained a ~1B-parameter decoder-only transformer from scratch on billions of customer transactions, and when they fine-tuned it for fraud classification it beat their production XGBoost model on both detection rate and false positives. They say the real secret sauce is the tokenization and the post-training.",
    "summary": "Lecue argues that a bank's tabular transaction data deserves its own foundation model rather than being pushed through a language model. Wells Fargo built one shared backbone, trained to predict the next transaction, that can be fine-tuned for fraud detection, credit line increase, anti-money laundering and marketing. That replaces many separate, siloed models. The architecture and training recipe were standard; the hard part was designing a vocabulary that turns about 100 raw features across five dimensions into tokens. On fraud detection the model flagged more fraud with fewer false positives than XGBoost, stayed well inside the 250 ms decline window, and is worth millions of dollars in savings.",
    "key_points": [
      "The model was pre-trained from scratch: the team used an existing transformer architecture but no open weights, and trained it on their own GPU cluster. The first run took a couple of weeks; pre-training now takes a couple of days.",
      "The data is billions of transactions a month across millions of accounts, with hundreds of transactions per client per month from debit, credit, check and digital channels. It is modeled on five dimensions: who (customer or merchant), what, how (channel), when and where.",
      "Instead of the heavily hand-crafted feature engineering in existing production models, they went back to about 100 raw features, cut down from thousands, and let the transformer blocks learn how they relate.",
      "Tokenization was the hardest part. They had to decide which values become tokens and keep the vocabulary small enough to train. For example, zip codes are grouped by distribution into about 2,800 tokens instead of one token per US zip code. Transaction amount, time between transactions, IP address and categoricals with very different cardinalities each needed their own treatment.",
      "They sized the model with Chinchilla scaling. The starting rule of 20 tokens per parameter turned out to be too few: experiments showed they needed roughly 10 times that.",
      "The model is a 1B-parameter decoder-only transformer, the smallest size that gave the biggest boost. An 11B version is in progress. Training used standard practice: causal and masked language modeling objectives, mixed precision, distributed data parallel, the largest micro-batch that fits, RoPE, flash attention and activation checkpointing.",
      "Lecue's explanation for the gain is that XGBoost loses the relationship between the transaction being scored and the customer's past transactions, while attention lets the model look at that history directly.",
      "Fraud classification has to finish within 250 ms for a decline. The model ran about 10 times faster than that. Compared with production XGBoost, it improved detection rate and also lowered false positives, which means fewer customer calls, lower operating cost and fewer lost legitimate transactions."
    ],
    "takeaways": [
      "For large proprietary tabular or sequential data, consider pre-training a domain foundation model as one shared backbone, then fine-tuning it per task instead of keeping separate siloed models.",
      "Put most of your effort into the vocabulary and tokenization: choose the raw information units, group high-cardinality fields such as zip codes by distribution, and keep the vocabulary small enough to train.",
      "Begin from raw features rather than downstream hand-engineered ones, and let attention learn the relationships across time and across customers.",
      "Don't take Chinchilla's 20 tokens per parameter as fixed: test it, because non-language data may need far more tokens per parameter.",
      "Judge success with metrics that map to money, such as detection rate together with false-positive rate, and check against a hard latency budget like 250 ms."
    ],
    "topics": [
      "foundation models",
      "fraud detection",
      "tabular data",
      "tokenization",
      "financial services",
      "pre-training",
      "transformers",
      "scaling laws"
    ],
    "tools": [
      "Wells Fargo",
      "XGBoost",
      "Claude",
      "ChatGPT",
      "Chinchilla",
      "RoPE",
      "Flash Attention"
    ],
    "quotes": [
      {
        "text": "We pre-train transformer blocks at scale from scratch using our transaction data.",
        "at": "05:02",
        "url": "https://www.youtube.com/watch?v=UB6hoWyh3B0&t=302s"
      },
      {
        "text": "the recipe or the secret sauce is really how you do the tokenization and then how you going to do the post the post training",
        "at": "12:24",
        "url": "https://www.youtube.com/watch?v=UB6hoWyh3B0&t=744s"
      },
      {
        "text": "we managed to get more weak flag more fraudulent activities and we even reduced the false positive",
        "at": "15:49",
        "url": "https://www.youtube.com/watch?v=UB6hoWyh3B0&t=949s"
      },
      {
        "text": "it's basically huge it's millions of dollar savings",
        "at": "16:25",
        "url": "https://www.youtube.com/watch?v=UB6hoWyh3B0&t=985s"
      }
    ],
    "words": 3379
  },
  "summary_url": "/#UB6hoWyh3B0",
  "transcript": {
    "html": "/transcripts/UB6hoWyh3B0.html",
    "txt": "/transcripts/UB6hoWyh3B0.txt",
    "vtt": "/transcripts/UB6hoWyh3B0.vtt"
  }
}