{
  "video": {
    "id": "gonAewBZxfU",
    "title": "Your Fridge Is a Valid Payment Device — Eleonora Drykina & Nina Žižakić, In The Pocket",
    "duration": 1502,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "gonAewBZxfU",
    "title": "Your Fridge Is a Valid Payment Device — Eleonora Drykina & Nina Žižakić, In The Pocket",
    "one_liner": "For a RAG system over 270 table-heavy financial rulebooks, three things moved accuracy: keeping document structure (non-overlapping chunks with section paths), flattening tables to CSV, and giving an agent four retrieval tools from narrow to wide instead of a fixed search-then-generate pipeline.",
    "summary": "In The Pocket built a rules assistant for Belgium's largest payment system operator, covering about 270 PDFs (about 6,000 pages, about 3,000 tables). The answers have to cite a specific section and page, and must not use outside knowledge about other payment systems. The speakers walk through the pipeline. Indexing produces structure-aware chunks with rich metadata and tables converted to CSV. Hybrid retrieval combines dense and BM25 search using Reciprocal Rank Fusion (RRF). An agent with four retrieval tools takes a median of 38 seconds per answer, and streaming makes that wait tolerable. They close with lessons on evaluation, logging and real usage since the June launch.",
    "key_points": [
      "Chunking: text is extracted per page, split into paragraphs and grouped into snippets of about 2,000 characters, never splitting a table across snippets. Regexes detect section headings (e.g. 'section three', '4.2.1') to give each chunk a section path. Chunks don't overlap, so sorting a section's chunks by chunk index rebuilds the original section.",
      "Metadata on each chunk: rulebook title and page number (for citations and source cards), section path, chunk index, rulebook ID, an LLM-generated summary/table of contents of the whole rulebook, and business fields such as which rules it applies to and whether it contains a table. The corpus ends up at about 4,000 chunks.",
      "Tables were the overlooked problem. Naive PDF text extraction garbled merged cells (e.g. 'domestic' / 'cross-border' / 'refunds' spanning rows). Converting tables to CSV and repeating the merged values in every row 'really improved the accuracy of the answers significantly'.",
      "Hybrid retrieval: top 50 from dense search and top 50 from BM25 are merged with RRF into a top 30, reranked by section content, and the top 10 go to the agent. RRF needs no calibration between the two scoring methods. On 63 questions, retrieval success rose from 92% to 97% compared with dense-only search, and it found every question dense search had found.",
      "The agent has four tools, from narrow to wide: search the rulebooks (top 10 chunks), fetch a whole section by its path, fetch a whole rulebook (small ones only), and list the available rulebooks. Safeguards: a rulebook over 100K tokens is refused and the tool returns its section list with a warning; a section name that doesn't match returns the valid sections; and a prompt instruction makes the agent ask itself before the 6th call (most questions take 2–4) whether it is finding something new or just rephrasing a search that already failed. That instruction came from watching traces where the agent repeated the same search.",
      "Their first UI had the agent fill in Pydantic-model UI components, but a partial Pydantic object can't be streamed, which meant 38 seconds of blank screen. The fix was to stream Markdown token by token, styled live, and add the source card as a separate component once the answer is complete.",
      "Logging uses Azure Table Storage: questions, tool calls, responses, thumbs up/down, source clicks and copies. Conversation history is deliberately capped at the last 20 messages.",
      "Evals: about 86 end-to-end tests in Promptfoo, scored by an LLM against criteria plus a check that the correct source was cited, alongside a retrieval-only eval. Behaviour rules include no advice, neutrality towards competitors, and saying clearly when something isn't in the rulebooks. Usage since the June launch (internal users only): about 900 questions, 100+ conversations, about 34 users. The first month included red teaming; by the third month over 90% of questions were genuine. Users paste in customer emails and protocol messages."
    ],
    "takeaways": [
      "Keep document structure: make chunks non-overlapping and tag each with its section path and chunk index, so the agent can rebuild a whole section and the UI can cite it precisely.",
      "Give tables their own handling: convert them to CSV and repeat merged cells in every row so the model reads the right row and column, which matters most for things like fees.",
      "Replace a fixed retrieve-then-generate step with an agent that has retrieval tools at different levels of detail, and add guardrails: size limits, returning valid options when a name doesn't match, and a nudge against repeating failed searches.",
      "Stream plain Markdown and add structured UI components only once the answer is complete and only where the interface needs them. Long agent answers have to stream.",
      "Rerun LLM-judged evals before trusting a change (5 of their 12 changed outcomes didn't repeat), check the grader as well as the model, and test with real users early. Since few users click like/dislike, infer satisfaction from follow-ups ('thank you' or a new topic counts as positive, asking the same thing again as negative)."
    ],
    "topics": [
      "rag",
      "agentic retrieval",
      "hybrid search",
      "chunking",
      "table extraction",
      "evals",
      "streaming ux",
      "fintech"
    ],
    "tools": [
      "In The Pocket",
      "BM25",
      "Reciprocal Rank Fusion",
      "Pydantic",
      "Azure Table Storage",
      "Promptfoo"
    ],
    "quotes": [
      {
        "text": "rule codes, message names, precise identifiers will look like noise to the embedding, even though to the user asking the question, it is often the essence of the question",
        "at": "12:08",
        "url": "https://www.youtube.com/watch?v=gonAewBZxfU&t=728s"
      },
      {
        "text": "This is not just a chat response. This is a small research task.",
        "at": "15:40",
        "url": "https://www.youtube.com/watch?v=gonAewBZxfU&t=940s"
      },
      {
        "text": "The problem is that you can't stream a partial Pydantic object.",
        "at": "16:30",
        "url": "https://www.youtube.com/watch?v=gonAewBZxfU&t=990s"
      },
      {
        "text": "sometimes the problem lies in the assessment system itself, not in the model.",
        "at": "19:38",
        "url": "https://www.youtube.com/watch?v=gonAewBZxfU&t=1178s"
      }
    ],
    "words": 3784
  },
  "summary_url": "/#gonAewBZxfU",
  "transcript": {
    "html": "/transcripts/gonAewBZxfU.html",
    "txt": "/transcripts/gonAewBZxfU.txt",
    "vtt": "/transcripts/gonAewBZxfU.vtt"
  }
}