Your Fridge Is a Valid Payment Device — Eleonora Drykina & Nina Žižakić, In The Pocket
https://www.youtube.com/watch?v=gonAewBZxfU
AI Engineer · 25 min · 204 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] Hello, welcome to our lecture titled "Your refrigerator is a valid payment device: RAG in production."
[00:11] Let me start by explaining the name.
[00:14] So , this is a page from one of our clients' rules.
[00:18] This is one of the many documents on which we built our RAG system.
[00:23] You expect to see dry financial documents, and for the most part they are just that .
[00:29] But there are also interesting moments.
[00:36] For example, here is this page, which lists devices that can be used to pay.
[00:45] There are the expected things, like phones or cards, but there are also more creative ones: cufflinks, a handbag, clothes, and even a refrigerator.
[00:56] I thought it was really cool to find such an interesting fact in documents that should have been very boring.
[01:02] Well , let's introduce ourselves.
[01:09] This is the only photo we have together .
[01:15] We don't usually wear VR headsets, but that day in the office we were trying out various interesting gadgets.
[01:23] I am Nina.
[01:25] I am an AI engineer.
[01:29] And I am Eleanor.
[01:30] I am a product engineer.
[01:31] We work at In The Pocket, a digital product studio headquartered in Belgium.
[01:37] We help our clients with AI transformation and software development .
[01:45] Today we will talk about the RAG project that we created for our client, a large Belgian fintech company.
[01:55] We will preserve her anonymity for this report.
[01:59] Essentially, this is a classic RAG problem.
[02:07] In this report, we want to show the technical decisions we had to make and delve into how we built it.
[02:16] This is a report that would be useful for us at the beginning of this project.
[02:22] So, what is our problem?
[02:33] Our client operates the largest payment system in Belgium, and all the rules are written in rulebooks.
[02:43] That's about 270 PDF files.
[02:47] That is about 6,000 pages.
[02:51] Most of the information is contained in tables.
[02:55] Approximately 3000 tables.
[02:58] This is how they structure their knowledge.
[03:00] A lot of important knowledge is contained in the tables themselves.
[03:05] Rulebooks often refer to each other.
[03:11] One collection refers to another, that to yet another, and so on.
[03:19] Working with RAG for a financial institution has its own challenges.
[03:26] The first of these is quoting.
[03:28] Of course, the answers must always be correct.
[03:32] But in a financial context, it is especially important.
[03:40] Therefore, each answer should contain a link to a specific section and page of the rules.
[03:45] So that users can see where this answer came from.
[03:49] The second point is justification.
[03:52] The LLM running at the core of the system is a generic model, so it is trained on general knowledge.
[04:00] So she knows a lot, including about other payment systems, how they work, and what their fees are.
[04:09] We had to decide and make sure what knowledge could be used.
[04:16] For example, knowing what a bank is is normal, it's logical , and it's not something that will be defined in the rules.
[04:24] But how much another payment system charges or how it works shouldn't be in the answer if it's not in the rules.
[04:33] This is what the solution we created looks like.
[04:39] This is a chatbot.
[04:41] You ask a question and it answers based on these rules.
[04:47] If the answer is in the table , it will also show that table.
[04:55] Below the answer is a list of sources used to form it, including the section and part of the rules, so that the user can open this document.
[05:05] In this example, I asked, " Can I pay with my refrigerator?"
[05:11] I got the answer, and then I can ask follow-up questions.
[05:15] This is what the whole system looks like.
[05:20] It consists of two parts.
[05:22] On the left is indexing, which is what we do offline.
[05:29] With each new rule release, we convert these PDF files into fragments and store them in a vector database.
[05:38] And on the right is service, that is, the online part for each user question.
[05:46] The agent searches this database, and then the response is streamed to the user.
[05:54] I'll talk about indexing, and Eleanor will talk about the maintenance part.
[06:02] So, about indexing: the setup is pretty simple.
[06:06] We take PDF files, break them into fragments, and add metadata to each of them.
[06:17] We perform special processing of the tables and then generate embeddings.
[06:23] We generate two types of embeddings: dense and BM25.
[06:29] So, I'll go into detail about the first three steps now, and Eleanor will talk about embeddings later.
[06:36] The first step is to split the PDF into fragments.
[06:47] We extract the text from each page, divide it into paragraphs, and group the paragraphs into a snippet until it reaches about 2000 characters, and we make sure not to break the tables between the two snippets.
[07:02] We also define section headings using regular expressions, for example: " section three" or " 4.2.1".
[07:14] This gives each fragment its path to the partition.
[07:19] For example , this fragment is in section four, paragraph 4.2. This path is used to add a link, as well as to retrieve the entire section when the agent needs more
[07:37] context.
[07:40] This works because we don't overlap the fragments.
[07:45] So, a chapter is simply a collection of fragments.
[07:49] If you put them in the correct order, they are restored to their original form.
[07:55] For the second step, we add metadata to each fragment.
[08:01] For example, here is a text fragment from this refrigerator user manual.
[08:07] Below are some metadata fields.
[08:13] We need the guide title and page number so we can display them in the link or on the source card for the user.
[08:26] This allows the user to go directly to the PDF file.
[08:33] We also use section paths both for citation and so that the agent can get the entire section at once.
[08:45] We also have a fragment index that allows an agent to piece together a section or even the entire guide .
[08:55] The agent uses the guide ID to get a list of all available guides.
[09:09] When indexing each manual, we use LLM to generate a summary table of contents, which is added to each fragment as a description of the manual.
[09:23] So, this way the agent knows which document the excerpt comes from.
[09:29] The rest of the metadata are business logic fields.
[09:37] That is, what is the rulebook, what rules does it apply to, whether it contains a table— this is what the agent uses or receives along with the search results.
[09:48] The third stage is table processing.
[09:56] This is important because, as I said, there is a lot of knowledge in the tables .
[10:04] At first, we somehow didn't realize this.
[10:06] The initial processing was just a simple text extraction from the PDF file, and it formatted the tables poorly.
[10:20] But in this example, you can see that the "domestic" field spans multiple rows, " cross-border" spans two more rows, and " refunds" spans both.
[10:37] So if it's not clearly translated into text format, the agent could get the wrong information, and especially with important things like commissions, it's important to get the right row and the right column.
[10:53] So we convert the tables to CSV format, essentially duplicating those fields, and then that's how we get the data to the agent, and when it gets it, it reads the correct
[11:16] row.
[11:18] Exactly the one he needs.
[11:20] And this really improved the accuracy of the answers significantly.
[11:27] So, it was about indexing.
[11:30] Now we have about 4,000 fragments.
[11:34] And now Eleanor will talk about the service part.
[11:39] Yes.
[11:42] So, the previous part was about building an index.
[11:48] Now this part will be about finding those same 10 fragments out of the 4,000 prepared.
[11:54] An obvious step would be to use dense embeddings when asking questions and convert each fragment into a vector for distance comparison.
[12:05] This is great for understanding the content, but rule codes, message names, precise identifiers will look like noise to the embedding, even though to the user asking the question, it is often the essence of the question .
[12:19] With BM25 we have a mirror image.
[12:24] Exact tokens will work, but paraphrasing won't.
[12:27] Therefore, we combined both approaches in one query and combined them using the Reciprocal Rank Fusion method.
[12:34] Um, dense vectors—top 50 , BM25—top 50, we combine them into top 30, rerank them by the content of the sections, and pass the top 10 to the agent.
[12:44] And a nice feature of RRF is that we don't need to calibrate the estimates between these two methods.
[12:54] When evaluating search results alone for 63 questions, compared to dense embeddings, the hybrid approach increased the success rate from 92 to 97% and did not miss any questions that dense
[13:07] embeddings could have found.
[13:11] So, we have 10 decent chunks, but not every question fits into 10 chunks.
[13:17] Some of them are interconnected .
[13:19] One reference links to another, and the answer will only be complete when you follow that link.
[13:25] Some of them require a broader context.
[13:28] For example, all the fees for this type of transaction will never be in the top 10 of anything.
[13:33] So instead of a fixed pipeline of "search, then generate," our agent has four tools.
[13:38] From narrow to wide.
[13:40] Directory searches always come first.
[13:42] These are the 10 fragments we talked about on the previous slide.
[13:47] If that's not enough, the directory partition extraction tool extracts the entire partition by its path.
[13:53] This is exactly the approach that Nina mentioned earlier.
[13:58] Storing the path to the partition.
[14:00] And this space here, when you delete it, it's just getting every fragment with this path, sorted by fragment index.
[14:09] Above that, getting a full directory only works for small directories.
[14:14] And a list of available directories for cases when the agent needs to know what even exists.
[14:23] So, a few safeguards helped make this work in practice.
[14:28] First, if the directory is too large— more than 100K tokens—the full directory tool refuses to load its contents into the context and instead returns a list of available sections with a warning that the directory is too long.
[14:45] Second, if the section name doesn't match, it returns the available sections so the agent can try again rather than giving up or trying to make up a name.
[14:57] And the last one is a tool called " disciplinary instruction" in the query.
[15:07] Since most questions require two to four calls, before the sixth call, the agent is instructed to ask themselves whether they are getting something truly new or simply reframing a search that has already failed.
[15:19] And this idea came about when we were watching real-world traces where an agent would run the same search over and over again, hoping that a number would eventually appear.
[15:32] So the price of all this is latency, with a median of 38 seconds per response.
[15:39] This is not just a chat response.
[15:42] This is a small research task.
[15:45] And we rely on streaming to make 38 seconds feel like a workflow, not a frozen chat.
[15:56] This brings me to the next slide.
[15:58] So, in the first iteration, we really wanted to move away from the basic chatbot interface.
[16:04] We prepared the interface components and described them as Pydantic models available to the agent, saying, “Here are the components you have .
[16:12] Use them in your answer along with the text.
[16:16] For example, a source card was a component, an event could be made a component, a collapsible list could be a component , etc. The idea was to have a somewhat AI-generated
[16:26] interface that would be more readable than a simple sheet of text.
[16:30] The problem is that you can't stream a partial Pydantic object.
[16:34] It becomes valid JSON only after the last token arrives.
[16:39] As you can see, for a system that takes 38 seconds to respond, that meant 38 seconds of blank screen, and then everything appeared at once.
[16:50] So, the solution that saved the user experience was to split the response into parts.
[16:55] The agent now streams Markdown text token by token, and the interface styles it in real time as it arrives.
[17:01] And then the source card becomes a separate element that is a real component of the interface.
[17:06] It doesn't contain information that you would want to click on before reading the entire answer.
[17:12] So we have nothing to lose by waiting for a complete answer before adding this component.
[17:19] This gives us an interactive card instead of a regular in-text link, which is more user-friendly.
[17:30] But once that answer appears on the screen , the next question is what the user will do with it.
[17:39] Click on the source, copy the response, continue the conversation, or leave.
[17:44] It is this user experience that we want to track, and this brings us to logging.
[17:53] So, for now, we have chosen the simplest logging solution—Azure Table Storage.
[17:58] We log every question, tool call, response, feedback ( positive or negative), as well as events like clicking on the source or copying a response , along with a conversation history that is intentionally kept
[18:12] small.
[18:12] These are just the last 20 posts as context.
[18:16] For the stage we are at now with this product, that is enough.
[18:21] As the product grows, we will understand what we will need at that time.
[18:26] And how do we know that all this really works?
[18:34] Our assessment questions come from real sources, we conducted two rounds of testing with real users, plus there are rules of conduct for our AI system regarding what it can and cannot
[18:45] do.
[18:47] For example, no advice, maintaining neutrality towards competitors, and clearly stating when something is not in the guides.
[18:57] We have about 86 end-to-end tests run through Promptfoo , a full evaluation by the LLM- agent against the criteria, as well as a check that the correct source has been cited, plus an evaluation of just the search from the previous
[19:10] stages.
[19:14] We learned two lessons from our experience that we want to share: first, LLM scores have background noise, and you need to rerun the tests before you can trust the results.
[19:31] For example, when we conducted our own evaluation, five out of 12 outcome changes did not repeat.
[19:38] And secondly, sometimes the problem lies in the assessment system itself, not in the model.
[19:44] We had a case with a " hallucination" of a date for which the referee kept giving fines, and then it turned out that this date was the real text in the rules.
[19:58] Yes.
[19:59] Now about the use.
[20:03] We launched this rules assistant in June.
[20:08] Initially, it was only available to our clients' internal users, and the goal is to provide access to their external participants and clients by the end of the year.
[20:28] So, at this point, we already have some usage metrics.
[20:36] We have received about 900 questions so far.
[20:41] They are distributed across over 100 conversations, and about 34 people actually use them.
[20:49] We also collected feedback through “likes ” and “dislikes,” but most people stopped using this feature after the first month.
[21:00] We also have data on how many times sources were clicked on or answers were copied .
[21:05] But that's all the numbers are.
[21:09] We also use AI for analytics.
[21:15] Here on the right we see how LLM interpreted it .
[21:21] We noticed that in the first month when we brought people on board, there was some “ red teaming” going on.
[21:32] People tried to hack the assistant, make it go off topic, or asked strange questions.
[21:39] But by the third month, over 90% of the questions became real and to the point.
[21:47] Secondly, people use it for real-world cases.
[21:54] We saw users inserting emails from customers and asking for help responding to them.
[22:00] Or they inserted a real message from the protocol, a table, etc., and then composed the answer they wanted to check.
[22:15] And third, since we didn't get a lot of feedback through " likes" and "dislikes," we tried to determine the tone of the conversations by analyzing the following questions.
[22:27] So if people say "thank you" or move on to a new topic, we, yes, count that as a positive interaction.
[22:41] And if they ask the same thing again, it 's most likely a negative interaction.
[22:48] And we saw that most of the conversations were successful.
[22:56] So , to summarize, what are the main conclusions from this project so far?
[23:02] When it comes to documents, we found that it is important to maintain their structure.
[23:13] So sections are important both for getting an answer and for the agent to be able to capture knowledge in some way .
[23:22] We also saw that tables require a separate approach and special handling.
[23:33] On the agent side, as mentioned, it was important to provide different levels of granularity and define the amount of context needed, and overall we benefited greatly when we moved from a one-time
[23:48] query to LLM to an agent search.
[23:53] We saw that the assessment scores increased significantly.
[24:02] From a UX perspective , it was the streaming of data that made this tool convenient and useful.
[24:13] And we only add structured components when the user interface requires it.
[24:28] And finally, we think it's important that we tested with real users early on, and also used those interviews to create evaluation prompts that help us shape the product and decide whether what we think is an improvement is actually
[24:45] one.
[24:53] Thank you for listening to our report.
[24:56] Thank you.