Your Fridge Is a Valid Payment Device — Eleonora Drykina & Nina Žižakić, In The Pocket

AI Engineer · 25 min · 204 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01Hello, welcome to our lecture titled "Your refrigerator is a valid payment device: RAG in production."
00:11Let me start by explaining the name.
00:14So , this is a page from one of our clients' rules.
00:18This is one of the many documents on which we built our RAG system.
00:23You expect to see dry financial documents, and for the most part they are just that .
00:29But there are also interesting moments.
00:36For example, here is this page, which lists devices that can be used to pay.
00:45There are the expected things, like phones or cards, but there are also more creative ones: cufflinks, a handbag, clothes, and even a refrigerator.
00:56I thought it was really cool to find such an interesting fact in documents that should have been very boring.
01:02Well , let's introduce ourselves.
01:09This is the only photo we have together .
01:15We don't usually wear VR headsets, but that day in the office we were trying out various interesting gadgets.
01:23I am Nina.
01:25I am an AI engineer.
01:29And I am Eleanor.
01:30I am a product engineer.
01:31We work at In The Pocket, a digital product studio headquartered in Belgium.
01:37We help our clients with AI transformation and software development .
01:45Today we will talk about the RAG project that we created for our client, a large Belgian fintech company.
01:55We will preserve her anonymity for this report.
01:59Essentially, this is a classic RAG problem.
02:07In this report, we want to show the technical decisions we had to make and delve into how we built it.
02:16This is a report that would be useful for us at the beginning of this project.
02:22So, what is our problem?
02:33Our client operates the largest payment system in Belgium, and all the rules are written in rulebooks.
02:43That's about 270 PDF files.
02:47That is about 6,000 pages.
02:51Most of the information is contained in tables.
02:55Approximately 3000 tables.
02:58This is how they structure their knowledge.
03:00A lot of important knowledge is contained in the tables themselves.
03:05Rulebooks often refer to each other.
03:11One collection refers to another, that to yet another, and so on.
03:19Working with RAG for a financial institution has its own challenges.
03:26The first of these is quoting.
03:28Of course, the answers must always be correct.
03:32But in a financial context, it is especially important.
03:40Therefore, each answer should contain a link to a specific section and page of the rules.
03:45So that users can see where this answer came from.
03:49The second point is justification.
03:52The LLM running at the core of the system is a generic model, so it is trained on general knowledge.
04:00So she knows a lot, including about other payment systems, how they work, and what their fees are.
04:09We had to decide and make sure what knowledge could be used.
04:16For example, knowing what a bank is is normal, it's logical , and it's not something that will be defined in the rules.
04:24But how much another payment system charges or how it works shouldn't be in the answer if it's not in the rules.
04:33This is what the solution we created looks like.
04:39This is a chatbot.
04:41You ask a question and it answers based on these rules.
04:47If the answer is in the table , it will also show that table.
04:55Below the answer is a list of sources used to form it, including the section and part of the rules, so that the user can open this document.
05:05In this example, I asked, " Can I pay with my refrigerator?"
05:11I got the answer, and then I can ask follow-up questions.
05:15This is what the whole system looks like.
05:20It consists of two parts.
05:22On the left is indexing, which is what we do offline.
05:29With each new rule release, we convert these PDF files into fragments and store them in a vector database.
05:38And on the right is service, that is, the online part for each user question.
05:46The agent searches this database, and then the response is streamed to the user.
05:54I'll talk about indexing, and Eleanor will talk about the maintenance part.
06:02So, about indexing: the setup is pretty simple.
06:06We take PDF files, break them into fragments, and add metadata to each of them.
06:17We perform special processing of the tables and then generate embeddings.
06:23We generate two types of embeddings: dense and BM25.
06:29So, I'll go into detail about the first three steps now, and Eleanor will talk about embeddings later.
06:36The first step is to split the PDF into fragments.
06:47We extract the text from each page, divide it into paragraphs, and group the paragraphs into a snippet until it reaches about 2000 characters, and we make sure not to break the tables between the two snippets.
07:02We also define section headings using regular expressions, for example: " section three" or " 4.2.1".
07:14This gives each fragment its path to the partition.
07:19For example , this fragment is in section four, paragraph 4.2. This path is used to add a link, as well as to retrieve the entire section when the agent needs more
07:37context.
07:40This works because we don't overlap the fragments.
07:45So, a chapter is simply a collection of fragments.
07:49If you put them in the correct order, they are restored to their original form.
07:55For the second step, we add metadata to each fragment.
08:01For example, here is a text fragment from this refrigerator user manual.
08:07Below are some metadata fields.
08:13We need the guide title and page number so we can display them in the link or on the source card for the user.
08:26This allows the user to go directly to the PDF file.
08:33We also use section paths both for citation and so that the agent can get the entire section at once.
08:45We also have a fragment index that allows an agent to piece together a section or even the entire guide .
08:55The agent uses the guide ID to get a list of all available guides.
09:09When indexing each manual, we use LLM to generate a summary table of contents, which is added to each fragment as a description of the manual.
09:23So, this way the agent knows which document the excerpt comes from.
09:29The rest of the metadata are business logic fields.
09:37That is, what is the rulebook, what rules does it apply to, whether it contains a table— this is what the agent uses or receives along with the search results.
09:48The third stage is table processing.
09:56This is important because, as I said, there is a lot of knowledge in the tables .
10:04At first, we somehow didn't realize this.
10:06The initial processing was just a simple text extraction from the PDF file, and it formatted the tables poorly.
10:20But in this example, you can see that the "domestic" field spans multiple rows, " cross-border" spans two more rows, and " refunds" spans both.
10:37So if it's not clearly translated into text format, the agent could get the wrong information, and especially with important things like commissions, it's important to get the right row and the right column.
10:53So we convert the tables to CSV format, essentially duplicating those fields, and then that's how we get the data to the agent, and when it gets it, it reads the correct
11:16row.
11:18Exactly the one he needs.
11:20And this really improved the accuracy of the answers significantly.
11:27So, it was about indexing.
11:30Now we have about 4,000 fragments.
11:34And now Eleanor will talk about the service part.
11:39Yes.
11:42So, the previous part was about building an index.
11:48Now this part will be about finding those same 10 fragments out of the 4,000 prepared.
11:54An obvious step would be to use dense embeddings when asking questions and convert each fragment into a vector for distance comparison.
12:05This is great for understanding the content, but rule codes, message names, precise identifiers will look like noise to the embedding, even though to the user asking the question, it is often the essence of the question .
12:19With BM25 we have a mirror image.
12:24Exact tokens will work, but paraphrasing won't.
12:27Therefore, we combined both approaches in one query and combined them using the Reciprocal Rank Fusion method.
12:34Um, dense vectors—top 50 , BM25—top 50, we combine them into top 30, rerank them by the content of the sections, and pass the top 10 to the agent.
12:44And a nice feature of RRF is that we don't need to calibrate the estimates between these two methods.
12:54When evaluating search results alone for 63 questions, compared to dense embeddings, the hybrid approach increased the success rate from 92 to 97% and did not miss any questions that dense
13:07embeddings could have found.
13:11So, we have 10 decent chunks, but not every question fits into 10 chunks.
13:17Some of them are interconnected .
13:19One reference links to another, and the answer will only be complete when you follow that link.
13:25Some of them require a broader context.
13:28For example, all the fees for this type of transaction will never be in the top 10 of anything.
13:33So instead of a fixed pipeline of "search, then generate," our agent has four tools.
13:38From narrow to wide.
13:40Directory searches always come first.
13:42These are the 10 fragments we talked about on the previous slide.
13:47If that's not enough, the directory partition extraction tool extracts the entire partition by its path.
13:53This is exactly the approach that Nina mentioned earlier.
13:58Storing the path to the partition.
14:00And this space here, when you delete it, it's just getting every fragment with this path, sorted by fragment index.
14:09Above that, getting a full directory only works for small directories.
14:14And a list of available directories for cases when the agent needs to know what even exists.
14:23So, a few safeguards helped make this work in practice.
14:28First, if the directory is too large— more than 100K tokens—the full directory tool refuses to load its contents into the context and instead returns a list of available sections with a warning that the directory is too long.
14:45Second, if the section name doesn't match, it returns the available sections so the agent can try again rather than giving up or trying to make up a name.
14:57And the last one is a tool called " disciplinary instruction" in the query.
15:07Since most questions require two to four calls, before the sixth call, the agent is instructed to ask themselves whether they are getting something truly new or simply reframing a search that has already failed.
15:19And this idea came about when we were watching real-world traces where an agent would run the same search over and over again, hoping that a number would eventually appear.
15:32So the price of all this is latency, with a median of 38 seconds per response.
15:39This is not just a chat response.
15:42This is a small research task.
15:45And we rely on streaming to make 38 seconds feel like a workflow, not a frozen chat.
15:56This brings me to the next slide.
15:58So, in the first iteration, we really wanted to move away from the basic chatbot interface.
16:04We prepared the interface components and described them as Pydantic models available to the agent, saying, “Here are the components you have .
16:12Use them in your answer along with the text.
16:16For example, a source card was a component, an event could be made a component, a collapsible list could be a component , etc. The idea was to have a somewhat AI-generated
16:26interface that would be more readable than a simple sheet of text.
16:30The problem is that you can't stream a partial Pydantic object.
16:34It becomes valid JSON only after the last token arrives.
16:39As you can see, for a system that takes 38 seconds to respond, that meant 38 seconds of blank screen, and then everything appeared at once.
16:50So, the solution that saved the user experience was to split the response into parts.
16:55The agent now streams Markdown text token by token, and the interface styles it in real time as it arrives.
17:01And then the source card becomes a separate element that is a real component of the interface.
17:06It doesn't contain information that you would want to click on before reading the entire answer.
17:12So we have nothing to lose by waiting for a complete answer before adding this component.
17:19This gives us an interactive card instead of a regular in-text link, which is more user-friendly.
17:30But once that answer appears on the screen , the next question is what the user will do with it.
17:39Click on the source, copy the response, continue the conversation, or leave.
17:44It is this user experience that we want to track, and this brings us to logging.
17:53So, for now, we have chosen the simplest logging solution—Azure Table Storage.
17:58We log every question, tool call, response, feedback ( positive or negative), as well as events like clicking on the source or copying a response , along with a conversation history that is intentionally kept
18:12small.
18:12These are just the last 20 posts as context.
18:16For the stage we are at now with this product, that is enough.
18:21As the product grows, we will understand what we will need at that time.
18:26And how do we know that all this really works?
18:34Our assessment questions come from real sources, we conducted two rounds of testing with real users, plus there are rules of conduct for our AI system regarding what it can and cannot
18:45do.
18:47For example, no advice, maintaining neutrality towards competitors, and clearly stating when something is not in the guides.
18:57We have about 86 end-to-end tests run through Promptfoo , a full evaluation by the LLM- agent against the criteria, as well as a check that the correct source has been cited, plus an evaluation of just the search from the previous
19:10stages.
19:14We learned two lessons from our experience that we want to share: first, LLM scores have background noise, and you need to rerun the tests before you can trust the results.
19:31For example, when we conducted our own evaluation, five out of 12 outcome changes did not repeat.
19:38And secondly, sometimes the problem lies in the assessment system itself, not in the model.
19:44We had a case with a " hallucination" of a date for which the referee kept giving fines, and then it turned out that this date was the real text in the rules.
19:58Yes.
19:59Now about the use.
20:03We launched this rules assistant in June.
20:08Initially, it was only available to our clients' internal users, and the goal is to provide access to their external participants and clients by the end of the year.
20:28So, at this point, we already have some usage metrics.
20:36We have received about 900 questions so far.
20:41They are distributed across over 100 conversations, and about 34 people actually use them.
20:49We also collected feedback through “likes ” and “dislikes,” but most people stopped using this feature after the first month.
21:00We also have data on how many times sources were clicked on or answers were copied .
21:05But that's all the numbers are.
21:09We also use AI for analytics.
21:15Here on the right we see how LLM interpreted it .
21:21We noticed that in the first month when we brought people on board, there was some “ red teaming” going on.
21:32People tried to hack the assistant, make it go off topic, or asked strange questions.
21:39But by the third month, over 90% of the questions became real and to the point.
21:47Secondly, people use it for real-world cases.
21:54We saw users inserting emails from customers and asking for help responding to them.
22:00Or they inserted a real message from the protocol, a table, etc., and then composed the answer they wanted to check.
22:15And third, since we didn't get a lot of feedback through " likes" and "dislikes," we tried to determine the tone of the conversations by analyzing the following questions.
22:27So if people say "thank you" or move on to a new topic, we, yes, count that as a positive interaction.
22:41And if they ask the same thing again, it 's most likely a negative interaction.
22:48And we saw that most of the conversations were successful.
22:56So , to summarize, what are the main conclusions from this project so far?
23:02When it comes to documents, we found that it is important to maintain their structure.
23:13So sections are important both for getting an answer and for the agent to be able to capture knowledge in some way .
23:22We also saw that tables require a separate approach and special handling.
23:33On the agent side, as mentioned, it was important to provide different levels of granularity and define the amount of context needed, and overall we benefited greatly when we moved from a one-time
23:48query to LLM to an agent search.
23:53We saw that the assessment scores increased significantly.
24:02From a UX perspective , it was the streaming of data that made this tool convenient and useful.
24:13And we only add structured components when the user interface requires it.
24:28And finally, we think it's important that we tested with real users early on, and also used those interviews to create evaluation prompts that help us shape the product and decide whether what we think is an improvement is actually
24:45one.
24:53Thank you for listening to our report.
24:56Thank you.