WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=gonAewBZxfU NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/gonAewBZxfU.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.663 --> 00:00:09.786 Hello, welcome to our lecture titled "Your refrigerator is a valid payment device: RAG in production." s2 00:00:11.452 --> 00:00:14.983 Let me start by explaining the name. s3 00:00:14.983 --> 00:00:18.129 So , this is a page from one of our clients' rules. s4 00:00:18.129 --> 00:00:23.929 This is one of the many documents on which we built our RAG system. s5 00:00:23.929 --> 00:00:29.560 You expect to see dry financial documents, and for the most part they are just that . s6 00:00:29.560 --> 00:00:35.349 But there are also interesting moments. s7 00:00:36.254 --> 00:00:42.873 For example, here is this page, which lists devices that can be used to pay. s8 00:00:45.122 --> 00:00:54.519 There are the expected things, like phones or cards, but there are also more creative ones: cufflinks, a handbag, clothes, and even a refrigerator. s9 00:00:56.965 --> 00:01:02.320 I thought it was really cool to find such an interesting fact in documents that should have been very boring. s10 00:01:02.320 --> 00:01:09.149 Well , let's introduce ourselves. s11 00:01:09.149 --> 00:01:14.207 This is the only photo we have together . s12 00:01:15.657 --> 00:01:23.016 We don't usually wear VR headsets, but that day in the office we were trying out various interesting gadgets. s13 00:01:23.016 --> 00:01:25.051 I am Nina. s14 00:01:25.051 --> 00:01:29.010 I am an AI engineer. s15 00:01:29.010 --> 00:01:30.250 And I am Eleanor. s16 00:01:30.250 --> 00:01:31.990 I am a product engineer. s17 00:01:31.990 --> 00:01:37.622 We work at In The Pocket, a digital product studio headquartered in Belgium. s18 00:01:37.622 --> 00:01:43.885 We help our clients with AI transformation and software development . s19 00:01:45.407 --> 00:01:53.889 Today we will talk about the RAG project that we created for our client, a large Belgian fintech company. s20 00:01:55.703 --> 00:01:59.346 We will preserve her anonymity for this report. s21 00:01:59.346 --> 00:02:06.748 Essentially, this is a classic RAG problem. s22 00:02:07.425 --> 00:02:16.525 In this report, we want to show the technical decisions we had to make and delve into how we built it. s23 00:02:16.525 --> 00:02:22.871 This is a report that would be useful for us at the beginning of this project. s24 00:02:22.871 --> 00:02:32.177 So, what is our problem? s25 00:02:33.450 --> 00:02:40.399 Our client operates the largest payment system in Belgium, and all the rules are written in rulebooks. s26 00:02:43.196 --> 00:02:47.380 That's about 270 PDF files. s27 00:02:47.380 --> 00:02:50.774 That is about 6,000 pages. s28 00:02:51.377 --> 00:02:55.046 Most of the information is contained in tables. s29 00:02:55.046 --> 00:02:58.238 Approximately 3000 tables. s30 00:02:58.238 --> 00:03:00.498 This is how they structure their knowledge. s31 00:03:00.498 --> 00:03:05.942 A lot of important knowledge is contained in the tables themselves. s32 00:03:05.942 --> 00:03:11.729 Rulebooks often refer to each other. s33 00:03:11.729 --> 00:03:17.625 One collection refers to another, that to yet another, and so on. s34 00:03:19.583 --> 00:03:25.362 Working with RAG for a financial institution has its own challenges. s35 00:03:26.272 --> 00:03:28.624 The first of these is quoting. s36 00:03:28.624 --> 00:03:32.906 Of course, the answers must always be correct. s37 00:03:32.906 --> 00:03:40.087 But in a financial context, it is especially important. s38 00:03:40.105 --> 00:03:45.330 Therefore, each answer should contain a link to a specific section and page of the rules. s39 00:03:45.330 --> 00:03:49.860 So that users can see where this answer came from. s40 00:03:49.860 --> 00:03:52.631 The second point is justification. s41 00:03:52.631 --> 00:04:00.946 The LLM running at the core of the system is a generic model, so it is trained on general knowledge. s42 00:04:00.946 --> 00:04:09.719 So she knows a lot, including about other payment systems, how they work, and what their fees are. s43 00:04:09.719 --> 00:04:15.351 We had to decide and make sure what knowledge could be used. s44 00:04:16.294 --> 00:04:24.270 For example, knowing what a bank is is normal, it's logical , and it's not something that will be defined in the rules. s45 00:04:24.685 --> 00:04:33.590 But how much another payment system charges or how it works shouldn't be in the answer if it's not in the rules. s46 00:04:33.590 --> 00:04:39.316 This is what the solution we created looks like. s47 00:04:39.316 --> 00:04:41.201 This is a chatbot. s48 00:04:41.201 --> 00:04:47.609 You ask a question and it answers based on these rules. s49 00:04:47.609 --> 00:04:52.522 If the answer is in the table , it will also show that table. s50 00:04:55.961 --> 00:05:05.916 Below the answer is a list of sources used to form it, including the section and part of the rules, so that the user can open this document. s51 00:05:05.916 --> 00:05:11.262 In this example, I asked, " Can I pay with my refrigerator?" s52 00:05:11.262 --> 00:05:15.176 I got the answer, and then I can ask follow-up questions. s53 00:05:15.176 --> 00:05:20.781 This is what the whole system looks like. s54 00:05:20.781 --> 00:05:22.816 It consists of two parts. s55 00:05:22.816 --> 00:05:28.824 On the left is indexing, which is what we do offline. s56 00:05:29.482 --> 00:05:38.284 With each new rule release, we convert these PDF files into fragments and store them in a vector database. s57 00:05:38.747 --> 00:05:46.180 And on the right is service, that is, the online part for each user question. s58 00:05:46.180 --> 00:05:54.919 The agent searches this database, and then the response is streamed to the user. s59 00:05:54.919 --> 00:06:02.299 I'll talk about indexing, and Eleanor will talk about the maintenance part. s60 00:06:02.299 --> 00:06:06.292 So, about indexing: the setup is pretty simple. s61 00:06:06.292 --> 00:06:17.531 We take PDF files, break them into fragments, and add metadata to each of them. s62 00:06:17.531 --> 00:06:23.122 We perform special processing of the tables and then generate embeddings. s63 00:06:23.957 --> 00:06:29.201 We generate two types of embeddings: dense and BM25. s64 00:06:29.201 --> 00:06:36.981 So, I'll go into detail about the first three steps now, and Eleanor will talk about embeddings later. s65 00:06:36.981 --> 00:06:42.117 The first step is to split the PDF into fragments. s66 00:06:47.362 --> 00:07:00.549 We extract the text from each page, divide it into paragraphs, and group the paragraphs into a snippet until it reaches about 2000 characters, and we make sure not to break the tables between the two snippets. s67 00:07:02.340 --> 00:07:14.007 We also define section headings using regular expressions, for example: " section three" or " 4.2.1". s68 00:07:14.007 --> 00:07:19.259 This gives each fragment its path to the partition. s69 00:07:19.259 --> 00:07:36.968 For example , this fragment is in section four, paragraph 4.2. This path is used to add a link, as well as to retrieve the entire section when the agent needs more s70 00:07:37.655 --> 00:07:39.655 context. s71 00:07:40.342 --> 00:07:45.244 This works because we don't overlap the fragments. s72 00:07:45.244 --> 00:07:49.086 So, a chapter is simply a collection of fragments. s73 00:07:49.227 --> 00:07:55.024 If you put them in the correct order, they are restored to their original form. s74 00:07:55.024 --> 00:08:01.035 For the second step, we add metadata to each fragment. s75 00:08:01.035 --> 00:08:07.558 For example, here is a text fragment from this refrigerator user manual. s76 00:08:07.558 --> 00:08:12.046 Below are some metadata fields. s77 00:08:13.987 --> 00:08:25.691 We need the guide title and page number so we can display them in the link or on the source card for the user. s78 00:08:26.855 --> 00:08:32.359 This allows the user to go directly to the PDF file. s79 00:08:33.439 --> 00:08:45.115 We also use section paths both for citation and so that the agent can get the entire section at once. s80 00:08:45.115 --> 00:08:55.614 We also have a fragment index that allows an agent to piece together a section or even the entire guide . s81 00:08:55.721 --> 00:09:04.936 The agent uses the guide ID to get a list of all available guides. s82 00:09:09.430 --> 00:09:19.852 When indexing each manual, we use LLM to generate a summary table of contents, which is added to each fragment as a description of the manual. s83 00:09:23.462 --> 00:09:29.486 So, this way the agent knows which document the excerpt comes from. s84 00:09:29.486 --> 00:09:37.430 The rest of the metadata are business logic fields. s85 00:09:37.430 --> 00:09:48.314 That is, what is the rulebook, what rules does it apply to, whether it contains a table— this is what the agent uses or receives along with the search results. s86 00:09:48.314 --> 00:09:56.795 The third stage is table processing. s87 00:09:56.795 --> 00:10:03.585 This is important because, as I said, there is a lot of knowledge in the tables . s88 00:10:04.129 --> 00:10:06.598 At first, we somehow didn't realize this. s89 00:10:06.598 --> 00:10:20.183 The initial processing was just a simple text extraction from the PDF file, and it formatted the tables poorly. s90 00:10:20.183 --> 00:10:35.877 But in this example, you can see that the "domestic" field spans multiple rows, " cross-border" spans two more rows, and " refunds" spans both. s91 00:10:37.852 --> 00:10:50.955 So if it's not clearly translated into text format, the agent could get the wrong information, and especially with important things like commissions, it's important to get the right row and the right column. s92 00:10:53.895 --> 00:11:16.143 So we convert the tables to CSV format, essentially duplicating those fields, and then that's how we get the data to the agent, and when it gets it, it reads the correct s93 00:11:16.143 --> 00:11:18.143 row. s94 00:11:18.807 --> 00:11:20.924 Exactly the one he needs. s95 00:11:20.924 --> 00:11:27.244 And this really improved the accuracy of the answers significantly. s96 00:11:27.244 --> 00:11:30.547 So, it was about indexing. s97 00:11:30.547 --> 00:11:34.401 Now we have about 4,000 fragments. s98 00:11:34.401 --> 00:11:39.887 And now Eleanor will talk about the service part. s99 00:11:39.887 --> 00:11:41.887 Yes. s100 00:11:42.737 --> 00:11:48.067 So, the previous part was about building an index. s101 00:11:48.748 --> 00:11:54.091 Now this part will be about finding those same 10 fragments out of the 4,000 prepared. s102 00:11:54.091 --> 00:12:05.557 An obvious step would be to use dense embeddings when asking questions and convert each fragment into a vector for distance comparison. s103 00:12:05.557 --> 00:12:19.220 This is great for understanding the content, but rule codes, message names, precise identifiers will look like noise to the embedding, even though to the user asking the question, it is often the essence of the question . s104 00:12:19.220 --> 00:12:24.325 With BM25 we have a mirror image. s105 00:12:24.325 --> 00:12:27.355 Exact tokens will work, but paraphrasing won't. s106 00:12:27.355 --> 00:12:34.459 Therefore, we combined both approaches in one query and combined them using the Reciprocal Rank Fusion method. s107 00:12:34.459 --> 00:12:44.626 Um, dense vectors—top 50 , BM25—top 50, we combine them into top 30, rerank them by the content of the sections, and pass the top 10 to the agent. s108 00:12:44.626 --> 00:12:52.653 And a nice feature of RRF is that we don't need to calibrate the estimates between these two methods. s109 00:12:54.454 --> 00:13:07.906 When evaluating search results alone for 63 questions, compared to dense embeddings, the hybrid approach increased the success rate from 92 to 97% and did not miss any questions that dense s110 00:13:07.906 --> 00:13:11.602 embeddings could have found. s111 00:13:11.602 --> 00:13:17.325 So, we have 10 decent chunks, but not every question fits into 10 chunks. s112 00:13:17.325 --> 00:13:19.520 Some of them are interconnected . s113 00:13:19.520 --> 00:13:25.224 One reference links to another, and the answer will only be complete when you follow that link. s114 00:13:25.224 --> 00:13:28.091 Some of them require a broader context. s115 00:13:28.091 --> 00:13:33.252 For example, all the fees for this type of transaction will never be in the top 10 of anything. s116 00:13:33.252 --> 00:13:38.538 So instead of a fixed pipeline of "search, then generate," our agent has four tools. s117 00:13:38.538 --> 00:13:40.140 From narrow to wide. s118 00:13:40.140 --> 00:13:42.168 Directory searches always come first. s119 00:13:42.168 --> 00:13:47.122 These are the 10 fragments we talked about on the previous slide. s120 00:13:47.122 --> 00:13:53.758 If that's not enough, the directory partition extraction tool extracts the entire partition by its path. s121 00:13:53.758 --> 00:13:58.276 This is exactly the approach that Nina mentioned earlier. s122 00:13:58.276 --> 00:14:00.150 Storing the path to the partition. s123 00:14:00.150 --> 00:14:09.370 And this space here, when you delete it, it's just getting every fragment with this path, sorted by fragment index. s124 00:14:09.370 --> 00:14:14.622 Above that, getting a full directory only works for small directories. s125 00:14:14.622 --> 00:14:23.880 And a list of available directories for cases when the agent needs to know what even exists. s126 00:14:23.880 --> 00:14:28.794 So, a few safeguards helped make this work in practice. s127 00:14:28.794 --> 00:14:45.836 First, if the directory is too large— more than 100K tokens—the full directory tool refuses to load its contents into the context and instead returns a list of available sections with a warning that the directory is too long. s128 00:14:45.836 --> 00:14:57.930 Second, if the section name doesn't match, it returns the available sections so the agent can try again rather than giving up or trying to make up a name. s129 00:14:57.930 --> 00:15:06.006 And the last one is a tool called " disciplinary instruction" in the query. s130 00:15:07.052 --> 00:15:19.960 Since most questions require two to four calls, before the sixth call, the agent is instructed to ask themselves whether they are getting something truly new or simply reframing a search that has already failed. s131 00:15:19.960 --> 00:15:32.136 And this idea came about when we were watching real-world traces where an agent would run the same search over and over again, hoping that a number would eventually appear. s132 00:15:32.136 --> 00:15:39.015 So the price of all this is latency, with a median of 38 seconds per response. s133 00:15:39.015 --> 00:15:42.264 This is not just a chat response. s134 00:15:42.264 --> 00:15:45.685 This is a small research task. s135 00:15:45.685 --> 00:15:56.065 And we rely on streaming to make 38 seconds feel like a workflow, not a frozen chat. s136 00:15:56.065 --> 00:15:58.925 This brings me to the next slide. s137 00:15:58.925 --> 00:16:04.527 So, in the first iteration, we really wanted to move away from the basic chatbot interface. s138 00:16:04.527 --> 00:16:12.955 We prepared the interface components and described them as Pydantic models available to the agent, saying, “Here are the components you have . s139 00:16:12.955 --> 00:16:16.128 Use them in your answer along with the text. s140 00:16:16.128 --> 00:16:26.995 For example, a source card was a component, an event could be made a component, a collapsible list could be a component , etc. The idea was to have a somewhat AI-generated s141 00:16:26.995 --> 00:16:30.998 interface that would be more readable than a simple sheet of text. s142 00:16:30.998 --> 00:16:34.731 The problem is that you can't stream a partial Pydantic object. s143 00:16:34.731 --> 00:16:39.463 It becomes valid JSON only after the last token arrives. s144 00:16:39.463 --> 00:16:50.244 As you can see, for a system that takes 38 seconds to respond, that meant 38 seconds of blank screen, and then everything appeared at once. s145 00:16:50.244 --> 00:16:55.102 So, the solution that saved the user experience was to split the response into parts. s146 00:16:55.102 --> 00:17:01.517 The agent now streams Markdown text token by token, and the interface styles it in real time as it arrives. s147 00:17:01.517 --> 00:17:06.460 And then the source card becomes a separate element that is a real component of the interface. s148 00:17:06.460 --> 00:17:12.598 It doesn't contain information that you would want to click on before reading the entire answer. s149 00:17:12.598 --> 00:17:19.655 So we have nothing to lose by waiting for a complete answer before adding this component. s150 00:17:19.655 --> 00:17:27.919 This gives us an interactive card instead of a regular in-text link, which is more user-friendly. s151 00:17:30.621 --> 00:17:39.115 But once that answer appears on the screen , the next question is what the user will do with it. s152 00:17:39.115 --> 00:17:44.778 Click on the source, copy the response, continue the conversation, or leave. s153 00:17:44.778 --> 00:17:52.447 It is this user experience that we want to track, and this brings us to logging. s154 00:17:53.867 --> 00:17:58.813 So, for now, we have chosen the simplest logging solution—Azure Table Storage. s155 00:17:58.813 --> 00:18:12.671 We log every question, tool call, response, feedback ( positive or negative), as well as events like clicking on the source or copying a response , along with a conversation history that is intentionally kept s156 00:18:12.671 --> 00:18:12.903 small. s157 00:18:12.903 --> 00:18:16.334 These are just the last 20 posts as context. s158 00:18:16.334 --> 00:18:21.864 For the stage we are at now with this product, that is enough. s159 00:18:21.864 --> 00:18:26.627 As the product grows, we will understand what we will need at that time. s160 00:18:26.627 --> 00:18:34.190 And how do we know that all this really works? s161 00:18:34.190 --> 00:18:45.737 Our assessment questions come from real sources, we conducted two rounds of testing with real users, plus there are rules of conduct for our AI system regarding what it can and cannot s162 00:18:45.737 --> 00:18:47.666 do. s163 00:18:47.666 --> 00:18:56.419 For example, no advice, maintaining neutrality towards competitors, and clearly stating when something is not in the guides. s164 00:18:57.578 --> 00:19:10.826 We have about 86 end-to-end tests run through Promptfoo , a full evaluation by the LLM- agent against the criteria, as well as a check that the correct source has been cited, plus an evaluation of just the search from the previous s165 00:19:10.826 --> 00:19:12.826 stages. s166 00:19:14.189 --> 00:19:29.147 We learned two lessons from our experience that we want to share: first, LLM scores have background noise, and you need to rerun the tests before you can trust the results. s167 00:19:31.300 --> 00:19:38.038 For example, when we conducted our own evaluation, five out of 12 outcome changes did not repeat. s168 00:19:38.832 --> 00:19:44.828 And secondly, sometimes the problem lies in the assessment system itself, not in the model. s169 00:19:44.828 --> 00:19:56.466 We had a case with a " hallucination" of a date for which the referee kept giving fines, and then it turned out that this date was the real text in the rules. s170 00:19:58.868 --> 00:19:59.949 Yes. s171 00:19:59.949 --> 00:20:03.921 Now about the use. s172 00:20:03.921 --> 00:20:08.363 We launched this rules assistant in June. s173 00:20:08.602 --> 00:20:28.724 Initially, it was only available to our clients' internal users, and the goal is to provide access to their external participants and clients by the end of the year. s174 00:20:28.724 --> 00:20:36.615 So, at this point, we already have some usage metrics. s175 00:20:36.838 --> 00:20:41.192 We have received about 900 questions so far. s176 00:20:41.192 --> 00:20:49.808 They are distributed across over 100 conversations, and about 34 people actually use them. s177 00:20:49.904 --> 00:21:00.555 We also collected feedback through “likes ” and “dislikes,” but most people stopped using this feature after the first month. s178 00:21:00.555 --> 00:21:05.726 We also have data on how many times sources were clicked on or answers were copied . s179 00:21:05.726 --> 00:21:09.639 But that's all the numbers are. s180 00:21:09.639 --> 00:21:15.742 We also use AI for analytics. s181 00:21:15.742 --> 00:21:21.849 Here on the right we see how LLM interpreted it . s182 00:21:21.849 --> 00:21:32.884 We noticed that in the first month when we brought people on board, there was some “ red teaming” going on. s183 00:21:32.884 --> 00:21:39.047 People tried to hack the assistant, make it go off topic, or asked strange questions. s184 00:21:39.047 --> 00:21:47.278 But by the third month, over 90% of the questions became real and to the point. s185 00:21:47.278 --> 00:21:53.813 Secondly, people use it for real-world cases. s186 00:21:54.036 --> 00:22:00.658 We saw users inserting emails from customers and asking for help responding to them. s187 00:22:00.658 --> 00:22:10.712 Or they inserted a real message from the protocol, a table, etc., and then composed the answer they wanted to check. s188 00:22:15.242 --> 00:22:25.806 And third, since we didn't get a lot of feedback through " likes" and "dislikes," we tried to determine the tone of the conversations by analyzing the following questions. s189 00:22:27.584 --> 00:22:39.921 So if people say "thank you" or move on to a new topic, we, yes, count that as a positive interaction. s190 00:22:41.000 --> 00:22:47.604 And if they ask the same thing again, it 's most likely a negative interaction. s191 00:22:48.125 --> 00:22:55.599 And we saw that most of the conversations were successful. s192 00:22:56.886 --> 00:23:02.587 So , to summarize, what are the main conclusions from this project so far? s193 00:23:02.587 --> 00:23:13.137 When it comes to documents, we found that it is important to maintain their structure. s194 00:23:13.910 --> 00:23:21.708 So sections are important both for getting an answer and for the agent to be able to capture knowledge in some way . s195 00:23:22.628 --> 00:23:31.867 We also saw that tables require a separate approach and special handling. s196 00:23:33.260 --> 00:23:48.510 On the agent side, as mentioned, it was important to provide different levels of granularity and define the amount of context needed, and overall we benefited greatly when we moved from a one-time s197 00:23:48.510 --> 00:23:52.050 query to LLM to an agent search. s198 00:23:53.766 --> 00:24:02.210 We saw that the assessment scores increased significantly. s199 00:24:02.210 --> 00:24:13.178 From a UX perspective , it was the streaming of data that made this tool convenient and useful. s200 00:24:13.178 --> 00:24:28.379 And we only add structured components when the user interface requires it. s201 00:24:28.693 --> 00:24:45.471 And finally, we think it's important that we tested with real users early on, and also used those interviews to create evaluation prompts that help us shape the product and decide whether what we think is an improvement is actually s202 00:24:45.471 --> 00:24:47.471 one. s203 00:24:53.606 --> 00:24:56.612 Thank you for listening to our report. s204 00:24:56.612 --> 00:24:59.172 Thank you.