WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=XovaGv4f39A NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/XovaGv4f39A.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:03.929 --> 00:00:05.929 [music] s2 00:00:10.080 --> 00:00:15.200 Hi, I'm Luis Romero Sevilla and I'm the VP of AI at the Orbifold operation. s3 00:00:15.200 --> 00:00:19.800 I'm on a mission to solve knowledge representation when all context matters. s4 00:00:19.800 --> 00:00:24.160 So, let's start with a very specific example. s5 00:00:24.160 --> 00:00:34.480 Let's say we have a large number of documents and all documents represent an event and all documents in the collection are relevant to answer a set of questions that the user has. s6 00:00:34.480 --> 00:00:38.200 Not only that, there's one more challenge. s7 00:00:38.200 --> 00:00:44.920 The document in the collection becomes obsolete very fast and all documents get replaced with new information. s8 00:00:44.920 --> 00:00:46.640 Let's start with a simple approach. s9 00:00:46.640 --> 00:00:48.320 We could start with a simple rag. s10 00:00:48.320 --> 00:00:53.360 For that, we just need a vector database and an embedding model. s11 00:00:53.360 --> 00:01:00.040 An embedding model takes the documents and turns them into a learned numerical representation, a vector. s12 00:01:00.040 --> 00:01:07.400 Now, we take those vectors and we store them in a database optimized for performing operations with the vectors. s13 00:01:07.400 --> 00:01:08.200 Perfect. s14 00:01:08.200 --> 00:01:10.085 Now, we can take all of our questions, s15 00:01:10.085 --> 00:01:10.320 [music] s16 00:01:10.320 --> 00:01:17.360 turn them into vectors and then look for a vectors that are similar to the initial query. s17 00:01:18.640 --> 00:01:26.880 Those vectors that are within the trip similarity threshold are retrieved and we can pass them to the LLM to answer the question. s18 00:01:27.440 --> 00:01:30.840 Inserting to a vector database, it's relatively fast. s19 00:01:30.840 --> 00:01:36.480 So, whenever a collection becomes obsolete, we can just replace it with a new one. s20 00:01:36.480 --> 00:01:40.920 We still have one problem with our very specific scenario. s21 00:01:41.240 --> 00:01:45.760 All the documents in the collection are relevant for us to answer the question. s22 00:01:45.760 --> 00:01:51.640 So, we can't just take all the documents in the collection and pass them to LLM. s23 00:01:51.640 --> 00:01:55.000 That's just one of the many limitations with this approach. s24 00:01:55.000 --> 00:01:55.281 Wow. s25 00:01:55.281 --> 00:01:55.840 [screaming] s26 00:01:55.840 --> 00:01:58.680 Now, let's get a bit more sophisticated. s27 00:01:58.680 --> 00:02:01.600 All documents are relevant to answer a global question. s28 00:02:01.600 --> 00:02:08.399 Therefore, there must be some connections and relationships between the details within a document in the collections. s29 00:02:08.399 --> 00:02:12.840 For us to map out those relationships, we're going to need a knowledge graph. s30 00:02:12.840 --> 00:02:16.160 And one implementation we could try is GraphRAG. s31 00:02:16.160 --> 00:02:18.206 GraphRAG has many steps, but basically, s32 00:02:18.206 --> 00:02:18.240 [music] s33 00:02:18.240 --> 00:02:27.040 it uses an LLM to read through all the documents and extract key entities and relationships between them. s34 00:02:27.040 --> 00:02:32.640 It constructs a network, a knowledge graph, where all those connections and details are tied together. s35 00:02:32.640 --> 00:02:39.800 Then, when a question is asked, it navigates this graph to synthesize a complete answer drawn across the entire collection. s36 00:02:39.800 --> 00:02:51.720 If your collection of documents isn't changed very often, GraphRAG is an excellent approach for finding those relationships within details to answer the user's question. s37 00:02:51.720 --> 00:03:01.120 However, our very specific scenario states that our data is not only deeply interconnected, but also the data gets replaced very often. s38 00:03:01.120 --> 00:03:08.480 Recomputing a knowledge graph every time the data gets replaced is computationally very expensive, and it takes relatively long time. s39 00:03:08.480 --> 00:03:13.120 Okay, what if we take an even simpler approach and we continue to build on top of it? s40 00:03:13.120 --> 00:03:21.200 If we were to use something like GraphRAG, each document needs to pass through an LLM [clears throat] for the entity and relationship structure anyway. s41 00:03:21.200 --> 00:03:24.760 Why can't we just throw all the documents into context? s42 00:03:24.760 --> 00:03:35.640 This approach would look something like "cache augmented generation" (CAG), where we use a model with a large context window, load the documents into the context, and cache the context by storing s43 00:03:35.640 --> 00:03:38.080 the model's KB matrix. s44 00:03:38.080 --> 00:03:46.560 The problem here is that the context window is limited, and if you fill the context window too much, the quality of the answer gets degraded, too. s45 00:03:46.560 --> 00:03:53.760 The solution: what if we use more CAGs in parallel and distribute the documents across different context buckets. s46 00:03:54.080 --> 00:03:57.760 Now, each cache can answer questions regarding its content. s47 00:03:57.760 --> 00:04:03.040 And now we just need something to ask the right questions to the right buckets. s48 00:04:03.040 --> 00:04:11.040 So, for this, we can use a smarter model to interrogate each bucket and eventually synthesize an answer. s49 00:04:11.040 --> 00:04:13.280 How do we distribute the documents? s50 00:04:13.280 --> 00:04:20.440 It sounds tempting to organize the documents by domains and tell the supervisor, "Hey, here are the different categories." s51 00:04:20.440 --> 00:04:30.320 But in practice, with very dense relationship between documents, the supervisor tends to ignore domains that at first glance seem irrelevant. s52 00:04:30.320 --> 00:04:33.919 For this reason, all documents are distributed in no particular order. s53 00:04:33.919 --> 00:04:40.560 The only requirement is to balance the number of documents in a way that the least amount of documents are needed. s54 00:04:40.560 --> 00:04:47.600 Then the supervisor model start exploring the buckets and progressively builds its internal understanding. s55 00:04:47.600 --> 00:04:52.560 And if it finds something interesting, it can ask a specific bucket follow-up questions. s56 00:04:52.560 --> 00:05:06.600 Because all caches can be loaded in parallel, the knowledge building process is significantly faster than graph rag while providing more accurate answers than a simple rag. s57 00:05:06.600 --> 00:05:10.040 And you're probably thinking, "KV cache can be pretty expensive." s58 00:05:10.040 --> 00:05:12.080 And you're absolutely right. s59 00:05:12.080 --> 00:05:17.320 But there are ways to reduce that cost by optimizing how long each cache lives. s60 00:05:17.320 --> 00:05:26.600 And at the end, there are many retrieval strategies, and all of them have their trade-offs, whether it's compute, cost, speed. s61 00:05:26.600 --> 00:05:29.760 Currently, there is no one-solution-fits-all. s62 00:05:29.760 --> 00:05:34.880 So, each type fits our solution to our very specific problem. s63 00:05:34.880 --> 00:05:35.640 Thank you for watching. s64 00:05:35.640 --> 00:05:42.560 And for any questions or continuing this conversation, going to leave my details here. s65 00:05:51.600 --> 00:05:53.600 Mhm.