When All Context Matters: Extended Cache Augmented Generation - Luis Romero-Sevilla, Orbis
https://www.youtube.com/watch?v=XovaGv4f39A
AI Engineer · 5 min · 65 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:03] [music]
[00:10] Hi, I'm Luis Romero Sevilla and I'm the VP of AI at the Orbifold operation.
[00:15] I'm on a mission to solve knowledge representation when all context matters.
[00:19] So, let's start with a very specific example.
[00:24] Let's say we have a large number of documents and all documents represent an event and all documents in the collection are relevant to answer a set of questions that the user has.
[00:34] Not only that, there's one more challenge.
[00:38] The document in the collection becomes obsolete very fast and all documents get replaced with new information.
[00:44] Let's start with a simple approach.
[00:46] We could start with a simple rag.
[00:48] For that, we just need a vector database and an embedding model.
[00:53] An embedding model takes the documents and turns them into a learned numerical representation, a vector.
[01:00] Now, we take those vectors and we store them in a database optimized for performing operations with the vectors.
[01:07] Perfect.
[01:08] Now, we can take all of our questions,
[01:10] [music]
[01:10] turn them into vectors and then look for a vectors that are similar to the initial query.
[01:18] Those vectors that are within the trip similarity threshold are retrieved and we can pass them to the LLM to answer the question.
[01:27] Inserting to a vector database, it's relatively fast.
[01:30] So, whenever a collection becomes obsolete, we can just replace it with a new one.
[01:36] We still have one problem with our very specific scenario.
[01:41] All the documents in the collection are relevant for us to answer the question.
[01:45] So, we can't just take all the documents in the collection and pass them to LLM.
[01:51] That's just one of the many limitations with this approach.
[01:55] Wow.
[01:55] [screaming]
[01:55] Now, let's get a bit more sophisticated.
[01:58] All documents are relevant to answer a global question.
[02:01] Therefore, there must be some connections and relationships between the details within a document in the collections.
[02:08] For us to map out those relationships, we're going to need a knowledge graph.
[02:12] And one implementation we could try is GraphRAG.
[02:16] GraphRAG has many steps, but basically,
[02:18] [music]
[02:18] it uses an LLM to read through all the documents and extract key entities and relationships between them.
[02:27] It constructs a network, a knowledge graph, where all those connections and details are tied together.
[02:32] Then, when a question is asked, it navigates this graph to synthesize a complete answer drawn across the entire collection.
[02:39] If your collection of documents isn't changed very often, GraphRAG is an excellent approach for finding those relationships within details to answer the user's question.
[02:51] However, our very specific scenario states that our data is not only deeply interconnected, but also the data gets replaced very often.
[03:01] Recomputing a knowledge graph every time the data gets replaced is computationally very expensive, and it takes relatively long time.
[03:08] Okay, what if we take an even simpler approach and we continue to build on top of it?
[03:13] If we were to use something like GraphRAG, each document needs to pass through an LLM [clears throat] for the entity and relationship structure anyway.
[03:21] Why can't we just throw all the documents into context?
[03:24] This approach would look something like "cache augmented generation" (CAG), where we use a model with a large context window, load the documents into the context, and cache the context by storing
[03:35] the model's KB matrix.
[03:38] The problem here is that the context window is limited, and if you fill the context window too much, the quality of the answer gets degraded, too.
[03:46] The solution: what if we use more CAGs in parallel and distribute the documents across different context buckets.
[03:54] Now, each cache can answer questions regarding its content.
[03:57] And now we just need something to ask the right questions to the right buckets.
[04:03] So, for this, we can use a smarter model to interrogate each bucket and eventually synthesize an answer.
[04:11] How do we distribute the documents?
[04:13] It sounds tempting to organize the documents by domains and tell the supervisor, "Hey, here are the different categories."
[04:20] But in practice, with very dense relationship between documents, the supervisor tends to ignore domains that at first glance seem irrelevant.
[04:30] For this reason, all documents are distributed in no particular order.
[04:33] The only requirement is to balance the number of documents in a way that the least amount of documents are needed.
[04:40] Then the supervisor model start exploring the buckets and progressively builds its internal understanding.
[04:47] And if it finds something interesting, it can ask a specific bucket follow-up questions.
[04:52] Because all caches can be loaded in parallel, the knowledge building process is significantly faster than graph rag while providing more accurate answers than a simple rag.
[05:06] And you're probably thinking, "KV cache can be pretty expensive."
[05:10] And you're absolutely right.
[05:12] But there are ways to reduce that cost by optimizing how long each cache lives.
[05:17] And at the end, there are many retrieval strategies, and all of them have their trade-offs, whether it's compute, cost, speed.
[05:26] Currently, there is no one-solution-fits-all.
[05:29] So, each type fits our solution to our very specific problem.
[05:34] Thank you for watching.
[05:35] And for any questions or continuing this conversation, going to leave my details here.
[05:51] Mhm.