When All Context Matters: Extended Cache Augmented Generation - Luis Romero-Sevilla, Orbis

AI Engineer · 5 min · 65 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:03[music]
2. 00:10Hi, I'm Luis Romero Sevilla and I'm the VP of AI at the Orbifold operation.
3. 00:15I'm on a mission to solve knowledge representation when all context matters.
4. 00:19So, let's start with a very specific example.
5. 00:24Let's say we have a large number of documents and all documents represent an event and all documents in the collection are relevant to answer a set of questions that the user has.
6. 00:34Not only that, there's one more challenge.
7. 00:38The document in the collection becomes obsolete very fast and all documents get replaced with new information.
8. 00:44Let's start with a simple approach.
9. 00:46We could start with a simple rag.
10. 00:48For that, we just need a vector database and an embedding model.
11. 00:53An embedding model takes the documents and turns them into a learned numerical representation, a vector.
12. 01:00Now, we take those vectors and we store them in a database optimized for performing operations with the vectors.
13. 01:07Perfect.
14. 01:08Now, we can take all of our questions,
15. 01:10[music]
16. 01:10turn them into vectors and then look for a vectors that are similar to the initial query.
17. 01:18Those vectors that are within the trip similarity threshold are retrieved and we can pass them to the LLM to answer the question.
18. 01:27Inserting to a vector database, it's relatively fast.
19. 01:30So, whenever a collection becomes obsolete, we can just replace it with a new one.
20. 01:36We still have one problem with our very specific scenario.
21. 01:41All the documents in the collection are relevant for us to answer the question.
22. 01:45So, we can't just take all the documents in the collection and pass them to LLM.
23. 01:51That's just one of the many limitations with this approach.
24. 01:55Wow.
25. 01:55[screaming]
26. 01:55Now, let's get a bit more sophisticated.
27. 01:58All documents are relevant to answer a global question.
28. 02:01Therefore, there must be some connections and relationships between the details within a document in the collections.
29. 02:08For us to map out those relationships, we're going to need a knowledge graph.
30. 02:12And one implementation we could try is GraphRAG.
31. 02:16GraphRAG has many steps, but basically,
32. 02:18[music]
33. 02:18it uses an LLM to read through all the documents and extract key entities and relationships between them.
34. 02:27It constructs a network, a knowledge graph, where all those connections and details are tied together.
35. 02:32Then, when a question is asked, it navigates this graph to synthesize a complete answer drawn across the entire collection.
36. 02:39If your collection of documents isn't changed very often, GraphRAG is an excellent approach for finding those relationships within details to answer the user's question.
37. 02:51However, our very specific scenario states that our data is not only deeply interconnected, but also the data gets replaced very often.
38. 03:01Recomputing a knowledge graph every time the data gets replaced is computationally very expensive, and it takes relatively long time.
39. 03:08Okay, what if we take an even simpler approach and we continue to build on top of it?
40. 03:13If we were to use something like GraphRAG, each document needs to pass through an LLM [clears throat] for the entity and relationship structure anyway.
41. 03:21Why can't we just throw all the documents into context?
42. 03:24This approach would look something like "cache augmented generation" (CAG), where we use a model with a large context window, load the documents into the context, and cache the context by storing
43. 03:35the model's KB matrix.
44. 03:38The problem here is that the context window is limited, and if you fill the context window too much, the quality of the answer gets degraded, too.
45. 03:46The solution: what if we use more CAGs in parallel and distribute the documents across different context buckets.
46. 03:54Now, each cache can answer questions regarding its content.
47. 03:57And now we just need something to ask the right questions to the right buckets.
48. 04:03So, for this, we can use a smarter model to interrogate each bucket and eventually synthesize an answer.
49. 04:11How do we distribute the documents?
50. 04:13It sounds tempting to organize the documents by domains and tell the supervisor, "Hey, here are the different categories."
51. 04:20But in practice, with very dense relationship between documents, the supervisor tends to ignore domains that at first glance seem irrelevant.
52. 04:30For this reason, all documents are distributed in no particular order.
53. 04:33The only requirement is to balance the number of documents in a way that the least amount of documents are needed.
54. 04:40Then the supervisor model start exploring the buckets and progressively builds its internal understanding.
55. 04:47And if it finds something interesting, it can ask a specific bucket follow-up questions.
56. 04:52Because all caches can be loaded in parallel, the knowledge building process is significantly faster than graph rag while providing more accurate answers than a simple rag.
57. 05:06And you're probably thinking, "KV cache can be pretty expensive."
58. 05:10And you're absolutely right.
59. 05:12But there are ways to reduce that cost by optimizing how long each cache lives.
60. 05:17And at the end, there are many retrieval strategies, and all of them have their trade-offs, whether it's compute, cost, speed.
61. 05:26Currently, there is no one-solution-fits-all.
62. 05:29So, each type fits our solution to our very specific problem.
63. 05:34Thank you for watching.
64. 05:35And for any questions or continuing this conversation, going to leave my details here.
65. 05:51Mhm.