Why Your Company Needs a Context Graph (and How to Build It) — Gil Feig, Merge

AI Engineer · 15 min · 228 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Right.
00:13Thank you all for for coming.
00:15Um my name is Gil Feyg.
00:16I am the co-founder and CTO of Merge.
00:20Merge helps companies build production AI very easily.
00:24We are the connective infrastructure uh to make everything come together.
00:28Uh today I'm going to be talking about why you need a context graph and how to build it.
00:32What I'm not going to do is talk about specific technology you should be using.
00:36Uh you can talk to your agent about that.
00:37It will help you.
00:38There's a lot of good technologies that can back this.
00:41Uh the focus of this is uh why you need a context graph and then sort of in a in a higher level how you should structure it, what the components should be, uh and how things come together
00:50to build the best possible uh brain for your company.
00:56If you haven't gotten a hat yet, by the way, my team is walking around passing out hats.
01:00They're They say well connected.
01:02Um so let's dive in here.
01:04So I'm not going to spend much talking about Merge, but just a quick overview of us.
01:08Um we have over 400 enterprise customers, OpenAI, Ramp, Netflix, Perplexity, a lot of companies across the board using us for both internal and customer-facing AI use cases.
01:20Uh we have three different products, Unified, Agent Handler, and Gateway that cover a lot of the components of it.
01:25Um we were founded 6 years ago by me and my co-founder Shen Shi.
01:28We've raised three rounds.
01:29Uh we're 130 employees in the three most beautiful cities in the world.
01:35All right.
01:36So quick agenda here.
01:38Step one or number one, we will talk about what a context graph even is and why it's so important that every company has one.
01:44We'll go into the different tiers of context, what what types of data, what types of of uh sort of logic from your company should be making it into your graph.
01:53We'll talk about context selection strategy.
01:55How are we picking the most relevant content for our agents or contexts for our agents at any moment in time?
02:01Traceability and provenance.
02:03How do we know where that data came from, who requested it, when it when it's from, is it fresh, all of that.
02:08And then lastly, some final key takeaways.
02:11At the end, I'll stick around if you do have any questions.
02:14Uh so, feel free to to come up to me afterwards.
02:18All right.
02:19So, what is a context graph in this talk?
02:21We're not going to go into the nodes and the edges that connect each node in a true graph infrastructure.
02:28We're going to talk about the company brain.
02:31Your company context graph consists of a lot of different components.
02:35It has third-party data.
02:37What are the other systems you use, like NetSuite, uh like like Jira.
02:41Um it has structured context.
02:44Uh so, that could be uh data coming in from other third parties.
02:47That could be data that that you all have put into your context layer.
02:52Uh things like things like static documents that you want to always be passed to your agents.
02:57We have memories.
02:58These are things that are formed.
02:59We all know memory, things that are formed as you're using the agent, telling it to remember things, things that your agent thinks that it should remember on its own.
03:06And then we also have skills.
03:08Uh and skills are a very very important part of your context graph, which we'll get to.
03:15All right.
03:16I see some picture Okay, cool.
03:17Uh so, moving on.
03:18Let's get Let's go through an example here.
03:20We're going to use this example to guide why it's not it's not enough to just connect a single MCP server or a couple MCP servers and call it a day.
03:29So, let's ask this simple question.
03:31One of one of your employees or or customers asks your agent, "Why was customer A upset last week?"
03:38Pretty simple question to answer.
03:40The agent goes, it sees it has access to your Zendesk ticketing, and it can check and say, "Hey, what it what tickets were filed by this customer last week?"
03:48Great.
03:48We saw that they saw that the API sync was broken, um and specific details about what went wrong.
03:54We also see the customer filed three tickets total, all of that.
03:58Great, that answers that question.
04:00But what if we want to extend that?
04:01And what if we want to say, "Which customers were upset last week?"
04:05That becomes a lot harder because you now have to have your agent load in every single ticket from Zendesk that happened in the last week.
04:12What if we're saying, "Which customers were upset in the last year?"
04:15We should be able to answer that.
04:16Our agent should be able to answer that, but it can't if it has to live live look up and pull tickets 100 at a time, thousands and thousands of those until it times out and it just can't answer the question.
04:26So, MCP is not fully solving a lot of these problems because the underlying access patterns of the APIs of these other platforms don't support a live look up methodology.
04:38If you could ask your ticketing system something semantically, if Jira exposed an endpoint that said, you know, that you could query in English, "Which customers were upset last week?"
04:47and it returned that data, that would be great.
04:49But no platforms incentivize to do that because it costs them money to vectorize it and it makes it so you don't have to visit their platform as often.
04:59So, you need a sinking layer on top of this.
05:01You can't just use MCP to live look up for things that you want to go deeper into, that you want to semantically question, you need to actually sync a copy of the data locally.
05:11Okay?
05:11This doesn't apply to everything.
05:13So, you might have you know, let's say static look ups to Stripe.
05:17We're always going to be asking Stripe a question about a specific customer.
05:20We're always going to be taking a single action in Stripe.
05:22That can be MCP.
05:23That can be live.
05:24But if you need any form of semantic look ups, any form of analysis that is beyond a simple look up, simple question, but requires looking across a a vast data set,
05:33you have to sync a copy.
05:35So, that becomes the next part of your sinking layer of your of your, uh, context layer.
05:40So, you have your MCPs, your live look ups, you have your synced or your cached look ups as well.
05:46And that you're pulling in from a third-party, you're putting in a vector database locally.
05:50Again, AI can build that vector DB for you relatively easily.
05:54Uh you got to you got to you know, work on your strategy of how you store data in the context layer and how you retrieve from it uh for that sync to cache layer.
06:01Um but that's sort of the next component here.
06:04But the next problem is, all right, now we have all this relevant data.
06:07We're we're pulling from our vector database to get, you know, semantic data and we're doing live lookups for other data we might want on top.
06:14Now, we need to bring all that data together cuz we can't just dump a massive amount of retrieved context back to our agent.
06:21So, that's where we introduce the router and the summarizer.
06:23The router is sending things to the right place at the beginning.
06:26The prompt comes in.
06:27It decides where, you know, who should be handling this, where it should be sent.
06:31Once that data's all aggregated in there, our summarizer then takes that data from from the the synced context and from the live context, summarizes it, and sends it back to the agent that sent the prompt over.
06:44But you all notice, we have one thing here that we haven't talked about, and that's memory and skills.
06:49That could live separately.
06:50Why is it important for your context layer?
06:52The reason is that getting context is actually a set of company principles and processes in and of itself.
07:00For example, which customer was upset last week can mean something very different to one company versus another.
07:06It could be number of tickets that are opened.
07:08Um but actually, what if those tickets were positive?
07:10What if actually we don't use a ticketing system and that we know that for customer happiness, we need to actually always go look at Qualtrics first and see survey data cuz that's just how our company works.
07:19We're different.
07:20That's why skills are important because you need skills like how do we determine if a customer is happy?
07:25So, now your agent should first check the skills before it then goes and starts checking your synced layer, your your uh live layer for what data to get and then summarizing and returning it.
07:36So, the flow here is prompt comes in, gets routed to the right agent.
07:40The agent looks to see if it has any memory or skills associated with the question or the task at hand.
07:47Based on that, it then knows where to look in the synced cache DB and from the live MCP lookups, summarize, send back.
07:57There's no magic here.
07:58There really is none.
07:59It's all physics as long as you know how to calculate, you know, within within your own sort of use case, you're great.
08:05This flow here where we looked at the number of open tickets again would not have been possible with live lookups, but with a local lookup it was really simple and we were able to basically gather all customers,
08:15number of tickets they have, summarize their tickets, and then give a status on each one.
08:20So, it's a mix of some live lookups, some cache lookups.
08:23That's all coming together here to build really really rich data for any question that gets asked.
08:31All right.
08:32And so, there are different tiers of context and some might debate whether prompt skills and memory are context.
08:38Context is anything that's being passed into the agent.
08:41We're going to put this all here because they all belong in your context layer.
08:45So, you have your prompt skills and memory.
08:46We talked about how those are going to really drive the the where do we get data from?
08:50You have your live API calls.
08:52These are easy easier to build, easier to add on, they're cheaper to run than synced context.
08:57So, you should prefer these when you know you're going to have simple lookups, simple retrievals.
09:02Cached data is absolutely essential if you want that that sort of semantic data, the complex lookups, deep deep questions about about any data.
09:12But, keep in mind it's a little bit more complicated to set up and build.
09:16It's more expensive to run because you have to have a vector database and you're constantly having to sync that data.
09:21So, it is tougher to keep up to date.
09:24And then lastly, something we didn't quite discuss yet is derived context, which is basically any additional context you create when you're pulling in data from the third party.
09:33So, this could be, you You as we're live pulling in all tickets, we're also for every time a customer's full set of tickets comes in, we're summarizing those and we're actually storing
09:43in our stored context, our structured stored context, we're storing a summary of the data that we pulled in.
09:50So, that in the future, if we have a common question, we don't have to then also go go re-look up and re-summarize everything.
09:58Awesome.
09:58So, all four of those come together.
10:00You saw the the chart for how they do that.
10:02Um and yeah, those are the tiers.
10:04So, quick flowchart of how this works when when a query comes in or a prompt comes in.
10:09So, first thing here is do we have a skill or memory match?
10:13If so, yes, we want to execute that skill.
10:16Does the skill need external data?
10:18No, then we're just generating a response and we're sending that right back.
10:22Unfortunately, it's not always this easy.
10:24So, what happens if we don't have a skill or memory match?
10:27No?
10:28Well, then let's go right to to data selection here.
10:31Uh and same thing for if that skill needs external data.
10:33So, we say yes, the first thing we're doing is checking our cache or our synced data store and we're saying, is the data here fresh?
10:40Is it within our SLAs?
10:41Can we use this data cuz it's viewed as fresh enough?
10:44Yes, then let's use that cache context.
10:47No?
10:48Then, unfortunately, we have to go down and we have to call the live API.
10:52Now, once we do have enough context, we're going to loop through these a few times till we build up the context we need.
10:58Once we do have enough context, we might want to store a memory of of whatever we just we just discovered or or figured out, uh store that to memory and then generate the response and send it back.
11:09And then, same thing here, if we didn't have enough context, I I just showed that, but we have to call the live API and then we will we will uh generate that response back to the client.
11:21I'll leave it here for a second.
11:22I I see some people looking and taking pictures.
11:30All right.
11:31So, last piece here, pretty much at the end, traceability and provenance.
11:35Doesn't seem important at front up front.
11:38Do not drop where the data is coming from.
11:41Always store it.
11:42You you have that from MCP.
11:44You also have that in if for anything you're syncing in Vector DBs, there's something called metadata.
11:48You want to store it in there.
11:49It's associated with the data.
11:51It is critical.
11:52It's similar to, you know, the way that this surface is in ChatGPT or any surface that you're using is when it tells you a fact or it says something, at the end it says three sources and it links to where it came from.
12:02That is even more important when you're making business critical decisions off of that data.
12:06When that data is is versioned, when that data can become outdated, when you're letting it make a decision on data that it synced a year ago.
12:12You don't want that.
12:13So, you need to know where it came from.
12:15Where did we fetch it?
12:16When did we fetch it?
12:17Which is not here.
12:17Under what idea under what ID and what scopes?
12:21Who accessed the data?
12:22What permissions were used to access it?
12:24Was it transformed?
12:25What did it originally look like?
12:27And then lastly, what should invalidate it?
12:28That could be age, for example.
12:30The data is more than more than a month old, we don't want to consider this ever.
12:35Really important for showing in your platform and if not for showing in your internal platform or whatever you're building, even for for being able to investigate when something does go wrong,
12:44cuz it will.
12:47So, putting it all together, here's another example of one that we might use.
12:51So, give me a one paragraph status update on our customer Acme.
12:55Include renewal risk and last support interaction.
12:58First, we're going to check the cache for the account summary.
13:00We're going to assume there's no skills here.
13:02So, we'll check that.
13:04If it's still or missing, we're going to call the CRM using a live API lookup.
13:09Then, we'll query the ticketing system for the last support interaction.
13:12Again, that's cached if within the SLA, otherwise it might have to do a live lookup.
13:16We're only we only want to pull five tickets.
13:18We can't pull the full history because again, when we have live lookups, we run into timeout risk.
13:23And then lastly, we'll generate a derived summary.
13:26We might store that derived summary, and we'll pass that back.
13:33So, key takeaways, tools are not context.
13:36A context graph is going to help you reason across the whole company.
13:39It's not just tools.
13:40It's data.
13:40It's synced data.
13:42It's processes that you have as a business.
13:44Who is documenting, you know, when a customer complains, what do we do?
13:48How does it know to always route to the right place?
13:50That is a skill.
13:52These It's your company's skills.
13:53It's your processes.
13:54It's your data, and it's your tools and actions you need to take.
13:58More data, not always better.
13:59You want fresh, high-quality data.
14:01You want to pull only what you need.
14:03When you do have to sync data, you want to structure it properly, so it's accessed in the right way.
14:08And then lastly, every answer has to be traceable.
14:11You do not want to be left in a position where your AI made a bad decision, and you have no idea why it did that.
14:19So, last piece, plugging merch here.
14:22Enterprise AI runs on us.
14:23We power these features for a lot of companies.
14:25We have Gateway, which is an LLM router.
14:27We have Unified, which helps you sync data from third parties out of your customers' platforms.
14:32And then we have Agent Handler, which is live MCP connections pre-built across hundreds of tools, governed identity all built in, uh including provenance, uh and yeah, it's plug and play for both internal and customer-facing use cases.
14:46So, if you want to build an easy context layer, hit us up.
14:51Thank you, everyone, for listening.
15:09[music]