Pinecone 2.0 — Edo Liberty, Pinecone

AI Engineer · 19 min · 193 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:13All right, welcome everyone.
00:18I want to tell you a little bit about uh the new the new uh knowledge layer and I'll start with telling you a little bit about theory of mind.
00:26Uh how many of you uh know what theory of mind is?
00:31All right, so a few folks.
00:33It's a uh term in psychology about uh our capacity to understand uh and have a mental state or a model of a mental state for other people.
00:42What do they know?
00:43What do they believe in?
00:44How do they act?
00:46That's what allows us to socialize.
00:48That's what allows us to lie to each other and so on.
00:50Uh but it's a crucial developmental stage for humans.
00:55uh in a necessary function.
00:57I'll I'll sort of explain it in the best possible way with a funny story just to break the ice.
01:02Uh about 15 16 years ago I uh worked at a company called Yahoo um that had a product called Yahoo Answers which was sort of like a Q&A forum.
01:13Uh we used then AI which is the stone age of AI you know it's like uh NLP and other statistical models to answer questions.
01:22But the funny thing is people would come to uh Yahoo and ask questions like am I fat?
01:28Which uh is funny because obviously uh you know Q&A forum would wouldn't have any clue whether you're you're too fat or not.
01:38Uh but this is a funny way to say yeah people have a very bad theory of mind on sort of what the products that they use what information they have and what they don't have.
01:48And I want to argue that in some sense we are making that mistake with AI today in a more subtle and less sort of like uh silly way but in a just as fundamental of a mismatch.
02:01And so I will categorize roughly three kinds of knowledge.
02:05I think there there are many more but you know in broad stro in broad broad strokes this is not a bad way to categorize.
02:12First of is general knowledge.
02:15Think about that is what's available in the public domain.
02:18Things that models are trained on and we expect an LLM to just know out the gate.
02:23Uh things about just you know improving your Rust code uh asking you like you know explaining stuff about the legal terms or uh sort of exploring use cases for for products.
02:36Um we know how to do that.
02:38Models are really good at that.
02:40When we use our agents they have that information.
02:42there is specific knowledge [clears throat] the companies of course the models wouldn't have uh any clue about whether we've uh implemented something specific in our codebase what a contract of a specific customer has of course this is information that's private to the company
03:00uh but we know how to deal with that and rag and search and of course pine cone veto dbs and so on uh solved these problems a long time ago there's still a lot to go but we by and large know how to do it
03:13and then I would say there is tribal knowledge.
03:16Okay, tribal knowledge is the sort of general knowledge inside the company.
03:21This is the kind of information that uh an employ a seasoned employee has that a new hireer does not.
03:28Okay, it's roughly how we do things, what our processes are, who's in charge of what, uh what are the guiding principles, what are the you know sort of how we get stuff done in this company.
03:39Okay.
03:41um that is not a that's not general knowledge because it's it's specific to your company but it's also not a specific piece of knowledge because you can't just get it by searching.
03:50You can't just get it by plucking it off one place and say yes this document says uh that this is our culture.
03:56It's the sum total of a lot of uh things.
04:00Um and so if our theory of mind of most of our agents in big companies today should be something like a new hire.
04:08They're brilliant.
04:09They have tools.
04:10They have we have very high expectations of them.
04:13But they are by and large clueless.
04:15Okay.
04:16They have no idea about your company's goals, culture, priorities, processes.
04:21Um they have to read a bunch of documents uh to be able to get any semblance of understanding of where they are in the world.
04:30Um and really they start most tasks from scratch.
04:36Okay.
04:36they really start kind of wake up and every time you start your your session, uh, they're like a new hireer on their first day on the job.
04:47So, I would argue that what they need is not necessarily to be smarter, to have better tools, but to have something new, which I would call a knowledge layer going forward.
04:57Um, and I have to explain to you what I think a knowledge layer has to actually do.
05:02But just I'll define it generally before explain to you what we do.
05:06Um so at the very least it needs to be persistent.
05:11We talked about this is the company knowledge, processes, capabilities and goals and so on.
05:16It has to be persistent.
05:18Uh it sure it uh feels like putting this thing together is going to be a monumental effort.
05:24So you want to do it once or at least very rarely.
05:27Uh and of course all your agents in company need to have access to this thing.
05:32The second thing is that you would argue that this [clears throat] knowledge layer needs to be uh highly specialized.
05:38If your agent uh is in charge of HR practices or optimizing your uh your kernels in your codebase or uh resolving customer issues that they would care about different things they would want to you know address they would want to have different information available to them.
05:56Um, and I would argue irrelevant of of specifically how it's created, I would argue that it's a good practice to have the domain experts in the company own those and have being able to interact with them and make sure that they actually
06:10uh are aligned with what the company wants to do.
06:14And finally, which is the hardest thing is they need to stay up to date and they need to stay concurrent constantly because companies change and you need to make information
06:26uh uh deprecated.
06:28When it becomes deprecated, if the company replaces its CEO, the very next day, everybody in the company knows who the CEO is.
06:36Okay?
06:36Even though 9,000 documents going from, you know, a day before yesterday to uh the last 10 years say it's somebody else.
06:45This has become common tribal knowledge immediately.
06:49And everything else, even you know, every rag search will tell you otherwise.
06:54This is now the overriding fact.
06:58Okay.
07:00Um I will argue before we move on that I'm not suggesting that this knowledge layer should replace everything else we're doing.
07:09Our agents are already very capable and should keep being very capable.
07:14Uh they have access to LLMs of course tooling local files search of course like rag vector search text search and so on.
07:21And I would argue that uh the knowledge layer needs to be a separate entity that all these agents have access to as as I explained why this is sort of a fundamentally
07:32different kind of data source.
07:35So I want to tell you more about Nexus.
07:38Um um Nexus is a product that we're uh uh of course it's a knowledge layer but I need I want to explain to you what it does.
07:49uh of course so the the four parts are roughly connectors what I would call context uh uh tasks and queries I'll explain each one of them uh now so connectors of course do the obvious things there's nothing much to talk about them
08:08um uh they bring data into the platform uh respect access uh rights and uh and control and so on um very dull as a topic of discussion but incredibly important in enterprises,
08:24incredibly important to do right.
08:26Uh and everybody who's built that knows how painstaking it is to get this thing to work well.
08:32Um the interesting sort of the first properly interesting part is um the context.
08:41So the context think about that as the um the sum of assets that the that Nexus or this knowledge layer saves about a specific topic.
08:53Think about this as a context for HR, for operations, for engineering and so on.
09:00Okay, of course it has the sources, the connections to the data and so on.
09:05But then it has the manifest.
09:07Okay, the manifest is the first truly novel idea and and a important uh entity.
09:13Okay, the way that we today give context to our agents is with skills and plugins and uh commands and sort of markdown files we write ourselves.
09:27In a knowledge layer, you don't do that anymore.
09:30Okay?
09:30You tell the you tell the knowledge layer, you tell Nexus what you care about in a manifest.
09:36You say what kind of tasks you're you want to complete, what entities you want to track, what kind of information you care about, and it's it's the knowledge layer's work to keep track of those files.
09:49Okay?
09:49And and organize its own data.
09:52Um uh and then it has the knowledge files themselves.
09:56So think about the manifest is the meta knowledge file or set of files.
10:02Um and then there's the knowledge the knowledge itself.
10:05uh think about this as five types of contents.
10:10One of them the the first kind is a semantic map.
10:13Okay, this is a file that organizes the catalog of where everything else is, the key schemas, the glosseries, all the information that you need to be able to operate this context.
10:27Then there are of course a bunch of markdown files that contain information, decisions, memories, uh all sorts of unstructured information.
10:36If you've uh who here has heard about uh LLM wiki okay so LLM wiki is this uh idea that Andre Karpathy suggested to have agents basically manage some stash of markdown files to sort of keep their own notes of stuff.
10:55This is a very similar idea.
10:57Okay, sort of on steroids but similar concept but that's not enough.
11:03Context contains also a bunch of SQL tables for tabular data for aggregation for facts that they need to memorize.
11:10uh a vector database to be able to semantically search and text search and filter a bunch of information and all the information that it itself decides is necessary to keep and and remember
11:21and graph entities and graphics to remember uh again entities relationships uh causal chains and so on.
11:30Okay.
11:31And all of that is managed and created for you.
11:35You don't touch any of this.
11:36You don't have to uh know how any of this is created.
11:40It's all maintained through the manifest.
11:44So these are this is the context.
11:46Now what are the tasks?
11:47What are the actions?
11:48These are the nouns.
11:48What are what are the verbs?
11:50What can we do with those uh objects?
11:53Uh first of all is of course import which is bring new information to the system, new files, new events, new uh concepts.
12:02Um curate is probably the most and most elaborate one of the two most elaborate functions.
12:09it's uh job is to take um to take the uh the say call it a file it doesn't have to be a file but let's say it's a new PDF or a new meeting transcription
12:20or a new uh presentation for the board and say now I need to update my view of the world okay so it takes that it takes the manifest which tells okay now
12:30what what things do I care about what concepts do I need to track what needs to be uh uh what do I need to memorize here and then it takes the
12:41semantic map that I told you contains all the like organizes where information is and understands okay fine I need to go update these tables these relationships my graph uh this
12:52uh these text indexes and so on um and when it's done it has essentially said okay fine I have assimulated the new information it's already it's it's sort of persisted in your context
13:06and we can move on Okay, search is sort of a internal function very important.
13:13It basically abstracts with simple functions all the access to that information so that the agent that I'll tell you in a second doesn't have to know uh how to sort of access the internal parts of the context.
13:27Now comes uh the second I think most uh most uh interesting uh part of of the system which is noql.
13:38Noql is our way to to uh specify how we want information out of the context.
13:45It's not enough to just ask questions.
13:47If you're an agent, it's really important for you to give us the budget in tokens or in dollars.
13:52how much time you want us to invest in this sort of like do you want us to just a quick off-the- cuff kind of immediate answer or do you want us to go full tilt
14:01and scour everything we know to make sure we give you the most accurate answer uh and so on the structure and and so on.
14:09It's a you know it's an agentic tool.
14:11This is not like a chat interface.
14:13Okay.
14:15Um and of course what you get back is like something that's agent friendly.
14:21This is uh you know grounded text and and kind of structured in the right way format.
14:28This again is not designed to be like a chat interface.
14:31This is really what your agents expect.
14:33Okay.
14:35Um so this is this is a uh um this is a uh trying to figure out how much time I have.
14:46Okay, I need to go a little bit faster.
14:48So uh this is a sophisticated crowd.
14:52I want to sort of uh pop the hood and show you a little bit about how things work.
14:57So uh if anything this is the interface I want to there are many many cool ideas uh that make this work but I want to tell you a little bit about just one of them.
15:09Okay.
15:10What you're seeing is the interface for Nexus.
15:13Okay.
15:14You will see at the top a question that the uh the agent issued and you see you see on the left uh the different steps that it went through.
15:24Okay.
15:25What you see on the bottom right is probably the most interesting.
15:28You see generated code on the bottom right.
15:31What Nexus does which is very different than other systems.
15:35It's what we call a runtime coding agent.
15:38Okay.
15:39Unlike other coding systems in in that you're used to uh where the task is take a large code base and then help me edit it right but when the edits are done
15:54what I have is a piece of software and that is deployed that is the artifact okay what is running does not contain the model anymore it's just the code okay this is not what's happening here the in query time you should think about the engine essentially
16:11building something like a Jupyter notebook, something like uh like a Python ripple, right?
16:16And they literally write code and execute it and write code and execute and if they get the answer, they know what to do with it.
16:23And if it's, you know, if it's not what they expected, they rewrite that piece of the code and what you get in the end is a is a piece of code that completes the task that gets the information that you want to get.
16:36Okay, that is savable.
16:38that's rerunnable.
16:40Okay.
16:40But it's also incredibly flexible because now the the answer is got not by by a query to a database.
16:50It's it's written by code which is of course incredibly flexible.
16:55Okay, that code is by the way, interestingly enough, if you replace if you write software this way, the the amount the amount of prompting that you need is significantly reduced.
17:08We went down from having something like 150,000 tokens to give our agents all the tooling they need to less than a thousand tokens to specify all the interfaces they need to get information out of Nexus.
17:19Okay.
17:20And so the the so what it looks like when you run it is like this.
17:28Okay.
17:28So on the left of course you'll see Nexus with uh NoQL answering the question and then on the right is sort the same uh task given to an agent with all the tooling to complete the task.
17:42Okay, but remember this on the right it's uh severely handicapped.
17:46It's like the employee on the first day on the job.
17:49They have to read a bunch.
17:50They have to figure out where the data is.
17:52They have to figure out what the task is.
17:53They have to write a bunch of code.
17:55Uh it's just it's just severely handicapped and there's I don't know if you okay stopped but um needless to say this is by far slower.
18:06It's a lot more expensive in terms of token consumption.
18:10Uh but interestingly enough it's also a hell of a lot less accurate because again they don't have the right context, the right objectives and so on in your company.
18:19Um, I'll just say that [clears throat] we work uh obviously with a bunch of early access customers already.
18:27Uh, you see some results here, but I'll just sort of fly through them and say that across different domains, across different enterprises, across different kinds of task, you we universally see the same thing that if you
18:42shift to this paradigm, you get significant cost savings.
18:48uh 77 90 you know percent token consump token reduction is pretty common.
18:55Um uh it gets a hell of a lot faster anywhere from 20 30% to sometimes uh uh uh 77% faster.
19:06Okay.
19:06But and the most important thing is it actually becomes a lot more accurate as well in the same time.
19:11So this is really uh it's a slam dunk.
19:15I'll just wrap up by saying that uh um the that Nexus is coming out of early access and into public preview literally tomorrow.
19:27So go uh try it out.