On AI and Knowledge — Pablo Castro, Distinguished Engineer & CVP for AI Knowledge, Microsoft

AI Engineer · 17 min · 162 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Now taking the stage is CVP and distinguished engineer at Microsoft, Pablo Castro.
00:22Hello everyone.
00:25Hello everyone.
00:26Good morning.
00:28It's great to be back here at the AI Engineer World's Fair.
00:33Now, my job at Microsoft is to connect the dots between AI and knowledge.
00:40As an information retrieval nerd, like that's great for me.
00:43Like I spend a lot of time on looking at knowledge representation, extraction, search, and whatnot.
00:50And thinking about agents and knowledge really invites to reflect on, you know, what it means to know something.
00:56And uh, you know, the the the nature of how do we get things done based on what we know.
01:02Next slide.
01:09All right.
01:14There.
01:15So, this morning what I thought we would do is spend a little bit of time talking about the nature of knowledge and split it into these three categories of intrinsic,
01:23extrinsic, and learned.
01:27Intrinsic knowledge is just the knowledge that comes with the models.
01:30You know, it's what we uh train the models on, the training data, and what um uh stored in the models' kind of parametric memory.
01:38And while it's kind of the obvious thing, I would argue this is the knowledge that actually threw us into the exponential we are in today.
01:45It's what started many of the scenarios that then grew on all the things we're doing with agents today.
01:52Let me give you an example with code.
01:54So, I wrote these two pieces of code about 25 years apart.
01:59And yet, the process to put this thing together was surprisingly similar.
02:04Like I had to sit down with what I knew or what I had to go look up and then just write it up.
02:11And while, you know, I'm illustrating this with knowledge, you could say the same thing about, you know, writing an email or creating a summary of a document.
02:19Now, you can see this exponential at play in tasks like these where, you know, I'm sure you can go further back.
02:25But an interesting point in time to start looking at this would be when Microsoft introduced IntelliSense.
02:30That was in '96.
02:32And, you know, it was great.
02:32You didn't have to remember function signatures anymore and whatnot.
02:36It takes 22 years from there to go for the next step where machine learning helps us actually rank the options we give you in IntelliSense, so it's quicker to pick the right choice.
02:47Just 3 years after that, GitHub Copilot launches.
02:51And that was one key inflection point.
02:54This was even before ChatGPT was announced.
02:57And, you know, I would argue that GitHub Copilot, ChatGPT, that sort of experiences were heavily grounded on this intrinsic memory, what the models already knew.
03:06From there, of course, things shifted.
03:08You know, a couple of years later, Cursor launches, GitHub Copilot X launches, and how we do things kind of evolved really quick, which takes us to kind of late last year.
03:18Opus 4.5 ships.
03:19And then rapid succession, you know, GPT, Opus, and other models keep getting better and better at coding.
03:25Which takes us to early this year, where incredibly successful software like like Open Claw comes out to existence with not a single line of code written by hand.
03:35So, this is the shape of the exponential we're in.
03:38And a lot of this was powered by the by the intrinsic knowledge in models and of course their ability to reason.
03:47Now, in the context of Microsoft, we want to make available all these models and make it easy for you to integrate them into the agents you're building.
03:54We do this from our agent platform that starts in GitHub where we all go and build.
04:00It has a contextualization system so you can ground your agents.
04:03And when it comes to agent hosting, observability, and management, we all do all of these in Foundry.
04:10Microsoft Foundry is also where we offer thousands of models in our model catalog so you can pick whatever is the right model for the right task.
04:19And we keep adding more every day.
04:21In fact, just yesterday we announced that Claude in Microsoft Foundry is generally available so you can use all the capabilities of Claude in the context of the unified experience in Foundry.
04:34So you get best of both worlds.
04:38Now, an interesting model got us here, but it only gets you so far if you're building a system that or an agent that needs to participate in what's happening in an organization
04:48or a company.
04:49And you know, as an industry we realized this early and we, you know, we saw the the rag pattern emerge.
04:56That started as a pretty low-tech technique, but quickly evolved and what we do today with context engineering and you know, it became a pretty sophisticated system for connecting agents and the knowledge they need to get their job done.
05:12Of the many dimensions of of which this got kind of complicated, I'm going to pick on two.
05:17One is kind of the evolution from simple and isolated data sets to whole company-wide grounding.
05:25And the other one is how we started with simple vector search and whatnot and we really saw this evolve into fairly complicated retrieval systems.
05:33So let's start with company grounding.
05:36Like at Microsoft, you know, spending time with customers, one of the things we saw early was that whenever you build an agent, you you always have the knowledge you care about for that agent and you'll manage that yourself,
05:48but you also need to ground the agent often on the kind of ambient data of your organization, you know, whenever the agent leaves.
05:54This includes maybe your documents, your emails, your chat threads, or the information in your data warehouse and whatnot.
06:03So, we built Microsoft IQ as a way to give you a single entry point into all these kind of ambient data that agents need to get the job done in addition to the specific information
06:14that you build into the agent.
06:16Microsoft IQ is not one feature, it's more like a set of capabilities that goes from work IQ that connects your agents to all the documents in say SharePoint, all the emails, calendar,
06:26your chats, and the connections between people to fabric IQ that gives you access to all the all your analytics assets, you know, from data warehouses and data lakes to Power BI reports,
06:37and foundry IQ, which is what you use for your all agents where you can push your own data and then use it for grounding.
06:43And of course, sometimes you have your agents need to go out to the web to ground on data maybe not yours, it's public information, but but you need to use it to complete
06:55the picture of what the agent world's view is, and for that we have web IQ.
07:01Now, this first part allows agents to ground on the kind of these ambient data.
07:06Now, the second dimension I mentioned before is the evolution of the actual retrieval systems.
07:12You know, when drive first emerged, I think, you know, what we saw is like an initial adoption for vector databases that really unblocked us from getting a lot of these systems off the ground, and that was great.
07:25Um, I think, you know, for a hot second as an industry, we thought that if we could get really, really good at computing cosine similarity between vectors, we were all set for retrieval.
07:35It turns out, you know, things never are are never that easy.
07:38Uh, so, you know, what evaluations show over and over again is how, you know, if you combine methods, you just get better results.
07:46Like in this case, this is an evaluation from actually I search, the search technology behind Foundry IQ.
07:52And you can see how individual methods don't do as well as combined methods, particularly when you apply them to real-world customer scenarios.
08:01Now, the trick is how you build a platform that allows you to combine all these building blocks without putting the complexity right in front of you.
08:08You like let you opt into it when you need control, but when you have a scenario that is clear, then you can have an easy system.
08:15So, in Foundry IQ, that was one of our core design goals.
08:19And the way we do this is we actually layer the system.
08:22So, you can start at the top, you can go to Foundry and say, "Hey, I have a bunch of, I don't know, PDFs or pictures over there.
08:28Just deal with them."
08:29And then we'll do everything under the covers, do like, I don't know, chunking, vectorization, deal with relevance and ranking, deal with agentic retrieval and whatnot.
08:37Now, if you're an expert and you want control, you can also do that.
08:41You can go to the bottom of the stack, you want to build vector indexes and tell us how to quantize the vectors or control lexical retrieval and whatnot.
08:48You can do all of that and you can do it in the same stack, which means you can go up and down as you as your needs change.
08:56Now, on on top of the core retrieval system, we also introduced an agentic retrieval stack because we see that for easy cases, like, you know, quick single-shot retrieval is great, but for more sophisticated cases, you do want
09:10a system that can reflect on on what's in the data set and decide whether or not we've satisfied the information need as stated in the input before we come back with results.
09:24Of course, we see a lot of patterns like this emerge and they always the question is, is this actually useful?
09:29Like, are the results better?
09:31Our experience in our own evaluations is for for difficult cases, agentic retrieval can a difference.
09:38You across the many metrics that we we track, you know, things like um the actual actual evidence recall or answer completeness, we see like the agentic retrieval approach continuously does better than than simple that's
09:52uh individual simple parts.
09:54Now, let me show you some of these in action if you can go to laptop.
10:06Can we switch to the laptop?
10:12There you go.
10:14So, here I'm in I'm in Foundry, and uh Foundry is where you you manage your agents, manage models, but also the place where you can manage all the knowledge that you give your agents in order to
10:26uh do their jobs.
10:27Um when here, you can you can create knowledge bases as the kind of the entry point of energy agents into the knowledge you care about.
10:34In this case, I'll create a knowledge base.
10:36I have a data set about movies.
10:38These are agentic retrieval systems, so I'll give it a model to power the retrieval workflow.
10:43And I can say how much effort you want the model to uh to make or the system to make.
10:48And this is effectively a trade-off between latency and quality.
10:51I can configure a number of other things, but critically, I want to say where the data I want to ground is coming from.
10:57And uh I can start from scratch, or in this case, I have a bunch of unstructured data like PDFs and whatnot in in blob storage.
11:05I have structured, you know, parquet tables with statistics, and I also want to ground on the web.
11:10So, if I take these three steps, and then I save this knowledge base, now I have this asset, this knowledge base that I can connect to an uh Foundry agent right here, and it'll take a second.
11:20Uh but also it's a standalone asset that if I have already a harness that I'm using in other in other places, every knowledge base is an MCP server, so you can just connect to it uh without having to write any glue code in the middle.
11:32Now, a knowledge base like this has uh you know, has a bunch of parts.
11:37Some of them, like uh for example, this uh storage uh content, you usually build indexes uh and you you know, vectorize these things and whatnot.
11:45Uh and if you want control over that, like if I if you don't, you can just use it here.
11:49But, if you do, let me just switch to Azure and show you the service behind that particular instance.
11:55Where if I go to knowledge bases, this is the knowledge base we just created a second ago.
11:59And I can go peek inside.
12:01For example, I can go fish out the indexes that back this particular uh piece of content.
12:06And in that index, I can see what is the structure of the index.
12:09Uh if I'm opinionated about, I don't know, maybe the quantization uh approach I want to use or which indexing algorithm I want for my vectors, I can say all of that.
12:18And of course, I can actually go and explore the data.
12:21And you know, see what's inside, how chunks were organized, and and whatnot.
12:26So, the goal of this is to again give you high product a highly productive environment when you need uh when you don't need uh the sophistication, and when you need it to make sure you have it
12:36to get your job done.
12:37We go back to slides.
12:45And of course, the other aspect of this is, you know, top of mind these days for all of us is token uh is token efficiency.
12:51And uh so, uh we carefully evaluate this system to make sure that we give you the most information dense answer that has the fewest tokens uh so that you you know, the
13:02the your consumption of tokens has a high value when it comes to all retrieval tasks.
13:08The last category of knowledge I wanted to talk about is learned knowledge.
13:12Now, learned knowledge is the result of us doing the work we do as individuals and as organizations every day.
13:19And the the idea that we can actually observe the processes and get better at them by reflecting and improving every step of it is something that is really uh changed now that we have agents doing the work and we can go tune the agents automatically.
13:34Satya wrote about this recently and reflected on the fact that people and agents can really compound in in how they do the work and how they can create this learning loop
13:44that effectively captures what's unique about the company or or the organization you're working on and inputs that to work to differentiate the work that you do.
13:54Now in Foundry we wanted to offer like a material a materialized version of this that you can use today.
14:00So we built a component called the agent optimizer that effectively goes through this process and allows you to evaluate a baseline, generate candidates, and then you know, evaluate the new candidates and we have a strong result,
14:13then deploy that to production.
14:15Let me give you a kind of a quick flavor of what this looks like if we can switch back to the laptop.
14:25All right.
14:25So here I'm I mean VS Code, I have the Foundry toolkit installed.
14:30And I have a simple agent, it doesn't matter how you write your agent as long as you externalize configuration like you know, your instructions, tool definitions, skills, and whatnot.
14:40So once you have one of those, it takes two key steps to do this.
14:44So first oops.
14:50Um I can actually so usually you have an evaluation already, but if you don't, you can actually say eval generate and what we'll do is we'll look at what we know about the agent like traces and instructions and whatnot and we'll produce a task adherence focused evaluation for you.
15:06In this case I run this a little bit earlier.
15:08So just to give you a flavor of what this looks like, you you have a bunch of tasks and then you know, the questions and the criteria and whatnot.
15:17Once you have a dataset you can evaluate then next step is you can say uh optimize.
15:24And uh I could just run optimize on its own and that will run In this case, this run for maybe 45 minutes or so and you get an optimized version by effectively hill climbing the metric that's established from by evaluation.
15:36Um so I run this earlier and so let me show you the output for this particular one.
15:42Where you can see that, you know, we established the baseline first and then we kept iterating on candidates uh using different combinations using a J power style kind of loop uh and uh looking for options that perform better given
15:54uh the rubric that we have.
15:55And uh the interesting thing is that once you found one that is that is better, then you can simply just say optimize apply and what this does is since you externalize the configuration,
16:05it allows you to swap one configuration for the other.
16:07Um if I if if we look here, you can see that for example, I have baseline and the one we just applied.
16:13And just to pick on instructions, these are just the uh trivial instructions for this example agent.
16:18But if I look at the optimized one, then you can see like a bunch of instructions that are not handwritten but that that they emerged out of the hill climbing process to get to make this particular
16:29um agent better given what we have in terms of instructions and skills and tools, but also based on reflecting on the actual uh traces from the agent as users are using it.
16:41So this is a real learning loop materialized in practice.
16:45You can go back to slides.
16:48So this was like a very quick overview about how do we think about knowledge in the context of AI and how do how we think we can enable these learning loops that will capture,
16:59you know, these differentiated capability that lives in each one of the companies and organizations we work on.
17:05If you want to try anything of what I talked about or showed today, you can head to ai.azure.com and get going.
17:11And with that, thank you all for listening this morning.
17:13I hope you have a great rest of the event.
17:15Thanks.