LLM Knowledge Bases: a practical guide — Ben Holmes, Warp

AI Engineer · 21 min · 331 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12All right.
00:13Are we ready to go?
00:15Okay, we're ready to go.
00:17Are y'all ready to go?
00:19We ready?
00:20All right.
00:21Let's do this thing.
00:22Uh, all right.
00:23Well, hello everyone.
00:24I'm Ben Holmes, developer relations lead at Warp.
00:28Uh, you might have heard of Warp as a terminal that's really nice to use for you and your coding agents.
00:33You may have also heard of it as a cloud platform that helps you build out software factories.
00:38If you haven't, we have a booth in the expo.
00:40You can go talk to us about it.
00:42Uh, but today I'm actually going to be talking about something called LLM knowledge bases.
00:47Uh, maybe a show of hands.
00:48How many people have an Apple Notes folder that's a complete disorganized mess that you don't know what to do with?
00:55Okay, a lot of people.
00:56Uh, and how many of y'all have used Obsidian before?
01:00Maybe.
01:00And how many of you use it now regularly?
01:04It's a little less people.
01:05That's all right.
01:06Uh, I'm actually not going to be showing youall too much on there, but if you're an Obsidian head, this talk is definitely going to be for you.
01:13Uh, so we're going to be talking about the ways that you can organize all of your sort of raw thoughts and notes that you're taking both between you and your LLMs
01:21into something that you can actually navigate and make sense of and even remember and discover ideas you might have forgotten about.
01:28Um, so our goal is to go from head over to this guy, uh, jumbled mess of scattered thoughts like this all the way to something that's nice, organized, and viewable like this.
01:41But I'm getting ahead of myself.
01:43Uh, our goal is to initially start with just a raw folder of all of your thoughts.
01:49We're going to be tracking our notes as markdown for this example.
01:53And I'll be showing you the app that I actually built myself called Hubble that'll let you navigate and write notes easily inside of there.
02:00Uh, but you can use any Markdown viewer, Warp, Obsidian, whatever you like.
02:05Uh then we'll talk about how you might turn that raw set of thoughts into something that's a bit more organized and interconnected.
02:11So enriching notes to connect related backlinks and tagging things based on categories so that you can find them or so agents can find them.
02:19Then we'll go a little bit further into wikis actually generating uh something that you can click around around areas of research that you have personal interest etc. uh that
02:30qualify the people concepts and sources that all of your notes contain.
02:34Then finally, we'll talk a bit about visualizations like that fancy graph view I was showing you.
02:38It's very easy to do these days now that agents can access your machine and use HTML.
02:43Um, so the first thing I wanted to talk about is how you get raw thoughts down actually.
02:48Uh, because it is pretty easy now to whip out Apple Notes and write something.
02:53Uh, but I would argue it should be even easier uh to get everything down.
02:58So, something that I'm going to show you here, uh, maybe another show of hands.
03:01How many people use Whisper Flow or something like that?
03:04Okay, that's not enough hands.
03:06It's honestly not enough hands.
03:08Uh, because you may not be aware, but voice dictation, even though it is pretty awkward to talk into your computer with a bunch of co-workers around, it is the fastest way to get your thoughts onto paper.
03:19It's going to be like 200 words per minute, I believe, is the average.
03:22Unless you're an absolute Olympic typist, it will be faster than any other method that you have.
03:27So, I suggest getting some sort of voice input tool uh just to get down context.
03:32There are a number of these.
03:34And by the way, for the people who aren't paying for Whisper Flow subscriptions, you're in luck.
03:38You don't have to pay anymore for this kind of stuff.
03:40There are local models that can do this.
03:42So, a bit of a side tangent on tool chains that you should probably check out.
03:46Uh I love highlighting this one.
03:48It's called Handy.
03:49Uh it's an open- source tool for voice dictation that just uses a local model, stays on device, very easy to use.
03:55Uh there's another one called Voice Inc. which is very similar.
03:59Uh this one I believe is like a $20 lifetime fee just to get updates in the app.
04:03Might as well do it just to support the developer.
04:05That's the one that I use.
04:07And it gives you a hot key uh as well as a mobile app.
04:10So if you wanted to get something down, you can just hold either the option key or the function key on your computer and it'll start writing down everything that you're saying and it will turn it into a formatted
04:21um set of sentences, paragraphs, etc. on the other side.
04:25Cut off a little at the end there, but you can see the punctuations are coming in.
04:28Great.
04:29And this whole uh set here with all these paragraph breaks, this was just one long transcription I wrote down after listening to a podcast.
04:37This was uh just me listening to the Acquired podcast about the founding of Walt Disney.
04:42Uh little bit offbeat for me.
04:44Uh but I thought it was an interesting read just about uh how these sorts of businesses are founded.
04:50Uh so I decided to take out a voice dictation app and write it down.
04:53And that's what I've been doing for all of my notes at this point.
04:56Either come to my computer or on my phone, I just kind of type into it to get all the raw data.
05:00And the reason I'm suggesting you do this is if you want to get to a point where you can actually have LMS generate wiks, visualizations, etc., you need a lot of raw data.
05:11You need a lot of raw materials.
05:13So don't worry if you're being a little bit scrappy, a little bit rambly.
05:16You're not formatting things with perfect bullet points.
05:19That's fine.
05:20The goal should just be get down as many thoughts in the moment as possible.
05:24Maybe it's a meeting transcript, maybe it's research notes after you've read an important passage.
05:29Whatever it is, just being able to get your idea down on the paper uh makes it very effective for LMS to go access it later.
05:36Um, so now let's talk a little bit about turning those raw materials into something more useful.
05:40Uh, this is another note that I took about the founding of Ferrari.
05:43I went down a little rabbit hole when I started listening to this guy.
05:46Uh, and this is what you can get to if you just ask an LLM to do a little bit of recon for you.
05:53So, you'll notice that in the center is the same note, same sort of style of like a voice dictation transcript, but we've got some more details now.
06:01We can see the tags that are for the topic.
06:04So, we can see this was a podcast.
06:06It's related to these focus areas.
06:08You can also see the source.
06:09It was able to pull down the URL and find it.
06:11It also wrote down when it was enriched.
06:14We'll talk about why I do that in a second.
06:16And at the bottom, it also finds some back links of those.
06:19This is related to that other podcast you were listening to.
06:21Let's go ahead and link them together.
06:23And that web gets tighter and tighter the more things that you get down.
06:26Uh, and the way that you can get from this to something like this is through an agent skill as you might expect.
06:34So, I have a skill for that right over here.
06:37And it's just called enrich note.
06:39Enrich note.
06:40So right here it's just showing how you might instruct an agent to take one of those note files and turn into something that's more useful.
06:47So here it's just saying let's go ahead and enrich that note.
06:50Put a little time stamp on there so if we ask the agent to do another pass it remembers that some other agent did it in the past.
06:57Uh have it generate some tags.
06:59I actually put all of my tags into this little uh reference folder over here.
07:04That way the agent isn't inventing new tags every time it goes through and tries to add more detail.
07:09This gives it a concrete list of things to look through.
07:13I actually instruct the agent to be reluctant to add new tags because Claude loves to get creative.
07:19Uh so just telling it please don't do that.
07:21Uh use the instructions, but if you really find a pattern, go ahead and add something to this list.
07:25You can go ahead and do that.
07:27Uh also telling it to go research the source with web tools.
07:30At this point, all the agents have web search.
07:32If not, exa is a great little tool that you can add into your harness to go find that for you.
07:37And the last one's those related notes.
07:39to find back links, use some file calls, find related notes using key term search, and put them into the bottom.
07:45So then we have that interconnected web.
07:48Uh, and to run this skill, you can of course do it manually with whatever harness.
07:52Uh, I'm going to be using warp just to show you here because totally unbiased.
07:57I'm a really big fan of warp.
07:58Can you believe it?
07:59Uh, but it also does give you access to any model including OpenWeight.
08:03So if you've ever been wanting to experiment with like GLM 5.2, too, just very good at this sort of task.
08:08You certainly can.
08:09In fact, I'll let's go ahead and flip over to that just to try it.
08:13And I'll go ahead and call this skill by hopping into our little note here.
08:19I might go ahead and call enrich note.
08:21And then I'll pull in this note from earlier.
08:24Grab that file path and just tell it to go ahead and add more detail to this guy.
08:29And I can show you a completed run of this, too.
08:32Uh right over here we can see uh this was well we'll talk about this in a moment but asking it to do it on a sort of automation across all of your notes.
08:41Uh it go ahead and it will go ahead and find any notes that weren't enriched in the past.
08:47So remember that at the top here we gave it a little timestamp every time it gets enriched.
08:51So the next time we ask an agent to do it, it was look for anything that's not tagged yet.
08:55It'll go through run that whole flow across all those notes.
08:58It'll make some modifications to the files and then on the other side you get a more organized notebook than you had before.
09:04It's very easy to do.
09:06We'll let this guy cook in the background while I talk a little bit about wikis.
09:11So the next step after we've gone ahead and combined everything into related ideas.
09:16You might be fine with this actually.
09:18I'll show you how useful this is just on its own.
09:20So this is like a total uh mind dump of everything in here.
09:24Like this was me reading The Left Hand of Darkness, which is a very good book, by the way.
09:28And you can see I was just taking spare notes across different chapters whenever I found something interesting.
09:33And it was able to combine all those related notes, so I can just click through and find them.
09:37Uh so I'm not hunting around where did I save in my Apple Notes the last time I wrote about this like two weeks ago.
09:42It's able to find it by related um topics.
09:45So if you're someone that likes following Wikipedia rabbit holes, it kind of invents a Wikipedia rabbit hole of your own thoughts.
09:52It's kind of fun.
09:53Uh, but if we want to make this a little bit more browsable, we might tell it to generate a wiki on top of this.
09:59So, take that set of raw ingredients and cook it together into something a little bit more u hardy.
10:05I don't know, it's a weird metaphor.
10:07Uh, let's go ahead into these wikis.
10:09So, these are a couple that I generated around topics that I'm interested in.
10:13Uh, one of them is probably predictable.
10:15You know, latest in AI news.
10:17anytime I listen to podcast or uh read a research document about the like system cards of the latest models for example uh I want to have a wiki that actually pulls all of that in and groups them by related topics.
10:31So this wiki that you're seeing right here uh if we click into it'll give you a set of people, places, and things basically.
10:41So we get all of our sources.
10:43So these are some of the topics that we're most interested in.
10:47Uh we can see a few in here, some of them expected, some of them not.
10:50We see like a Ralph loop rabbit hole for example.
10:53We some see a number of concepts that are pulled out of all of these ideas.
10:57Uh and then down here we see some people all the people you might expect some maybe not like this is something that I pulled together from a intersection with AI and music.
11:05Adam Neely is actually a jazz musician that's been talking about how AI has affected the music industry.
11:11So I wrote that down.
11:12Found it was a little bit interesting.
11:13Pulled it into our latest AI wiki so that we can find it easily.
11:17Also organizations.
11:19You would expect all of these I assume.
11:21This one more music related.
11:22Uh this one the thing you should go try after this talk etc. Uh and all of that is generated programmatically.
11:30I didn't write any of this because all I have time to do is generate the raw ingredients not connecting it all together myself.
11:37Uh so to have to have an agent generate this this is actually uh gist from Andre Carpathy.
11:43This is where the LLM knowledgebased idea kind of came together.
11:45You can find this really easily now if you just search his name and then wiki.
11:49It's kind of uh uh got a lot of usage.
11:53So this is the simple principle of it.
11:56We want to take a raw directory which is where we're taking all of our spare Apple notes from earlier and we want to combine it into whatever focus area that we care about.
12:06So, we grab the raw sources, we create a wiki, and uh really you can tweak this to taste as much as you want.
12:13He even calls this an idea.
12:14So, what I did to generate that latest AI wiki was just describing these are the things that I care about.
12:20They're in this raw folder.
12:22Here's the rest of his gist.
12:23Go ahead and generate a wiki for me.
12:25And on the other side, you end up with something very organized like this.
12:29Uh I was able to do this for another topic, too, that's a little offbeat.
12:32Uh, this one was uh because I was listening to the Bible in a year podcast.
12:37This is something I wanted to study recently, totally outside of my wheelhouse with a number of characters that I forget the name of.
12:43If anyone's read religious texts, you know that all the names sound very similar and can be hard to pronounce.
12:48Uh, so having something that can pull together all the people with references to what each of them did is very useful.
12:55So I can click through here and it generated sort of an entry for who that character is.
13:00You can imagine this is also useful in the workspace.
13:02If you take a lot of meeting notes and you want to have a people section of all the people they met with, interested clients if you're in customer success,
13:09it can generate all of those for you.
13:11And then it can create back links over to any related meetings you've had with them.
13:15Maybe the source links.
13:17So going back to our raw notes, if you want to go to the raw materials, you could click through and find those.
13:23Uh really all we're trying to do is make sense of all these notes that we take every day and make us something a little bit more browsable, a little bit more readable.
13:31Um so the next thing I want to talk about beyond that is uh it's really nice that you can ask an agent to generate these on demand, but as we can see with our little agent running in the background, it can take time
13:42uh for an agent to go generate these things.
13:45So ideally we have a flow that will sort of do this in the background for us.
13:51maybe while we sleep, maybe on an automated schedule or something like that.
13:55Uh so that's kind of the next thing to talk about, which is taking these skills that enrich the notes, generate wikis, update them for you over time, and turn them into something that runs on a cadence,
14:07maybe daily, maybe weekly.
14:09U and there's a little platform over here.
14:11There's a number of ones that do it.
14:12By the way, if you've played with like Codex app, they have an automations tool that spins up tasks to run on your machine every day, but it means your laptop has to be cracked open when it runs because it's a local automation.
14:23If you want something that runs in the cloud, uh I will shout out that we do have something called oz.dev.
14:29If you go search for that in your web browser, it'll help you set up with automated schedules.
14:35Uh as well as like other triggers like maybe a Slack message, iMessage, whatever you want to set up.
14:40Uh these I set up on schedules.
14:42So this was one that I set up with the uh wiki skill that I showed you a little bit earlier and I encoded it into this set of instructions right here.
14:52Uh so the way this guy works and I'll just go ahead and whiteboard this for you live and show you how it works is uh we create an automation
14:59that takes our folder of markdown syncs it into a box maybe like a cloud sandbox powered by Docker if you need it uh otherwise you don't and then sync it back up when the agent's done.
15:10Uh so the way that I've set it up is using uh the Obsidian headless CLI.
15:15So Obsidian has a really nice tool that lets you take a bunch of markdown on your computer and sync it somewhere else and then pull it back down.
15:22You could also just do like a git clone too if you want to be a little bit less creative, put all of your notes in a GitHub folder so that way a cloud agent could pull it down and do it for you.
15:31I prefer using Obsidian CLI for this just because it avoids having to like push and pull your notes.
15:38No one has time for that.
15:39I just want it to sync in the background.
15:40That's what this does.
15:42Uh so it's a tool that you can install into a cloud sandbox.
15:46It'll sync down that folder of markdown.
15:48Uh you can run a process inside of it.
15:51Maybe you have like your enriched note skill that I was showing you earlier.
15:54Just instruct the agent run enriched note uh across end notes that are uh not enriched yet and then let it do all of its tool calls and code diffs and then by the time it's done
16:06it'll sync it back up.
16:07That's really the flow.
16:08It's super simple.
16:10Uh, and that's what we've instructed this cloud runner to do.
16:13So, if you're unfamiliar with Oz, no problem.
16:15Uh, all this is is a cloud sandbox that'll stand up every day and it'll instruct the agent to do this.
16:21So, this is the prompt and the prompt is I gave you the Obsidian CLI in your environment.
16:26Use that, pull down that wiki, go ahead and update it, and I put in some special instructions for my own setup, and then sync it back up.
16:34So that way when I come back to my computer in the morning, I wake up to a perfectly fresh wiki that I can review.
16:40It's like the daily paper, but it's your own, which is so exciting.
16:44Uh, we see another schedule that set up over here for the enrich note phase, generating those back links, making it easier for you to navigate around everything.
16:51I made that a skill.
16:52Again, we tell it to sync everything down.
16:55We tell it to run in rich note, and then we push it back up.
16:58So by the time I go on my computer, there everything is.
17:01Uh, and I can show you one of these runs.
17:03You can actually view it in your browser.
17:05Uh so over here, this is where it was running in rich node across all of our nodes from earlier.
17:10And we can see that same thing I was showing you, but this was from the cloud.
17:14So while I was sleeping, it looked at all the notes that I took throughout the day.
17:17And it tied them all together in something that was easy to browse.
17:21Uh which is super clean, super easy to use.
17:24Uh we can also take a look back if we want to at that run from earlier just to see if it did its job.
17:30Yeah, look at that.
17:31Um, so this was that note earlier that was just a jumbled mess.
17:33Now we're seeing here's the podcast.
17:35I did some web search.
17:36I found it.
17:36So if you want to go listen to it again, there it is.
17:39We have a heading now.
17:40And we have some related notes.
17:41The Ferrari um podcast.
17:43We have some notes from some other founder podcast.
17:45There we go.
17:46Don't have to do any work anymore.
17:47All you have to do is voice dictate in your phone.
17:49And all this stuff comes out on the other side.
17:52The last thing I want to talk about uh just to expand your mind a bit beyond the markdown doing something a little bit more exciting is turning all of your thoughts into visualizations,
18:03graph views, charts, whatever you want to do.
18:07Uh and that's something that I've set up inside of this uh notes app I've been showing you this whole time.
18:11You can also look at HTML pages inside of here.
18:14So I told it because I wanted to like, hey, I want to take all these markdown files and I want you to just build with HTML and Tailwind some sort of graph view.
18:23So instead of me clicking around a wiki and following a bunch of links, I want to see a bird's eye view of everything that we're writing down here.
18:30Figure out common patterns and common areas of interest.
18:33And we can look at them right here.
18:35And this is not a tool that you have to install.
18:37By the way, I told an agent, build this for me.
18:39So you can do that now.
18:41Um, so we see some stuff in the center that's very stray thoughts.
18:44I'm a scattered person.
18:45It makes sense.
18:46Uh, we have some books.
18:47Clearly, I need to read more.
18:49We have some interest in like startup founding here.
18:52We have AI and engineering.
18:54Of course, we have faith in scripture because I was reading Bible in a year.
18:56As I mentioned earlier, we see all of that grouped up inside of here.
19:00And I also told it, let's make these nodes something clickable.
19:03So now I can see whatever note that was.
19:06I can see how it's connected to something else.
19:08So we can see this is connected to a number of ideas.
19:10You just kind of walk around the graph.
19:12It's useful just to get an idea of what you're actually interested in and where you have gaps in your thinking, but it's also useful if you want to drill down.
19:20And also just to show you this is something that you control.
19:23It's like can can you put it in space?
19:25I don't know.
19:26Let's put in space.
19:27How about that?
19:27Now it's in space.
19:28Uh so it looks a little bit more like a star constellation.
19:31So we can click in there.
19:32We can sort of look around.
19:34See how everything is connected.
19:36It's really nice.
19:37You could also connect into like a burndown chart.
19:40If you're trying to build up a habit tracker of how often you're writing notes on certain things, you can invent one of those.
19:45This is kind of like a GitHub chart.
19:47Clearly, I don't have a super consistent habit, but it's building and we can track it inside of here.
19:52Uh, whatever you want to put together, you can just ask an agent to do it.
19:57Um, and that's really that's really the end of the talk.
19:59That's everything I wanted to share with y'all.
20:01If I have any sort of parting gifts or things to check out, I'll first call out like if you want that automation platform oz.dev.
20:08Uh it's built by warp, warp.dev/os, however you want to look it up.
20:12Uh this is the tool that you can use to set up schedules to update notes in the background.
20:17And if you want to try that little notes app I was showing you, hub.md, it's free and open source.
20:22We're open to contributions.
20:23There's a lot of things missing as you can imagine.
20:26Uh so if you're looking for an Apple notes that's agent accessible, here you go.
20:30This is what you can go try out.
20:31I hope to see you in the issue logs.
20:34Uh, with that, it's been great chatting with y'all.
20:37I will u Oh, I'm supposed to plug myself, aren't I?
20:40Let me plug myself.
20:41There we go.
20:41If you need to follow me, there I am.
20:44I'm a developer relations at Warp.
20:46You can find the whole team uh both at the booth and also on Twitter and everywhere else.
20:50Uh, and also our side event which we have tonight.
20:53Um, so yeah, with that, I'll let y'all go.
20:55Hope you enjoy the rest of the conference.
20:56Thank you so much.
20:58[applause]