WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=ZRM_TfEZcIo

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/ZRM_TfEZcIo.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:00.000 --> 00:00:04.320
I spent 18 months turning my second brain into my living research memory.

s2
00:00:04.320 --> 00:00:05.720
Let me explain.

s3
00:00:05.720 --> 00:00:15.080
So, within my second brain, I currently have over 5,000 notes in Obsidian and another 5,000 notes in Readwise and some scattered in Notion and Google Drive.

s4
00:00:15.080 --> 00:00:19.680
And all of this is growing on every week 250 files per month.

s5
00:00:19.680 --> 00:00:20.880
And this is what I want.

s6
00:00:20.880 --> 00:00:24.320
On the left, you can see my whole Obsidian vault, this huge mass.

s7
00:00:24.320 --> 00:00:35.560
And whenever I start working on something such as an article, a new project, a new code base, a new feature, or whatever, I want to actually pull high-signal notes that are actually useful

s8
00:00:35.560 --> 00:00:37.840
for my current work.

s9
00:00:37.840 --> 00:00:42.120
And you would ask yourself, why not use directly Codex Cloud or Notebook LM?

s10
00:00:42.120 --> 00:00:43.760
And I think it's that I am.

s11
00:00:43.760 --> 00:00:49.200
But you need a system that sits between those harnesses and your second brain.

s12
00:00:49.200 --> 00:00:54.760
Okay, so let's go back to the root of my problem, which is that I'm always losing my research.

s13
00:00:54.760 --> 00:00:57.240
For example, my reading list is a graveyard.

s14
00:00:57.240 --> 00:01:05.040
When I'm scrolling social media and I save that cool X post, a new article, a new new YouTube video, a GitHub repository, it doesn't matter.

s15
00:01:05.040 --> 00:01:17.080
Whenever I actually want to start working on something, I never recall what I have in my second brain or I have to spend a ton of time actually finding meaningful notes that I can use

s16
00:01:17.080 --> 00:01:18.520
in my work, right?

s17
00:01:18.520 --> 00:01:27.960
And another problem that I have is that I want the system to actually be anchored into my personal notes, into my personal values, into my personal faith.

s18
00:01:27.960 --> 00:01:32.600
I want the system to be personal to reflect my own thoughts, right?

s19
00:01:32.600 --> 00:01:38.800
And that's why in today's video, Louis François and I will teach you how to build your own AI research OS.

s20
00:01:38.800 --> 00:01:43.480
This also comes with code, so you can also try it out yourself.

s21
00:01:43.480 --> 00:01:45.480
And I'm Paul Yushin.

s22
00:01:45.480 --> 00:01:51.560
I'm the founder and CEO of Decoding AI, where I do a ton of content on courses on how to ship AI products.

s23
00:01:51.560 --> 00:01:55.240
And I'm also the co-author of the LM Engineers Handbook bestseller.

s24
00:01:55.240 --> 00:02:01.760
And the system, the AI research OS that I will teach you in this video is the system that I use in my daily work.

s25
00:02:01.760 --> 00:02:05.600
And now I will pass the torch to Luis François.

s26
00:02:05.600 --> 00:02:06.520
Thanks, Paul.

s27
00:02:06.520 --> 00:02:08.320
So, I'm Luis François Bouchard.

s28
00:02:08.320 --> 00:02:12.880
I'm the co-founder and CTO of Towards AI, where we build educational courses.

s29
00:02:12.880 --> 00:02:19.000
And I'm also the creator of What's AI, a YouTube channel where I explain AI engineering techniques.

s30
00:02:19.000 --> 00:02:21.880
I used to explain AI research before.

s31
00:02:21.880 --> 00:02:24.280
Now, I'm focusing on AI engineering.

s32
00:02:24.280 --> 00:02:28.120
I'm also the author of the book Building AI systems for production.

s33
00:02:28.120 --> 00:02:31.080
And before that, I was a PhD student.

s34
00:02:31.080 --> 00:02:34.760
So, I honestly make research for a living.

s35
00:02:34.760 --> 00:02:40.080
I used to do a PhD, as I said, in AI and doing tons of research and research work.

s36
00:02:40.080 --> 00:02:46.680
Now, I build courses, I write videos, I research for videos, I build trainings for companies for a living.

s37
00:02:46.680 --> 00:02:52.680
And all of these things that I do start with a very good research.

s38
00:02:52.680 --> 00:03:01.120
And also leveraging tons of knowledge and insights that we get at Towards AI from building for clients.

s39
00:03:01.120 --> 00:03:04.080
So, I have tons of notes as well, just like Paul.

s40
00:03:04.080 --> 00:03:07.640
And we try to leverage them the best possible.

s41
00:03:07.640 --> 00:03:17.760
And as you'll see, we'll build some sort of tool to leverage our second brain, where as you'll see, there will be some differences between how I use it and how Paul uses it.

s42
00:03:17.760 --> 00:03:24.400
And that's the core goal of the repository that we built and on this project is that we want you to adapt it for your needs.

s43
00:03:24.400 --> 00:03:31.400
The whole goal is how can we make research better, but more specifically, how can we better leverage what we have?

s44
00:03:31.400 --> 00:03:33.080
So, let's dive into it.

s45
00:03:33.080 --> 00:03:42.959
And first, we need to figure out which tool to use and when, because this whole research system that we built is not for every query.

s46
00:03:42.959 --> 00:03:52.560
If you just need a fast answer, like a few quick questions or just something where that that you would just Google, basically.

s47
00:03:52.560 --> 00:03:57.640
Well, obviously, just Google it or ask ChatGPT, Cloud, whichever system you want.

s48
00:03:57.640 --> 00:04:07.520
But, the problem when doing that is that if you have a lot of following up question or it's a bigger project that you need to build on and have basically

s49
00:04:07.520 --> 00:04:12.640
a very long context or tons of information to share, relying on ChatGPT isn't ideal.

s50
00:04:12.640 --> 00:04:20.680
And it also means that you are fully dependent on the architecture that OpenAI or ChatGPT's team built.

s51
00:04:20.680 --> 00:04:31.919
So, the next step here is to ask yourself for a more complex problem, do you need to act quickly or do you want to build some next feature and and do something very difficult?

s52
00:04:31.919 --> 00:04:45.360
If you just have a small repo for a quick change or write one article, just do one thing that you know won't be repeatable that much, definitely use Codex or Cloud Code or some agent that you trust.

s53
00:04:45.360 --> 00:04:50.960
Sometimes, you need to keep on digging to make it better, to improve efficiency, optimize it more.

s54
00:04:50.960 --> 00:05:00.600
And so, typically, when you have to do that, you want your research sources, your research to stick and to be able to refer to them in the future.

s55
00:05:00.600 --> 00:05:11.440
So, if you want a process like this where the sources that you find, the notes that you take stick around in time and have an agent be able to leverage that efficiently

s56
00:05:11.440 --> 00:05:18.080
and being able to come back to these information, to ask follow-up questions, to digest content even more.

s57
00:05:18.080 --> 00:05:30.800
And right now, for instance, when I make a new video, I want also the agent and the system to understand the previous videos I made to not duplicate content, to not repeat myself, and to refer to some other content.

s58
00:05:30.800 --> 00:05:42.080
In this case, there are some tools that are very interesting that you might have tried before, like NotebookLM, that is super powerful to do research, to digest content efficiently, and to come back to it.

s59
00:05:42.080 --> 00:05:46.840
But, the problem with NotebookLM is that is Well, first, the main problem is that you don't own it.

s60
00:05:46.840 --> 00:05:48.440
You cannot do anything you want with it.

s61
00:05:48.440 --> 00:05:51.360
You cannot personalize as much as possible.

s62
00:05:51.360 --> 00:05:53.520
It's not agent native.

s63
00:05:53.520 --> 00:05:58.920
And it's obviously weak for coding tasks since it's just browser based.

s64
00:05:58.920 --> 00:06:07.000
So, it's far from ideal from something that Paul and I needed and that most AI engineers need in general.

s65
00:06:07.000 --> 00:06:22.240
So, if you need your agents to be able to leverage all you do, uh whether it is a big research, a new video, whatever you write, you do, you code, you typically want your other agents, your other projects to be able to leverage what you learn from what you just did.

s66
00:06:22.240 --> 00:06:32.640
And one thing that we advise especially for production, obviously for product, is to build some sort of retrieval rag pipeline with vector databases.

s67
00:06:32.640 --> 00:06:35.200
But, this needs an infrastructure.

s68
00:06:35.200 --> 00:06:40.840
It's not really human-friendly to be able to digest quickly, to check notes, to make edits.

s69
00:06:40.840 --> 00:06:42.320
It's hard to inspect by hand.

s70
00:06:42.320 --> 00:06:44.200
You need to build everything around it.

s71
00:06:44.200 --> 00:06:49.080
It's definitely far from ideal for just something I want to use on a daily basis.

s72
00:06:49.080 --> 00:06:54.400
Obviously, it's super powerful at scale, very interesting especially in a product.

s73
00:06:54.400 --> 00:06:57.560
But, as I said, this project is for us.

s74
00:06:57.560 --> 00:07:01.360
And I don't want something live, super professional as a product.

s75
00:07:01.360 --> 00:07:08.080
I just want something I will use and that my agents and different projects can leverage as best as possible.

s76
00:07:08.080 --> 00:07:25.240
So, the last question to ask ourselves here is that if you want everything there but more personalization, so a personalized research assistant that builds some sort of Wikipedia that compounds over time and and is easily

s77
00:07:25.240 --> 00:07:40.400
inspectable and usable where you have tons of sources, documents, videos, comparisons, implementations, new research, new topics that you keep on adding and that you keep on wanting to leverage

s78
00:07:40.400 --> 00:07:44.680
and review easily, this is where you may want to build something yourself.

s79
00:07:44.680 --> 00:07:53.120
And in our case, we built a personalized research OS that we will share in this talk with exactly what we built and how.

s80
00:07:53.120 --> 00:07:59.360
But, the downside is that it definitely needs a bit more setup than just opening Cloud Code.

s81
00:07:59.360 --> 00:08:11.240
Right now, the main problem with using Cloud Code and other agentic tool is that you give Codex links, PDFs, and different information, for example, my most recent Loop Engineering

s82
00:08:11.240 --> 00:08:19.880
video, and then the next session you use Codex, you have to paste it all again or ask it to use skills.

s83
00:08:19.880 --> 00:08:34.479
And whatever structure that Codex or ChatGPT, whatever tool that you use, build on the fly to leverage what you did, the scripts it ran, the scripts it had, you all lost it or kept it inside a skill that you have to ask it to reuse,

s84
00:08:34.479 --> 00:08:39.080
and that usually isn't ideal and just grows and grows over time.

s85
00:08:39.080 --> 00:08:45.080
And the problem is that all this information that you give to the model is not the bottleneck.

s86
00:08:45.080 --> 00:08:48.240
The bottleneck is how can you leverage it in the future?

s87
00:08:48.240 --> 00:08:56.400
Meaning that with an agent, the context window becomes everything, the database, the file system, the memory, the reasoning space.

s88
00:08:56.400 --> 00:09:01.080
It has to do it all, and when you stop the conversation, it loses everything.

s89
00:09:01.080 --> 00:09:07.560
And the thing is that we don't need necessarily to provide more and more and more context for a better research.

s90
00:09:07.560 --> 00:09:16.400
You need a proper memory and context management, and ideally some personality with it, especially in my case when I do videos.

s91
00:09:16.400 --> 00:09:26.000
So, what we did is that we decided to build a system with plain files, mostly markdown files, that we can leverage easily and that agents can leverage easily.

s92
00:09:26.000 --> 00:09:30.440
I won't detail it very much here because Paul will talk about it in depth.

s93
00:09:30.440 --> 00:09:34.320
And as I said, Paul has like 5,000 or something notes.

s94
00:09:34.320 --> 00:09:40.800
I have just a few hundred, but that's just to say that we need to consider that we didn't start from nothing.

s95
00:09:40.800 --> 00:09:44.200
We already both had some sort of large database.

s96
00:09:44.200 --> 00:09:48.440
In my case, I made hundreds of videos and I take many notes.

s97
00:09:48.440 --> 00:10:04.400
So, I still need to leverage these years of content that I already made and tons of meetings that I have with my team, with clients when we build for them that I want to leverage as well because we learn a lot by building for people.

s98
00:10:04.400 --> 00:10:14.160
We have highlights from interesting tweets that I see, interesting articles that I see, and I want all my projects to be able to leverage my agent skills.

s99
00:10:14.160 --> 00:10:26.840
So, I decided to pivot and instead of having a folder for Cloud Code skills and having all my meeting recaps in Granola and having years of notes on Apple Notes and the tweets on the saved Chrome tab,

s100
00:10:26.840 --> 00:10:30.640
instead, I moved everything automatically into Obsidian.

s101
00:10:30.640 --> 00:10:34.000
It's just a note reader, obviously, so you don't have to use that.

s102
00:10:34.000 --> 00:10:49.120
You can just save it locally, but I used Codex to set up everything so that Granola is automatically saved there, my notes are now on Obsidian just because it's a nice UI, I like it, and I can use it from my phone, my computer, my Windows, Mac, everything.

s103
00:10:49.120 --> 00:11:01.839
So, anyways, I moved everything to Obsidian, which means that it's saved locally in my file system, which means it's basically my companion for researching and building everything I build now.

s104
00:11:01.839 --> 00:11:04.640
And what we built, obviously, leverages that.

s105
00:11:04.640 --> 00:11:18.560
We built a repo called AI Research OS for this workshop where it's basically just skills for Cloud Code and Codex with plugins to be able to do a very deep research about a topic

s106
00:11:18.560 --> 00:11:23.520
or a simpler search or distillation, different tools that you can use.

s107
00:11:23.520 --> 00:11:29.960
The most useful and complete one will be the research tool that I use, for example, when I kick off a new video topic.

s108
00:11:29.960 --> 00:11:39.600
And the goal of this repo is to have you implement it, install the cloud plugins from it, and tune it to your needs.

s109
00:11:39.600 --> 00:11:44.160
Right now, it can connect to, as I said, Obsidian with my local notes.

s110
00:11:44.160 --> 00:11:55.200
It can use Readwise, Notebook LM, your GitHub repos, any links that you send for GitHub or YouTube videos, and web links, obviously, and documents.

s111
00:11:55.200 --> 00:12:02.240
But there are tons of things missing, as we will discuss in the end, that you can easily implement, just asking Cloud Coder or Codex to do so.

s112
00:12:02.240 --> 00:12:07.720
Like for example, I implemented the YouTube video transcript in honestly a few seconds, just one prompt.

s113
00:12:07.720 --> 00:12:10.880
It's super easy for Codex to implement it.

s114
00:12:10.880 --> 00:12:17.920
So, the thing is that this whole repository and this whole project is a very useful companion for my own work.

s115
00:12:17.920 --> 00:12:28.320
But, as I said, it implements tons of features and state-of-the-art context management and memory management techniques that I believe AI engineers need to know.

s116
00:12:28.320 --> 00:12:42.120
And now, Paul will dive into all this three-layer system that we built with the raw content, the index that I mentioned, and the wiki-like synthesized version of all your notes, all your research, all your work.

s117
00:12:42.120 --> 00:12:48.040
So, he'll cover everything we did, how it ended up, and show how to use it.

s118
00:12:48.040 --> 00:12:48.600
Okay.

s119
00:12:48.600 --> 00:12:58.080
So, now I want to go over the three versions of our system and how it progressed over time, and most importantly, why we added more complexity.

s120
00:12:58.080 --> 00:13:04.839
So, in the first version, we wanted to scope it just to create lessons for our agent engineering course.

s121
00:13:04.839 --> 00:13:11.160
So, we wanted to keep it super simple, where we had as input a topic and a research MD as output.

s122
00:13:11.160 --> 00:13:17.880
So, within the input, we had the topic plus a set of golden links, which were manually handpicked by us.

s123
00:13:17.880 --> 00:13:23.920
We applied this deep research algorithm, and we had as output a static research MD file.

s124
00:13:23.920 --> 00:13:32.760
And if we go zoom into the architecture, we first scraped the links of the golden links, right?

s125
00:13:32.760 --> 00:13:45.360
Because we already know them and we use them as seed for context for the deep research algorithm, which was a really powerful technique because we had more context on how to frame our questions.

s126
00:13:45.360 --> 00:13:59.640
And during the query rounds, we basically used the very classic deep research algorithm where we had one main agent, the orchestrator, which created multiple questions based on the initial topic and the scraped

s127
00:13:59.640 --> 00:14:00.920
context.

s128
00:14:00.920 --> 00:14:19.040
And each agent managed its own question and used Gemini grounded in Google to to query basically Google and gather multiple resources and each agent gather these resources, which returned multiple links and created some executive summaries

s129
00:14:19.040 --> 00:14:20.360
of each link.

s130
00:14:20.360 --> 00:14:20.679
And then

s131
00:14:20.679 --> 00:14:20.800
[snorts]

s132
00:14:20.800 --> 00:14:34.280
it passed all this information back to the agent to the main agent where the main agent basically aggregated all this information into a summarized way so it did not exploded the context.

s133
00:14:34.280 --> 00:14:36.920
And we did this for three rounds.

s134
00:14:36.920 --> 00:14:46.200
So basically after three rounds of generating six queries per round, we ended up with like 40-50 links in total.

s135
00:14:46.200 --> 00:14:50.240
So you can imagine that there's a lot of noise over there.

s136
00:14:50.240 --> 00:14:59.240
So that's why we also applied a ranking algorithm where we wanted to like find the the information with the highest signal.

s137
00:14:59.240 --> 00:15:04.920
And basically we compared each source against the topic, the initial topic of the user.

s138
00:15:04.920 --> 00:15:11.080
And like that, we fully scraped only the top K elements based on the ranking score.

s139
00:15:11.080 --> 00:15:16.240
And for the rest of the of the links, we just kept the summaries.

s140
00:15:16.240 --> 00:15:27.120
And then we compiled everything into this research MD file as a single flat file which we used for each lesson of our course in in our particular use case.

s141
00:15:27.120 --> 00:15:29.920
But as you can imagine, it was pretty limited.

s142
00:15:29.920 --> 00:15:32.240
For the course, it worked great right away.

s143
00:15:32.240 --> 00:15:37.120
We generated 35 lessons really quick, but we wanted more.

s144
00:15:37.120 --> 00:15:45.520
So, we started to aim this deep research loop to the second brain as well, right?

s145
00:15:45.520 --> 00:15:55.720
Before, it was targeting only the public web, which made this a pretty generic, and we had to manually find all those golden links.

s146
00:15:55.720 --> 00:16:08.160
So, by aiming this deep research loop to the second brain, where we basically organically keep track of all the information that of all the research that we really want

s147
00:16:08.160 --> 00:16:17.320
and is filtered by us, we can organically gather all those golden rings into our deep research algorithm.

s148
00:16:17.320 --> 00:16:20.720
So, let's look at how this new algorithm looks like.

s149
00:16:20.720 --> 00:16:23.120
It's basically the same loop, right?

s150
00:16:23.120 --> 00:16:28.120
But now we target our own sources instead of just the public web.

s151
00:16:28.120 --> 00:16:32.600
Now, for input, we have only the topic because we don't need the golden links.

s152
00:16:32.600 --> 00:16:38.080
As I said, the golden links are actually a reflection of our second brain system.

s153
00:16:38.080 --> 00:16:48.920
In theory, you can also add them if you really want to, but that's the beauty of this new strategy because you can just put as input some topic and we'll find everything that it needs.

s154
00:16:48.920 --> 00:16:56.240
And then we use this topic only as seed for for for the context to generate the the queries.

s155
00:16:56.240 --> 00:16:59.720
And now we do the same deep research algorithm, right?

s156
00:16:59.720 --> 00:17:11.720
The same query rounds, but instead of targeting only the public web, now we plugged in all our second brains, such as the our Obsidian, our Readwise, our Notebook LM, our GitHub.

s157
00:17:11.720 --> 00:17:20.760
You can also use, for example, Gemini deep research for this, like similar to how we we use notebook LM or you can extend this with whatever you want.

s158
00:17:20.760 --> 00:17:26.560
For example, YouTube, uh Google Drive, Notion, or whatever makes sense on your infrastructure.

s159
00:17:26.560 --> 00:17:36.560
The idea is that now we are target our queries from the deep research algorithm to our second brain plus the public web.

s160
00:17:36.560 --> 00:17:45.880
And after we apply the same algorithm such as ranking, fully scraping, summaries, and compile everything into this research MD file.

s161
00:17:45.880 --> 00:17:48.000
But now we have another problem, right?

s162
00:17:48.000 --> 00:17:50.840
This this research MD file is static.

s163
00:17:50.840 --> 00:17:53.000
It's a pile of static data.

s164
00:17:53.000 --> 00:17:55.760
And usually research is not static, right?

s165
00:17:55.760 --> 00:18:06.200
So, after you end up with with this file, you most often realize that you want to ask another question or some information is stale and you don't need it anymore.

s166
00:18:06.200 --> 00:18:10.320
Or or basically, you want more out of this research MD file.

s167
00:18:10.320 --> 00:18:13.160
And we which means that you need to start all of this from scratch.

s168
00:18:13.160 --> 00:18:18.480
And the operation that I showed you uh above is an extensive operation.

s169
00:18:18.480 --> 00:18:20.920
It consumes a lot of tokens and it takes a lot of time.

s170
00:18:20.920 --> 00:18:24.520
So, you don't want to run it from scratch.

s171
00:18:24.520 --> 00:18:28.600
And that's why you need to add a wiki layer on top of it.

s172
00:18:28.600 --> 00:18:38.880
And that's why V3 of this system is actually a deep research algorithm plus an LM knowledge base on top of it, aka the wiki layer.

s173
00:18:38.880 --> 00:18:41.560
So, the new algorithm looks like this.

s174
00:18:41.560 --> 00:18:45.120
So, we have sources in and a wiki out.

s175
00:18:45.120 --> 00:18:53.320
And the sources, as I said before, can be like Obsidian, notebook LM, Google Drive, or YouTube, Notion, even custom URLs, right?

s176
00:18:53.320 --> 00:19:03.480
That that's also powerful as well, where you use tools such as a Bright Data to to parse basically any single page application, any type of site, any type of public

s177
00:19:03.480 --> 00:19:04.560
information that's out there.

s178
00:19:04.560 --> 00:19:05.920
We can put it in.

s179
00:19:05.920 --> 00:19:09.320
And then you apply the same deep research algorithm.

s180
00:19:09.320 --> 00:19:12.680
You store everything into raw files, right?

s181
00:19:12.680 --> 00:19:19.240
Instead of compiling everything into research and D file, now we store each file individually.

s182
00:19:19.240 --> 00:19:23.280
And we create an index out of all these files.

s183
00:19:23.280 --> 00:19:27.960
And ultimately, we generate a wiki on top of it, and which we can query.

s184
00:19:27.960 --> 00:19:31.080
We can query basically the wiki plus the index.

s185
00:19:31.080 --> 00:19:35.000
Okay, so this is just the high-level architecture of the new system.

s186
00:19:35.000 --> 00:19:37.080
Let's zoom into it.

s187
00:19:37.080 --> 00:19:47.760
So, what do I want to start with is that you should forget the infra structure you think you need, such as vector databases, knowledge graphs, semantic search, text search.

s188
00:19:47.760 --> 00:19:58.360
All that is beautiful, but add a lot of complexity, especially for like this personal wikis, personal research operating systems that you want to use very lightly.

s189
00:19:58.360 --> 00:20:01.680
So, I want a system just based on files, right?

s190
00:20:01.680 --> 00:20:06.600
A simple mechanism that's that's very rooted into how your computer works.

s191
00:20:06.600 --> 00:20:12.080
And that's why we'll create all the system just based on files and just based on references.

s192
00:20:12.080 --> 00:20:17.240
So, no database, just a simple index based on references.

s193
00:20:17.240 --> 00:20:18.400
And how how this works?

s194
00:20:18.400 --> 00:20:30.360
We have an agent that reads an index.yaml file that's basically a catalog of your all your data plus the summaries of of each source and some metadata around it.

s195
00:20:30.360 --> 00:20:46.800
For example, here on the right, you can see part of an index.yaml file that contains 10 sources and 38 wiki pages as derivatives of these sources, where uh we can see there into the sources list of the YAML file the first

s196
00:20:46.800 --> 00:20:47.960
uh source, for example.

s197
00:20:47.960 --> 00:20:56.480
And as you can see, it has like the the the link to the original file plus some metadata, such the origin, the title, the authors, the the the publication

s198
00:20:56.480 --> 00:21:01.760
date, the summary, and and things things like this, which can be flexible, right?

s199
00:21:01.760 --> 00:21:11.720
And the next step is that based on this index.yaml file, we need to point to all the wiki pages, to all the wiki derivatives, to all the raw sources.

s200
00:21:11.720 --> 00:21:16.920
So, basically, this index.yaml file is an entry point for our agent, right?

s201
00:21:16.920 --> 00:21:23.040
It's what we will give to our agent to actually reason on how to find our data.

s202
00:21:23.040 --> 00:21:26.240
It's an index, ultimately, right?

s203
00:21:26.240 --> 00:21:30.680
So, the next step is to understand how the wiki actually looks like.

s204
00:21:30.680 --> 00:21:38.120
So, on the left, you can see the high-level structure of the wiki, where we have the raw folder, the wiki folder, and the index.

s205
00:21:38.120 --> 00:21:41.600
In the raw folder, we actually just have the raw data, which is immutable.

s206
00:21:41.600 --> 00:21:43.440
You don't want to touch that.

s207
00:21:43.440 --> 00:21:46.720
The And the index points to everything that we need.

s208
00:21:46.720 --> 00:21:58.760
And in the wiki, we actually have derivatives created by the LLM, which contains things such as comparisons between multiple concepts, entities, or just simple notes as a reflection of our questions

s209
00:21:58.760 --> 00:22:05.560
or repositories that we ingested, and we can create multiple notes based on a on on a repository, right?

s210
00:22:05.560 --> 00:22:11.080
Or open questions that based on our questions that LLM couldn't answer yet.

s211
00:22:11.080 --> 00:22:16.160
And everything that you can analyze on top of your raw data.

s212
00:22:16.160 --> 00:22:23.000
And on the right, based on Obsidian, we can see like the sub graph reflected just based on in this index.

s213
00:22:23.000 --> 00:22:32.200
And this is just like the first iteration, but as the the the the wiki grows, you can see connections made between entities and concepts.

s214
00:22:32.200 --> 00:22:42.040
For example, concepts are things such as tool registry, context compaction, sandboxing, or entities are open code, closed code, MCP, right?

s215
00:22:42.040 --> 00:22:49.560
So, as you can see, you can beautifully can start visually and practically create connections.

s216
00:22:49.560 --> 00:22:53.840
Now, the next obvious question is how do we actually query this wiki?

s217
00:22:53.840 --> 00:23:04.720
So, as I said, the agent will have as input this index.yaml file, which contains summaries and metadata about each source.

s218
00:23:04.720 --> 00:23:06.760
But what happens next, right?

s219
00:23:06.760 --> 00:23:11.040
The next step is actually to look into the source wiki page.

s220
00:23:11.040 --> 00:23:15.680
Where the source wiki page is like an executive summary of each page.

s221
00:23:15.680 --> 00:23:21.360
Which is basically not just a summary, but a more expanded summary of of each source.

s222
00:23:21.360 --> 00:23:26.480
And sometimes the agent just looks into this, gets what it needs, and goes back.

s223
00:23:26.480 --> 00:23:29.560
Which is very token efficient, right?

s224
00:23:29.560 --> 00:23:41.720
And if it doesn't find within this uh source wiki page, we also need links into the wiki derivative, such as concepts, entities, notes, comparisons, and so and so forth.

s225
00:23:41.720 --> 00:23:50.200
And only if it doesn't find the necessary information up to this point, it needs and it actually reads the whole raw source, right?

s226
00:23:50.200 --> 00:23:55.480
Which basically contains the whole article, the whole paper, the the the whole video, or whatever.

s227
00:23:55.480 --> 00:24:03.560
And this makes just through pure referencing and creating this simple hierarchy, this makes everything very token efficient.

s228
00:24:03.560 --> 00:24:07.200
Now, the beautiful part is that this wiki is actually alive, right?

s229
00:24:07.200 --> 00:24:11.120
For example, every question leaves a trace into your wiki.

s230
00:24:11.120 --> 00:24:16.100
So for every question, the LLM can create a new concept file, a new notes

s231
00:24:16.100 --> 00:24:16.400
[snorts]

s232
00:24:16.400 --> 00:24:18.440
file, a new comparison file.

s233
00:24:18.440 --> 00:24:21.480
And every question is is tracked into a log.

s234
00:24:21.480 --> 00:24:31.800
So basically, the the the wiki doesn't evolve only when you ingest new data or do a deep research round, it actually evolves as you start talking with it, right?

s235
00:24:31.800 --> 00:24:33.880
That's the beautiful part, actually.

s236
00:24:33.880 --> 00:24:42.400
And like that, you can see a true reflection of of yourself, of what you haven't understood, of all your questions from the past.

s237
00:24:42.400 --> 00:24:46.200
And the beautiful part is that the the wiki is never frozen, right?

s238
00:24:46.200 --> 00:24:48.560
Similar to the research entity files.

s239
00:24:48.560 --> 00:24:55.280
At any point, you can ingest a new custom link that you think that you need into the wiki, or even run a new deep research round.

s240
00:24:55.280 --> 00:25:01.240
Or as I said previously, the wiki keeps evolving just purely based on your questions.

s241
00:25:01.240 --> 00:25:09.360
And another important thing to understand is that this wiki doesn't sit on top your of your entire second brain, right?

s242
00:25:09.360 --> 00:25:20.160
For example, in my particular use case, I use the PARA method coined by Tiago Forte where all my data is structured between project, areas, resources, and archive.

s243
00:25:20.160 --> 00:25:30.280
Where all my notes resources that I save, sources that I save yet article or whatever are just piped directly to the resource a flat list.

s244
00:25:30.280 --> 00:25:37.240
And whenever I need something, I just references them into projects and areas, right?

s245
00:25:37.240 --> 00:25:43.400
And like this, Obsidian is just an immutable snapshot that a LLM never touches, right?

s246
00:25:43.400 --> 00:25:44.520
So, this is my data.

s247
00:25:44.520 --> 00:25:51.440
I don't really want the LLM to touch my personal notes that I manually write, right?

s248
00:25:51.440 --> 00:25:55.080
So, then how can we actually put this wiki to use, right?

s249
00:25:55.080 --> 00:26:02.240
So, as I said, we have the big Obsidian snapshot which is our global second brain.

s250
00:26:02.240 --> 00:26:15.120
And then whenever we start to work on a new project, we reference this second brain through this deep research algorithm that I explained, and we scope it down to our own project,

s251
00:26:15.120 --> 00:26:15.360
right?

s252
00:26:15.360 --> 00:26:26.520
So, basically, whenever we want to start working on something, we run this deep research loop or we start ingesting some particular repositories, articles, notes, and so on and so forth.

s253
00:26:26.520 --> 00:26:32.120
And we usually do that through a set of skills plugged into a harness.

s254
00:26:32.120 --> 00:26:38.560
And a project can be basically anything such as writing a new article, doing a new video, doing a set of slides.

s255
00:26:38.560 --> 00:26:41.280
I I applied this technique doing this slide.

s256
00:26:41.280 --> 00:26:50.520
Or you can even apply it for something more complex such as writing a book, doing a course, or or keeping track of a whole code base, right?

s257
00:26:50.520 --> 00:26:52.320
You can also you use it for that.

s258
00:26:52.320 --> 00:27:00.520
So, basically, a project can be anything where you want, as I said initially, to transform research into work.

s259
00:27:00.520 --> 00:27:04.560
The project is the work, and your second brain is the research.

s260
00:27:04.560 --> 00:27:06.600
So, now I want to show you a few demos.

s261
00:27:06.600 --> 00:27:17.440
So, what you need to do is go to the AI research OS workshop repository, and here you can find all the skills required to run what we presented into this presentation.

s262
00:27:17.440 --> 00:27:24.000
And everything is packed as a cloud code plugin, but you can very easily tweak it and install it with any other harness.

s263
00:27:24.000 --> 00:27:34.880
And also in the read me, you can find details on how to install all all the other dependencies, because the thing is that this system is dependent on tools such as Obsidian, Readwise notebook elements, and so on and so forth.

s264
00:27:34.880 --> 00:27:39.040
So, you need to set up specific CLIs or authentication issues.

s265
00:27:39.040 --> 00:27:45.720
But, I don't really want to waste any of your time with setup issues, and I want to go straight directly into the examples.

s266
00:27:45.720 --> 00:27:48.280
So, I prepared here three examples.

s267
00:27:48.280 --> 00:28:00.680
The first one is a research on one of my previous articles on agentic AI engineering, and within these files, I have a brand dump of everything that I knew I wanted to talk on this subject.

s268
00:28:00.680 --> 00:28:08.440
And on top of that, I also added a few references that I knew 100% that I want to add into this wiki.

s269
00:28:08.440 --> 00:28:21.000
And what we need to do to actually trigger the the algorithm on top of these files is to open up a cloud session, and then just call the skill, the research skill, and point it it to this file.

s270
00:28:21.000 --> 00:28:21.800
And that's it.

s271
00:28:21.800 --> 00:28:24.640
Everything else is baked directly into the skill.

s272
00:28:24.640 --> 00:28:38.520
It will understand my intent that I want to create a wiki on agentic harness engineering just looking at these files and looking at the topic, and it will know that before starting the deep research algorithm, it actually needs to scrape

s273
00:28:38.520 --> 00:28:44.920
this information to to use it as context when it frames the questions for the deep research algorithm.

s274
00:28:44.920 --> 00:28:49.320
Now, we need to wait a bit for for the agent to reason on top of it.

s275
00:28:49.320 --> 00:28:53.840
And I will actually just put it on auto mode to speed speed this up, right?

s276
00:28:53.840 --> 00:29:01.480
I use this hundreds of times, so I know it won't delete anything from my computer or it won't do anything weird.

s277
00:29:01.480 --> 00:29:04.600
Okay, so now this is the most important part, right?

s278
00:29:04.600 --> 00:29:09.520
So, it asked me how deep I want the deep research algorithm to be.

s279
00:29:09.520 --> 00:29:12.000
We have light, deep, fast.

s280
00:29:12.000 --> 00:29:19.880
This mostly controls how many questions you want to run per one round and how many rounds you want to run.

s281
00:29:19.880 --> 00:29:27.400
And usually light or fast is more than enough because remember this process consumes a lot of tokens.

s282
00:29:27.400 --> 00:29:35.640
So, you kind of need to do the deep one only when you you really want to look over tons and tons of notes.

s283
00:29:35.640 --> 00:29:40.040
And for this use case, I will just pick the light one to to to speed this up.

s284
00:29:40.040 --> 00:29:44.480
And in this use case, it just does one round of three queries, right?

s285
00:29:44.480 --> 00:29:48.960
And for the fast one, it does two rounds of three queries.

s286
00:29:48.960 --> 00:29:52.800
So, I would just keep it keep it around that spectrum.

s287
00:29:52.800 --> 00:30:05.880
And now the process will will take around 10 to 20 minutes to actually look around my Obsidian, to look around my Readwise, my Notebook LM, and run those queries on top of this.

s288
00:30:05.880 --> 00:30:09.280
And I actually run this, right?

s289
00:30:09.280 --> 00:30:17.880
And now let's open this wiki that we created based on the prompt before in Obsidian and let's look what's inside the the wiki.

s290
00:30:17.880 --> 00:30:31.680
So, we have three big objects, the raw files, which are basically a raw copy of what we found, the index, which contains all the references to towards the wiki, right?

s291
00:30:31.680 --> 00:30:38.160
We can also in Obsidian have this beautiful a subgraph where we can very quickly understand what is going on.

s292
00:30:38.160 --> 00:30:41.840
This is created purely based on this this file.

s293
00:30:41.840 --> 00:30:45.600
And then the most interesting part is inside the wiki.

s294
00:30:45.600 --> 00:30:52.360
Where we have comparisons, concepts, entities, and sources.

s295
00:30:52.360 --> 00:30:56.840
The sources contain the executive summary of of our raw sources.

s296
00:30:56.840 --> 00:31:06.360
So, the LLM doesn't really need to every time when it reads them, and it needs them to to read the raw sources, but it needs just the executive summaries,

s297
00:31:06.360 --> 00:31:10.000
which are computed just one time during the ingestion.

s298
00:31:10.000 --> 00:31:21.400
And for example, for the comparisons, it understood out of the box that it needs to do comparisons between like adjective rag versus file systems or compaction versus recursive language models

s299
00:31:21.400 --> 00:31:25.480
or or or anything of interest based on on the sources.

s300
00:31:25.480 --> 00:31:28.480
And the most interesting part is actually the concepts.

s301
00:31:28.480 --> 00:31:35.680
So, it automatically extracted all the concepts that we need to understand from this pile of resources.

s302
00:31:35.680 --> 00:31:48.440
For example, if we open the agent loop resources, we automatically can look and get this beautiful summary containing like graphs, tags, and explaining us everything that we need on this topic.

s303
00:31:48.440 --> 00:31:58.440
And we can do the same on all the concepts from from from this wiki or all the entities from the wiki and so and so forth.

s304
00:31:58.440 --> 00:32:01.280
Okay, so now let's go to the second example.

s305
00:32:01.280 --> 00:32:07.360
It It's a simple example where I want to learn more on harness engineering and how harnesses work.

s306
00:32:07.360 --> 00:32:16.160
So, in in this prompt over here, I just want to ingest the three open source repositories on open code, Pi, and Hermes.

s307
00:32:16.160 --> 00:32:19.400
And I don't want to do deep research at all, right?

s308
00:32:19.400 --> 00:32:31.040
I just want to ingest those repositories and explore topics such as the general architecture, agents architecture, sub agents, memory system, and the agent permission flow.

s309
00:32:31.040 --> 00:32:35.680
And let's run the research on top of this prompt.

s310
00:32:35.680 --> 00:32:48.000
And now what the research will do will clone automatically all these repositories and will explore all the repositories on the topics that I gave here and it will create notes

s311
00:32:48.000 --> 00:32:58.160
at the individual level of each repository on how they work, on how how the architecture works at the repository level and then we can create higher level notes, right?

s312
00:32:58.160 --> 00:33:13.960
Within the wiki derivatives and compare all the architectures or create aggregate architectures on on like the general trends of all those harnesses and basically explore and learn everything that we want about harness engineering directly

s313
00:33:13.960 --> 00:33:15.560
from the code.

s314
00:33:15.560 --> 00:33:22.400
And again, usually I just do auto mode and let it do its own gist, but I already run this, right?

s315
00:33:22.400 --> 00:33:33.400
So here is the wiki for these GitHub repositories and again, we have the raw files and we have the index and the wiki.

s316
00:33:33.400 --> 00:33:52.240
And here probably within the repos we can see all the three repositories and for example, inside the open core repositories we can see empty files explaining everything that we need on particular topics such as the permission flow, the memory system and so on and so forth and we can see this

s317
00:33:52.240 --> 00:34:00.440
in all the other repositories and the most interesting part here is actually that we have comparisons on all of these, right?

s318
00:34:00.440 --> 00:34:14.639
So we can actually understand the differences within the architecture within these harnesses or we also have all the concepts extracted from these repositories and we can understand what are the key architectural decisions

s319
00:34:14.639 --> 00:34:17.000
from these repositories.

s320
00:34:17.000 --> 00:34:24.919
So we can go crazy with this and this is super useful if you want to, for example, write your own harness.

s321
00:34:24.919 --> 00:34:33.760
And the third examples is the simplest one in in reality, which is just based on ingesting some some simple links, right?

s322
00:34:33.760 --> 00:34:41.120
And again, I will just exit the the previous run and I want to run this example from scratch.

s323
00:34:41.120 --> 00:34:52.760
And here, I just want to show you that you can use this also with a very basic setup where I just want to ingest three custom random links.

s324
00:34:52.760 --> 00:34:59.560
And as before, I just pass this prompt and it will start the research process.

s325
00:34:59.560 --> 00:35:10.640
And I want to highlight that you can run example two ending on the GitHub repositories and this example without any other setup like without setting up Obsidian, Readwise, or anything else.

s326
00:35:10.640 --> 00:35:21.920
You can just install this plugin and run this examples because it here is not dependent on any other service than Git and using curl to get this this URLs.

s327
00:35:21.920 --> 00:35:33.960
And again, here if we go into Obsidian and explore the wiki created out of this example three, we we can see the index, we can see the wiki itself

s328
00:35:33.960 --> 00:35:35.720
with all the sources, right?

s329
00:35:35.720 --> 00:35:42.560
One executive summary for for each source and all the concepts, extracted entities, and so on and so forth.

s330
00:35:42.560 --> 00:35:48.760
And the idea is that as you start asking questions on top of this, everything starts to get more interesting.

s331
00:35:48.760 --> 00:35:56.920
So now, let's assume that we want to ask for example a question on harness engineering based on the wiki created out of the GitHub repositories.

s332
00:35:56.920 --> 00:36:08.920
So what we have to do is just again hit the research field pointed to the wiki that we just created, and then just ask our question.

s333
00:36:08.920 --> 00:36:26.640
And let's assume that I want to learn more on sandboxing, more exactly how remote sandboxing works and how is plugged into the heart.

s334
00:36:26.920 --> 00:36:30.000
This can be basically any any any other question.

s335
00:36:30.000 --> 00:36:46.520
And now what it will happen, it will basically query this wiki, it will give you an answer, and based on this, you can also start creating notes, comparisons, or maybe it will extract and find new entities that you care about.

s336
00:36:46.520 --> 00:36:49.960
So basically, it will start updating the wiki.

s337
00:36:49.960 --> 00:36:50.240
All right.

s338
00:36:50.240 --> 00:36:52.640
So now, where is this project going?

s339
00:36:52.640 --> 00:36:53.400
What is it?

s340
00:36:53.400 --> 00:36:54.600
What should you do?

s341
00:36:54.600 --> 00:36:57.080
First, it's still rough at some points.

s342
00:36:57.080 --> 00:36:59.360
Like it needs more connectors, as you saw.

s343
00:36:59.360 --> 00:37:04.520
We need to add Google Drive, Notion, Slack, and tons of other connectors that could be useful to you.

s344
00:37:04.520 --> 00:37:08.600
But to be honest, it's not really useful to me and my current workflow or to Paul.

s345
00:37:08.600 --> 00:37:15.720
So we didn't add them yet, because the core of this project is to be useful for us and for you to take over and add whatever you need.

s346
00:37:15.720 --> 00:37:20.960
And the other main goal of this project is to teach memory and context management.

s347
00:37:20.960 --> 00:37:25.520
So all these extra features aren't really useful towards that purpose.

s348
00:37:25.520 --> 00:37:34.280
There are other some weaknesses, like it's hard to know which sources are outdated or weak or strong compared to some other system that we built.

s349
00:37:34.280 --> 00:37:39.080
So we know we can improve this, but again, it's not really the priority here.

s350
00:37:39.080 --> 00:37:42.160
And lastly, it's still obviously a builder workflow.

s351
00:37:42.160 --> 00:37:47.880
You use it through cloud code and Codex, and it's just to me in the terminal and I just really like it.

s352
00:37:47.880 --> 00:37:52.240
So it's not a final polished product with a nice UI, nice UX.

s353
00:37:52.240 --> 00:37:54.280
And honestly, that's by design.

s354
00:37:54.280 --> 00:37:58.760
So we don't really care about this, because our goal is to teach AI engineering.

s355
00:37:58.760 --> 00:38:02.600
It's not to build the next best product.

s356
00:38:02.600 --> 00:38:15.880
Still, we have a few next improvements we want to do very shortly, from having a stronger linting to a better memory compaction, because that's a big issue and it's just very complicated in general to manage memory correctly,

s357
00:38:15.880 --> 00:38:18.640
and the state of the art is always progressing there.

s358
00:38:18.640 --> 00:38:27.840
We, as I said, need better source provenance to trust the sources and be able to rank them properly and reuse them properly if needed and be able to know

s359
00:38:27.840 --> 00:38:33.200
access quickly as a user if this source is relevant or not.

s360
00:38:33.200 --> 00:38:37.400
And we have other next improvements to do.

s361
00:38:37.400 --> 00:38:41.080
But those are mostly for optimization and for the future.

s362
00:38:41.080 --> 00:38:49.680
And the thing is that we actually build all of that into another product that you can even build yourself.

s363
00:38:49.680 --> 00:39:04.600
Because we created a course called Agent Engineering where we build a similar deep research system with a writing and research agent where we build a system with the same goal to be able to learn best AI engineering practices.

s364
00:39:04.600 --> 00:39:14.520
It's a very in-depth course where I assume it takes around 60 hours to complete with a final project being the multi-agent system I just described that you'll build for yourself.

s365
00:39:14.520 --> 00:39:26.440
So if this presentation and the demo repo that you saw was interesting, please consider checking out the Towards AI Academy with our courses on there including the Agent Engineering

s366
00:39:26.440 --> 00:39:32.960
course to learn more on the best practices when building around and with agents.