WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=WP3hjUXd918

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/WP3hjUXd918.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:13.360 --> 00:00:14.400
All right.

s2
00:00:14.400 --> 00:00:16.800
Good afternoon everyone.

s3
00:00:16.800 --> 00:00:21.039
Thank you for joining and not watching the game.

s4
00:00:21.039 --> 00:00:31.840
Uh I hope it will be a bit more interesting or at least you will learn something compared to to uh hopefully Germany winning or some uh anyways.

s5
00:00:31.840 --> 00:00:33.840
Yeah.

s6
00:00:34.960 --> 00:00:35.200
All right.

s7
00:00:35.200 --> 00:00:36.000
Is it fine?

s8
00:00:36.000 --> 00:00:38.000
Okay.

s9
00:00:38.160 --> 00:00:38.480
All right.

s10
00:00:38.480 --> 00:00:45.200
So I'm here to talk about we are here to talk about context engineering in 2026.

s11
00:00:45.200 --> 00:00:54.960
And more specifically, we are here because we've all lived that that situation where you try to do things with an agent and ultimately it does just exactly the thing that

s12
00:00:54.960 --> 00:00:56.320
you don't want it to do.

s13
00:00:56.320 --> 00:01:04.799
And it in my case it usually ends up like this where I'm super mad and I just type back hoping it it learns.

s14
00:01:04.799 --> 00:01:19.680
And uh usually the problem here is not that the the model just got dumber and you need to switch to cloud or to codeex or or whatever the the harness that you're using, but it's more that the the context is filling up and it's getting worse and worse.

s15
00:01:19.680 --> 00:01:22.479
The results are getting worse because of it.

s16
00:01:22.479 --> 00:01:29.119
In our case, this is important because we build courses and trainings for AI engineers specifically.

s17
00:01:29.119 --> 00:01:36.159
And one of the features that we provide is an AI tutor to help answer questions based on our lessons.

s18
00:01:36.159 --> 00:01:45.600
And if the interaction is just like the one before and they are super mad at us, they might just ask for refund and it could end up like this.

s19
00:01:45.600 --> 00:01:48.640
So that's not what we want.

s20
00:01:48.640 --> 00:02:01.360
And so what we did for this workshop and just for the AI tutor in general is to run many different experiments in order to figure out how in our case we can fix context rut or at least

s21
00:02:01.360 --> 00:02:08.800
improve the i tutor as much as possible and reduce the cost as well of running the tutor.

s22
00:02:08.800 --> 00:02:16.720
Uh the QR code here is a link to a hugging face space where you have all these experiments that you can see and also the AI tutor is open source.

s23
00:02:16.720 --> 00:02:20.000
who will share another code for the repo but it's also linked on the hugging face.

s24
00:02:20.000 --> 00:02:28.160
So everything is open source you can access everything uh and even see the experiments online and use the AI tutorial online as well.

s25
00:02:28.160 --> 00:02:42.000
In the next 80 minutes, we will I will start talking about compaction, memory retrieval and um everything that you can do in 2026 that usually works and then my colleagues will jump in with

s26
00:02:42.000 --> 00:02:55.519
our the architecture of our AI tutor, our decisions, what we built and the evaluations that we built, the how we built them and what we decided to evaluate and then the results and what we took out of this.

s27
00:02:55.519 --> 00:03:07.360
So of of course it's applied to our use case and our AI tutor but hopefully you can get away some interesting insights at least from from this and some best practices that we learned throughout

s28
00:03:07.360 --> 00:03:10.239
more specifically we is towards AI.

s29
00:03:10.239 --> 00:03:18.239
Um I founded the company with my partners in in a few years ago and we've always been focused around education.

s30
00:03:18.239 --> 00:03:22.720
Obviously back in the day it was more about computer vision and and more basic machine learning.

s31
00:03:22.720 --> 00:03:29.440
Now it's towards AI engineering and agents, anything that works for the industry.

s32
00:03:29.440 --> 00:03:38.400
And I'm joined by my colleagues helping me develop this AI tutor and our courses, Omar and Samidi that we'll jump in later on.

s33
00:03:38.400 --> 00:03:41.280
And here more specifically towards AI is quite large.

s34
00:03:41.280 --> 00:03:44.480
But what one one of the things that we do is our academy.

s35
00:03:44.480 --> 00:03:57.519
So the towards AI academy where we build courses technical courses for AI engineers to upskill towards AI engineering and as I said we provide an AI tutor for the students.

s36
00:03:57.599 --> 00:04:04.239
So the the AI tutor specifically uh will will be like our baseline for all our experiments.

s37
00:04:04.239 --> 00:04:11.599
We will use that to test all the different features based on real user interactions and to have the best results possible.

s38
00:04:11.599 --> 00:04:16.639
We had five requirements we wanted to ensure that the chatbot follows.

s39
00:04:16.639 --> 00:04:24.240
The first one obviously we want the answers of the chatbot to be grounded in our content not just its own knowledge.

s40
00:04:24.240 --> 00:04:32.880
We need the the tutor to be based in the current students and current lesson to not be a because we have multiple courses.

s41
00:04:32.880 --> 00:04:39.280
So we just need to ensure that it answers from this course content.

s42
00:04:39.280 --> 00:04:51.759
Then it needs to hold long help sessions in case the student is debugging or or just iterating a lot with the the tutor and uh obviously handled code because it's for AI engineers.

s43
00:04:51.759 --> 00:04:59.280
So we just code a lot and it needs to have somewhat of a low latency to not be frustrating to use.

s44
00:04:59.280 --> 00:05:02.960
And all of this is related to context engineering.

s45
00:05:02.960 --> 00:05:13.919
And here we'll be talking about uh what is context engineering in 2026 or at least what we figured out from this from these experiments.

s46
00:05:13.919 --> 00:05:20.960
And since everything here is in the context in the in the context of models, it creates two problems.

s47
00:05:20.960 --> 00:05:23.919
First, the context window is finite.

s48
00:05:23.919 --> 00:05:30.400
Uh everything the model will see from instructions to to lessons to code will lie in the same space that is limited.

s49
00:05:30.400 --> 00:05:37.440
And the more you pile things in this space, the worse the results will be and the more expensive it will be because you pay for more tokens.

s50
00:05:37.440 --> 00:05:41.199
So that's one of the the main problem we're trying to fix.

s51
00:05:41.199 --> 00:05:44.240
And the second problem is that the model is stateless.

s52
00:05:44.240 --> 00:05:51.440
So when a student reopens the AI tutorially if you don't build anything around it, the model will have no idea what's going on.

s53
00:05:51.440 --> 00:05:53.199
It's just starting from zero.

s54
00:05:53.199 --> 00:05:56.800
So this creates two things we have to work on.

s55
00:05:56.800 --> 00:06:10.880
the context management which means within one session and the memory aspect of these models which means across sessions in these experiments and in this workshop we focus on the first part the context management

s56
00:06:10.880 --> 00:06:15.520
because you cannot have multiple sessions is if one session is is shitty.

s57
00:06:15.520 --> 00:06:23.120
So we try to really optimize for this and maybe in a future workshop we will do one about memory hopefully.

s58
00:06:23.120 --> 00:06:27.360
So for context management what does the the tutor sees in our case?

s59
00:06:27.360 --> 00:06:41.120
It sees uh many things from the system prompt to being for being a tutor and the context of our courses and stuff to tool definitions uh so which tools it can use how to use them when to use them.

s60
00:06:41.120 --> 00:06:51.440
The chat history if it's an ongoing discussion any old tool outputs that was called course chunks that we retrieved to answer the students and finally the user questions.

s61
00:06:51.440 --> 00:07:00.639
So it's just not the user question that we send obviously and typically it's the smallest part but it can also have contain a lot of code to debug and to help

s62
00:07:00.639 --> 00:07:01.680
or error logs.

s63
00:07:01.680 --> 00:07:10.479
So it can be large as well and all of this together just ends up costing more and more to us.

s64
00:07:10.479 --> 00:07:15.120
So we really want to optimize this this context management aspect.

s65
00:07:15.120 --> 00:07:26.560
And what we've seen just quickly is that the the main bottleneck or the main problem scaling the context is the old tool outputs which contains like any old chunks retrieved

s66
00:07:26.560 --> 00:07:37.199
all the tool calls and tool results pairs uh from from all the tools called over the the many sessions that it could have the many turns that that one session could have have

s67
00:07:37.199 --> 00:07:44.400
and any uh file or searches that it did to in its own memory.

s68
00:07:44.400 --> 00:07:48.800
And the main problem is not is not that it's more costly.

s69
00:07:48.800 --> 00:07:58.720
It's also that the quality degrades as we know with a program called context route because of the way how large language models are trained to handle longer context with

s70
00:07:58.720 --> 00:08:11.360
we just inject facts into large compass which doesn't tell them to manage the whole the whole context together or understand the global context and uh so so one of the reason is to help with quality.

s71
00:08:11.360 --> 00:08:14.240
We want to reduce the context as much as possible.

s72
00:08:14.240 --> 00:08:22.479
And the others are that if you reququery the model in a in a discussion, you resend every previous tokens.

s73
00:08:22.479 --> 00:08:27.360
So you pay for them as well again, which is far from ideal.

s74
00:08:27.360 --> 00:08:36.800
And you increase the the latency, so the time to first token or TTFT uh which makes which creates a very bad user experience.

s75
00:08:36.800 --> 00:08:51.040
So you may want to in our case manage our context for speed and for spending not just because the quality drops as we will see in our experiments and and in the results in the near future.

s76
00:08:51.040 --> 00:08:54.720
So how we do that to manage uh spending and and speed?

s77
00:08:54.720 --> 00:08:56.480
How do we manage our context?

s78
00:08:56.480 --> 00:08:58.959
We use compaction.

s79
00:08:58.959 --> 00:09:00.640
And the idea is just very simple.

s80
00:09:00.640 --> 00:09:02.480
Uh the idea of compaction is very simple.

s81
00:09:02.480 --> 00:09:13.440
is just to try to have the smallest context possible that contains the information to be able to answer the question and you drop or save the rest somewhere.

s82
00:09:13.440 --> 00:09:19.680
And to do compaction, even a good compaction, you don't necessarily have to have large language models.

s83
00:09:19.680 --> 00:09:30.800
You can start quite cheap with trivial tools like uh if you have if you use tools in your system um like a web search or just executing code you can automatically

s84
00:09:30.800 --> 00:09:32.320
truncate outliers.

s85
00:09:32.320 --> 00:09:43.600
So if you have like once tool that that produce 300 lines instead of 10 usually you can just truncate almost everything except the the head and tail the the beginning and the end and just

s86
00:09:43.600 --> 00:09:50.560
re write that it's truncated so that the model in the future can recall the tool if it's if it feels it lacks context.

s87
00:09:50.560 --> 00:09:56.880
You can use the the simplest approach that works the best to just use a sliding window or just trim.

s88
00:09:56.880 --> 00:10:16.160
Basically, use the last n number of of turns that the user sent which uh you need to determine by based on your your own system and your users and you can clear uh for for some specific tools depending on those that you implement you can just always clear most of the outputs.

s89
00:10:16.160 --> 00:10:19.760
So that's for when that's not even using language models.

s90
00:10:19.760 --> 00:10:25.519
And then you can use language models to basically spend tokens to save even more tokens.

s91
00:10:25.519 --> 00:10:29.680
And in this case, you don't even have to to use large language models.

s92
00:10:29.680 --> 00:10:37.839
You can even use smaller ones or even low super small local ones that that runs in like one MacBook to do a few techniques.

s93
00:10:37.839 --> 00:10:41.120
There are many techniques that exist for for compacting.

s94
00:10:41.120 --> 00:10:53.040
the those that had the most impact in our experiments where selective retention where the language models will just decide based on where the discussion go is going what to keep what to discard.

s95
00:10:53.040 --> 00:10:55.600
Then the simplest one here summarization.

s96
00:10:55.600 --> 00:11:07.120
So just summarize continuously summarize the some previous terms depending on your on on your own application and in the end you can do here what cloud code does.

s97
00:11:07.120 --> 00:11:14.399
So when it reaches the limit you just sum produce a summary of everything and reset completely with that summary.

s98
00:11:14.399 --> 00:11:19.120
And there are many more techniques I highlighted here those that work the best.

s99
00:11:19.120 --> 00:11:26.880
We will discuss them later on in this workshop and the actual results and experiment setup but those are the most successful techniques.

s100
00:11:26.880 --> 00:11:35.600
And I want to highlight also delta summarization that cloud code uses that is very useful when you spawn sub aents when you use sub aents.

s101
00:11:35.600 --> 00:11:45.600
It just means to keep a summary and updating the summary based on the new summary that you produce over time and then the the sub agent will just give that to the main agent.

s102
00:11:45.600 --> 00:11:51.519
But in our case, we don't use sub agents because the tutor works really well with just one main.

s103
00:11:51.519 --> 00:11:55.120
So we don't need to add complexity for that.

s104
00:11:55.760 --> 00:12:01.440
And lastly, after spending token to to save more tokens, you can also offload things.

s105
00:12:01.440 --> 00:12:05.120
Right now it's obviously memory and skills are super popular.

s106
00:12:05.120 --> 00:12:08.320
So you can of course offload to your memory.

s107
00:12:08.320 --> 00:12:14.959
So just saving text in your documentations and you can use what's been there for years now.

s108
00:12:14.959 --> 00:12:18.480
Uh retrieve augmented generation which is very powerful.

s109
00:12:18.480 --> 00:12:21.360
And as a side note we also compared with graph rag.

s110
00:12:21.360 --> 00:12:27.040
So everything even if I not I don't mention them we compared them and I highlight just the the best results here.

s111
00:12:27.040 --> 00:12:37.839
So we we compared graph rag with with rag here and it's in our case it just ended up being way costlier to set up and just tie on the results

s112
00:12:37.839 --> 00:12:42.880
because it basically was 100% based on our real user evaluations.

s113
00:12:42.880 --> 00:12:47.760
So we don't need to use graph rag but it's it depends on your own case.

s114
00:12:47.760 --> 00:12:57.680
If you have a very large data set with inter with relations and interconnected topics and things, it might be worth implementing.

s115
00:12:57.680 --> 00:13:01.120
So you definitely want to still test it.

s116
00:13:01.120 --> 00:13:17.040
And uh speaking of memory and of up offloading to files, this is uh as I said, basically just saving them locally or or in a server for for you to use, which means it's fully reversible because you don't lose anything.

s117
00:13:17.040 --> 00:13:19.120
you don't lose any ongoing discussion.

s118
00:13:19.120 --> 00:13:28.480
You just save them ready to be referred to in the future if the same student comes back and asks related questions.

s119
00:13:28.480 --> 00:13:40.800
And it's basically the the Tpetes idea of the LLM wiki uh which if you link with some sort of uh chunks so a version of rag it's pretty powerful

s120
00:13:40.800 --> 00:13:47.360
and it makes your system become quite cheap and durable and it's easy to inspect from both humans and agents.

s121
00:13:47.360 --> 00:13:49.200
So it's really interesting.

s122
00:13:49.200 --> 00:13:51.760
More specifically, it looks like this in our case.

s123
00:13:51.760 --> 00:13:54.160
So we have just chunks.

s124
00:13:54.160 --> 00:14:03.360
We save everything into chunks and we cross-link the chunks with pointers and then these chunks are linked to raw data files from where they come from.

s125
00:14:03.360 --> 00:14:07.600
Then we have one index that will just map all these chunks.

s126
00:14:07.600 --> 00:14:14.160
So just a link to all the chunks and some context of what it is about and the agent will just see this index.

s127
00:14:14.160 --> 00:14:17.519
So it sees it's like I think it was 450 tokens.

s128
00:14:17.519 --> 00:14:37.120
So it's very small and it just sees that index and then uh based on the user question if it seems to be related to some user specific question that it may exists in the memory it will scan the index it will go back to the chunk if it's enough it will answer based on the junk the chunk

s129
00:14:37.120 --> 00:14:42.320
retrieved if it's not enough it can even go back to the raw data to have even more information.

s130
00:14:42.320 --> 00:14:47.839
So basically it's just the best way to pull context accordingly to the task complexity.

s131
00:14:47.839 --> 00:14:54.399
So if the task is complex you will pull more and if it's simple you'll p pull less.

s132
00:14:54.800 --> 00:15:06.399
And a parenthesis on this what we've seen working with our clients and just building this in general is that right now everyone is is converging towards having more and smaller

s133
00:15:06.399 --> 00:15:07.120
skills.

s134
00:15:07.120 --> 00:15:26.320
So you have it's it's way better to build small very precise skills that refer to each other's skills to uh to save on context and just load skills one by one and even be able to spawn a sub agent with one dedicated skill context instead of just basically to save context.

s135
00:15:26.320 --> 00:15:28.399
It's a well the idea of progressive disclosure.

s136
00:15:28.399 --> 00:15:32.079
So you just load what you need right now.

s137
00:15:32.240 --> 00:15:34.800
Okay, just to go back now on on compaction.

s138
00:15:34.800 --> 00:15:40.560
This is what the the talk is about because memory we have made some experiments but we couldn't really fit here.

s139
00:15:40.560 --> 00:15:42.480
It was a bit too much.

s140
00:15:42.480 --> 00:16:01.440
And uh when talking about compaction there's an important problem or solution that uh that appeared recently uh well not recently but was way more popularized is the uh the the main problem is that first the main problem is that you when you ask a follow-up question

s141
00:16:01.440 --> 00:16:05.040
you need to recomputee all previous tokens every time.

s142
00:16:05.040 --> 00:16:12.399
So you just end up paying twice for the same tokens or three times or four times if the conversation is going which is obviously far from ideal.

s143
00:16:12.399 --> 00:16:17.199
So what providers do nowadays is to offer prom caching.

s144
00:16:17.199 --> 00:16:35.759
So you will it they will save the embedding and and KV cache and I won't enter into the details but they will premputee they will have saved a lot of the the compute for some tokens and you can just reload them and what's interesting for us is that these already sent token that you reuse are much much

s145
00:16:35.759 --> 00:16:49.279
much cheaper uh specifically it can go up to 50 times cheaper with some API like deepseek uh which we will discuss in the experiments and what that means is that If you send a very long context,

s146
00:16:49.279 --> 00:16:59.839
you will pay just one 15th of the the price per token and just pay the full price of the qu the new user question or the new interaction.

s147
00:16:59.839 --> 00:17:11.360
And that's a problem for compaction because when you are compacting, summarizing, doing any transformation to this context, the the provider cannot use the cache because it it's a new context.

s148
00:17:11.360 --> 00:17:16.160
It doesn't the model is not intelligent enough to understand it's the same topic change.

s149
00:17:16.160 --> 00:17:17.600
it it just cannot use the cache.

s150
00:17:17.600 --> 00:17:20.959
So you will pay full price for these new transform tokens.

s151
00:17:20.959 --> 00:17:27.360
So here what it means is that you need to to compress to for for compaction to be worthwhile.

s152
00:17:27.360 --> 00:17:32.160
You need to compress by more than 50 times the context.

s153
00:17:32.160 --> 00:17:38.480
So it can be quite difficult in some cases without losing quality.

s154
00:17:38.480 --> 00:17:53.360
So caching is truly a gamecher especially because nowadays almost all APIs offer it very easy to use and typically they they save cost on like on 90% of the cost when you use caching

s155
00:17:53.360 --> 00:18:04.559
but in some case as I said it can go up to way more than that and not only that it helps with cost but caching cache tokens are also already computed so it's way faster to get

s156
00:18:04.559 --> 00:18:12.480
the first the answer So ultimately it means that summarization is potentially a trap.

s157
00:18:12.480 --> 00:18:22.720
You may not want to use it at all or you you may want to just use it very specifically which is what the most serious harnesses do nowadays.

s158
00:18:22.720 --> 00:18:35.840
cloud code, C codeex and all of them use context caching but also use a different method of compactions and we know that because obviously of the leak and then because codeex is open source

s159
00:18:35.840 --> 00:18:43.039
and then the APIs also provide ways to manage context directly and caching directly.

s160
00:18:43.039 --> 00:18:54.799
So it's it becomes when you build yourself a harness like we do with the AI tutor, it becomes really interesting to understand when to use which technique and test them obviously.

s161
00:18:55.600 --> 00:18:57.360
So where is this going?

s162
00:18:57.360 --> 00:19:08.880
It mean it means that you don't you don't want to just compact you don't want to summarize everything any time because it may kill the cache and you won't be able to use it.

s163
00:19:08.880 --> 00:19:22.160
So some best guidance that we found is that obvious obviously when the user seems to talk about a very different topic, you may want to refresh the session to clean it

s164
00:19:22.160 --> 00:19:23.679
to clear it.

s165
00:19:23.679 --> 00:19:33.840
Um when you scope or files, use what I described with progressive disclosure to just show the smallest amount of context possible.

s166
00:19:33.919 --> 00:19:38.640
You want to clear every old tool outputs that are not useful anymore.

s167
00:19:38.640 --> 00:19:40.720
You may want to compact in some case.

s168
00:19:40.720 --> 00:19:45.200
We will see that in the experiments in a few minutes.

s169
00:19:45.200 --> 00:19:48.720
And you may want to optimize for cache hits.

s170
00:19:48.720 --> 00:19:51.760
So just having the model use be able to use its cache.

s171
00:19:51.760 --> 00:19:57.360
And regarding that providers are constantly improving their feature set to manage cache.

s172
00:19:57.360 --> 00:20:02.160
So you just need to uh stay current and follow what what the API allows nowadays.

s173
00:20:02.160 --> 00:20:13.919
But they all provide different methods and um you will have access to the slides but there's I put a link earlier in the in the first few slides on a very interesting article regarding regarding

s174
00:20:13.919 --> 00:20:16.320
prompt caching that I recommend checking out.

s175
00:20:16.320 --> 00:20:23.200
It's like on the sixth or seventh slide but you will have the link to the slides.

s176
00:20:23.200 --> 00:20:31.120
And lastly, you may want to use a model router to optimize especially cost on various tasks.

s177
00:20:31.280 --> 00:20:37.520
And most importantly, and what the majority of people don't do, you want to log everything.

s178
00:20:37.520 --> 00:20:38.159
It's super easy.

s179
00:20:38.159 --> 00:20:40.880
You just ask cloud to implement opic and track everything.

s180
00:20:40.880 --> 00:20:42.960
It's you don't have anything to do.

s181
00:20:42.960 --> 00:20:48.640
So it's definitely worthwhile to to implement and you can track cache it rate.

s182
00:20:48.640 --> 00:20:52.320
You can track user frustration which we've seen that cloud does.

s183
00:20:52.320 --> 00:20:56.320
So I keep uh telling it when I'm not happy.

s184
00:20:56.320 --> 00:21:07.200
And uh you you may want to log for some um abnormally long outputs or any weird behavior that a small language model could detect.

s185
00:21:07.200 --> 00:21:18.240
And so all of this together is the is context engineering which basically means to decide what the model sees every time you call it.

s186
00:21:18.240 --> 00:21:31.520
And to us for the AI tutor, it means to decide what to keep in the current context window in order to optimize the caching what to drop or compact and when to do that.

s187
00:21:31.520 --> 00:21:42.480
Uh and those two first things are exactly what what we studied in many experiments that we that my colleague Omar will share with you right now.

s188
00:21:42.480 --> 00:21:47.039
And uh so you can follow along with the experiments on the the QR code link.

s189
00:21:47.039 --> 00:21:55.520
It's the hugging face that I mentioned earlier and in there there's also a link to the repo and everything.

s190
00:21:56.480 --> 00:21:56.880
All right.

s191
00:21:56.880 --> 00:22:00.880
So, welcome to this second part of the of the workshop.

s192
00:22:00.880 --> 00:22:12.880
So now that we have an overall idea of what context engineering is and what are the different techniques that we can apply to our agents uh now what we want to do here is

s193
00:22:12.880 --> 00:22:18.960
see how they actually perform in in our case for our AI tutor.

s194
00:22:18.960 --> 00:22:23.760
I will first start by describing a little bit about how the AI tutor works.

s195
00:22:23.760 --> 00:22:29.760
So the system design and then I will follow up with the initial experiments that we did.

s196
00:22:34.799 --> 00:22:35.280
All right.

s197
00:22:35.280 --> 00:22:38.240
So, the AI tutor is actually very simple.

s198
00:22:38.240 --> 00:22:48.159
It's one agent that it's a React type of agent that just loops over over tool calls and thinking blocks.

s199
00:22:48.159 --> 00:22:52.880
And here we just create a very simple one using the lang chain library.

s200
00:22:52.880 --> 00:23:06.000
So we use the create agent method with the in-memory saver because we in this case we don't save past conversations we just use the the current history and to customize

s201
00:23:06.000 --> 00:23:17.919
this agent we use the middleware feature of lang chain where you we can add different uh features that can change the behavior of the agent at runtime.

s202
00:23:17.919 --> 00:23:30.559
So in this case, we want to summarize for example and clear the tool outputs or also in our case have the the user be able to choose the the source

s203
00:23:30.559 --> 00:23:38.480
uh uh the sources the like which lessons which courses to to use to to answer.

s204
00:23:38.880 --> 00:23:41.840
To do that we also add two different tools.

s205
00:23:41.840 --> 00:23:49.679
So we have the first one the retrieve tutor context which which uses a very classic hybrid search pipeline.

s206
00:23:49.679 --> 00:23:59.679
So um semantic search along with keyword search and we combine both results to get the the best possible list of chunks.

s207
00:23:59.679 --> 00:24:13.440
And then recently we also added the second one which is letting the agent actually browse the file system the knowledge base uh just like we can just like coding agents can in the

s208
00:24:13.440 --> 00:24:26.320
browsing your codebase for example and we also borrow from the idea of karpathy by creating a wiki and helping the agent basically browse uh this knowledge base more easily.

s209
00:24:26.320 --> 00:24:30.320
I will come back to these tools in a few in a few me moments.

s210
00:24:30.320 --> 00:24:42.559
We also add a fast API app to wrap the whole system with an endpoint and we add a nextgs UI to let students use the DII tutor.

s211
00:24:43.279 --> 00:24:45.600
So I will talk a little bit about the first tool.

s212
00:24:45.600 --> 00:24:47.679
So we have a large corpus.

s213
00:24:47.679 --> 00:24:53.919
So we have all of the lessons from all the different courses that we created over the past two years.

s214
00:24:53.919 --> 00:25:06.159
uh and also documentation from various public open source libraries like lang chain, lama index and even documentation from uh openai.

s215
00:25:06.159 --> 00:25:16.159
So they they make available available all of the markdown files from like how to use the open API, how to use codeex and we also have like cloud code documentation.

s216
00:25:16.159 --> 00:25:23.520
So it's a very big corpus that has over 8 million tokens and of course this cannot fit in a single context window.

s217
00:25:23.520 --> 00:25:30.080
So we have to store it in a way where we can retrieve the most important or most relevant information.

s218
00:25:30.080 --> 00:25:38.480
So in this case the agent receives a question and the user can then choose well beforehand the user can choose specific source.

s219
00:25:38.480 --> 00:25:41.279
So we can filter this knowledge base.

s220
00:25:41.279 --> 00:25:45.440
It makes it better for um to improve recall.

s221
00:25:45.440 --> 00:25:49.520
So precision to get the most relevant information.

s222
00:25:49.520 --> 00:25:51.600
Then we do hybrid search.

s223
00:25:51.600 --> 00:25:54.799
So this is very classic um hybrid search.

s224
00:25:54.799 --> 00:25:57.760
We use embedding model.

s225
00:25:57.760 --> 00:26:09.520
In this case it's a coherent coherent model with BM25 for the keyword index to get the the most relevant top 30 uh chunks.

s226
00:26:09.520 --> 00:26:25.520
Then we merge the two results from the semantic similarity and the keyword search and we then rerank to the top five most relevant chunks and that's that's what return to the agent.

s227
00:26:25.520 --> 00:26:27.840
We also have a limit of 100,000 tokens.

s228
00:26:27.840 --> 00:26:29.440
So we don't want to.

s229
00:26:29.440 --> 00:26:40.000
So let's say for example we we go over we just remove the last few chunk uh the last chunks that make it so that we don't cross that threshold.

s230
00:26:40.000 --> 00:26:43.279
So these numbers the this configuration actually is not random.

s231
00:26:43.279 --> 00:26:48.000
We did experiments uh to optimize this pipeline.

s232
00:26:48.000 --> 00:26:52.720
I'm not going to talk about it but it's in one of the courses that we share.

s233
00:26:52.720 --> 00:26:57.600
Uh basically we just want to try as many configurations as possible and improve recall.

s234
00:26:57.600 --> 00:27:06.559
So we measure did we retrieve the correct page uh in our knowledge base and we just choose the best uh settings.

s235
00:27:07.120 --> 00:27:19.440
So like I said it uh yeah in this case it's very precise very it's very good but what if the agent needs to browse the whole knowledge base to get the best possible answer.

s236
00:27:19.440 --> 00:27:31.600
So let's say I want to learn about codeex and I want to learn about cloud code how can I best use uh those two tools and also like if we need to have v various documentation

s237
00:27:31.600 --> 00:27:47.120
pages in context is this the best possible uh tool and there's a paper uh I I link it here in the in in the slides but it's a yeah it's a paper that shows that basically letting the agent browse

s238
00:27:47.120 --> 00:27:53.200
the the knowledge base can be very bene beneicial uh so you can look at it if you want afterwards.

s239
00:27:53.200 --> 00:27:57.360
Very very slow to load.

s240
00:27:57.679 --> 00:28:13.200
Uh so that's why we ended up creating this second tool the run keep run knowledgebased command where the agent can browse u the knowledge base the file system using bash commands.

s241
00:28:13.200 --> 00:28:19.200
So first of all what we did first is create these uh three different u folders.

s242
00:28:19.200 --> 00:28:23.039
So we have the raw folder with all the different markdown files.

s243
00:28:23.039 --> 00:28:30.559
So all the lessons from the different courses the documentation from the different open open source libraries.

s244
00:28:30.559 --> 00:28:46.720
We we also have this generated uh folder with this is the generated automatically with basically just the the the titles of each of the markdown files so it's easier for the agent to to find

s245
00:28:46.720 --> 00:28:48.799
um relevant information.

s246
00:28:48.799 --> 00:28:54.240
And then we also have the wiki where uh in this case it was cl code that created this.

s247
00:28:54.240 --> 00:29:04.799
It basically reads all of the raw files in the in the folder in the raw folder and it creates a very uh concise uh set of files.

s248
00:29:04.799 --> 00:29:10.080
So topics, frameworks uh from the different source uh sources.

s249
00:29:10.080 --> 00:29:12.960
So for example, I can have a topic related to finetuning.

s250
00:29:12.960 --> 00:29:18.320
So in this case, the agent will be able to find all of the different raw files related to finetuning.

s251
00:29:18.320 --> 00:29:22.960
For example, here we let the agent only read.

s252
00:29:22.960 --> 00:29:24.960
So this is when we actually deploy it.

s253
00:29:24.960 --> 00:29:28.559
So this is the one you can try on the on the space here.

s254
00:29:28.559 --> 00:29:30.880
The agent can only read the knowledge base.

s255
00:29:30.880 --> 00:29:32.640
It cannot modify it.

s256
00:29:32.640 --> 00:29:39.120
Uh so we only allow these bash commands that basically cannot uh modify the the file system.

s257
00:29:39.120 --> 00:29:41.760
We also put some limits around this.

s258
00:29:41.760 --> 00:29:52.000
So for example, if a command lasts over 8 seconds, we can just return an error or let the agent execute something else because it's taking too much time.

s259
00:29:52.000 --> 00:29:56.799
And we also cap the tool outputs to 40,000 uh characters.

s260
00:29:56.799 --> 00:30:05.039
So in this case, for example, if a lesson is over this amount, what the agent can do is then okay, so I just got the first 40,000.

s261
00:30:05.039 --> 00:30:10.080
Let me let me do a follow-up command to get the last uh piece of the of the lesson.

s262
00:30:10.080 --> 00:30:14.960
For example, uh we also limit the number of the of commands.

s263
00:30:14.960 --> 00:30:26.799
We actually never see the agent go over this limit of 20 commands per turn, but this is just a a fallback uh in case it takes too too much time to to answer.

s264
00:30:26.799 --> 00:30:33.919
And we also sandbox of course the agent to only browse the knowledge base this specific folder.

s265
00:30:33.919 --> 00:30:36.080
Like I said, we create this offline.

s266
00:30:36.080 --> 00:30:51.360
So the the three different file folders we create this once or every time we want to add a new uh a new course for example we we we tell cloud code can you add a new uh can you add this to the raw raw folder and also create new topics

s267
00:30:51.360 --> 00:30:53.840
around this new new content.

s268
00:30:53.840 --> 00:31:02.080
So we do this once and then we deploy it so that the agent u can can browse it with our current system prompt.

s269
00:31:02.080 --> 00:31:06.240
We can see that it's uh used almost every time.

s270
00:31:06.240 --> 00:31:12.240
So for almost 90% of the terms we can tweak this to make it use it less uh or more.

s271
00:31:12.240 --> 00:31:15.919
We didn't optimize for for this specifically.

s272
00:31:15.919 --> 00:31:30.320
And I guess the most interesting aspect of doing this is that we measured the precision the recall of using this tool uh actually turning turning it off and we actually got the same amount of

s273
00:31:30.320 --> 00:31:30.880
recall.

s274
00:31:30.880 --> 00:31:43.200
So just using the first tool was enough to get all the relevant information and just having this second tool was just 50% slower.

s275
00:31:43.200 --> 00:31:48.240
Basically, it's faster without this tool because it does less tool calls.

s276
00:31:48.240 --> 00:31:58.799
Um, and yeah, we we basically didn't see any improvement on uh on re on answering with the correct uh documentation.

s277
00:31:58.799 --> 00:32:10.240
It it was fun to add but we yeah we didn't see any any benefit and one reason for that is that we tested using real world well basically the questions we get from students

s278
00:32:10.240 --> 00:32:17.760
and basically those questions were complex enough we guess to actually benefit from using this new new setup.

s279
00:32:17.760 --> 00:32:27.200
Um so this is what this is how initially uh our AI tutor managed its context.

s280
00:32:27.200 --> 00:32:33.919
Uh we we started with this because uh it looked fast, it looked good.

s281
00:32:33.919 --> 00:32:36.240
Uh we didn't actually measure anything.

s282
00:32:36.240 --> 00:32:37.840
It was just like oh it looks good.

s283
00:32:37.840 --> 00:32:40.640
Okay, we will just set it set it like this.

s284
00:32:40.640 --> 00:32:54.480
And so basically we have these three different um context engineering techniques where we clear the outputs uh after 5,000 tokens uh but we at least keep the last five ones.

s285
00:32:54.480 --> 00:33:01.840
So this was basically a way to keep a very small context uh during a conversation.

s286
00:33:01.840 --> 00:33:04.799
We also added the the capacity to summarize.

s287
00:33:04.799 --> 00:33:17.600
So after 30,000 tokens uh the two well the system summarizes the uh the history but we at least keep the last 20 messages to make sure that uh those are very accurate

s288
00:33:17.600 --> 00:33:19.200
those messages.

s289
00:33:19.200 --> 00:33:22.159
And we also have the source preference.

s290
00:33:22.159 --> 00:33:32.080
So that that just allows people to choose what uh sources to use uh when the thetutor answers.

s291
00:33:32.399 --> 00:33:39.279
But like I said, these are unproven unproven defaults and we actually want to know what actually works best.

s292
00:33:39.279 --> 00:33:46.240
Um so I guess you you can Lou showed this at the beginning, but you can access the tutor live.

s293
00:33:46.240 --> 00:33:49.200
I'm just going to show this very quickly.

s294
00:33:49.200 --> 00:33:51.519
This is the huging face.

s295
00:33:51.519 --> 00:33:55.039
So this is a separate UI just to show you.

s296
00:33:55.039 --> 00:34:02.080
We also have the the chat bubble one on the lessons on the course themselves itself.

s297
00:34:02.080 --> 00:34:11.839
Uh and here on the on the left you can choose the different uh sources to use, enable them or disable them and then you can uh yeah send your your query.

s298
00:34:11.839 --> 00:34:27.440
So as you can see here I can just uh send a new request and we see Gemini in this case Gemini 3.5 flash use its reasoning and use its uh capacity to to do tool calling and to answer

s299
00:34:27.440 --> 00:34:58.320
and I guess uh here there's a I think it's just the internet bugging But we should have a response and yeah the code is open source so you can use your favorite coding agent to explore the codebase

s300
00:34:58.320 --> 00:35:03.200
and uh and learn about how we implemented this specifically.

s301
00:35:03.200 --> 00:35:09.839
So we have the different activity u the the activity that the the the model did.

s302
00:35:09.839 --> 00:35:16.720
So for tool calls it thought four times and it use 10 sources and we have the the final answer.

s303
00:35:19.040 --> 00:35:25.200
So everything that uh so every time the student uses this chatbot we actually log everything.

s304
00:35:25.200 --> 00:35:39.760
So we for every single turn we have uh the input tokens, the the output tokens, how many of them were cached, uh what was the cost, what time it took to to get the first token, how many tool calls it did

s305
00:35:39.760 --> 00:35:44.720
and if we actually if the system actually did something around summarization.

s306
00:35:44.720 --> 00:35:52.000
So this is very useful and that's uh what we are going to use when uh measuring the uh the different techniques.

s307
00:35:52.000 --> 00:35:54.000
So why do we need to measure?

s308
00:35:54.000 --> 00:35:59.040
Well, because the the techniques that Lou showed all sound very smart.

s309
00:35:59.040 --> 00:36:04.720
So you you might think that they are very uh useful, but sometimes they're not.

s310
00:36:04.720 --> 00:36:10.800
And uh as we as we discovered with our experiments, actually it might be detrimental.

s311
00:36:10.800 --> 00:36:23.440
So because of the the way APIs cache the tokens when they are um when they are sent and it's also difficult to to know in advance what what is best to use.

s312
00:36:23.440 --> 00:36:34.400
So before I go into the experiments, I just want to define a few words because we're I'm going to use these words throughout the uh the the presentation.

s313
00:36:34.400 --> 00:36:40.480
So a preset is basically uh the way the AI tutor was set up in the in the experiment.

s314
00:36:40.480 --> 00:36:49.680
So for example, it can be in this preset we did summarization at this amount of tokens or in this preset we we used sliding window for example.

s315
00:36:49.680 --> 00:36:51.920
So that's what a preset is.

s316
00:36:51.920 --> 00:36:53.680
we have the different tasks.

s317
00:36:53.680 --> 00:37:12.320
So at task type uh for these initial experiments we only did two tasks single turn and multiple turns also sessions uh called sessions and one run is basically just running one preset on a task and then you get the run.

s318
00:37:12.320 --> 00:37:16.400
Um the bundle is just the the result.

s319
00:37:16.400 --> 00:37:23.440
So it's just a JSON file with all the different uh metrics that we save to the disk.

s320
00:37:24.079 --> 00:37:26.960
So first task single turn.

s321
00:37:26.960 --> 00:37:32.000
Uh so these are question and answers and we didn't generate this.

s322
00:37:32.000 --> 00:37:33.280
It's not synthetic.

s323
00:37:33.280 --> 00:37:48.720
Uh we actually I I had codeex scrape all of the uh questions and responses from our website where students can ask questions and get answers from uh members of the staff.

s324
00:37:48.720 --> 00:37:54.720
Uh and that's how I got this initial we got this initial um data set.

s325
00:37:54.720 --> 00:38:05.680
uh we cleaned the data set and only used 60 60 pairs uh because we saw that some of the questions weren't good for the type of task for example

s326
00:38:05.680 --> 00:38:19.440
uh there were old questions about previous versions of some libraries and we we might not I mean the right now if the tutor answers it's not going to be using this old version of the library so we just

s327
00:38:19.440 --> 00:38:30.800
uh remove some of the questions some duplicate ones and and and we got this first uh data set and what we measure is the retrieval.

s328
00:38:30.800 --> 00:38:44.720
So did we retrieve the correct is did the total retrieve the correct lesson for example this is done automatically we can see uh just by looking at the code did we did we use the correct uh lesson or not

s329
00:38:44.720 --> 00:38:51.440
and we also look at did do we have the correct facts in the answer or the right kind of response in the answer.

s330
00:38:51.440 --> 00:38:57.440
These two uh are actually graded using using an LLM.

s331
00:38:57.440 --> 00:39:07.040
Uh you can use APIs to do it but uh right now I think the best way to do it is to just use use your uh code code subscription or your codec

s332
00:39:07.040 --> 00:39:11.839
subscription because it's cheaper than using the APIs.

s333
00:39:12.720 --> 00:39:15.920
We also have the second task the session.

s334
00:39:15.920 --> 00:39:20.800
So multi-turn um conversations back and forth.

s335
00:39:20.800 --> 00:39:29.359
uh here what we want to know is is the AI tutor able to recall facts after multiple terms.

s336
00:39:29.359 --> 00:39:35.760
So this is a bit uh in this case we do use some generated uh content.

s337
00:39:35.760 --> 00:39:39.440
So we do in generate facts that we put at the beginning.

s338
00:39:39.440 --> 00:39:44.079
So we have a student like a fake student state uh fact.

s339
00:39:44.079 --> 00:39:56.079
So for example, I want to learn about rag and then we stuff the conversation with just filler messages because we just want yeah we just want to have a lot of messages

s340
00:39:56.079 --> 00:40:04.640
and then we have um a probe which is just the student asking a question again.

s341
00:40:04.640 --> 00:40:09.040
So for example it can be uh what should I learn today?

s342
00:40:09.040 --> 00:40:17.920
And since the uh I'm going to go through it in the next slide, but uh yeah, let's let's let's the example.

s343
00:40:17.920 --> 00:40:25.440
So for example, here at turn one, we we uh we have the student the fake student implement a state of fact.

s344
00:40:25.440 --> 00:40:28.880
So it the student wants to learn about rag evaluation.

s345
00:40:28.880 --> 00:40:32.000
Um and that that's basically the fact.

s346
00:40:32.000 --> 00:40:45.040
Then we just add a lot of messages uh filler filler messages and then at at some point we have the student say for example what topic should I learn about today

s347
00:40:45.040 --> 00:40:52.960
and then what we expect the AI tutor to to to say is that uh the student should learn about rag evaluation.

s348
00:40:52.960 --> 00:40:57.359
So more specifically heat rate and MR for example.

s349
00:40:57.359 --> 00:41:05.440
We also have a gate part in the evaluation where let's say we are testing the summarization technique.

s350
00:41:05.440 --> 00:41:09.599
Uh we actually just want to know did summarization actually happen or not.

s351
00:41:09.599 --> 00:41:18.560
Um so this is this is this is like one example of one task in the sessions uh data set in the sessions task.

s352
00:41:18.560 --> 00:41:21.200
Now we have the evaluation hardness.

s353
00:41:21.200 --> 00:41:34.079
Um so the main uh harness I guess the the main function is the run the run task the run battery function that just uh runs this uh this task.

s354
00:41:34.079 --> 00:41:47.040
Um we have the grading like I said we it can either be a code check so did we retrieve the correct lesson or not and we can also have the LM as a judge and in this case we use

s355
00:41:47.040 --> 00:41:49.839
the subscription of code.

s356
00:41:49.839 --> 00:41:52.000
We also have the check triggers aspect.

s357
00:41:52.000 --> 00:41:57.280
So this is just a check to to see that if the evaluation went uh good or not.

s358
00:41:57.280 --> 00:41:58.800
Did we actually compact or not?

s359
00:41:58.800 --> 00:42:05.760
This is this is just to make sure that the the run is actually good and we can save it.

s360
00:42:05.760 --> 00:42:17.119
And then we have a generated report uh to see like what was the latency, what was the time to first token and every metric that we can uh measure.

s361
00:42:17.599 --> 00:42:21.680
So yeah, basically we can evaluate everything and then grade it afterwards.

s362
00:42:21.680 --> 00:42:25.839
Uh we we can run it once and grade it afterwards.

s363
00:42:25.920 --> 00:42:34.800
So what what we run so we run 11 presets uh and we change them uh for for each experiment.

s364
00:42:34.800 --> 00:42:37.520
So we have the the full history.

s365
00:42:37.520 --> 00:42:40.000
So these are the the main ones the full history.

s366
00:42:40.000 --> 00:42:43.839
So this is the case where we don't don't touch the context.

s367
00:42:43.839 --> 00:42:47.440
We we leave everything as is in the history.

s368
00:42:47.440 --> 00:42:53.200
And then we also have the production that I showed at the beginning the defaults that we have.

s369
00:42:53.200 --> 00:42:58.560
Uh so these are the reference points and then we have these six techniques that I want to compare.

s370
00:42:58.560 --> 00:43:12.880
So sliding window prompt compression uh selective retention and the other ones and I what I want to see is just what uh memory recall recall do I get if I keep everything else fixed.

s371
00:43:12.880 --> 00:43:18.960
So I use the same model the same prompt the same tools the same data set what's the difference?

s372
00:43:18.960 --> 00:43:22.400
Uh now ju now just doing this was a bit expensive.

s373
00:43:22.400 --> 00:43:28.480
I I didn't expect this to to get over $500, but it did.

s374
00:43:28.480 --> 00:43:34.720
And that's one of the reason we did afterwards follow-up experiments using cheaper models.

s375
00:43:34.720 --> 00:43:38.079
But my colleague Sani will talk about this.

s376
00:43:38.079 --> 00:43:43.280
Uh so what ARM so what basically what preset actually won.

s377
00:43:43.280 --> 00:43:47.599
Uh, and so these are our um our results.

s378
00:43:47.599 --> 00:44:02.640
And as you can see, h we weren't expecting this, but basically not touching um the context was actually the best the best strategy for recovering this fact over time over multiple

s379
00:44:02.640 --> 00:44:04.240
uh messages.

s380
00:44:04.240 --> 00:44:13.200
Um you can see that the production so the the defaults that we thought were good enough were actually not the best.

s381
00:44:13.200 --> 00:44:15.920
Uh not doing anything is actually better.

s382
00:44:15.920 --> 00:44:23.920
H we did two uh two different experiments where one was just one trial and the second was um two trials.

s383
00:44:23.920 --> 00:44:42.880
So we have like a more statist I I guess it's it's better but the uh the numbers are not like can might not be accurate because it's just one trial and two trials but I guess the most interesting thing is just the order in which the techniques ended up being in the table.

s384
00:44:42.880 --> 00:44:49.359
So in this case keeping everything wins on the memory side but what about the cost?

s385
00:44:49.359 --> 00:44:54.560
Uh this is what the production cost uh was for the single turn and the session.

s386
00:44:54.560 --> 00:45:02.960
So almost 50 cents uh for a single turn and 24 cents uh for the multi-turn or each turn.

s387
00:45:02.960 --> 00:45:15.280
Uh we actually had very good memory uh recall for pretty much all the techniques in the single turn because in single turn you don't have enough tokens to actually fire up the different strategies.

s388
00:45:15.280 --> 00:45:21.760
Uh so for one response you don't need to do summarization compaction or anything like that.

s389
00:45:21.760 --> 00:45:25.760
Uh so that's why you you see um high numbers.

s390
00:45:25.760 --> 00:45:35.680
Uh but as you can see after in the multi-turn uh task you can see that the quality degraded to 38%.

s391
00:45:35.760 --> 00:45:39.760
uh and we if we compare this to the full history.

s392
00:45:39.760 --> 00:45:51.520
So here we don't touch the context uh the history of the model we can see that not touching uh is actually cheaper, it's faster and we have better recall overall.

s393
00:45:51.520 --> 00:45:57.760
So keeping everything wins on on all of these three fronts.

s394
00:45:57.760 --> 00:46:16.800
Uh so why does it so why is it actually uh like why is it uh why do we have less latency like what's we want to we wanted to understand that and it's basically because if you

s395
00:46:16.800 --> 00:46:26.240
remove the tool outputs consistently then the agent needs to rerieve uh afterwards for information it already had.

s396
00:46:26.240 --> 00:46:40.319
So you're just making the agent uh do more tool calls and that's why it ended up costing more uh and yeah using more tokens and having less uh uh less memory recall.

s397
00:46:40.319 --> 00:46:51.280
So these are the results we initially got using uh Gemini 3.5 with the uh this this data set of 11 to 13 turns is not huge.

s398
00:46:51.280 --> 00:46:53.599
Uh there's not many messages.

s399
00:46:53.599 --> 00:47:03.520
That's why we wanted we now want uh in the follow-up part do more different experiments that my colleagues already will will show you.

s400
00:47:03.520 --> 00:47:07.760
So yeah, let me in let me introduce you to something.

s401
00:47:07.760 --> 00:47:09.760
Yeah.

s402
00:47:10.480 --> 00:47:21.040
Um, so this is going to be my part and um, as we ended on the note where Omar just said that it costed it c um, it costed us almost $600

s403
00:47:21.040 --> 00:47:24.560
to um, run the evals that we ran.

s404
00:47:24.560 --> 00:47:35.760
One question we were trying to um, answer with this extended evaluation was um, so when do you uh, when does compaction actually matter or should you actually compact or not?

s405
00:47:35.760 --> 00:47:44.960
because we clearly saw that when we have um you know the full answers in the window it works really well but it costs a lot of money.

s406
00:47:44.960 --> 00:47:49.839
So um I tried to you know do this evaluation in like three sorts of context.

s407
00:47:49.839 --> 00:47:52.400
Uh first one was cash chats.

s408
00:47:52.400 --> 00:47:57.520
So you know like when you're chatting with Gemini you'd be able to see that is the cash chat option.

s409
00:47:57.520 --> 00:48:00.160
The second version is going to be document plus tool.

s410
00:48:00.160 --> 00:48:10.640
So if you're pasting like a long document in the AI tutor or if there's just tool output what happens then and finally you know if you go local or if you scale this evaluation

s411
00:48:10.640 --> 00:48:23.359
how well is it going to work out so um should you ever compact so before my part Omar just showed that on Gemini 3.5 flash um keeping everything one

s412
00:48:23.359 --> 00:48:33.680
um but uh you know the why did we come up with that question was because um you know full history on a frontier model like Gemini would be very very expensive.

s413
00:48:33.680 --> 00:48:41.440
Um so um you know with all this extended experiment we are trying to figure out was it actually worth it or not.

s414
00:48:41.440 --> 00:48:47.920
But uh before we start that I wanted to just talk about the different context we see in our AI tutor application.

s415
00:48:47.920 --> 00:48:59.359
Um so first one is going to be like a long chat history uh where you know we have um a long chat but all of the details what the the students are asking or they're chatting about they can get buried in.

s416
00:48:59.359 --> 00:49:01.520
Um the second is going to be a pasted document.

s417
00:49:01.520 --> 00:49:05.839
So you know we do have a lot of students who are going to just like copy paste a lot of documents there.

s418
00:49:05.839 --> 00:49:10.000
Um so you know um and also because we have like limited uh context window.

s419
00:49:10.000 --> 00:49:11.839
How does that fit?

s420
00:49:11.839 --> 00:49:15.760
And then it's going to be you know different tools that we use internally.

s421
00:49:15.760 --> 00:49:26.079
Um but this is just going to be like bunch of logs that the tools have and since you know these are like different context each of them need some sort of different fix.

s422
00:49:26.079 --> 00:49:32.079
So you know we had to evaluate it for all different um contexts that we had.

s423
00:49:32.079 --> 00:49:45.359
So um you know the first obvious thing uh looking at the cost uh we were like okay let's try out a cheaper model and see you know um does it do better you know what sort of techniques work on that does compaction work on it or not.

s424
00:49:45.359 --> 00:50:00.960
So um deepseek v4 flash was an obvious option and we were like we will uh try this out on this and see um how well it works out and you know you can see there is um you know a drastic cost difference between Gemini and Deepseek here.

s425
00:50:00.960 --> 00:50:11.920
So here we can see um you know there's a drastic difference between cost when we checked the performance on deepseek v4 and the main reason was that that we were getting a cash discount.

s426
00:50:11.920 --> 00:50:22.240
So the cash discount on GE um deepseek was um you know 50x as compared to Gemini and even in this um you know setting we saw that keeping all

s427
00:50:22.240 --> 00:50:24.160
um you know all of the context still one.

s428
00:50:24.160 --> 00:50:30.559
So you know we were just getting the best performance um in that case for deepseek as well.

s429
00:50:30.559 --> 00:50:41.119
So, but you know the the plus thing from this experiment was that we figured out a cheaper model but um you know uh now we wanted to see that even though we have everything

s430
00:50:41.119 --> 00:50:51.440
uh you know keeping everything uh makes it cheaper but does it remember better you know does uh keeping all of the context remember all of the details the student might be asking us.

s431
00:50:51.440 --> 00:50:55.280
So um you know this is how I tested out the memory of the model.

s432
00:50:55.280 --> 00:51:05.760
Um so we have conversation within our system where you know students are asking questions like you know about their setup about the errors that they're seeing what whatever they've already tried.

s433
00:51:05.760 --> 00:51:18.240
So I just took these questions um so I just took these chats and you know asked questions about specific details just to see if the model is able to like figure that out if it is able to give me an output for that or not.

s434
00:51:18.240 --> 00:51:28.480
And you know the results I saw is that 95% of the time if the model was able to uh you know give us the right exact details that I was trying to look for

s435
00:51:28.480 --> 00:51:39.040
and even when it is keeping all of the details whereas if I summarize first or if I compact um the context I had it only gave me the answer back 32%

s436
00:51:39.040 --> 00:51:39.680
of the time.

s437
00:51:39.680 --> 00:51:47.440
And um if you think about it um the reason for that is that when you summarize you you know you uh remove all of the necessary details.

s438
00:51:47.440 --> 00:51:52.800
So you're not able to uh you know keep all of those details and the model keeps on missing those out.

s439
00:51:52.800 --> 00:51:57.359
So uh you know we are able to see that keeping everything wins in terms of cost.

s440
00:51:57.359 --> 00:52:02.240
If you have like a you know model like deepseeek um then it also is remembering things.

s441
00:52:02.240 --> 00:52:06.240
So you know it's not that if you have a long conversation it is not able to remember things.

s442
00:52:06.240 --> 00:52:08.720
So you know it is correct 95% of the time.

s443
00:52:08.720 --> 00:52:12.640
So the next thing we wanted to see is the cost part of it.

s444
00:52:12.640 --> 00:52:16.559
Uh you know how does it actually um does it cost most?

s445
00:52:16.559 --> 00:52:17.599
Does it cost the lease?

s446
00:52:17.599 --> 00:52:19.920
What happens in terms of tokens?

s447
00:52:19.920 --> 00:52:27.200
So, um you know on Deep Seek we saw um the setup that was sending the most tokens is actually the cheapest to run.

s448
00:52:27.200 --> 00:52:37.359
So the full history u you know setup that we had was sending the most tokens but uh we were still getting the best results out of it because 97%

s449
00:52:37.359 --> 00:52:44.960
of the tokens that we had were cached and uh you know as you as I as we just saw in the previous slide cash tokens are really cheap.

s450
00:52:44.960 --> 00:52:47.839
they are uh you know they are charged separately.

s451
00:52:47.839 --> 00:53:00.960
So um summarizing uh works the other way uh and every turn it makes the model reads and writes uh new tokens and you know this is something that I ran on like 36 turn conversation

s452
00:53:00.960 --> 00:53:14.079
and it was about 1.78 million tokens and keeping everything still came out ahead and um so you know this answers the question that it wasn't just u you know Gemini

s453
00:53:14.079 --> 00:53:29.680
it it works same on deepseek as well So um the cheaper model is u what brought the cost down but the results are the same and um you know in in respect of this I wanted the next thing I wanted to ask is

s454
00:53:29.680 --> 00:53:39.920
what happens when the conversations really grow because all of these were tested out on like short conversation if I really really uh you know increase the length of the conversation how does that work out

s455
00:53:39.920 --> 00:53:51.839
so um you know when I tried to you know run this entire experiment on a longer context I saw saw that um you know if even if I'm pulling like one specific detail out of it

s456
00:53:51.839 --> 00:53:53.760
um the model performed really well.

s457
00:53:53.760 --> 00:53:59.920
So on the top part the green line those are all of the distinctive distinctive facts that the model is able to find out.

s458
00:53:59.920 --> 00:54:06.079
So we can see that up until 800k tokens as well the model was not missing out on those facts.

s459
00:54:06.079 --> 00:54:11.599
It was giving me like good and consistent results for uh you know some ambiguous facts.

s460
00:54:11.599 --> 00:54:23.200
um the performance dropped um you know to half of what I observed on the distinct distinctive facts but for overall our for our AI tutor this uh you know result um was really good.

s461
00:54:23.200 --> 00:54:38.319
So you know we saw that um the model even when you're not compacting anything holds really well even if you have a really long conversation so uh but so far everything that I've talked about is only per turn.

s462
00:54:38.319 --> 00:54:51.839
So you know every cost has been per turn but does it help when you know we scale it because um a chatbot is not something um that you know it's not like one turn uh situation for a chatbot.

s463
00:54:51.839 --> 00:55:00.000
Um the tutor is a long-term service where you know students are asking um you know questions on on a massive scale.

s464
00:55:00.000 --> 00:55:15.040
So let's say we have uh you know if I I calculated this using this like if we have 100,000 to a million turns of questions every day what would you know be the cost on deepseek or so you know for deepseek the cost was uh approximately

s465
00:55:15.040 --> 00:55:34.160
somewhere from 18,000 to like 180,000 a month and uh you know even though we don't see that sort of volume as of now but paying per token starts to add up and so the the alternative for that was going over to a local model just to see if we are getting the same sort of performance on local

s466
00:55:34.160 --> 00:55:36.160
model or not.

s467
00:55:36.160 --> 00:55:49.280
And um so one way um you know we were thinking that because local models cache as well can you know we use the same sort of setup on a local model and will we get the same result.

s468
00:55:49.280 --> 00:55:55.839
So uh you know because uh we had like hardware limitations we just tested it out on a MacBook.

s469
00:55:55.839 --> 00:56:13.359
the maximum context window that we could go up to was 32K and uh so we thought that can we uh you know can we do that locally now uh but we can't because uh you know the the sort of uh lessons that we had are had are bigger than like 32k

s470
00:56:13.359 --> 00:56:24.400
context window on their own and once the conversation doesn't fit in the window um caching was no longer helpful for us and we uh you know we have to make

s471
00:56:24.400 --> 00:56:31.119
um the context smaller either by compressing it or by retrieving only the parts that we need.

s472
00:56:31.119 --> 00:56:39.599
So uh you know the next question we were trying to answer is can we uh you know once you have to compact locally what actually works?

s473
00:56:39.599 --> 00:56:56.400
So uh you know for um the chat memory going local uh and trying to keep everything stops winning because you can't keep everything and you know a simple uh question here would be that why can't you just keep increasing the length of the model you know why can't you use a bigger model because

s474
00:56:56.400 --> 00:57:08.079
of course we have hardware limitations for us it was uh you know a MacBook but you know GPUs also have like hardware limitations but you know we went from like a 7B model 8B model to a 32-B model

s475
00:57:08.079 --> 00:57:22.480
uh but here we landed Ed on the uh you know conclusion that even though you keep increasing the length of the model uh it is not going to increase your context window you know you cannot repair that part you have to you know make a choice there

s476
00:57:22.480 --> 00:57:36.559
uh but um you know this is only for the chat history and what happens when you are trying to deal with like documents locally so um this was a little surprising because if you are retrieving results with like local documents

s477
00:57:36.559 --> 00:57:40.960
um you know it was really good we got like 100% in accuracy in that case.

s478
00:57:40.960 --> 00:57:51.920
So if uh you know when students are pasting something which is too big uh you know rag is a good option there you can use that and it can help you retrieve the exact uh data that you're looking for.

s479
00:57:51.920 --> 00:57:56.079
Also the processing time in this case was anywhere from 25 to 65 seconds.

s480
00:57:56.079 --> 00:58:01.359
So you know which is uh pretty good in terms of the you know the output that we're getting.

s481
00:58:01.359 --> 00:58:07.280
Uh but you know if you are trying to like stuff the window with like more context than you have.

s482
00:58:07.280 --> 00:58:18.880
We saw that you know it took us like approximately 340 seconds um to get the output when you know our um conversations were really long and the output that we got was a single token.

s483
00:58:18.880 --> 00:58:24.640
So you know you are um not getting anything but you're also wasting a lot of time when you are trying to do it.

s484
00:58:24.640 --> 00:58:31.119
So um you know you have to be careful about what option you choose in this case.

s485
00:58:31.119 --> 00:58:38.640
So um so you know you also have like different type of retrieval strategies that you could use.

s486
00:58:38.640 --> 00:58:48.960
The default retrieval is of course semantic search where you're just trying to match uh you know the meaning of the text and that is the dense rag uh heading that you can see on the chart.

s487
00:58:48.960 --> 00:58:58.960
Um it mostly works but uh you know we saw we tried to make it work from like uh 50k token to 200k and we saw that you know dense um rag worked really well.

s488
00:58:58.960 --> 00:59:00.720
you know it was like 80%.

s489
00:59:00.720 --> 00:59:11.839
But when we increased it to like 400k tokens it was not able to facts that were buried in the middle and it started giving us like 0% recall whereas uh you know something like BM25

s490
00:59:11.839 --> 00:59:14.640
it still got 100% every time.

s491
00:59:14.640 --> 00:59:25.520
So semantic search uh on its own is not enough and that's why you know when Omar talked about our setup in the AI tutor we're actually using a hybrid search we're using a mix of both

s492
00:59:25.520 --> 00:59:38.559
u you know dense and um you know BM25 we're using a combination of both those um so you know after all of this we came up we had like one other question which was like

s493
00:59:38.559 --> 00:59:47.359
uh how does all of this um local setup compare to cloud because that's the uh real way we'll see the result.

s494
00:59:47.359 --> 00:59:58.559
So we wanted to put them side by side just to see uh what is the output and for chat uh you know the local setup was not up to the par of um the cloud you know the cloud setup

s495
00:59:58.559 --> 01:00:03.839
on cloud keeping everything scores you know somewhere from from 92 to 95%.

s496
01:00:03.839 --> 01:00:10.000
But locally it was stuck at like 33% and the context window was a limitation here.

s497
01:00:10.000 --> 01:00:21.359
Also you can see that um you know local um local models actually work because um there is no cost like you cannot see because you already own the hardware

s498
01:00:21.359 --> 01:00:25.359
though um you know there's a throughput limitation there.

s499
01:00:25.359 --> 01:00:33.359
Uh but if you use a technique like retrieval uh you know you get like good accuracy even on a local setup.

s500
01:00:33.359 --> 01:00:50.400
So um in our case what we found is that on memory uh you know keeping the whole chat recalled about like 95% of the details that we were providing it it was able to you know give us correct answer 95% of the time versus it was uh like just 32%

s501
01:00:50.400 --> 01:00:58.000
if you summarize it um on long context uh you know finding a single fact uh is easy for the model.

s502
01:00:58.000 --> 01:01:05.119
we went up to 800k tokens and you know we did not um see any sort of context rot in that case.

s503
01:01:05.119 --> 01:01:17.599
Um on cost per turn we saw that the cheapest run is actually the one which is sending the most tokens because caching makes um resending the same context uh you know very cheap

s504
01:01:17.599 --> 01:01:28.240
and on no cost at scale um um you know it scales up for like let's say if we have like thousand students uh you know Gemini costs us about like $40,000

s505
01:01:28.240 --> 01:01:38.480
40,000 a month whereas Deep Seek Deep Seek was around 1,900 a month um so um going local saves us on a cost a bit more.

s506
01:01:38.480 --> 01:01:42.880
So the main thing to take away is that um do not compact by default.

s507
01:01:42.880 --> 01:01:48.640
You have to name the constraint that you have and then you know look for a better alternative.

s508
01:01:48.799 --> 01:01:53.760
So what did we finally decide after all of these different experiments that we ran?

s509
01:01:53.760 --> 01:01:59.040
Um so we decided on deep uh V4 flash because uh we had hardware limitations.

s510
01:01:59.040 --> 01:02:02.160
So for us um the cloud structure worked out really well.

s511
01:02:02.160 --> 01:02:06.240
It is also the cheapest considering the current um intake of students we have.

s512
01:02:06.240 --> 01:02:08.400
So you know it works out well for us.

s513
01:02:08.400 --> 01:02:17.359
Um and we're using on top of that model we are using a mix of u you know using hybrid retrieval to uh you know get good results.

s514
01:02:17.359 --> 01:02:20.640
For memory uh you know we have chosen to keep everything.

s515
01:02:20.640 --> 01:02:31.200
uh we have got um you know a default limit that after 30k tokens we are going to you know have compaction but up until that we're planning to like keep everything

s516
01:02:31.200 --> 01:02:46.400
and uh you know that is the tutor setup that we have so you know because we had like time limitations so these are all the evaluations and experiments that I could pack into this time but if you would like to you know learn more about these evaluations

s517
01:02:46.400 --> 01:02:54.880
or you would want to build a tutor yourself um this is the fullstack AI engineering course uh on academy.towardsai.net.

s518
01:02:55.119 --> 01:02:59.359
Um so you can go to this link and uh you know go through the course.

s519
01:02:59.359 --> 01:03:00.880
Uh but thank you so much everyone.

s520
01:03:00.880 --> 01:03:04.960
Uh and now we can take any questions.

s521
01:03:07.908 --> 01:03:09.908
[applause]

s522
01:03:22.318 --> 01:03:24.318
[music]