Context Engineering in 2026 — Louis-François Bouchard, Omar Solano & Samridhi Vaid, Towards AI

AI Engineer · 63 min · 522 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:13All right.
00:14Good afternoon everyone.
00:16Thank you for joining and not watching the game.
00:21Uh I hope it will be a bit more interesting or at least you will learn something compared to to uh hopefully Germany winning or some uh anyways.
00:31Yeah.
00:34All right.
00:35Is it fine?
00:36Okay.
00:38All right.
00:38So I'm here to talk about we are here to talk about context engineering in 2026.
00:45And more specifically, we are here because we've all lived that that situation where you try to do things with an agent and ultimately it does just exactly the thing that
00:54you don't want it to do.
00:56And it in my case it usually ends up like this where I'm super mad and I just type back hoping it it learns.
01:04And uh usually the problem here is not that the the model just got dumber and you need to switch to cloud or to codeex or or whatever the the harness that you're using, but it's more that the the context is filling up and it's getting worse and worse.
01:19The results are getting worse because of it.
01:22In our case, this is important because we build courses and trainings for AI engineers specifically.
01:29And one of the features that we provide is an AI tutor to help answer questions based on our lessons.
01:36And if the interaction is just like the one before and they are super mad at us, they might just ask for refund and it could end up like this.
01:45So that's not what we want.
01:48And so what we did for this workshop and just for the AI tutor in general is to run many different experiments in order to figure out how in our case we can fix context rut or at least
02:01improve the i tutor as much as possible and reduce the cost as well of running the tutor.
02:08Uh the QR code here is a link to a hugging face space where you have all these experiments that you can see and also the AI tutor is open source.
02:16who will share another code for the repo but it's also linked on the hugging face.
02:20So everything is open source you can access everything uh and even see the experiments online and use the AI tutorial online as well.
02:28In the next 80 minutes, we will I will start talking about compaction, memory retrieval and um everything that you can do in 2026 that usually works and then my colleagues will jump in with
02:42our the architecture of our AI tutor, our decisions, what we built and the evaluations that we built, the how we built them and what we decided to evaluate and then the results and what we took out of this.
02:55So of of course it's applied to our use case and our AI tutor but hopefully you can get away some interesting insights at least from from this and some best practices that we learned throughout
03:07more specifically we is towards AI.
03:10Um I founded the company with my partners in in a few years ago and we've always been focused around education.
03:18Obviously back in the day it was more about computer vision and and more basic machine learning.
03:22Now it's towards AI engineering and agents, anything that works for the industry.
03:29And I'm joined by my colleagues helping me develop this AI tutor and our courses, Omar and Samidi that we'll jump in later on.
03:38And here more specifically towards AI is quite large.
03:41But what one one of the things that we do is our academy.
03:44So the towards AI academy where we build courses technical courses for AI engineers to upskill towards AI engineering and as I said we provide an AI tutor for the students.
03:57So the the AI tutor specifically uh will will be like our baseline for all our experiments.
04:04We will use that to test all the different features based on real user interactions and to have the best results possible.
04:11We had five requirements we wanted to ensure that the chatbot follows.
04:16The first one obviously we want the answers of the chatbot to be grounded in our content not just its own knowledge.
04:24We need the the tutor to be based in the current students and current lesson to not be a because we have multiple courses.
04:32So we just need to ensure that it answers from this course content.
04:39Then it needs to hold long help sessions in case the student is debugging or or just iterating a lot with the the tutor and uh obviously handled code because it's for AI engineers.
04:51So we just code a lot and it needs to have somewhat of a low latency to not be frustrating to use.
04:59And all of this is related to context engineering.
05:02And here we'll be talking about uh what is context engineering in 2026 or at least what we figured out from this from these experiments.
05:13And since everything here is in the context in the in the context of models, it creates two problems.
05:20First, the context window is finite.
05:23Uh everything the model will see from instructions to to lessons to code will lie in the same space that is limited.
05:30And the more you pile things in this space, the worse the results will be and the more expensive it will be because you pay for more tokens.
05:37So that's one of the the main problem we're trying to fix.
05:41And the second problem is that the model is stateless.
05:44So when a student reopens the AI tutorially if you don't build anything around it, the model will have no idea what's going on.
05:51It's just starting from zero.
05:53So this creates two things we have to work on.
05:56the context management which means within one session and the memory aspect of these models which means across sessions in these experiments and in this workshop we focus on the first part the context management
06:10because you cannot have multiple sessions is if one session is is shitty.
06:15So we try to really optimize for this and maybe in a future workshop we will do one about memory hopefully.
06:23So for context management what does the the tutor sees in our case?
06:27It sees uh many things from the system prompt to being for being a tutor and the context of our courses and stuff to tool definitions uh so which tools it can use how to use them when to use them.
06:41The chat history if it's an ongoing discussion any old tool outputs that was called course chunks that we retrieved to answer the students and finally the user questions.
06:51So it's just not the user question that we send obviously and typically it's the smallest part but it can also have contain a lot of code to debug and to help
07:00or error logs.
07:01So it can be large as well and all of this together just ends up costing more and more to us.
07:10So we really want to optimize this this context management aspect.
07:15And what we've seen just quickly is that the the main bottleneck or the main problem scaling the context is the old tool outputs which contains like any old chunks retrieved
07:26all the tool calls and tool results pairs uh from from all the tools called over the the many sessions that it could have the many turns that that one session could have have
07:37and any uh file or searches that it did to in its own memory.
07:44And the main problem is not is not that it's more costly.
07:48It's also that the quality degrades as we know with a program called context route because of the way how large language models are trained to handle longer context with
07:58we just inject facts into large compass which doesn't tell them to manage the whole the whole context together or understand the global context and uh so so one of the reason is to help with quality.
08:11We want to reduce the context as much as possible.
08:14And the others are that if you reququery the model in a in a discussion, you resend every previous tokens.
08:22So you pay for them as well again, which is far from ideal.
08:27And you increase the the latency, so the time to first token or TTFT uh which makes which creates a very bad user experience.
08:36So you may want to in our case manage our context for speed and for spending not just because the quality drops as we will see in our experiments and and in the results in the near future.
08:51So how we do that to manage uh spending and and speed?
08:54How do we manage our context?
08:56We use compaction.
08:58And the idea is just very simple.
09:00Uh the idea of compaction is very simple.
09:02is just to try to have the smallest context possible that contains the information to be able to answer the question and you drop or save the rest somewhere.
09:13And to do compaction, even a good compaction, you don't necessarily have to have large language models.
09:19You can start quite cheap with trivial tools like uh if you have if you use tools in your system um like a web search or just executing code you can automatically
09:30truncate outliers.
09:32So if you have like once tool that that produce 300 lines instead of 10 usually you can just truncate almost everything except the the head and tail the the beginning and the end and just
09:43re write that it's truncated so that the model in the future can recall the tool if it's if it feels it lacks context.
09:50You can use the the simplest approach that works the best to just use a sliding window or just trim.
09:56Basically, use the last n number of of turns that the user sent which uh you need to determine by based on your your own system and your users and you can clear uh for for some specific tools depending on those that you implement you can just always clear most of the outputs.
10:16So that's for when that's not even using language models.
10:19And then you can use language models to basically spend tokens to save even more tokens.
10:25And in this case, you don't even have to to use large language models.
10:29You can even use smaller ones or even low super small local ones that that runs in like one MacBook to do a few techniques.
10:37There are many techniques that exist for for compacting.
10:41the those that had the most impact in our experiments where selective retention where the language models will just decide based on where the discussion go is going what to keep what to discard.
10:53Then the simplest one here summarization.
10:55So just summarize continuously summarize the some previous terms depending on your on on your own application and in the end you can do here what cloud code does.
11:07So when it reaches the limit you just sum produce a summary of everything and reset completely with that summary.
11:14And there are many more techniques I highlighted here those that work the best.
11:19We will discuss them later on in this workshop and the actual results and experiment setup but those are the most successful techniques.
11:26And I want to highlight also delta summarization that cloud code uses that is very useful when you spawn sub aents when you use sub aents.
11:35It just means to keep a summary and updating the summary based on the new summary that you produce over time and then the the sub agent will just give that to the main agent.
11:45But in our case, we don't use sub agents because the tutor works really well with just one main.
11:51So we don't need to add complexity for that.
11:55And lastly, after spending token to to save more tokens, you can also offload things.
12:01Right now it's obviously memory and skills are super popular.
12:05So you can of course offload to your memory.
12:08So just saving text in your documentations and you can use what's been there for years now.
12:14Uh retrieve augmented generation which is very powerful.
12:18And as a side note we also compared with graph rag.
12:21So everything even if I not I don't mention them we compared them and I highlight just the the best results here.
12:27So we we compared graph rag with with rag here and it's in our case it just ended up being way costlier to set up and just tie on the results
12:37because it basically was 100% based on our real user evaluations.
12:42So we don't need to use graph rag but it's it depends on your own case.
12:47If you have a very large data set with inter with relations and interconnected topics and things, it might be worth implementing.
12:57So you definitely want to still test it.
13:01And uh speaking of memory and of up offloading to files, this is uh as I said, basically just saving them locally or or in a server for for you to use, which means it's fully reversible because you don't lose anything.
13:17you don't lose any ongoing discussion.
13:19You just save them ready to be referred to in the future if the same student comes back and asks related questions.
13:28And it's basically the the Tpetes idea of the LLM wiki uh which if you link with some sort of uh chunks so a version of rag it's pretty powerful
13:40and it makes your system become quite cheap and durable and it's easy to inspect from both humans and agents.
13:47So it's really interesting.
13:49More specifically, it looks like this in our case.
13:51So we have just chunks.
13:54We save everything into chunks and we cross-link the chunks with pointers and then these chunks are linked to raw data files from where they come from.
14:03Then we have one index that will just map all these chunks.
14:07So just a link to all the chunks and some context of what it is about and the agent will just see this index.
14:14So it sees it's like I think it was 450 tokens.
14:17So it's very small and it just sees that index and then uh based on the user question if it seems to be related to some user specific question that it may exists in the memory it will scan the index it will go back to the chunk if it's enough it will answer based on the junk the chunk
14:37retrieved if it's not enough it can even go back to the raw data to have even more information.
14:42So basically it's just the best way to pull context accordingly to the task complexity.
14:47So if the task is complex you will pull more and if it's simple you'll p pull less.
14:54And a parenthesis on this what we've seen working with our clients and just building this in general is that right now everyone is is converging towards having more and smaller
15:06skills.
15:07So you have it's it's way better to build small very precise skills that refer to each other's skills to uh to save on context and just load skills one by one and even be able to spawn a sub agent with one dedicated skill context instead of just basically to save context.
15:26It's a well the idea of progressive disclosure.
15:28So you just load what you need right now.
15:32Okay, just to go back now on on compaction.
15:34This is what the the talk is about because memory we have made some experiments but we couldn't really fit here.
15:40It was a bit too much.
15:42And uh when talking about compaction there's an important problem or solution that uh that appeared recently uh well not recently but was way more popularized is the uh the the main problem is that first the main problem is that you when you ask a follow-up question
16:01you need to recomputee all previous tokens every time.
16:05So you just end up paying twice for the same tokens or three times or four times if the conversation is going which is obviously far from ideal.
16:12So what providers do nowadays is to offer prom caching.
16:17So you will it they will save the embedding and and KV cache and I won't enter into the details but they will premputee they will have saved a lot of the the compute for some tokens and you can just reload them and what's interesting for us is that these already sent token that you reuse are much much
16:35much cheaper uh specifically it can go up to 50 times cheaper with some API like deepseek uh which we will discuss in the experiments and what that means is that If you send a very long context,
16:49you will pay just one 15th of the the price per token and just pay the full price of the qu the new user question or the new interaction.
16:59And that's a problem for compaction because when you are compacting, summarizing, doing any transformation to this context, the the provider cannot use the cache because it it's a new context.
17:11It doesn't the model is not intelligent enough to understand it's the same topic change.
17:16it it just cannot use the cache.
17:17So you will pay full price for these new transform tokens.
17:20So here what it means is that you need to to compress to for for compaction to be worthwhile.
17:27You need to compress by more than 50 times the context.
17:32So it can be quite difficult in some cases without losing quality.
17:38So caching is truly a gamecher especially because nowadays almost all APIs offer it very easy to use and typically they they save cost on like on 90% of the cost when you use caching
17:53but in some case as I said it can go up to way more than that and not only that it helps with cost but caching cache tokens are also already computed so it's way faster to get
18:04the first the answer So ultimately it means that summarization is potentially a trap.
18:12You may not want to use it at all or you you may want to just use it very specifically which is what the most serious harnesses do nowadays.
18:22cloud code, C codeex and all of them use context caching but also use a different method of compactions and we know that because obviously of the leak and then because codeex is open source
18:35and then the APIs also provide ways to manage context directly and caching directly.
18:43So it's it becomes when you build yourself a harness like we do with the AI tutor, it becomes really interesting to understand when to use which technique and test them obviously.
18:55So where is this going?
18:57It mean it means that you don't you don't want to just compact you don't want to summarize everything any time because it may kill the cache and you won't be able to use it.
19:08So some best guidance that we found is that obvious obviously when the user seems to talk about a very different topic, you may want to refresh the session to clean it
19:22to clear it.
19:23Um when you scope or files, use what I described with progressive disclosure to just show the smallest amount of context possible.
19:33You want to clear every old tool outputs that are not useful anymore.
19:38You may want to compact in some case.
19:40We will see that in the experiments in a few minutes.
19:45And you may want to optimize for cache hits.
19:48So just having the model use be able to use its cache.
19:51And regarding that providers are constantly improving their feature set to manage cache.
19:57So you just need to uh stay current and follow what what the API allows nowadays.
20:02But they all provide different methods and um you will have access to the slides but there's I put a link earlier in the in the first few slides on a very interesting article regarding regarding
20:13prompt caching that I recommend checking out.
20:16It's like on the sixth or seventh slide but you will have the link to the slides.
20:23And lastly, you may want to use a model router to optimize especially cost on various tasks.
20:31And most importantly, and what the majority of people don't do, you want to log everything.
20:37It's super easy.
20:38You just ask cloud to implement opic and track everything.
20:40It's you don't have anything to do.
20:42So it's definitely worthwhile to to implement and you can track cache it rate.
20:48You can track user frustration which we've seen that cloud does.
20:52So I keep uh telling it when I'm not happy.
20:56And uh you you may want to log for some um abnormally long outputs or any weird behavior that a small language model could detect.
21:07And so all of this together is the is context engineering which basically means to decide what the model sees every time you call it.
21:18And to us for the AI tutor, it means to decide what to keep in the current context window in order to optimize the caching what to drop or compact and when to do that.
21:31Uh and those two first things are exactly what what we studied in many experiments that we that my colleague Omar will share with you right now.
21:42And uh so you can follow along with the experiments on the the QR code link.
21:47It's the hugging face that I mentioned earlier and in there there's also a link to the repo and everything.
21:56All right.
21:56So, welcome to this second part of the of the workshop.
22:00So now that we have an overall idea of what context engineering is and what are the different techniques that we can apply to our agents uh now what we want to do here is
22:12see how they actually perform in in our case for our AI tutor.
22:18I will first start by describing a little bit about how the AI tutor works.
22:23So the system design and then I will follow up with the initial experiments that we did.
22:34All right.
22:35So, the AI tutor is actually very simple.
22:38It's one agent that it's a React type of agent that just loops over over tool calls and thinking blocks.
22:48And here we just create a very simple one using the lang chain library.
22:52So we use the create agent method with the in-memory saver because we in this case we don't save past conversations we just use the the current history and to customize
23:06this agent we use the middleware feature of lang chain where you we can add different uh features that can change the behavior of the agent at runtime.
23:17So in this case, we want to summarize for example and clear the tool outputs or also in our case have the the user be able to choose the the source
23:30uh uh the sources the like which lessons which courses to to use to to answer.
23:38To do that we also add two different tools.
23:41So we have the first one the retrieve tutor context which which uses a very classic hybrid search pipeline.
23:49So um semantic search along with keyword search and we combine both results to get the the best possible list of chunks.
23:59And then recently we also added the second one which is letting the agent actually browse the file system the knowledge base uh just like we can just like coding agents can in the
24:13browsing your codebase for example and we also borrow from the idea of karpathy by creating a wiki and helping the agent basically browse uh this knowledge base more easily.
24:26I will come back to these tools in a few in a few me moments.
24:30We also add a fast API app to wrap the whole system with an endpoint and we add a nextgs UI to let students use the DII tutor.
24:43So I will talk a little bit about the first tool.
24:45So we have a large corpus.
24:47So we have all of the lessons from all the different courses that we created over the past two years.
24:53uh and also documentation from various public open source libraries like lang chain, lama index and even documentation from uh openai.
25:06So they they make available available all of the markdown files from like how to use the open API, how to use codeex and we also have like cloud code documentation.
25:16So it's a very big corpus that has over 8 million tokens and of course this cannot fit in a single context window.
25:23So we have to store it in a way where we can retrieve the most important or most relevant information.
25:30So in this case the agent receives a question and the user can then choose well beforehand the user can choose specific source.
25:38So we can filter this knowledge base.
25:41It makes it better for um to improve recall.
25:45So precision to get the most relevant information.
25:49Then we do hybrid search.
25:51So this is very classic um hybrid search.
25:54We use embedding model.
25:57In this case it's a coherent coherent model with BM25 for the keyword index to get the the most relevant top 30 uh chunks.
26:09Then we merge the two results from the semantic similarity and the keyword search and we then rerank to the top five most relevant chunks and that's that's what return to the agent.
26:25We also have a limit of 100,000 tokens.
26:27So we don't want to.
26:29So let's say for example we we go over we just remove the last few chunk uh the last chunks that make it so that we don't cross that threshold.
26:40So these numbers the this configuration actually is not random.
26:43We did experiments uh to optimize this pipeline.
26:48I'm not going to talk about it but it's in one of the courses that we share.
26:52Uh basically we just want to try as many configurations as possible and improve recall.
26:57So we measure did we retrieve the correct page uh in our knowledge base and we just choose the best uh settings.
27:07So like I said it uh yeah in this case it's very precise very it's very good but what if the agent needs to browse the whole knowledge base to get the best possible answer.
27:19So let's say I want to learn about codeex and I want to learn about cloud code how can I best use uh those two tools and also like if we need to have v various documentation
27:31pages in context is this the best possible uh tool and there's a paper uh I I link it here in the in in the slides but it's a yeah it's a paper that shows that basically letting the agent browse
27:47the the knowledge base can be very bene beneicial uh so you can look at it if you want afterwards.
27:53Very very slow to load.
27:57Uh so that's why we ended up creating this second tool the run keep run knowledgebased command where the agent can browse u the knowledge base the file system using bash commands.
28:13So first of all what we did first is create these uh three different u folders.
28:19So we have the raw folder with all the different markdown files.
28:23So all the lessons from the different courses the documentation from the different open open source libraries.
28:30We we also have this generated uh folder with this is the generated automatically with basically just the the the titles of each of the markdown files so it's easier for the agent to to find
28:46um relevant information.
28:48And then we also have the wiki where uh in this case it was cl code that created this.
28:54It basically reads all of the raw files in the in the folder in the raw folder and it creates a very uh concise uh set of files.
29:04So topics, frameworks uh from the different source uh sources.
29:10So for example, I can have a topic related to finetuning.
29:12So in this case, the agent will be able to find all of the different raw files related to finetuning.
29:18For example, here we let the agent only read.
29:22So this is when we actually deploy it.
29:24So this is the one you can try on the on the space here.
29:28The agent can only read the knowledge base.
29:30It cannot modify it.
29:32Uh so we only allow these bash commands that basically cannot uh modify the the file system.
29:39We also put some limits around this.
29:41So for example, if a command lasts over 8 seconds, we can just return an error or let the agent execute something else because it's taking too much time.
29:52And we also cap the tool outputs to 40,000 uh characters.
29:56So in this case, for example, if a lesson is over this amount, what the agent can do is then okay, so I just got the first 40,000.
30:05Let me let me do a follow-up command to get the last uh piece of the of the lesson.
30:10For example, uh we also limit the number of the of commands.
30:14We actually never see the agent go over this limit of 20 commands per turn, but this is just a a fallback uh in case it takes too too much time to to answer.
30:26And we also sandbox of course the agent to only browse the knowledge base this specific folder.
30:33Like I said, we create this offline.
30:36So the the three different file folders we create this once or every time we want to add a new uh a new course for example we we we tell cloud code can you add a new uh can you add this to the raw raw folder and also create new topics
30:51around this new new content.
30:53So we do this once and then we deploy it so that the agent u can can browse it with our current system prompt.
31:02We can see that it's uh used almost every time.
31:06So for almost 90% of the terms we can tweak this to make it use it less uh or more.
31:12We didn't optimize for for this specifically.
31:15And I guess the most interesting aspect of doing this is that we measured the precision the recall of using this tool uh actually turning turning it off and we actually got the same amount of
31:30recall.
31:30So just using the first tool was enough to get all the relevant information and just having this second tool was just 50% slower.
31:43Basically, it's faster without this tool because it does less tool calls.
31:48Um, and yeah, we we basically didn't see any improvement on uh on re on answering with the correct uh documentation.
31:58It it was fun to add but we yeah we didn't see any any benefit and one reason for that is that we tested using real world well basically the questions we get from students
32:10and basically those questions were complex enough we guess to actually benefit from using this new new setup.
32:17Um so this is what this is how initially uh our AI tutor managed its context.
32:27Uh we we started with this because uh it looked fast, it looked good.
32:33Uh we didn't actually measure anything.
32:36It was just like oh it looks good.
32:37Okay, we will just set it set it like this.
32:40And so basically we have these three different um context engineering techniques where we clear the outputs uh after 5,000 tokens uh but we at least keep the last five ones.
32:54So this was basically a way to keep a very small context uh during a conversation.
33:01We also added the the capacity to summarize.
33:04So after 30,000 tokens uh the two well the system summarizes the uh the history but we at least keep the last 20 messages to make sure that uh those are very accurate
33:17those messages.
33:19And we also have the source preference.
33:22So that that just allows people to choose what uh sources to use uh when the thetutor answers.
33:32But like I said, these are unproven unproven defaults and we actually want to know what actually works best.
33:39Um so I guess you you can Lou showed this at the beginning, but you can access the tutor live.
33:46I'm just going to show this very quickly.
33:49This is the huging face.
33:51So this is a separate UI just to show you.
33:55We also have the the chat bubble one on the lessons on the course themselves itself.
34:02Uh and here on the on the left you can choose the different uh sources to use, enable them or disable them and then you can uh yeah send your your query.
34:11So as you can see here I can just uh send a new request and we see Gemini in this case Gemini 3.5 flash use its reasoning and use its uh capacity to to do tool calling and to answer
34:27and I guess uh here there's a I think it's just the internet bugging But we should have a response and yeah the code is open source so you can use your favorite coding agent to explore the codebase
34:58and uh and learn about how we implemented this specifically.
35:03So we have the different activity u the the activity that the the the model did.
35:09So for tool calls it thought four times and it use 10 sources and we have the the final answer.
35:19So everything that uh so every time the student uses this chatbot we actually log everything.
35:25So we for every single turn we have uh the input tokens, the the output tokens, how many of them were cached, uh what was the cost, what time it took to to get the first token, how many tool calls it did
35:39and if we actually if the system actually did something around summarization.
35:44So this is very useful and that's uh what we are going to use when uh measuring the uh the different techniques.
35:52So why do we need to measure?
35:54Well, because the the techniques that Lou showed all sound very smart.
35:59So you you might think that they are very uh useful, but sometimes they're not.
36:04And uh as we as we discovered with our experiments, actually it might be detrimental.
36:10So because of the the way APIs cache the tokens when they are um when they are sent and it's also difficult to to know in advance what what is best to use.
36:23So before I go into the experiments, I just want to define a few words because we're I'm going to use these words throughout the uh the the presentation.
36:34So a preset is basically uh the way the AI tutor was set up in the in the experiment.
36:40So for example, it can be in this preset we did summarization at this amount of tokens or in this preset we we used sliding window for example.
36:49So that's what a preset is.
36:51we have the different tasks.
36:53So at task type uh for these initial experiments we only did two tasks single turn and multiple turns also sessions uh called sessions and one run is basically just running one preset on a task and then you get the run.
37:12Um the bundle is just the the result.
37:16So it's just a JSON file with all the different uh metrics that we save to the disk.
37:24So first task single turn.
37:26Uh so these are question and answers and we didn't generate this.
37:32It's not synthetic.
37:33Uh we actually I I had codeex scrape all of the uh questions and responses from our website where students can ask questions and get answers from uh members of the staff.
37:48Uh and that's how I got this initial we got this initial um data set.
37:54uh we cleaned the data set and only used 60 60 pairs uh because we saw that some of the questions weren't good for the type of task for example
38:05uh there were old questions about previous versions of some libraries and we we might not I mean the right now if the tutor answers it's not going to be using this old version of the library so we just
38:19uh remove some of the questions some duplicate ones and and and we got this first uh data set and what we measure is the retrieval.
38:30So did we retrieve the correct is did the total retrieve the correct lesson for example this is done automatically we can see uh just by looking at the code did we did we use the correct uh lesson or not
38:44and we also look at did do we have the correct facts in the answer or the right kind of response in the answer.
38:51These two uh are actually graded using using an LLM.
38:57Uh you can use APIs to do it but uh right now I think the best way to do it is to just use use your uh code code subscription or your codec
39:07subscription because it's cheaper than using the APIs.
39:12We also have the second task the session.
39:15So multi-turn um conversations back and forth.
39:20uh here what we want to know is is the AI tutor able to recall facts after multiple terms.
39:29So this is a bit uh in this case we do use some generated uh content.
39:35So we do in generate facts that we put at the beginning.
39:39So we have a student like a fake student state uh fact.
39:44So for example, I want to learn about rag and then we stuff the conversation with just filler messages because we just want yeah we just want to have a lot of messages
39:56and then we have um a probe which is just the student asking a question again.
40:04So for example it can be uh what should I learn today?
40:09And since the uh I'm going to go through it in the next slide, but uh yeah, let's let's let's the example.
40:17So for example, here at turn one, we we uh we have the student the fake student implement a state of fact.
40:25So it the student wants to learn about rag evaluation.
40:28Um and that that's basically the fact.
40:32Then we just add a lot of messages uh filler filler messages and then at at some point we have the student say for example what topic should I learn about today
40:45and then what we expect the AI tutor to to to say is that uh the student should learn about rag evaluation.
40:52So more specifically heat rate and MR for example.
40:57We also have a gate part in the evaluation where let's say we are testing the summarization technique.
41:05Uh we actually just want to know did summarization actually happen or not.
41:09Um so this is this is this is like one example of one task in the sessions uh data set in the sessions task.
41:18Now we have the evaluation hardness.
41:21Um so the main uh harness I guess the the main function is the run the run task the run battery function that just uh runs this uh this task.
41:34Um we have the grading like I said we it can either be a code check so did we retrieve the correct lesson or not and we can also have the LM as a judge and in this case we use
41:47the subscription of code.
41:49We also have the check triggers aspect.
41:52So this is just a check to to see that if the evaluation went uh good or not.
41:57Did we actually compact or not?
41:58This is this is just to make sure that the the run is actually good and we can save it.
42:05And then we have a generated report uh to see like what was the latency, what was the time to first token and every metric that we can uh measure.
42:17So yeah, basically we can evaluate everything and then grade it afterwards.
42:21Uh we we can run it once and grade it afterwards.
42:25So what what we run so we run 11 presets uh and we change them uh for for each experiment.
42:34So we have the the full history.
42:37So these are the the main ones the full history.
42:40So this is the case where we don't don't touch the context.
42:43We we leave everything as is in the history.
42:47And then we also have the production that I showed at the beginning the defaults that we have.
42:53Uh so these are the reference points and then we have these six techniques that I want to compare.
42:58So sliding window prompt compression uh selective retention and the other ones and I what I want to see is just what uh memory recall recall do I get if I keep everything else fixed.
43:12So I use the same model the same prompt the same tools the same data set what's the difference?
43:18Uh now ju now just doing this was a bit expensive.
43:22I I didn't expect this to to get over $500, but it did.
43:28And that's one of the reason we did afterwards follow-up experiments using cheaper models.
43:34But my colleague Sani will talk about this.
43:38Uh so what ARM so what basically what preset actually won.
43:43Uh, and so these are our um our results.
43:47And as you can see, h we weren't expecting this, but basically not touching um the context was actually the best the best strategy for recovering this fact over time over multiple
44:02uh messages.
44:04Um you can see that the production so the the defaults that we thought were good enough were actually not the best.
44:13Uh not doing anything is actually better.
44:15H we did two uh two different experiments where one was just one trial and the second was um two trials.
44:23So we have like a more statist I I guess it's it's better but the uh the numbers are not like can might not be accurate because it's just one trial and two trials but I guess the most interesting thing is just the order in which the techniques ended up being in the table.
44:42So in this case keeping everything wins on the memory side but what about the cost?
44:49Uh this is what the production cost uh was for the single turn and the session.
44:54So almost 50 cents uh for a single turn and 24 cents uh for the multi-turn or each turn.
45:02Uh we actually had very good memory uh recall for pretty much all the techniques in the single turn because in single turn you don't have enough tokens to actually fire up the different strategies.
45:15Uh so for one response you don't need to do summarization compaction or anything like that.
45:21Uh so that's why you you see um high numbers.
45:25Uh but as you can see after in the multi-turn uh task you can see that the quality degraded to 38%.
45:35uh and we if we compare this to the full history.
45:39So here we don't touch the context uh the history of the model we can see that not touching uh is actually cheaper, it's faster and we have better recall overall.
45:51So keeping everything wins on on all of these three fronts.
45:57Uh so why does it so why is it actually uh like why is it uh why do we have less latency like what's we want to we wanted to understand that and it's basically because if you
46:16remove the tool outputs consistently then the agent needs to rerieve uh afterwards for information it already had.
46:26So you're just making the agent uh do more tool calls and that's why it ended up costing more uh and yeah using more tokens and having less uh uh less memory recall.
46:40So these are the results we initially got using uh Gemini 3.5 with the uh this this data set of 11 to 13 turns is not huge.
46:51Uh there's not many messages.
46:53That's why we wanted we now want uh in the follow-up part do more different experiments that my colleagues already will will show you.
47:03So yeah, let me in let me introduce you to something.
47:07Yeah.
47:10Um, so this is going to be my part and um, as we ended on the note where Omar just said that it costed it c um, it costed us almost $600
47:21to um, run the evals that we ran.
47:24One question we were trying to um, answer with this extended evaluation was um, so when do you uh, when does compaction actually matter or should you actually compact or not?
47:35because we clearly saw that when we have um you know the full answers in the window it works really well but it costs a lot of money.
47:44So um I tried to you know do this evaluation in like three sorts of context.
47:49Uh first one was cash chats.
47:52So you know like when you're chatting with Gemini you'd be able to see that is the cash chat option.
47:57The second version is going to be document plus tool.
48:00So if you're pasting like a long document in the AI tutor or if there's just tool output what happens then and finally you know if you go local or if you scale this evaluation
48:10how well is it going to work out so um should you ever compact so before my part Omar just showed that on Gemini 3.5 flash um keeping everything one
48:23um but uh you know the why did we come up with that question was because um you know full history on a frontier model like Gemini would be very very expensive.
48:33Um so um you know with all this extended experiment we are trying to figure out was it actually worth it or not.
48:41But uh before we start that I wanted to just talk about the different context we see in our AI tutor application.
48:47Um so first one is going to be like a long chat history uh where you know we have um a long chat but all of the details what the the students are asking or they're chatting about they can get buried in.
48:59Um the second is going to be a pasted document.
49:01So you know we do have a lot of students who are going to just like copy paste a lot of documents there.
49:05Um so you know um and also because we have like limited uh context window.
49:10How does that fit?
49:11And then it's going to be you know different tools that we use internally.
49:15Um but this is just going to be like bunch of logs that the tools have and since you know these are like different context each of them need some sort of different fix.
49:26So you know we had to evaluate it for all different um contexts that we had.
49:32So um you know the first obvious thing uh looking at the cost uh we were like okay let's try out a cheaper model and see you know um does it do better you know what sort of techniques work on that does compaction work on it or not.
49:45So um deepseek v4 flash was an obvious option and we were like we will uh try this out on this and see um how well it works out and you know you can see there is um you know a drastic cost difference between Gemini and Deepseek here.
50:00So here we can see um you know there's a drastic difference between cost when we checked the performance on deepseek v4 and the main reason was that that we were getting a cash discount.
50:11So the cash discount on GE um deepseek was um you know 50x as compared to Gemini and even in this um you know setting we saw that keeping all
50:22um you know all of the context still one.
50:24So you know we were just getting the best performance um in that case for deepseek as well.
50:30So, but you know the the plus thing from this experiment was that we figured out a cheaper model but um you know uh now we wanted to see that even though we have everything
50:41uh you know keeping everything uh makes it cheaper but does it remember better you know does uh keeping all of the context remember all of the details the student might be asking us.
50:51So um you know this is how I tested out the memory of the model.
50:55Um so we have conversation within our system where you know students are asking questions like you know about their setup about the errors that they're seeing what whatever they've already tried.
51:05So I just took these questions um so I just took these chats and you know asked questions about specific details just to see if the model is able to like figure that out if it is able to give me an output for that or not.
51:18And you know the results I saw is that 95% of the time if the model was able to uh you know give us the right exact details that I was trying to look for
51:28and even when it is keeping all of the details whereas if I summarize first or if I compact um the context I had it only gave me the answer back 32%
51:39of the time.
51:39And um if you think about it um the reason for that is that when you summarize you you know you uh remove all of the necessary details.
51:47So you're not able to uh you know keep all of those details and the model keeps on missing those out.
51:52So uh you know we are able to see that keeping everything wins in terms of cost.
51:57If you have like a you know model like deepseeek um then it also is remembering things.
52:02So you know it's not that if you have a long conversation it is not able to remember things.
52:06So you know it is correct 95% of the time.
52:08So the next thing we wanted to see is the cost part of it.
52:12Uh you know how does it actually um does it cost most?
52:16Does it cost the lease?
52:17What happens in terms of tokens?
52:19So, um you know on Deep Seek we saw um the setup that was sending the most tokens is actually the cheapest to run.
52:27So the full history u you know setup that we had was sending the most tokens but uh we were still getting the best results out of it because 97%
52:37of the tokens that we had were cached and uh you know as you as I as we just saw in the previous slide cash tokens are really cheap.
52:44they are uh you know they are charged separately.
52:47So um summarizing uh works the other way uh and every turn it makes the model reads and writes uh new tokens and you know this is something that I ran on like 36 turn conversation
53:00and it was about 1.78 million tokens and keeping everything still came out ahead and um so you know this answers the question that it wasn't just u you know Gemini
53:14it it works same on deepseek as well So um the cheaper model is u what brought the cost down but the results are the same and um you know in in respect of this I wanted the next thing I wanted to ask is
53:29what happens when the conversations really grow because all of these were tested out on like short conversation if I really really uh you know increase the length of the conversation how does that work out
53:39so um you know when I tried to you know run this entire experiment on a longer context I saw saw that um you know if even if I'm pulling like one specific detail out of it
53:51um the model performed really well.
53:53So on the top part the green line those are all of the distinctive distinctive facts that the model is able to find out.
53:59So we can see that up until 800k tokens as well the model was not missing out on those facts.
54:06It was giving me like good and consistent results for uh you know some ambiguous facts.
54:11um the performance dropped um you know to half of what I observed on the distinct distinctive facts but for overall our for our AI tutor this uh you know result um was really good.
54:23So you know we saw that um the model even when you're not compacting anything holds really well even if you have a really long conversation so uh but so far everything that I've talked about is only per turn.
54:38So you know every cost has been per turn but does it help when you know we scale it because um a chatbot is not something um that you know it's not like one turn uh situation for a chatbot.
54:51Um the tutor is a long-term service where you know students are asking um you know questions on on a massive scale.
55:00So let's say we have uh you know if I I calculated this using this like if we have 100,000 to a million turns of questions every day what would you know be the cost on deepseek or so you know for deepseek the cost was uh approximately
55:15somewhere from 18,000 to like 180,000 a month and uh you know even though we don't see that sort of volume as of now but paying per token starts to add up and so the the alternative for that was going over to a local model just to see if we are getting the same sort of performance on local
55:34model or not.
55:36And um so one way um you know we were thinking that because local models cache as well can you know we use the same sort of setup on a local model and will we get the same result.
55:49So uh you know because uh we had like hardware limitations we just tested it out on a MacBook.
55:55the maximum context window that we could go up to was 32K and uh so we thought that can we uh you know can we do that locally now uh but we can't because uh you know the the sort of uh lessons that we had are had are bigger than like 32k
56:13context window on their own and once the conversation doesn't fit in the window um caching was no longer helpful for us and we uh you know we have to make
56:24um the context smaller either by compressing it or by retrieving only the parts that we need.
56:31So uh you know the next question we were trying to answer is can we uh you know once you have to compact locally what actually works?
56:39So uh you know for um the chat memory going local uh and trying to keep everything stops winning because you can't keep everything and you know a simple uh question here would be that why can't you just keep increasing the length of the model you know why can't you use a bigger model because
56:56of course we have hardware limitations for us it was uh you know a MacBook but you know GPUs also have like hardware limitations but you know we went from like a 7B model 8B model to a 32-B model
57:08uh but here we landed Ed on the uh you know conclusion that even though you keep increasing the length of the model uh it is not going to increase your context window you know you cannot repair that part you have to you know make a choice there
57:22uh but um you know this is only for the chat history and what happens when you are trying to deal with like documents locally so um this was a little surprising because if you are retrieving results with like local documents
57:36um you know it was really good we got like 100% in accuracy in that case.
57:40So if uh you know when students are pasting something which is too big uh you know rag is a good option there you can use that and it can help you retrieve the exact uh data that you're looking for.
57:51Also the processing time in this case was anywhere from 25 to 65 seconds.
57:56So you know which is uh pretty good in terms of the you know the output that we're getting.
58:01Uh but you know if you are trying to like stuff the window with like more context than you have.
58:07We saw that you know it took us like approximately 340 seconds um to get the output when you know our um conversations were really long and the output that we got was a single token.
58:18So you know you are um not getting anything but you're also wasting a lot of time when you are trying to do it.
58:24So um you know you have to be careful about what option you choose in this case.
58:31So um so you know you also have like different type of retrieval strategies that you could use.
58:38The default retrieval is of course semantic search where you're just trying to match uh you know the meaning of the text and that is the dense rag uh heading that you can see on the chart.
58:48Um it mostly works but uh you know we saw we tried to make it work from like uh 50k token to 200k and we saw that you know dense um rag worked really well.
58:58you know it was like 80%.
59:00But when we increased it to like 400k tokens it was not able to facts that were buried in the middle and it started giving us like 0% recall whereas uh you know something like BM25
59:11it still got 100% every time.
59:14So semantic search uh on its own is not enough and that's why you know when Omar talked about our setup in the AI tutor we're actually using a hybrid search we're using a mix of both
59:25u you know dense and um you know BM25 we're using a combination of both those um so you know after all of this we came up we had like one other question which was like
59:38uh how does all of this um local setup compare to cloud because that's the uh real way we'll see the result.
59:47So we wanted to put them side by side just to see uh what is the output and for chat uh you know the local setup was not up to the par of um the cloud you know the cloud setup
59:58on cloud keeping everything scores you know somewhere from from 92 to 95%.
60:03But locally it was stuck at like 33% and the context window was a limitation here.
60:10Also you can see that um you know local um local models actually work because um there is no cost like you cannot see because you already own the hardware
60:21though um you know there's a throughput limitation there.
60:25Uh but if you use a technique like retrieval uh you know you get like good accuracy even on a local setup.
60:33So um in our case what we found is that on memory uh you know keeping the whole chat recalled about like 95% of the details that we were providing it it was able to you know give us correct answer 95% of the time versus it was uh like just 32%
60:50if you summarize it um on long context uh you know finding a single fact uh is easy for the model.
60:58we went up to 800k tokens and you know we did not um see any sort of context rot in that case.
61:05Um on cost per turn we saw that the cheapest run is actually the one which is sending the most tokens because caching makes um resending the same context uh you know very cheap
61:17and on no cost at scale um um you know it scales up for like let's say if we have like thousand students uh you know Gemini costs us about like $40,000
61:2840,000 a month whereas Deep Seek Deep Seek was around 1,900 a month um so um going local saves us on a cost a bit more.
61:38So the main thing to take away is that um do not compact by default.
61:42You have to name the constraint that you have and then you know look for a better alternative.
61:48So what did we finally decide after all of these different experiments that we ran?
61:53Um so we decided on deep uh V4 flash because uh we had hardware limitations.
61:59So for us um the cloud structure worked out really well.
62:02It is also the cheapest considering the current um intake of students we have.
62:06So you know it works out well for us.
62:08Um and we're using on top of that model we are using a mix of u you know using hybrid retrieval to uh you know get good results.
62:17For memory uh you know we have chosen to keep everything.
62:20uh we have got um you know a default limit that after 30k tokens we are going to you know have compaction but up until that we're planning to like keep everything
62:31and uh you know that is the tutor setup that we have so you know because we had like time limitations so these are all the evaluations and experiments that I could pack into this time but if you would like to you know learn more about these evaluations
62:46or you would want to build a tutor yourself um this is the fullstack AI engineering course uh on academy.towardsai.net.
62:55Um so you can go to this link and uh you know go through the course.
62:59Uh but thank you so much everyone.
63:00Uh and now we can take any questions.
63:07[applause]
63:22[music]