We Cut 94% of AI Coding Tokens With a Local Code Index - Rajkumar Sakthivel, Tesco
https://www.youtube.com/watch?v=dRmWYHuIJxM
AI Engineer · 10 min · 170 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] Hey, I'm Raj.
[00:02] I want to tell a story.
[00:03] Me and my friend Foss, we are building project together.
[00:08] We are using AI coding tools everyday.
[00:11] Cloud code, cursor, co-pilot, code X, normal stuff.
[00:15] One month our AI bill was fine.
[00:18] Next month, huge.
[00:20] We did nothing different.
[00:22] Same project, same tools, just more of it.
[00:25] We panicked.
[00:27] We looked what was happening and we found something surprising.
[00:31] Most of the money was not the AI thinking.
[00:35] Most of it was sending too much context.
[00:42] Files the AI don't need.
[00:45] Context is important.
[00:48] Code that was not relevant sent anyway every time.
[00:52] So, me and my friend Foss, we started to building something to fix it.
[00:57] In this talk, is about what we built and what we learned.
[01:05] Every AI coding tools uh do the same thing.
[01:11] It send your code to the model as a context.
[01:14] And the tools thinks more context is better.
[01:18] We measured typical query on our project.
[01:22] It was sending 45,000 tokens of context, but the part of actually mattered is about 5,000 only.
[01:29] Other 40,000 tokens are not useful, but we paid for them every single query.
[01:38] It's It's like ordering a pizza and paying for extra nine pizzas you don't eat every time.
[01:49] We tried three things before we found what works.
[01:54] First, we changed our prompt be short, only show relevant code.
[01:59] Sounds good.
[02:00] But, it does not work.
[02:01] The model already got 45,000 tokens before it read the prompt.
[02:06] Cost already happened.
[02:08] Second, we change the model setting like a max token temperature.
[02:12] Same problem.
[02:14] These changes the output, not the input.
[02:17] Uh money is in the input.
[02:19] Third, uh output compression.
[02:22] This one actually works.
[02:26] We told the model to write short answers.
[02:28] It cut the output 75%.
[02:31] But, output only uh about 10% of the cost.
[02:36] So, 75% of a small number still small number, not enough.
[02:42] We need to fix the input.
[02:46] This is the most important slide.
[02:48] 90% of your AI cost is input.
[02:51] Files, search results, context you send in.
[02:54] Only 10% is output.
[02:57] The code the AI writes back.
[03:00] So, if you cut the output by 75%, you can save about eight 8% total.
[03:07] But, if you cut input by 94%, you can save about 61% total.
[03:13] Same math, but different result.
[03:15] Fix the input.
[03:17] That's where your money goes.
[03:20] We built a local search layer.
[03:22] It sits between your code base and the AI.
[03:25] Instead of sending whole files, the AI search and index.
[03:29] It gets back only small piece of code actually it needs.
[03:34] Here how it works.
[03:35] Five steps.
[03:36] Step one, we read the code and break into small pieces, functions, classes, methods.
[03:44] Not a random chunks, proper piece that makes sense.
[03:47] Step two, we run two searches at at the same time.
[03:52] One such find the code by meaning.
[03:55] One such find the code by exact words.
[03:58] Then combine the results.
[04:00] This is where big saving comes from.
[04:03] Steps three, we can shrink their results even more.
[04:08] Keep only the function name and the description.
[04:10] Cut 50-line function down to five lines.
[04:14] Step four, uh we track the connection with the which function called which.
[04:20] So, if you find one piece of code, you can find everything connected to it.
[04:26] Step five, every results get score.
[04:29] If the score is too low, we don't send it.
[04:33] No bad context.
[04:34] Everything runs on your machine.
[04:36] Nothing goes to the cloud.
[04:38] This is the beauty.
[04:40] Why do we run two searches instead of one?
[04:44] Because each one has a weakness.
[04:47] Meaning-based search is good at finding related ideas, but it misses exact names.
[04:55] You will search for um authenticate user function, and it might show you different auth function instead, because they are similar meaning.
[05:05] The word-based search is good at exact names, but it misses related ideas.
[05:12] You search for login flow, and it misses everything that says sign in.
[05:17] By themselves, both searches miss about one in four results.
[05:23] Together, they miss about one in 10.
[05:26] They fix each other weakness spots.
[05:31] Here is the hard part Foss and I spend most of the time on.
[05:35] The search finds results, but they actually relevant?
[05:39] Sometimes search results returns 10 results, and none of them are right.
[05:45] If they use the bad results, it gives confident wrong answer.
[05:50] The worse than no answer.
[05:52] We tried asking AI to judge its own results, too slow.
[05:57] Add 2-3 seconds every time.
[05:59] We tried to fix the score limit, too simple.
[06:02] Short questions score low even when the match is perfect.
[06:06] It worked.
[06:08] Some simple formula, 50% meaning score, 30% keyword score, 20% how the recent code is.
[06:17] And the limit adjust based on the current result.
[06:20] It runs 0.4 milliseconds, no extra AI calls needed.
[06:26] The lesson we learned, simple formula beats the complex model most of the time.
[06:33] We need numbers, not just stories.
[06:36] So, here are ours.
[06:38] We tested an open-source real project, FastAPI, 53 files, 20 real questions a developer would ask.
[06:48] Without using our tool, 83 uh K tokens per questions.
[06:53] With our tools, uh 4.9 K tokens per questions.
[06:57] That is 94% less.
[07:00] With the extra compression on top, 523 tokens per question.
[07:06] And the accuracy still um find the right code 90% of the code.
[07:13] These numbers are real.
[07:15] Test is public.
[07:16] You can run it yourself.
[07:18] The command is on screen.
[07:22] I want to be honest about the limits.
[07:25] The 94% again the worst case, reading full files every time.
[07:31] In a real life, the tools like a cloud code already smarter than that.
[07:37] Real savings are lower than 94%.
[07:40] We use full file base because it is the only one we can measure the same way every time.
[07:49] Big mixed code base are hard.
[07:51] We tested on large projects with 396 files.
[07:57] The recall dropped almost zero.
[07:59] If your files each do one thing, it works well.
[08:03] If your files do many things, it struggles.
[08:07] We use a small fast model for search.
[08:10] It's quick.
[08:11] Re-indexing takes under a second.
[08:14] But the bigger model would find more.
[08:17] We chose speed over perfection.
[08:21] Simple choices work better than complex one.
[08:24] Small database instead of big infrastructure.
[08:28] Two searches instead of one fancy one.
[08:31] Local instead of cloud.
[08:33] Simple one.
[08:36] Here is something Foss pointed out earlier on.
[08:39] We use many tools.
[08:40] Cloud code for hard problems.
[08:42] Cursor for quick edits.
[08:44] Copilot for small completions.
[08:47] Each tool starts fresh every time.
[08:51] They do not share anything.
[08:54] You explain the same code base to three different days.
[08:58] We built one shared index so all your tools connect to it.
[09:03] Same search, same results for all of them.
[09:07] And we added memory.
[09:10] When one tool learns something about your project, that knowledge stays.
[09:16] Next session, different tool, same project, the context is already there.
[09:21] We explain the code base once.
[09:25] Every tool remembered.
[09:29] This is the saving report on real project.
[09:32] 247 queries.
[09:34] 12.4 million tokens saved.
[09:37] Nearly 186 not spent.
[09:40] Most of the savings, 84% came from search layer.
[09:45] The rest of them, compression.
[09:47] This is not an estimate.
[09:49] The tool tracks every query.
[09:51] It compares what would have been sent against what was sent.
[09:57] Then it multiplies by the model price.
[10:00] Run it on your project for a week.
[10:03] See your own number.
[10:05] Me and Foss built this because we had a huge bill and no good answer.
[10:10] The answer was not a better model.
[10:13] The answer was sending less.
[10:15] We argue about which model is best, Opus or Sonnet.
[10:18] But the models may be 30% of the cost, but other 70% is what you feed it.
[10:25] Fix the input, the model choice matters less than you think.
[10:29] One command to try it, CCE.
[10:33] It's free, open source.
[10:34] You will see the code on screen.
[10:36] Try it, see the number, tell us what you saved.
[10:39] Thanks.
[10:40] Happy coding.