We Cut 94% of AI Coding Tokens With a Local Code Index - Rajkumar Sakthivel, Tesco

AI Engineer · 10 min · 170 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01Hey, I'm Raj.
2. 00:02I want to tell a story.
3. 00:03Me and my friend Foss, we are building project together.
4. 00:08We are using AI coding tools everyday.
5. 00:11Cloud code, cursor, co-pilot, code X, normal stuff.
6. 00:15One month our AI bill was fine.
7. 00:18Next month, huge.
8. 00:20We did nothing different.
9. 00:22Same project, same tools, just more of it.
10. 00:25We panicked.
11. 00:27We looked what was happening and we found something surprising.
12. 00:31Most of the money was not the AI thinking.
13. 00:35Most of it was sending too much context.
14. 00:42Files the AI don't need.
15. 00:45Context is important.
16. 00:48Code that was not relevant sent anyway every time.
17. 00:52So, me and my friend Foss, we started to building something to fix it.
18. 00:57In this talk, is about what we built and what we learned.
19. 01:05Every AI coding tools uh do the same thing.
20. 01:11It send your code to the model as a context.
21. 01:14And the tools thinks more context is better.
22. 01:18We measured typical query on our project.
23. 01:22It was sending 45,000 tokens of context, but the part of actually mattered is about 5,000 only.
24. 01:29Other 40,000 tokens are not useful, but we paid for them every single query.
25. 01:38It's It's like ordering a pizza and paying for extra nine pizzas you don't eat every time.
26. 01:49We tried three things before we found what works.
27. 01:54First, we changed our prompt be short, only show relevant code.
28. 01:59Sounds good.
29. 02:00But, it does not work.
30. 02:01The model already got 45,000 tokens before it read the prompt.
31. 02:06Cost already happened.
32. 02:08Second, we change the model setting like a max token temperature.
33. 02:12Same problem.
34. 02:14These changes the output, not the input.
35. 02:17Uh money is in the input.
36. 02:19Third, uh output compression.
37. 02:22This one actually works.
38. 02:26We told the model to write short answers.
39. 02:28It cut the output 75%.
40. 02:31But, output only uh about 10% of the cost.
41. 02:36So, 75% of a small number still small number, not enough.
42. 02:42We need to fix the input.
43. 02:46This is the most important slide.
44. 02:4890% of your AI cost is input.
45. 02:51Files, search results, context you send in.
46. 02:54Only 10% is output.
47. 02:57The code the AI writes back.
48. 03:00So, if you cut the output by 75%, you can save about eight 8% total.
49. 03:07But, if you cut input by 94%, you can save about 61% total.
50. 03:13Same math, but different result.
51. 03:15Fix the input.
52. 03:17That's where your money goes.
53. 03:20We built a local search layer.
54. 03:22It sits between your code base and the AI.
55. 03:25Instead of sending whole files, the AI search and index.
56. 03:29It gets back only small piece of code actually it needs.
57. 03:34Here how it works.
58. 03:35Five steps.
59. 03:36Step one, we read the code and break into small pieces, functions, classes, methods.
60. 03:44Not a random chunks, proper piece that makes sense.
61. 03:47Step two, we run two searches at at the same time.
62. 03:52One such find the code by meaning.
63. 03:55One such find the code by exact words.
64. 03:58Then combine the results.
65. 04:00This is where big saving comes from.
66. 04:03Steps three, we can shrink their results even more.
67. 04:08Keep only the function name and the description.
68. 04:10Cut 50-line function down to five lines.
69. 04:14Step four, uh we track the connection with the which function called which.
70. 04:20So, if you find one piece of code, you can find everything connected to it.
71. 04:26Step five, every results get score.
72. 04:29If the score is too low, we don't send it.
73. 04:33No bad context.
74. 04:34Everything runs on your machine.
75. 04:36Nothing goes to the cloud.
76. 04:38This is the beauty.
77. 04:40Why do we run two searches instead of one?
78. 04:44Because each one has a weakness.
79. 04:47Meaning-based search is good at finding related ideas, but it misses exact names.
80. 04:55You will search for um authenticate user function, and it might show you different auth function instead, because they are similar meaning.
81. 05:05The word-based search is good at exact names, but it misses related ideas.
82. 05:12You search for login flow, and it misses everything that says sign in.
83. 05:17By themselves, both searches miss about one in four results.
84. 05:23Together, they miss about one in 10.
85. 05:26They fix each other weakness spots.
86. 05:31Here is the hard part Foss and I spend most of the time on.
87. 05:35The search finds results, but they actually relevant?
88. 05:39Sometimes search results returns 10 results, and none of them are right.
89. 05:45If they use the bad results, it gives confident wrong answer.
90. 05:50The worse than no answer.
91. 05:52We tried asking AI to judge its own results, too slow.
92. 05:57Add 2-3 seconds every time.
93. 05:59We tried to fix the score limit, too simple.
94. 06:02Short questions score low even when the match is perfect.
95. 06:06It worked.
96. 06:08Some simple formula, 50% meaning score, 30% keyword score, 20% how the recent code is.
97. 06:17And the limit adjust based on the current result.
98. 06:20It runs 0.4 milliseconds, no extra AI calls needed.
99. 06:26The lesson we learned, simple formula beats the complex model most of the time.
100. 06:33We need numbers, not just stories.
101. 06:36So, here are ours.
102. 06:38We tested an open-source real project, FastAPI, 53 files, 20 real questions a developer would ask.
103. 06:48Without using our tool, 83 uh K tokens per questions.
104. 06:53With our tools, uh 4.9 K tokens per questions.
105. 06:57That is 94% less.
106. 07:00With the extra compression on top, 523 tokens per question.
107. 07:06And the accuracy still um find the right code 90% of the code.
108. 07:13These numbers are real.
109. 07:15Test is public.
110. 07:16You can run it yourself.
111. 07:18The command is on screen.
112. 07:22I want to be honest about the limits.
113. 07:25The 94% again the worst case, reading full files every time.
114. 07:31In a real life, the tools like a cloud code already smarter than that.
115. 07:37Real savings are lower than 94%.
116. 07:40We use full file base because it is the only one we can measure the same way every time.
117. 07:49Big mixed code base are hard.
118. 07:51We tested on large projects with 396 files.
119. 07:57The recall dropped almost zero.
120. 07:59If your files each do one thing, it works well.
121. 08:03If your files do many things, it struggles.
122. 08:07We use a small fast model for search.
123. 08:10It's quick.
124. 08:11Re-indexing takes under a second.
125. 08:14But the bigger model would find more.
126. 08:17We chose speed over perfection.
127. 08:21Simple choices work better than complex one.
128. 08:24Small database instead of big infrastructure.
129. 08:28Two searches instead of one fancy one.
130. 08:31Local instead of cloud.
131. 08:33Simple one.
132. 08:36Here is something Foss pointed out earlier on.
133. 08:39We use many tools.
134. 08:40Cloud code for hard problems.
135. 08:42Cursor for quick edits.
136. 08:44Copilot for small completions.
137. 08:47Each tool starts fresh every time.
138. 08:51They do not share anything.
139. 08:54You explain the same code base to three different days.
140. 08:58We built one shared index so all your tools connect to it.
141. 09:03Same search, same results for all of them.
142. 09:07And we added memory.
143. 09:10When one tool learns something about your project, that knowledge stays.
144. 09:16Next session, different tool, same project, the context is already there.
145. 09:21We explain the code base once.
146. 09:25Every tool remembered.
147. 09:29This is the saving report on real project.
148. 09:32247 queries.
149. 09:3412.4 million tokens saved.
150. 09:37Nearly 186 not spent.
151. 09:40Most of the savings, 84% came from search layer.
152. 09:45The rest of them, compression.
153. 09:47This is not an estimate.
154. 09:49The tool tracks every query.
155. 09:51It compares what would have been sent against what was sent.
156. 09:57Then it multiplies by the model price.
157. 10:00Run it on your project for a week.
158. 10:03See your own number.
159. 10:05Me and Foss built this because we had a huge bill and no good answer.
160. 10:10The answer was not a better model.
161. 10:13The answer was sending less.
162. 10:15We argue about which model is best, Opus or Sonnet.
163. 10:18But the models may be 30% of the cost, but other 70% is what you feed it.
164. 10:25Fix the input, the model choice matters less than you think.
165. 10:29One command to try it, CCE.
166. 10:33It's free, open source.
167. 10:34You will see the code on screen.
168. 10:36Try it, see the number, tell us what you saved.
169. 10:39Thanks.
170. 10:40Happy coding.