WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=dRmWYHuIJxM

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/dRmWYHuIJxM.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.040 --> 00:00:02.640
Hey, I'm Raj.

s2
00:00:02.640 --> 00:00:03.840
I want to tell a story.

s3
00:00:03.840 --> 00:00:08.040
Me and my friend Foss, we are building project together.

s4
00:00:08.040 --> 00:00:11.200
We are using AI coding tools everyday.

s5
00:00:11.200 --> 00:00:15.800
Cloud code, cursor, co-pilot, code X, normal stuff.

s6
00:00:15.800 --> 00:00:18.640
One month our AI bill was fine.

s7
00:00:18.640 --> 00:00:20.600
Next month, huge.

s8
00:00:20.600 --> 00:00:22.080
We did nothing different.

s9
00:00:22.080 --> 00:00:25.600
Same project, same tools, just more of it.

s10
00:00:25.600 --> 00:00:27.080
We panicked.

s11
00:00:27.080 --> 00:00:31.400
We looked what was happening and we found something surprising.

s12
00:00:31.400 --> 00:00:35.280
Most of the money was not the AI thinking.

s13
00:00:35.280 --> 00:00:42.840
Most of it was sending too much context.

s14
00:00:42.840 --> 00:00:45.760
Files the AI don't need.

s15
00:00:45.760 --> 00:00:48.960
Context is important.

s16
00:00:48.960 --> 00:00:52.120
Code that was not relevant sent anyway every time.

s17
00:00:52.120 --> 00:00:57.120
So, me and my friend Foss, we started to building something to fix it.

s18
00:00:57.120 --> 00:01:04.680
In this talk, is about what we built and what we learned.

s19
00:01:05.760 --> 00:01:11.080
Every AI coding tools uh do the same thing.

s20
00:01:11.080 --> 00:01:14.760
It send your code to the model as a context.

s21
00:01:14.760 --> 00:01:18.640
And the tools thinks more context is better.

s22
00:01:18.640 --> 00:01:22.160
We measured typical query on our project.

s23
00:01:22.160 --> 00:01:29.920
It was sending 45,000 tokens of context, but the part of actually mattered is about 5,000 only.

s24
00:01:29.920 --> 00:01:38.480
Other 40,000 tokens are not useful, but we paid for them every single query.

s25
00:01:38.480 --> 00:01:48.920
It's It's like ordering a pizza and paying for extra nine pizzas you don't eat every time.

s26
00:01:49.480 --> 00:01:54.680
We tried three things before we found what works.

s27
00:01:54.680 --> 00:01:59.320
First, we changed our prompt be short, only show relevant code.

s28
00:01:59.320 --> 00:02:00.400
Sounds good.

s29
00:02:00.400 --> 00:02:01.920
But, it does not work.

s30
00:02:01.920 --> 00:02:06.960
The model already got 45,000 tokens before it read the prompt.

s31
00:02:06.960 --> 00:02:08.679
Cost already happened.

s32
00:02:08.679 --> 00:02:12.959
Second, we change the model setting like a max token temperature.

s33
00:02:12.959 --> 00:02:14.240
Same problem.

s34
00:02:14.240 --> 00:02:17.800
These changes the output, not the input.

s35
00:02:17.800 --> 00:02:19.640
Uh money is in the input.

s36
00:02:19.640 --> 00:02:22.760
Third, uh output compression.

s37
00:02:22.760 --> 00:02:26.000
This one actually works.

s38
00:02:26.000 --> 00:02:28.560
We told the model to write short answers.

s39
00:02:28.560 --> 00:02:31.600
It cut the output 75%.

s40
00:02:31.600 --> 00:02:36.400
But, output only uh about 10% of the cost.

s41
00:02:36.400 --> 00:02:42.240
So, 75% of a small number still small number, not enough.

s42
00:02:42.240 --> 00:02:45.160
We need to fix the input.

s43
00:02:46.280 --> 00:02:48.200
This is the most important slide.

s44
00:02:48.200 --> 00:02:51.560
90% of your AI cost is input.

s45
00:02:51.560 --> 00:02:54.880
Files, search results, context you send in.

s46
00:02:54.880 --> 00:02:57.640
Only 10% is output.

s47
00:02:57.640 --> 00:03:00.280
The code the AI writes back.

s48
00:03:00.280 --> 00:03:07.200
So, if you cut the output by 75%, you can save about eight 8% total.

s49
00:03:07.200 --> 00:03:13.239
But, if you cut input by 94%, you can save about 61% total.

s50
00:03:13.239 --> 00:03:15.400
Same math, but different result.

s51
00:03:15.400 --> 00:03:17.040
Fix the input.

s52
00:03:17.040 --> 00:03:20.000
That's where your money goes.

s53
00:03:20.360 --> 00:03:22.680
We built a local search layer.

s54
00:03:22.680 --> 00:03:25.800
It sits between your code base and the AI.

s55
00:03:25.800 --> 00:03:29.720
Instead of sending whole files, the AI search and index.

s56
00:03:29.720 --> 00:03:34.160
It gets back only small piece of code actually it needs.

s57
00:03:34.160 --> 00:03:35.360
Here how it works.

s58
00:03:35.360 --> 00:03:36.400
Five steps.

s59
00:03:36.400 --> 00:03:44.360
Step one, we read the code and break into small pieces, functions, classes, methods.

s60
00:03:44.360 --> 00:03:47.680
Not a random chunks, proper piece that makes sense.

s61
00:03:47.680 --> 00:03:52.519
Step two, we run two searches at at the same time.

s62
00:03:52.519 --> 00:03:55.160
One such find the code by meaning.

s63
00:03:55.160 --> 00:03:58.840
One such find the code by exact words.

s64
00:03:58.840 --> 00:04:00.760
Then combine the results.

s65
00:04:00.760 --> 00:04:03.640
This is where big saving comes from.

s66
00:04:03.640 --> 00:04:08.080
Steps three, we can shrink their results even more.

s67
00:04:08.080 --> 00:04:10.800
Keep only the function name and the description.

s68
00:04:10.800 --> 00:04:14.360
Cut 50-line function down to five lines.

s69
00:04:14.360 --> 00:04:20.680
Step four, uh we track the connection with the which function called which.

s70
00:04:20.680 --> 00:04:26.400
So, if you find one piece of code, you can find everything connected to it.

s71
00:04:26.400 --> 00:04:29.800
Step five, every results get score.

s72
00:04:29.800 --> 00:04:33.080
If the score is too low, we don't send it.

s73
00:04:33.080 --> 00:04:34.440
No bad context.

s74
00:04:34.440 --> 00:04:36.760
Everything runs on your machine.

s75
00:04:36.760 --> 00:04:38.640
Nothing goes to the cloud.

s76
00:04:38.640 --> 00:04:40.640
This is the beauty.

s77
00:04:40.640 --> 00:04:44.240
Why do we run two searches instead of one?

s78
00:04:44.240 --> 00:04:47.880
Because each one has a weakness.

s79
00:04:47.880 --> 00:04:55.880
Meaning-based search is good at finding related ideas, but it misses exact names.

s80
00:04:55.880 --> 00:05:05.960
You will search for um authenticate user function, and it might show you different auth function instead, because they are similar meaning.

s81
00:05:05.960 --> 00:05:12.040
The word-based search is good at exact names, but it misses related ideas.

s82
00:05:12.040 --> 00:05:17.800
You search for login flow, and it misses everything that says sign in.

s83
00:05:17.800 --> 00:05:23.360
By themselves, both searches miss about one in four results.

s84
00:05:23.360 --> 00:05:26.240
Together, they miss about one in 10.

s85
00:05:26.240 --> 00:05:30.840
They fix each other weakness spots.

s86
00:05:31.200 --> 00:05:35.360
Here is the hard part Foss and I spend most of the time on.

s87
00:05:35.360 --> 00:05:39.360
The search finds results, but they actually relevant?

s88
00:05:39.360 --> 00:05:45.360
Sometimes search results returns 10 results, and none of them are right.

s89
00:05:45.360 --> 00:05:50.880
If they use the bad results, it gives confident wrong answer.

s90
00:05:50.880 --> 00:05:52.800
The worse than no answer.

s91
00:05:52.800 --> 00:05:57.720
We tried asking AI to judge its own results, too slow.

s92
00:05:57.720 --> 00:05:59.800
Add 2-3 seconds every time.

s93
00:05:59.800 --> 00:06:02.600
We tried to fix the score limit, too simple.

s94
00:06:02.600 --> 00:06:06.840
Short questions score low even when the match is perfect.

s95
00:06:06.840 --> 00:06:08.000
It worked.

s96
00:06:08.000 --> 00:06:17.640
Some simple formula, 50% meaning score, 30% keyword score, 20% how the recent code is.

s97
00:06:17.640 --> 00:06:20.560
And the limit adjust based on the current result.

s98
00:06:20.560 --> 00:06:26.960
It runs 0.4 milliseconds, no extra AI calls needed.

s99
00:06:26.960 --> 00:06:33.520
The lesson we learned, simple formula beats the complex model most of the time.

s100
00:06:33.960 --> 00:06:36.880
We need numbers, not just stories.

s101
00:06:36.880 --> 00:06:38.560
So, here are ours.

s102
00:06:38.560 --> 00:06:48.000
We tested an open-source real project, FastAPI, 53 files, 20 real questions a developer would ask.

s103
00:06:48.000 --> 00:06:53.560
Without using our tool, 83 uh K tokens per questions.

s104
00:06:53.560 --> 00:06:57.919
With our tools, uh 4.9 K tokens per questions.

s105
00:06:57.919 --> 00:07:00.840
That is 94% less.

s106
00:07:00.840 --> 00:07:06.600
With the extra compression on top, 523 tokens per question.

s107
00:07:06.600 --> 00:07:13.800
And the accuracy still um find the right code 90% of the code.

s108
00:07:13.800 --> 00:07:15.600
These numbers are real.

s109
00:07:15.600 --> 00:07:16.800
Test is public.

s110
00:07:16.800 --> 00:07:18.360
You can run it yourself.

s111
00:07:18.360 --> 00:07:21.720
The command is on screen.

s112
00:07:22.720 --> 00:07:25.640
I want to be honest about the limits.

s113
00:07:25.640 --> 00:07:31.160
The 94% again the worst case, reading full files every time.

s114
00:07:31.160 --> 00:07:37.520
In a real life, the tools like a cloud code already smarter than that.

s115
00:07:37.520 --> 00:07:40.800
Real savings are lower than 94%.

s116
00:07:40.800 --> 00:07:49.160
We use full file base because it is the only one we can measure the same way every time.

s117
00:07:49.160 --> 00:07:51.520
Big mixed code base are hard.

s118
00:07:51.520 --> 00:07:57.680
We tested on large projects with 396 files.

s119
00:07:57.680 --> 00:07:59.800
The recall dropped almost zero.

s120
00:07:59.800 --> 00:08:03.800
If your files each do one thing, it works well.

s121
00:08:03.800 --> 00:08:07.440
If your files do many things, it struggles.

s122
00:08:07.440 --> 00:08:10.480
We use a small fast model for search.

s123
00:08:10.480 --> 00:08:11.360
It's quick.

s124
00:08:11.360 --> 00:08:14.600
Re-indexing takes under a second.

s125
00:08:14.600 --> 00:08:17.919
But the bigger model would find more.

s126
00:08:17.919 --> 00:08:21.280
We chose speed over perfection.

s127
00:08:21.280 --> 00:08:24.160
Simple choices work better than complex one.

s128
00:08:24.160 --> 00:08:28.000
Small database instead of big infrastructure.

s129
00:08:28.000 --> 00:08:31.360
Two searches instead of one fancy one.

s130
00:08:31.360 --> 00:08:33.240
Local instead of cloud.

s131
00:08:33.240 --> 00:08:35.680
Simple one.

s132
00:08:36.680 --> 00:08:39.240
Here is something Foss pointed out earlier on.

s133
00:08:39.240 --> 00:08:40.400
We use many tools.

s134
00:08:40.400 --> 00:08:42.719
Cloud code for hard problems.

s135
00:08:42.719 --> 00:08:44.440
Cursor for quick edits.

s136
00:08:44.440 --> 00:08:47.800
Copilot for small completions.

s137
00:08:47.800 --> 00:08:51.720
Each tool starts fresh every time.

s138
00:08:51.720 --> 00:08:54.440
They do not share anything.

s139
00:08:54.440 --> 00:08:58.760
You explain the same code base to three different days.

s140
00:08:58.760 --> 00:09:03.839
We built one shared index so all your tools connect to it.

s141
00:09:03.839 --> 00:09:07.880
Same search, same results for all of them.

s142
00:09:07.880 --> 00:09:10.960
And we added memory.

s143
00:09:10.960 --> 00:09:16.320
When one tool learns something about your project, that knowledge stays.

s144
00:09:16.320 --> 00:09:21.520
Next session, different tool, same project, the context is already there.

s145
00:09:21.520 --> 00:09:25.040
We explain the code base once.

s146
00:09:25.040 --> 00:09:27.800
Every tool remembered.

s147
00:09:29.400 --> 00:09:32.520
This is the saving report on real project.

s148
00:09:32.520 --> 00:09:34.680
247 queries.

s149
00:09:34.680 --> 00:09:37.480
12.4 million tokens saved.

s150
00:09:37.480 --> 00:09:40.880
Nearly 186 not spent.

s151
00:09:40.880 --> 00:09:45.520
Most of the savings, 84% came from search layer.

s152
00:09:45.520 --> 00:09:47.720
The rest of them, compression.

s153
00:09:47.720 --> 00:09:49.560
This is not an estimate.

s154
00:09:49.560 --> 00:09:51.480
The tool tracks every query.

s155
00:09:51.480 --> 00:09:57.200
It compares what would have been sent against what was sent.

s156
00:09:57.200 --> 00:10:00.640
Then it multiplies by the model price.

s157
00:10:00.640 --> 00:10:03.160
Run it on your project for a week.

s158
00:10:03.160 --> 00:10:05.960
See your own number.

s159
00:10:05.960 --> 00:10:10.240
Me and Foss built this because we had a huge bill and no good answer.

s160
00:10:10.240 --> 00:10:13.320
The answer was not a better model.

s161
00:10:13.320 --> 00:10:15.120
The answer was sending less.

s162
00:10:15.120 --> 00:10:18.520
We argue about which model is best, Opus or Sonnet.

s163
00:10:18.520 --> 00:10:25.640
But the models may be 30% of the cost, but other 70% is what you feed it.

s164
00:10:25.640 --> 00:10:29.920
Fix the input, the model choice matters less than you think.

s165
00:10:29.920 --> 00:10:33.360
One command to try it, CCE.

s166
00:10:33.360 --> 00:10:34.800
It's free, open source.

s167
00:10:34.800 --> 00:10:36.640
You will see the code on screen.

s168
00:10:36.640 --> 00:10:39.920
Try it, see the number, tell us what you saved.

s169
00:10:39.920 --> 00:10:40.320
Thanks.

s170
00:10:40.320 --> 00:10:42.640
Happy coding.