WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=fWXJM-J0ZB8

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/fWXJM-J0ZB8.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.360 --> 00:00:02.000
Hi there.

s2
00:00:02.000 --> 00:00:09.560
I'm Rachel Lee Neighbors, and today I'm here to talk with you about how to use local models to stop paying for frontier models.

s3
00:00:09.560 --> 00:00:11.360
Let's dig into it.

s4
00:00:11.360 --> 00:00:21.160
So, I've worked on standards that power today's web with Mozilla on Firefox DevTools and the W3C on web standards, and of course on Microsoft's Edge browser.

s5
00:00:21.160 --> 00:00:23.680
I've even been on the React team.

s6
00:00:23.680 --> 00:00:33.760
Now, I've spent the past 3 years consulting with AI startups and some of our favorite LLM and browser companies on all things web, AI, and UI.

s7
00:00:33.760 --> 00:00:36.200
And recently, I've joined Arize.

s8
00:00:36.200 --> 00:00:42.640
Have you ever had your CTO ruin your agentic workflow with a slight change of prompt or an LLM migration?

s9
00:00:42.640 --> 00:00:44.040
Have you ever been that CTO?

s10
00:00:44.040 --> 00:00:49.960
Well, you probably need Arize's observability platform for models and the agents who love them.

s11
00:00:49.960 --> 00:00:55.240
Anyway, I'll be actually using one of Arize's open-source projects today, Phoenix.

s12
00:00:55.240 --> 00:00:57.720
We'll talk more about that later.

s13
00:00:57.720 --> 00:01:02.040
But today, specifically, I'm here to talk with you about how AI is costing you.

s14
00:01:02.040 --> 00:01:10.400
Every time you reach for foundation models like GPT-5 or Claude, it's costing you, your users, and the environment.

s15
00:01:10.520 --> 00:01:16.200
Let's have a look at the costs of one-size-fits-all inference.

s16
00:01:17.560 --> 00:01:19.000
So, first there's security.

s17
00:01:19.000 --> 00:01:20.440
Security costs trust.

s18
00:01:20.440 --> 00:01:30.320
When you use a large LLM that's in the cloud, you're sending data to remote servers, and it always carries the risk of exposure, interception, and retention by third parties.

s19
00:01:30.320 --> 00:01:38.280
We have cases where the use of remote AI chatbots has led to sensitive business data being stored, breached, and leaked to the public.

s20
00:01:38.280 --> 00:01:40.480
Latency costs the user experience.

s21
00:01:40.480 --> 00:01:57.240
Now, there's been research on mitigating response delays in LLM chats in VR and found that 4 seconds is the limit of believability for users, and many calls that you will make to large models are going to take longer than 4 seconds, as we will see when we check Phoenix.

s22
00:01:57.600 --> 00:01:59.040
It cost your business.

s23
00:01:59.040 --> 00:02:04.880
Third-party inference costs are uncontrollable compared to API API costs.

s24
00:02:04.880 --> 00:02:13.120
Agency compounding levels of inference and this means even if the tokens are cheaper, you may be using more of them or if you're using four of them, they may be

s25
00:02:13.120 --> 00:02:14.520
more expensive.

s26
00:02:14.520 --> 00:02:25.000
And of course, if you aren't connected, remote models simply aren't going to work, which means that unless your software is connected to the web, nobody can use it.

s27
00:02:25.000 --> 00:02:28.000
Big inference going offline costs productivity.

s28
00:02:28.000 --> 00:02:35.000
If there's an outage, if you're in a place where you cannot reach Wi-Fi or you're in a very secure environment.

s29
00:02:35.200 --> 00:02:47.760
Now, token costs have been falling as of late, but total inference spend has been rising because agent can reasoning workloads consume tokens way faster than prices are dropping.

s30
00:02:48.200 --> 00:02:55.541
But we can completely eliminate most of these costs and it starts by asking ourselves exactly how much

s31
00:02:55.541 --> 00:02:56.080
[snorts]

s32
00:02:56.080 --> 00:02:57.160
is this costing?

s33
00:02:57.160 --> 00:03:02.920
Do we really need an LLM to do this job?

s34
00:03:02.959 --> 00:03:03.840
All right.

s35
00:03:03.840 --> 00:03:10.640
You can use task-specific models, which are small in size and power consumption compared to an AI foundation.

s36
00:03:10.640 --> 00:03:12.080
Uh this is my little cheat sheet.

s37
00:03:12.080 --> 00:03:16.840
If you're looking for an expert model, is a camera pointing at something?

s38
00:03:16.840 --> 00:03:20.280
Division uh you know, is is there a a vision component?

s39
00:03:20.280 --> 00:03:23.120
You can use a model like MobileNet, YOLO, MediaPipe.

s40
00:03:23.120 --> 00:03:25.200
Is it a microphone recording something?

s41
00:03:25.200 --> 00:03:29.040
There are audio models like Whisper and Wave2Vec2.

s42
00:03:29.200 --> 00:03:30.880
Chat translation or analysis?

s43
00:03:30.880 --> 00:03:36.160
This is where you might use something like a small language model like Gemma or Quinn.

s44
00:03:36.200 --> 00:03:42.920
These uh small models are called SLMs uh or smaller language models.

s45
00:03:42.920 --> 00:03:46.519
The the definition of small is up to debate here.

s46
00:03:46.519 --> 00:03:59.480
Uh these are great for times when we do need the language power of a generative pre-trained trans- transformer, a GPT, but we probably don't need the sum total of human knowledge in a black box at our disposal,

s47
00:03:59.480 --> 00:04:01.800
or we don't need multimodal capabilities.

s48
00:04:01.800 --> 00:04:05.640
We're not going to be analyzing images and audio at the same time.

s49
00:04:05.640 --> 00:04:11.640
Now, SLMs, smaller language models, contain millions to billions of parameters.

s50
00:04:11.640 --> 00:04:16.280
An LLM contains billions to, well, trillions.

s51
00:04:16.280 --> 00:04:17.760
So, you see the big dot there?

s52
00:04:17.760 --> 00:04:20.079
That's one of the smaller LLMs that's out there.

s53
00:04:20.079 --> 00:04:23.440
But, the little dot actually represents one of the larger SLMs.

s54
00:04:23.440 --> 00:04:33.080
So, you can see that there is a huge difference in parameter size, and this means that there is a vast difference in the size of machine you're going to need to run one of these models.

s55
00:04:33.080 --> 00:04:37.919
The good news is you don't need most of what's in the the big green dot there.

s56
00:04:37.919 --> 00:04:47.000
You don't need history, you don't need philosophy, you don't need all those Reddit chats, you don't need a lot of what the models have learned and been trained on.

s57
00:04:47.000 --> 00:04:54.640
Most of us are using our models for things like summarizing a chat thread, or detecting if this person's being a jerk right now.

s58
00:04:54.640 --> 00:05:01.040
And it takes a surprisingly smaller amount of parameters to determine those things.

s59
00:05:01.800 --> 00:05:09.680
Now, the nice thing about smaller language models is that they consume the same or less energy as large language models to produce correct uh awareness.

s60
00:05:09.680 --> 00:05:11.360
We have uh correct responses.

s61
00:05:11.360 --> 00:05:14.640
We have good good research that shows this.

s62
00:05:14.640 --> 00:05:20.720
Small language models um they come in all sizes and shapes.

s63
00:05:20.720 --> 00:05:30.280
Most small language models uh for mobile and web are deployed with quantization, that is to say, 8-bit, 4-bit, and that can have a quarter disk and memory requirements.

s64
00:05:30.280 --> 00:05:35.120
1 billion parameters fits on about 2 GB in FP16.

s65
00:05:35.120 --> 00:05:39.080
Uh this this grid, by the way, gives upper bound estimates.

s66
00:05:39.080 --> 00:05:45.120
So, these are just to give you an idea of what will and will not fit on different devices.

s67
00:05:45.120 --> 00:05:48.480
These are so lightweight that they can be put on devices.

s68
00:05:48.480 --> 00:05:51.280
Speaking of, my Pixel Pro ships with one.

s69
00:05:51.280 --> 00:05:59.720
And I I of course bought the the Pixel 10 Pro as soon as I could because I wanted to know what it would be like to work with an on-device model.

s70
00:05:59.720 --> 00:06:02.200
An SLM of my very own.

s71
00:06:02.200 --> 00:06:05.720
The good news is that small language models are production ready.

s72
00:06:05.720 --> 00:06:09.440
Nvidia called SLMs the future of agentic AI.

s73
00:06:09.440 --> 00:06:18.880
Once again, great research paper from 2025 that found that SLMs are sufficiently powerful for running agentic task loads.

s74
00:06:18.880 --> 00:06:22.200
And they consume less energy than language models.

s75
00:06:22.200 --> 00:06:26.680
Let me Let me give you a bar chart here to take a peek.

s76
00:06:26.680 --> 00:06:34.480
So, let's say this is the total amount of, you know, energy consumed to perform a task that an LLM would take.

s77
00:06:34.480 --> 00:06:38.320
An SLM takes about 25% of that.

s78
00:06:38.320 --> 00:06:41.880
And a task-specific model takes about half of that over.

s79
00:06:41.880 --> 00:06:50.120
So, as you can see, it's from an energy perspective much cheaper to run the smaller language models and task-specific models.

s80
00:06:50.120 --> 00:06:53.560
Um So, what are the benefits?

s81
00:06:53.560 --> 00:07:03.000
Once more, more secure, works offline, no fees, more efficient, and lower latency because it's on device, no round trips.

s82
00:07:03.000 --> 00:07:07.520
Now, I've been using local AI for some time.

s83
00:07:07.520 --> 00:07:10.320
I for instance first started using local models.

s84
00:07:10.320 --> 00:07:19.880
You see on the left you've got Claude, but on the right you have Goose, which is an open an open agent harness that has a a really nice interface

s85
00:07:19.880 --> 00:07:21.800
for chatting with models.

s86
00:07:21.800 --> 00:07:26.960
And this one here, it's running Gemma, and it's able to answer.

s87
00:07:26.960 --> 00:07:28.480
It takes a little longer.

s88
00:07:28.480 --> 00:07:34.680
Smaller models can be a little longer depending on what device you're running and how big the model is.

s89
00:07:34.680 --> 00:07:46.240
But one of my favorite questions to ask a model is how likely is it that ichthyosaurs, marine reptiles from the dinosaur era, had echolocation cuz if you look at their skeletons,

s90
00:07:46.240 --> 00:07:48.320
they look a lot like dolphins.

s91
00:07:48.320 --> 00:07:54.760
And it takes reasoning about biology to be able to determine whether or not that's possible.

s92
00:07:54.760 --> 00:08:09.520
And it's hilarious to me that even I think this was Gemma 3 was able to come up with a good a good response that there is no evidence in the fossil record of their having developed a melon or

s93
00:08:09.520 --> 00:08:14.080
the specialized bones required for conduction that dolphins have.

s94
00:08:14.080 --> 00:08:17.640
Of course, Claude hedged it by saying, "Well, we just can't know."

s95
00:08:17.640 --> 00:08:22.600
Uh about one out of every three times, which was interesting.

s96
00:08:22.600 --> 00:08:28.720
Claude has always been a little nervous about about coming forth with an opinion.

s97
00:08:28.840 --> 00:08:31.360
So, how do you pick the best model for the job?

s98
00:08:31.360 --> 00:08:32.840
This is the hard part.

s99
00:08:32.840 --> 00:08:43.159
Uh I recent I So, how do you pick the best model for the job?

s100
00:08:43.159 --> 00:08:48.440
I built a framework with Google that I use on my own projects.

s101
00:08:48.720 --> 00:08:52.280
You can find more about it at web.dev, but I'll run through it here.

s102
00:08:52.280 --> 00:08:56.320
Now, first off, I like to think of this as prototype big, deploy small.

s103
00:08:56.320 --> 00:08:57.480
Just repeat this to yourself.

s104
00:08:57.480 --> 00:08:58.480
Prototype big.

s105
00:08:58.480 --> 00:08:59.080
Think big.

s106
00:08:59.080 --> 00:08:59.640
Go big.

s107
00:08:59.640 --> 00:09:01.680
Deploy small.

s108
00:09:01.680 --> 00:09:12.520
Um you want to convert the parts of your system over to SLMs and specialized models for production, but you can prototype on a foundation model, no problem.

s109
00:09:12.520 --> 00:09:19.600
Uh the first step in this process is to determine whether or not what you're trying to do is even possible at all.

s110
00:09:19.600 --> 00:09:25.000
You use the largest, most capable model, and you you just see if you can get it to do the thing.

s111
00:09:25.000 --> 00:09:32.560
See if you can get the model to recognize people by their handwriting or the way they they write.

s112
00:09:32.560 --> 00:09:37.040
If that is possible, then you know that another model can probably handle it, too.

s113
00:09:37.040 --> 00:09:46.160
So, you could use like a foundation model like Gemini, or you could use a really really tough uh task-specific model for like handwriting recognition for instance.

s114
00:09:46.160 --> 00:09:51.400
Now, this is a feature of the product that I like to build on the side.

s115
00:09:51.400 --> 00:09:52.600
It's Mima.

s116
00:09:52.600 --> 00:09:56.040
It is a a client for all your social networks.

s117
00:09:56.040 --> 00:10:01.360
And I built a feature for it that summarizes long conversation threads.

s118
00:10:01.360 --> 00:10:06.680
Because I don't know about you, but sometimes I go to bed and I wake up and there's 50 comments on something I posted.

s119
00:10:06.680 --> 00:10:09.120
And I just want to know, are people angry at me?

s120
00:10:09.120 --> 00:10:16.480
And this is what uh this is a feature that saves me saves me a lot of heart attacks in the morning.

s121
00:10:16.480 --> 00:10:20.400
So, I first prototyped it out with Claude to prove that it's good enough.

s122
00:10:20.400 --> 00:10:30.680
So, you can see the little you can see the little summaries there, and says a pretty good job of of saying who's talking about what.

s123
00:10:30.680 --> 00:10:40.120
So, first step for this was to collect a set of inputs and outputs, that is to say in in my case, I wanted to collect a set of threads,

s124
00:10:40.120 --> 00:10:42.280
and then how I would summarize them.

s125
00:10:42.280 --> 00:10:45.760
Or in this case, how Claude summarized them seemed adequate to me.

s126
00:10:45.760 --> 00:10:49.160
So, I I exported a golden data set.

s127
00:10:49.160 --> 00:11:02.240
Now, a golden data set is a curated high-quality collection of preferably human-labeled input-output pairs that you're going to use as the ground truth to evaluate, validate, and benchmark your model.

s128
00:11:02.440 --> 00:11:04.760
This is what the the data looked like.

s129
00:11:04.760 --> 00:11:14.000
Um this is all public knowledge, but you know, you can see it's got the author handle, the content, created at, whether or not it came from me.

s130
00:11:14.000 --> 00:11:17.040
And I created a big JSONL of all of these.

s131
00:11:17.040 --> 00:11:24.240
So, there were 14 threads, and I evaluated each one for summaries and annotations.

s132
00:11:24.240 --> 00:11:31.200
Because some of the summaries actually link deeper to ex link deeper into the conversation.

s133
00:11:31.200 --> 00:11:35.000
So, it was the same threads, but had two different outcomes for them.

s134
00:11:35.000 --> 00:11:38.880
One was a short summary, and one was a summary with references.

s135
00:11:38.880 --> 00:11:39.760
All right.

s136
00:11:39.760 --> 00:11:43.760
So, that means 28 examples all together.

s137
00:11:43.760 --> 00:11:45.360
Um yeah.

s138
00:11:45.360 --> 00:11:49.360
So, these are the things that I was going to be measuring.

s139
00:11:49.360 --> 00:11:56.720
Um you need to before you start doing anything, you need to know what it is that you're measuring.

s140
00:11:56.720 --> 00:11:59.000
What is success?

s141
00:11:59.000 --> 00:12:05.480
In this case, was the JSON being output by the summarization process correct?

s142
00:12:05.480 --> 00:12:09.160
Um an easy way to test is to try JSON parse.

s143
00:12:09.160 --> 00:12:10.960
And if it works, yay.

s144
00:12:10.960 --> 00:12:12.920
If it doesn't, boo.

s145
00:12:12.920 --> 00:12:23.760
Um the reference structural validity, you know, if it's pointing to different parts of the conversation inside the summary, do those parts actually exist?

s146
00:12:23.760 --> 00:12:32.960
Factual consistency, that is to say that it's actually able to summarize the content of the threads within reasonableness.

s147
00:12:32.960 --> 00:12:37.560
So, if we're talking about cats, it doesn't give a summary saying we're talking about puppies.

s148
00:12:37.560 --> 00:12:45.160
Uh this is this is something that we need an LLM to judge or humans, but the LLMs will be cheaper, I assure you.

s149
00:12:45.160 --> 00:12:50.280
Um length compliance, making sure it stays within a certain word count, and of course checking the latency.

s150
00:12:50.280 --> 00:12:58.000
P50, this is, you know, the median across the eval set, and the P95 is the worst-case scenario.

s151
00:12:58.000 --> 00:13:00.080
All right.

s152
00:13:00.839 --> 00:13:12.800
So, you collect these, uh and now you're going to test and do different models, and compare how those models rank against the big models.

s153
00:13:12.800 --> 00:13:15.360
You test from small to large.

s154
00:13:15.360 --> 00:13:21.720
I had to choose a selection of models, but first I need to find a framework, a tool to do the measuring.

s155
00:13:21.720 --> 00:13:25.400
And I chose Phoenix, which happens to be created by Arize.

s156
00:13:25.400 --> 00:13:34.240
It is open source, it's free and the engineers who work on it are pretty darn awesome if I do say so myself.

s157
00:13:34.360 --> 00:13:39.600
We're going to perform first of all a what is known as a capability eval.

s158
00:13:39.600 --> 00:13:44.400
A capability eval asks, you know, what can this agent do well?

s159
00:13:44.400 --> 00:13:51.560
Where we compare the performance of the large model against the performance of a selection of smaller models.

s160
00:13:51.560 --> 00:13:56.560
We say this is what Claude Opus produced and this is what Gemma 4 produced.

s161
00:13:56.560 --> 00:13:59.440
How do these two stack up against each other?

s162
00:13:59.440 --> 00:14:03.120
Now, I used Claude Sonnet for the baseline.

s163
00:14:03.120 --> 00:14:08.640
And you can see actually there in the bottom number one, you can see the the baseline.

s164
00:14:08.640 --> 00:14:09.960
It's looking pretty good.

s165
00:14:09.960 --> 00:14:13.280
It's got an average latency of 2.9 seconds.

s166
00:14:13.280 --> 00:14:22.000
It costs about $0.22 $0.22 to run this to run 14 of these tasks.

s167
00:14:22.000 --> 00:14:29.280
So, I actually did some math and it turns out I'm using about a dollar worth of inference every day using Mima.

s168
00:14:29.280 --> 00:14:34.839
I don't have the money to pay for that many teenage girls using Mima every day.

s169
00:14:34.839 --> 00:14:38.160
So, we're going to have to run this on device.

s170
00:14:38.160 --> 00:14:46.560
Good news is the total cost column for all these small local models is absolutely zilch because that inference has been pushed to the consumer.

s171
00:14:46.560 --> 00:14:47.640
It runs on their device.

s172
00:14:47.640 --> 00:14:52.720
They're the one who has to charge the phone so that it can draw energy from the battery to run the model.

s173
00:14:52.720 --> 00:15:02.640
Of course, you want it to not suck suck the battery dry, but that is another conversation for something that we could be testing and evaluating in the future.

s174
00:15:02.640 --> 00:15:07.400
Now, I chose Let's look at our Let's look at our contestants here.

s175
00:15:07.400 --> 00:15:22.960
We've got Quen 2.5 Instruct weighing in at 1.5 billion parameters, only a handy 1 GB on disk, and its sister Quen 3, 1.7 billion, just a little bit bigger.

s176
00:15:22.960 --> 00:15:29.320
And Llama 3.2, weighing in at 3 billion parameters and a tidy 2 GB on disk.

s177
00:15:29.320 --> 00:15:42.120
And then there was Gemma 4 E2B, which is 5 billion parameters, a hefty 3.1 GB, but this was the one that so many engineers I spoke with when I was picking a model were like, "Oh, Gemma 4 is the best.

s178
00:15:42.120 --> 00:15:50.960
You got to use Gemma 4." And I think that's important here because if I had just gone with what my buddies told me, I may have given the user What?

s179
00:15:50.960 --> 00:15:51.360
Pardon.

s180
00:15:51.360 --> 00:15:52.160
Not may have.

s181
00:15:52.160 --> 00:15:59.840
I would have given the user an extremely different experience, not a good experience.

s182
00:15:59.920 --> 00:16:05.240
You're going to want to select the smallest model that gives acceptable responses for your use case.

s183
00:16:05.240 --> 00:16:10.680
Or as I like to call it, the SAGE model, the small and good enough model.

s184
00:16:10.680 --> 00:16:12.120
I'm trying to make this a thing.

s185
00:16:12.120 --> 00:16:12.920
Bear with me.

s186
00:16:12.920 --> 00:16:15.600
I hope we can make SAGE happen.

s187
00:16:15.600 --> 00:16:22.400
Now, at first, I thought I wanted the the Qwen 2.5 model because it was the fastest.

s188
00:16:22.400 --> 00:16:25.040
You can see that Qwen is all the way here.

s189
00:16:25.040 --> 00:16:25.839
Let's see, where'd it go?

s190
00:16:25.839 --> 00:16:29.800
Yeah, 2.5 is the gray circle in the lower left-hand corner.

s191
00:16:29.800 --> 00:16:35.400
It came in around 1 second total in latency on the P P50.

s192
00:16:35.400 --> 00:16:36.600
That that's amazing.

s193
00:16:36.600 --> 00:16:40.440
Like, that's really fast for for an AI summary.

s194
00:16:40.440 --> 00:16:46.920
The problem was that its accuracy was pretty pretty low compared to everybody else.

s195
00:16:46.920 --> 00:16:53.000
The orange square is Gemma 4 E2B and the blue diamond is Claude Sonnet, which is our ceiling.

s196
00:16:53.000 --> 00:16:58.040
It's the most accurate, but it's also a little pudgy, a little slow.

s197
00:16:58.040 --> 00:17:01.560
It's weighing in around 3 seconds in latency.

s198
00:17:01.560 --> 00:17:15.360
So, when we take accuracy into account, the winner was actually Llama 3.2, which is the big green circle that's weighing in right here around the 90% for accuracy.

s199
00:17:15.360 --> 00:17:31.240
And you can see that both uh both Claude and uh and Llama 3.2 are much faster than Gemma 4. Gemma 4 was coming in around 8 seconds.

s200
00:17:31.240 --> 00:17:38.040
Now, maybe that was because Gemma 4 needed a different kind of prompt, but this was pretty consistent.

s201
00:17:38.040 --> 00:17:43.320
I was testing each one of these three times and then averaging the results.

s202
00:17:43.320 --> 00:17:44.760
This is what the results look like.

s203
00:17:44.760 --> 00:17:52.080
I recommend when you're running evals with something like like like Phoenix, you open up the experiment and you actually take a look.

s204
00:17:52.080 --> 00:17:56.880
You can see what the raw responses were, uh what the expected responses were.

s205
00:17:56.880 --> 00:17:59.760
You can actually compare them against each other.

s206
00:17:59.760 --> 00:18:11.320
Um I found that in many cases, Llama's response was so close to Claude's response as to be pretty much indistinguishable from one another.

s207
00:18:11.320 --> 00:18:12.160
So, yeah.

s208
00:18:12.160 --> 00:18:16.640
Uh Llama 3.2 was the ultimate winner and I decided to move forward with them.

s209
00:18:16.640 --> 00:18:28.400
And this makes sense because Llama 3.2 is created by Meta and Meta has a strong interest in creating models that do a good job with human inputs and summarizing human things on a social network.

s210
00:18:28.400 --> 00:18:29.880
Makes sense, right?

s211
00:18:29.880 --> 00:18:31.400
I mean, it's a social thing.

s212
00:18:31.400 --> 00:18:34.720
Benefits from a a social model.

s213
00:18:34.720 --> 00:18:36.600
So, recap.

s214
00:18:36.600 --> 00:18:39.640
Here's how you right-size your model in four steps.

s215
00:18:39.640 --> 00:18:41.320
Number one, you prove it's possible.

s216
00:18:41.320 --> 00:18:46.560
You test whether what you're trying to accomplish is possible at all by using the largest possible model.

s217
00:18:46.560 --> 00:18:50.543
This could be a foundation model like Gemini or a task-specific model.

s218
00:18:50.543 --> 00:18:50.720
[snorts]

s219
00:18:50.720 --> 00:18:52.520
And you set success criteria.

s220
00:18:52.520 --> 00:19:00.600
You collect a set of inputs and outputs um that that you want to see coincide with one another.

s221
00:19:00.600 --> 00:19:02.120
This will be the bar.

s222
00:19:02.120 --> 00:19:03.640
And then you test from small to large.

s223
00:19:03.640 --> 00:19:15.280
You compare the outputs of small models against your test criteria and you work your way up from the smallest model until you you within an acceptable range of that small and good enough model, that sage model.

s224
00:19:15.280 --> 00:19:23.480
That's when you select your sage model, the smallest model that gives acceptable responses for your inputs.

s225
00:19:24.080 --> 00:19:26.840
But you're probably wondering, what are we going to do about that gap?

s226
00:19:26.840 --> 00:19:33.280
I mean, 90% accuracy, that sounds like it could be uh you know, that could be a deal breaker.

s227
00:19:33.280 --> 00:19:37.760
Well, we can squeeze better performance out of smaller models with prompt engineering.

s228
00:19:37.760 --> 00:19:41.520
This is important in cases where you can't control which model you're using.

s229
00:19:41.520 --> 00:19:48.760
Some people might, for instance, create a distilled model that's been trained to do this one task really well.

s230
00:19:48.760 --> 00:19:59.360
But if you're working, for instance, with a mobile app, you might not want to be using a distilled model because every time you add capabilities, you'll probably have to train the model slightly,

s231
00:19:59.360 --> 00:20:03.640
and then you'll be shipping a new 1 or 2 GB model every time to your users.

s232
00:20:03.640 --> 00:20:09.240
So, this is a great example of like, "Ooh, I don't think I'll be able to control that model.

s233
00:20:09.240 --> 00:20:12.640
Once it's on the user's device, I'm not going to ask them to download new ones.

s234
00:20:12.640 --> 00:20:14.960
That would eat up their data plan."

s235
00:20:14.960 --> 00:20:28.080
Um you might join a team, and they're already committed to Gemma 3, and they're just not going to be implementing Gemma 4 until they've rolled out an an an update that uh you know, addresses all of the evals.

s236
00:20:28.080 --> 00:20:32.240
So, let's take a look at closing the gap between Saona and Llama.

s237
00:20:32.240 --> 00:20:37.400
Now, the the green and blue dots in that upper left-hand quadrant.

s238
00:20:37.920 --> 00:20:49.920
So, the measures I I honed in on, the ones that they really seem to have different results with, were JSON reference uh JSON and reference structural validity, factual consistency,

s239
00:20:49.920 --> 00:20:53.360
um P50 latency, and P95 latency.

s240
00:20:53.360 --> 00:21:05.600
And I decided that the P50 latency the cutoff is, you know, one 1 and 1/2 seconds for P95 would be 3 and 1/2 because, remember, 4 seconds is that worst-case scenario for people

s241
00:21:05.600 --> 00:21:12.200
feeling disconnected from an AI powered experience as per the research.

s242
00:21:13.200 --> 00:21:16.800
So, where do we start?

s243
00:21:16.800 --> 00:21:20.920
I recommend that you optimize one step at a time.

s244
00:21:20.920 --> 00:21:31.880
You want to isolate one variable per prompt variant to test whether what you're trying to accomplish is moving the needle when you're using the different prompts.

s245
00:21:33.840 --> 00:21:36.160
So, I created five prompts.

s246
00:21:36.160 --> 00:21:40.120
Well, I created four prompts and I had the original prompt as the baseline.

s247
00:21:40.120 --> 00:21:41.760
The original prompt was pretty good.

s248
00:21:41.760 --> 00:21:44.200
V2 used number input.

s249
00:21:44.200 --> 00:21:49.120
The same prompt reformatted the threat the threat as numbered messages instead of using JSON.

s250
00:21:49.120 --> 00:21:58.080
The hypothesis being that smaller models could track natural language indexing better than array offsets in heap of JSON.

s251
00:21:58.080 --> 00:22:00.520
The second one was a few shot.

s252
00:22:00.520 --> 00:22:05.080
It added a couple of examples and outcomes to the prompt.

s253
00:22:05.080 --> 00:22:11.000
The hypothesis here being that small models learn format from examples faster than from rules.

s254
00:22:11.000 --> 00:22:13.200
Then there was the strict rules version.

s255
00:22:13.200 --> 00:22:16.080
This one was a house of no prompt.

s256
00:22:16.080 --> 00:22:18.480
It had explicit negative constraints.

s257
00:22:18.480 --> 00:22:19.640
No preamble.

s258
00:22:19.640 --> 00:22:21.760
Don't count words before responding.

s259
00:22:21.760 --> 00:22:23.800
I mean, do count words before responding.

s260
00:22:23.800 --> 00:22:29.880
And the hypothesis was that small models respond to literal commands and that they like to be bossed around a bit.

s261
00:22:29.880 --> 00:22:34.520
And then lastly, there was chain of thought which forced the model to identify key moments before writing.

s262
00:22:34.520 --> 00:22:39.520
The hypothesis just being that thinking out loud would improve grounding.

s263
00:22:39.600 --> 00:22:39.795
So,

s264
00:22:39.795 --> 00:22:41.240
[snorts]

s265
00:22:41.240 --> 00:22:50.080
ran the tests again with these new prompts and just just just to the llama 3.2 and we discovered a couple of things.

s266
00:22:50.080 --> 00:22:52.360
We I mean, myself.

s267
00:22:52.360 --> 00:22:56.640
I just ran this locally on my machine and was able to compare the results.

s268
00:22:56.640 --> 00:23:02.560
You can even put the results into something like Claude and have have conversation about the trades if you like.

s269
00:23:02.560 --> 00:23:09.120
So, stop that.

s270
00:23:09.120 --> 00:23:11.680
All right, we're back.

s271
00:23:12.320 --> 00:23:23.440
So, in this case, the baseline wasn't really good at at the determining how short it should be so it would fit within a certain section of content.

s272
00:23:23.440 --> 00:23:27.120
The ref accuracy was 91.2%.

s273
00:23:27.120 --> 00:23:33.000
It was factually correct 87.1% of the time and the latency was 1 second.

s274
00:23:33.000 --> 00:23:37.080
Reformatted input didn't really make a difference.

s275
00:23:37.080 --> 00:23:41.880
Explicit rules actually didn't made things worse.

s276
00:23:41.880 --> 00:23:47.080
The model responded very negatively to being told what it couldn't do couldn't couldn't do.

s277
00:23:47.080 --> 00:23:51.600
It um it was a a naughty child didn't like to take instructions.

s278
00:23:51.600 --> 00:24:05.240
Chain of thought didn't have that big a it it did a little bit better on the length and unfortunately that came at increasing the latency by 600 milliseconds.

s279
00:24:05.240 --> 00:24:10.880
The best performing one was the few shot one that provided a couple of threads and a couple of examples.

s280
00:24:10.880 --> 00:24:14.520
It was much better at getting the length right.

s281
00:24:14.520 --> 00:24:27.360
It was more accurate and in its references it agreed with the Claude model a little bit more about what was said and it only increased the the latency by 200 milliseconds.

s282
00:24:27.360 --> 00:24:31.080
So, that sounds like a pretty good deal, right?

s283
00:24:31.240 --> 00:24:36.560
So, it was the few shot prompt that one that made the biggest improvement.

s284
00:24:36.560 --> 00:24:41.840
Let's have a look at how it stacks up to the original.

s285
00:24:41.840 --> 00:24:43.080
Bum bum bum.

s286
00:24:43.080 --> 00:24:47.840
So, you can see the bar here Claude Sonnet versus Llama 3.2 3B.

s287
00:24:47.840 --> 00:24:56.680
Um I actually did a couple of things at this point because I wanted to close that gap completely.

s288
00:24:57.760 --> 00:25:12.960
So, Llama 3.2B with a few shot prompt was actually able to get within a reasonable error area of um a reasonable error margin of error.

s289
00:25:12.960 --> 00:25:19.080
The P50 latency went to less than 1 and 1/2 minutes, so it was totally green.

s290
00:25:19.080 --> 00:25:36.360
Uh the structural validity 91.7% factual consistency was also at 92.9% Uh the P95 latency was well under 750 milliseconds uh less than Claude.

s291
00:25:36.360 --> 00:25:39.920
In general, its latency was really good.

s292
00:25:39.920 --> 00:25:48.720
But, let's see about those uh those couple of those 10% here between structural validity and factual consistency.

s293
00:25:48.720 --> 00:25:53.880
This is why it's important to actually open up your e-vows and take a look at what's inside.

s294
00:25:53.880 --> 00:25:59.280
Uh when it came to factual consistency, it turned out that Claude was just being a very strict judge.

s295
00:25:59.280 --> 00:26:16.880
I was using Claude to judge the responses and comparing, you know, Claude Opus was comparing Claude Sonnet's response to uh Llama 3.2's response, and of course Claude was favoring its little sister and being like, "Yeah, um I think I think that uh

s296
00:26:16.880 --> 00:26:27.440
you know, I don't think your interpretation of what Jenna said is accurate because you said she was being angsty and uh she was actually being cross."

s297
00:26:27.440 --> 00:26:28.560
It was that sort of thing.

s298
00:26:28.560 --> 00:26:31.960
And this is why it's important to crack them open.

s299
00:26:31.960 --> 00:26:39.320
Now, as for rough consistency and length, those actually could be handled inside the harness and post-processing.

s300
00:26:39.320 --> 00:26:45.600
So, making sure that it's, you know, got the right number of references in the thread, that's something very simple to look at.

s301
00:26:45.600 --> 00:26:47.840
You can just see how long the thread is.

s302
00:26:47.840 --> 00:26:57.760
And if there are more refs than there are members of the thread, that's incorrect uh when it comes to, you know, how long the actual uh summary is.

s303
00:26:57.760 --> 00:27:01.120
If it's too long, you can just truncate it.

s304
00:27:01.120 --> 00:27:07.440
And so, when we added the post-processing, we're actually able to close that gap pretty solidly.

s305
00:27:07.440 --> 00:27:20.793
Now, we've got uh 100% JSON validity, structural validity was 100%, factual consistency, there's only a little bit of a disagreement there, and it turns out that it was the judge being too darn strict.

s306
00:27:20.793 --> 00:27:21.520
[sighs and gasps]

s307
00:27:21.520 --> 00:27:34.920
We got the P5 uh P50 latency down to around a 1 second, and P95 uh it was under 300 and uh three three three three and a half seconds.

s308
00:27:34.920 --> 00:27:35.760
So, that's pretty good.

s309
00:27:35.760 --> 00:27:46.880
It actually ended up meeting and beating Claude's on it after doing this little bit of extra effort, and I'm saving about a dollar a day in inference costs.

s310
00:27:47.320 --> 00:27:53.200
So, the important thing is, after you've done something like this, you don't want to lose one ground.

s311
00:27:53.200 --> 00:28:01.920
You'll want to make sure that uh you can update the prompt in the future or upgrade the model.

s312
00:28:01.920 --> 00:28:03.280
You don't want to lose one ground.

s313
00:28:03.280 --> 00:28:14.800
You want to ensure that you can upgrade the the model or change the prompt in the future, and it won't cause summaries to expand to be a paragraph long

s314
00:28:14.800 --> 00:28:18.680
or to start hallucinating things that are untrue.

s315
00:28:18.680 --> 00:28:29.120
You want to keep your evals running for that, and that would be a regression eval, and you run these sort of like you run um CICD tests, you know,

s316
00:28:29.120 --> 00:28:34.440
uh it's how you keep your CTO from blowing away your agentic experience by accident one morning.

s317
00:28:34.440 --> 00:28:38.200
True story, happened to a founder friend of mine.

s318
00:28:38.200 --> 00:28:40.000
So, where do you get started?

s319
00:28:40.000 --> 00:28:42.720
Uh how many Claude calls could be Llama calls?

s320
00:28:42.720 --> 00:28:55.400
I I challenge you to go home today and take a look at what you're sending to LLM's and ask yourself, is this something that a smaller model could handle and how much money would I save if I did that?

s321
00:28:55.520 --> 00:29:02.720
When working on AI projects, keep an eye out for SLM's and specialized models that might be on device already.

s322
00:29:02.720 --> 00:29:15.920
For instance, Chrome and the prompt API, they access Gemini Nano which ships with Chrome um natively and that can be really useful cuz that means you don't need to be shipping a model to anyone using the browser.

s323
00:29:15.920 --> 00:29:19.400
You can take advantage of what's right there.

s324
00:29:20.280 --> 00:29:24.680
Keep in mind that these models are more efficient and they meet your users where they are.

s325
00:29:24.680 --> 00:29:26.840
Their information stays on device.

s326
00:29:26.840 --> 00:29:37.520
You don't have to worry about PII and you know, the energy consumption is well, a blessing for the environment so to speak.

s327
00:29:38.080 --> 00:29:41.920
Remember to prototype big and deploy small.

s328
00:29:41.920 --> 00:29:50.720
You may want to prototype your system with a foundation model and then convert it parts of it to small language models and specialized models for production.

s329
00:29:50.800 --> 00:29:58.800
Keep in mind you want to prove it, define it, test it and then select your sage model that passes the test.

s330
00:29:59.000 --> 00:30:07.520
You can use prompt engineering to help get better results from SLM's and close that gap with remote models.

s331
00:30:08.080 --> 00:30:17.000
So, I challenge you to consider your current implementation of prototype and convert one feature to use a smaller local model.

s332
00:30:17.000 --> 00:30:21.560
Go home, try it out for yourself and you might be surprised.

s333
00:30:21.960 --> 00:30:24.320
Anyway, I look forward to see you out there on the web.

s334
00:30:24.320 --> 00:30:28.200
If you enjoyed talking about small local models or the attentive web, you can follow me.

s335
00:30:28.200 --> 00:30:30.000
I'm at nearestneighbors.com.

s336
00:30:30.000 --> 00:30:36.440
If you want to try running some evals with your current setup, you can start testing with Phoenix at phoenix.arise.com.

s337
00:30:36.520 --> 00:30:43.400
And lastly, you can sign up for uh the beta for Mima uh at mima.social.

s338
00:30:43.400 --> 00:30:46.800
I've been Rachel Neighbors, and it's been awesome chatting with you today.

s339
00:30:46.800 --> 00:30:51.440
Go forth and build your own inference stack.