WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=3axY1wz2OvA

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/3axY1wz2OvA.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.360 --> 00:00:14.640
Thank you everyone for taking the time to be here today.

s3
00:00:14.640 --> 00:00:16.040
My name is Oliver Cannon.

s4
00:00:16.040 --> 00:00:19.320
I'm a research analyst at SemiAnalysis.

s5
00:00:19.320 --> 00:00:23.520
Um time and attention are the most valuable things any of us have.

s6
00:00:23.520 --> 00:00:27.320
Uh unless of course you're an early stage investor in OpenAI or Anthropic.

s7
00:00:27.320 --> 00:00:30.520
Um with that, uh let's get into it.

s8
00:00:30.520 --> 00:00:32.720
Uh not all GPUs are deployed the same.

s9
00:00:32.720 --> 00:00:35.360
A GPU falls to the level of the operator.

s10
00:00:35.360 --> 00:00:38.040
Um buying hardware gives you potential.

s11
00:00:38.040 --> 00:00:43.800
Uh operating capability is what determines how much useful work you actually are able to deliver.

s12
00:00:43.800 --> 00:00:47.600
Uh that leads us to the question I will attempt to answer with all of you today.

s13
00:00:47.600 --> 00:00:51.200
Who can produce the intelligence you need at the lowest cost?

s14
00:00:51.200 --> 00:00:52.320
Cost matters.

s15
00:00:52.320 --> 00:01:04.320
Uh who uh in today's presentation I want to show you guys why and how operating capability changes that cost and why it matters in the commodity market that is compute.

s16
00:01:05.400 --> 00:01:07.440
What matters in a commodity market?

s17
00:01:07.440 --> 00:01:11.800
Um a lot of tech investors or technologists may not be familiar with this framework.

s18
00:01:11.800 --> 00:01:14.160
Um so a little crash course.

s19
00:01:14.160 --> 00:01:18.600
Uh in a commodity market producers compete to deliver comparable output.

s20
00:01:18.600 --> 00:01:22.840
Uh a lower production cost gives more room to compete when prices fall.

s21
00:01:22.840 --> 00:01:25.440
Uh that's what a cost curve shows.

s22
00:01:25.440 --> 00:01:30.000
Producers lined up by what they spend to deliver the same usable work.

s23
00:01:30.000 --> 00:01:36.160
If two producers earn the same price, the one spending less to deliver the work gets to keep more of it.

s24
00:01:36.160 --> 00:01:38.640
Pretty simple, right?

s25
00:01:39.600 --> 00:01:43.400
For us, intelligence is the economic output we care about.

s26
00:01:43.400 --> 00:01:49.400
Eventually, our North Star metric will be price per successfully completed task at the required quality.

s27
00:01:49.400 --> 00:01:51.360
Uh unfortunately, we're not quite there yet.

s28
00:01:51.360 --> 00:01:53.600
That's a pretty hard thing to standardize.

s29
00:01:53.600 --> 00:02:04.560
Um today price per token is a useful practical proxy, especially for comparable workloads and quality requirements, and gets us closer to delivered output.

s30
00:02:04.560 --> 00:02:13.680
Uh much closer than the often quoted uh all you finance bros out here will have heard this uh GPU uh dollar per hour price.

s31
00:02:13.680 --> 00:02:16.560
Um or as we view it, a cost.

s32
00:02:16.560 --> 00:02:23.840
Um We view GPU hours as a measure of input, not output, the cost versus price distinction.

s33
00:02:23.840 --> 00:02:29.520
Customer prices are different from the producer's cost of doing the work.

s34
00:02:31.320 --> 00:02:34.959
GPU hours are the input, intelligence is the output.

s35
00:02:34.959 --> 00:02:39.880
Uh on this slide, we can visualize the inputs needed to create useful intelligence.

s36
00:02:39.880 --> 00:02:45.520
Uh we start with hardware and electricity, GPUs uh for the the this presentation.

s37
00:02:45.520 --> 00:02:54.760
Software and operations can be thought of as a multiplier on those first two inputs, with the end product being useful intelligence.

s38
00:02:56.240 --> 00:03:02.959
The operator is who chooses and implements the software that serves the model and keeps requests moving through the system.

s39
00:03:03.440 --> 00:03:10.760
The machines might be available, but you still need someone to make them deliver responses customers can actually use.

s40
00:03:10.760 --> 00:03:16.959
What matters is how much useful work you get and whether it meets your requirements for the specified task.

s41
00:03:16.959 --> 00:03:20.040
The conversion can differ with the same kind of chip.

s42
00:03:20.040 --> 00:03:23.320
Let's look at that in the measured example.

s43
00:03:23.760 --> 00:03:28.519
Here's a concrete example from our InferenceX public InferenceX calculator.

s44
00:03:28.519 --> 00:03:31.880
You guys can all check it out, semi-analysis.com.

s45
00:03:31.880 --> 00:03:42.239
Uh both estimates use B200 GPUs serving the same model with roughly 8,000 input tokens and 1,000 output tokens, a pretty standard ratio.

s46
00:03:42.239 --> 00:03:46.560
The P50 target is 100 output tokens per second for each user.

s47
00:03:46.560 --> 00:03:50.360
Uh the serving stack is the software handling the model's responses.

s48
00:03:50.360 --> 00:03:59.160
In the P90 scenario, one of these stacks gives about 200 generated output tokens per second, while the other one gives about 300.

s49
00:03:59.160 --> 00:04:05.440
That's a big difference and comes down to the software, settings, and number of requests running.

s50
00:04:05.440 --> 00:04:08.959
All choices made by the operator.

s51
00:04:11.240 --> 00:04:16.720
Same chip, same target interactivity, about 50% more output.

s52
00:04:16.720 --> 00:04:17.600
That's a lot.

s53
00:04:17.600 --> 00:04:20.880
So, what does that mean for the buyer's bill?

s54
00:04:23.320 --> 00:04:26.840
What does that intelligence actually cost the customer?

s55
00:04:26.840 --> 00:04:33.480
For this this example, let's assume the hourly bill is the same for both deployments and both meet your requirements.

s56
00:04:33.480 --> 00:04:39.000
At roughly 1 and 1/2 times the output, your rental spend per acceptable token is proportionally lower.

s57
00:04:39.000 --> 00:04:45.560
That is to say, the hour buys you more usable and useful output.

s58
00:04:45.960 --> 00:04:48.400
What counts as useful intelligence?

s59
00:04:48.400 --> 00:04:51.360
Useful intelligence has to do the job you hired it to do.

s60
00:04:51.360 --> 00:04:52.840
Pretty simple, right?

s61
00:04:52.840 --> 00:04:58.640
The answer needs to be good enough for the task, arrive quickly enough, and be available when you need it.

s62
00:04:58.640 --> 00:05:03.200
Your information also needs to stay within the access boundaries you you've agreed upon.

s63
00:05:03.200 --> 00:05:06.720
AKA, it needs to be secure.

s64
00:05:06.720 --> 00:05:15.400
This is an overlooked requirement that is not captured in the chip's performance metrics, but it is essential for a chip to be usable and useful.

s65
00:05:15.520 --> 00:05:21.840
In our published security work, we found monitoring access that exposed another customer's logs and metrics.

s66
00:05:21.840 --> 00:05:31.160
This is a case, and an unfortunately common one at that, where the GPUs were fast, and in the case I'm referencing, really, really fast, which makes the whole thing sadder.

s67
00:05:31.160 --> 00:05:34.400
Uh and the service still failed horrifically.

s68
00:05:34.400 --> 00:05:41.560
Um ensuring cluster security takes ongoing work, just as delivering the promised performance does.

s69
00:05:42.919 --> 00:05:47.400
In most cases, it is the operator that owns that responsibility.

s70
00:05:48.440 --> 00:05:51.680
Someone has to make the system work in production.

s71
00:05:51.680 --> 00:05:54.440
Suppose a faster serving update becomes available.

s72
00:05:54.440 --> 00:06:03.880
Someone has to test it on the customer's workload, check whether the output still meets the requirements, and then decide when, where, and how to deploy it.

s73
00:06:03.880 --> 00:06:14.240
That means checking the actual responses, testing it under load, and keeping a way to return to previous version if the update causes problems once it's deployed.

s74
00:06:14.240 --> 00:06:19.240
Once it is deployed, someone has to maintain it and respond when issues come up.

s75
00:06:20.880 --> 00:06:25.560
That work may sit with your provider, your own team, or both.

s76
00:06:25.560 --> 00:06:32.440
Irregardless, you need to know who's responsible and whether they can actually do it.

s77
00:06:33.560 --> 00:06:35.600
So, how do we assess the operator?

s78
00:06:35.600 --> 00:06:46.960
Uh SemiAnalysis ClusterMax, we're in version 3.0 at this point, uh assesses operating cap- capability on managed GPU clusters, couple couple hundred, uh growing.

s79
00:06:46.960 --> 00:06:54.160
Please reach out if you want to be on ClusterMax 4.0. Um we look at provisioning, reliability, monitoring, and security.

s80
00:06:54.160 --> 00:07:04.400
Uh hands-on testing, documentation, and customer's feedback help us understand how the service runs, not just on a good day, but also when a machine fails.

s81
00:07:05.840 --> 00:07:13.840
I want to take the chance to reiterate that ClusterMax ratings apply to managed clusters, uh different from bare metal, very different.

s82
00:07:13.840 --> 00:07:17.440
Um with bare metal, more operating work usually sits with the customer.

s83
00:07:17.440 --> 00:07:24.240
Uh applying a managed service rating there does not automatically cover every product a provider sells.

s84
00:07:24.280 --> 00:07:27.400
The tiers are relative to peers.

s85
00:07:27.600 --> 00:07:28.640
Little bit of a tongue twister.

s86
00:07:28.640 --> 00:07:30.320
Uh I nailed it, pretty happy with that.

s87
00:07:30.320 --> 00:07:30.645
Uh

s88
00:07:30.645 --> 00:07:31.160
[laughter]

s89
00:07:31.160 --> 00:07:35.280
and pricing and availability are also important factors in this assessment.

s90
00:07:35.280 --> 00:07:42.400
Uh these are metrics that do not directly measure uh token output, uh but are incredibly important.

s91
00:07:42.400 --> 00:07:43.760
Obviously, very important.

s92
00:07:43.760 --> 00:07:46.400
Uh assessment helps us understand who can deliver.

s93
00:07:46.400 --> 00:07:48.280
Now we can move on to the next question.

s94
00:07:48.280 --> 00:07:51.320
What does it actually cost to produce useful intelligence?

s95
00:07:51.320 --> 00:07:56.920
Let's start with a compute bill and other costs that producers still has to cover.

s96
00:07:57.760 --> 00:08:00.160
Rental price and production cost are different.

s97
00:08:00.160 --> 00:08:08.840
In our example, GPU provider the GPU provider rents out hours, the inference a producer buys those hours and sells responses.

s98
00:08:08.840 --> 00:08:13.080
The rental bill is revenue to the provider and an input cost to the inference producer.

s99
00:08:13.080 --> 00:08:17.640
The GPU owner pays for the equipment, power, space, and support.

s100
00:08:17.640 --> 00:08:25.440
The inference producer's full cost includes rent plus whatever software, staff, and other services it pays for beyond that rental.

s101
00:08:25.440 --> 00:08:32.840
If the inference producer owns its own it equipment instead, we spread the equipment investment over its working life and add running costs.

s102
00:08:32.840 --> 00:08:36.240
What the producer charges and what it spends are separate numbers.

s103
00:08:36.240 --> 00:08:42.360
To understand the cost of useful intelligence, we need to know how much useful work the spending actually produces.

s104
00:08:42.360 --> 00:08:46.200
Uh better operations lowers that cost of useful intelligence.

s105
00:08:46.200 --> 00:08:50.520
The B200 comparison showed why the amount of output matters.

s106
00:08:50.520 --> 00:08:57.560
More acceptable responses from the same paid hours can spread the inference producer's rental expense over more useful work.

s107
00:08:57.560 --> 00:09:03.800
For a producer that owns its GPUs, the same logic spreads the equipment investment and running costs over more output.

s108
00:09:03.800 --> 00:09:13.200
Count all the spending needed to deliver the service over the same period, then divide by the output customers can actually use.

s109
00:09:13.200 --> 00:09:17.960
That gives us cost for acceptable token for a comparable workload.

s110
00:09:19.880 --> 00:09:22.800
Better operations also require investment.

s111
00:09:22.800 --> 00:09:30.240
Testing an update, deploying it safely, uh and keeping it working take people and money.

s112
00:09:30.240 --> 00:09:33.640
The extra output has to justify the extra spending.

s113
00:09:33.640 --> 00:09:42.240
Uh this advantage can persist when a team repeats that work as software and workloads change, but one good benchmark doesn't establish it.

s114
00:09:42.240 --> 00:09:45.040
You have to earn it every single day.

s115
00:09:45.040 --> 00:09:55.720
Uh each new workload still needs checking, but a repeatable operating process, from what we've observed, can support a lasting cost advantage and a price advantage, too.

s116
00:09:55.720 --> 00:10:01.040
Uh even if you do keep improving, though, your competitors are not sitting still.

s117
00:10:01.040 --> 00:10:04.040
Uh your position on the cost curve is relative.

s118
00:10:04.040 --> 00:10:10.520
This is This slide, out of all the slides, should make clear the importance of the commodity framework in all of this.

s119
00:10:10.520 --> 00:10:15.160
Um your position on the cost curve depends on what your competitors can deliver, not just yourself.

s120
00:10:15.160 --> 00:10:16.960
It's a relative measure.

s121
00:10:16.960 --> 00:10:18.680
Here's a simple hypothetical.

s122
00:10:18.680 --> 00:10:24.200
We're comparing the same task, the same required quality, using made-up cost units.

s123
00:10:24.200 --> 00:10:26.720
Uh I like to imagine bars of gold.

s124
00:10:26.720 --> 00:10:35.960
Uh your full cost per successful task falls from 10 to 8. Uh in the same period, your competitor's falls from 6 to 12.

s125
00:10:37.200 --> 00:10:38.160
You improved.

s126
00:10:38.160 --> 00:10:39.480
That's awesome.

s127
00:10:39.480 --> 00:10:40.800
But both of you did.

s128
00:10:40.800 --> 00:10:45.800
And in this example, you started as the cheaper producer and ended as the more expensive one.

s129
00:10:45.800 --> 00:10:52.760
That gives the competitor more room to lower its selling price, provide provided customers find the two services equally acceptable.

s130
00:10:52.760 --> 00:10:56.320
Uh getting better does not automatically move you ahead.

s131
00:10:56.320 --> 00:11:02.839
Applying the commodity framework makes you realize that you have to compare your your cost per useful work on a relative basis.

s132
00:11:02.839 --> 00:11:07.280
Comparing your own product with what your competitors can deliver.

s133
00:11:07.280 --> 00:11:15.280
That comparison can be easy to overlook in moments like today, where demand far outstrips supply and will for the foreseeable future.

s134
00:11:15.280 --> 00:11:19.040
Uh two to three years visibility.

s135
00:11:19.040 --> 00:11:21.040
Good.

s136
00:11:21.800 --> 00:11:24.000
Scarcity can obscure those differences.

s137
00:11:24.000 --> 00:11:30.000
When demand is strong and supply is constrained, producers with very different costs can can find customers.

s138
00:11:30.000 --> 00:11:33.040
Strong revenue can make those differences feel less urgent.

s139
00:11:33.040 --> 00:11:43.080
If you're a CEO, why would you focus on your precious time and energy and money shoring up your spot on the cost curve when you can be building a massive new data center?

s140
00:11:43.080 --> 00:11:47.160
Uh in moments like this, it's important to step back and have a little perspective.

s141
00:11:47.160 --> 00:11:49.000
Uh take a little historical perspective.

s142
00:11:49.000 --> 00:11:53.240
Uh every commodity in the history of humanity has had swings in supply and demand.

s143
00:11:53.240 --> 00:11:57.960
Just because we don't see when that will be yet does not mean that this cycle will be any different.

s144
00:11:57.960 --> 00:12:01.280
Uh unless you're a total AI maxi, which some of us are.

s145
00:12:01.280 --> 00:12:03.200
So, I'm very open to being wrong.

s146
00:12:03.200 --> 00:12:05.200
I'm wrong every day.

s147
00:12:05.200 --> 00:12:11.400
Um but I would say if you're a business with leverage, you need to be especially conscientious of this.

s148
00:12:11.400 --> 00:12:22.040
Being proactive with your built with building your cost advantage uh might be not be as sexy as making your top line bigger, but it matters deeply if you want to survive through a business cycle.

s149
00:12:22.040 --> 00:12:31.080
Uh that's why you want to build your operating capability and cost advantage while you have the opportunity uh when the sun is shining, when you can raise capital.

s150
00:12:31.080 --> 00:12:36.320
Uh they matter even more when customers have more choices to make and prices come under pressure.

s151
00:12:36.320 --> 00:12:41.280
So, how do we make clear what uh the buyer actually gets?

s152
00:12:41.680 --> 00:12:44.800
Comparable compute needs clear grades.

s153
00:12:44.800 --> 00:12:48.880
Now, let's look at the compute uh that an intelligence producer is buying.

s154
00:12:48.880 --> 00:12:54.080
We've already seen that the same chip can deliver different results depending on how it's operated.

s155
00:12:54.080 --> 00:13:01.680
Uh in two offers for B200 GPU hours can include different services and leave different operating jobs with the customer.

s156
00:13:01.680 --> 00:13:08.440
A commodity framework needs a way to describe these differences so buyers know which services they can treat as comparable.

s157
00:13:08.440 --> 00:13:11.400
That's the job of a grade.

s158
00:13:12.280 --> 00:13:16.960
The grade describes a particular compute service and what its buyers can expect.

s159
00:13:16.960 --> 00:13:20.839
It doesn't make intelligence interchangeable across every task.

s160
00:13:20.839 --> 00:13:26.320
Once we've defined that service, we can ask what its rental price tells us.

s161
00:13:27.040 --> 00:13:30.240
What does a rental price benchmark measure?

s162
00:13:30.240 --> 00:13:35.960
Uh a rental price benchmark summarizes the price of GPU hours within its defined market.

s163
00:13:35.960 --> 00:13:40.640
It isn't the producer's production cost or a measure of intelligence delivered.

s164
00:13:40.640 --> 00:13:42.840
We need to know where those prices come from.

s165
00:13:42.840 --> 00:13:47.200
An advertised offer and a completed trade are not the same thing.

s166
00:13:47.200 --> 00:13:56.520
Service scope, contract terms, uh whose prices are sampled and how they're combined all shape that reference point.

s167
00:13:56.520 --> 00:14:01.320
Uh operating quality can influence those prices without the index measuring it separately.

s168
00:14:01.320 --> 00:14:06.040
Running a rental reference doesn't tell you how much useful work your deployment will produce.

s169
00:14:06.040 --> 00:14:12.800
To use it as a hedge, you need to understand how it tracks the rent you actually pay or earn.

s170
00:14:13.440 --> 00:14:16.000
Uh what can compute futures hedge?

s171
00:14:16.000 --> 00:14:19.080
There's been a lot of excitement lately about compute futures.

s172
00:14:19.080 --> 00:14:25.560
For all of you that don't know, a futures contract sets a standardized term for future delivery or cash settlement.

s173
00:14:25.560 --> 00:14:27.760
Cash settlement depends on a price reference.

s174
00:14:27.760 --> 00:14:31.880
Buyers and sellers both face uncertainty about rental prices.

s175
00:14:31.880 --> 00:14:35.640
Think back to the serving stacks and monitor in the monitoring failure.

s176
00:14:35.640 --> 00:14:39.600
Assessing who runs the system helps describe the service buyers get.

s177
00:14:39.600 --> 00:14:44.320
That helps us define grade so buyers know which compute services are comparable.

s178
00:14:44.320 --> 00:14:48.520
The benchmark then needs to track the rent they actually pay or earn.

s179
00:14:48.520 --> 00:14:51.360
That's why I make this argument.

s180
00:14:51.760 --> 00:14:57.240
Rating the operator is an essential building block for building a credible compute futures market.

s181
00:14:57.240 --> 00:14:58.760
A rating informs that work.

s182
00:14:58.760 --> 00:15:06.160
It doesn't by itself define a futures grade, establish a sound benchmark, or create enough trading to buy and sell easily.

s183
00:15:06.160 --> 00:15:09.280
It also doesn't necessarily make a hedge effective.

s184
00:15:09.280 --> 00:15:12.640
A rental price hedge can help offset rental price changes.

s185
00:15:12.640 --> 00:15:17.800
It does not fix a failed deployment or ensure you that your customer pays.

s186
00:15:18.839 --> 00:15:23.440
Charging more than the index does not by itself make it a bad hedge.

s187
00:15:23.440 --> 00:15:27.400
What matters is whether changes in the reference track changes in your rent.

s188
00:15:27.400 --> 00:15:31.320
Unexpected changes in that relationship leave real price risk.

s189
00:15:31.320 --> 00:15:36.840
Those tool Those tools can help you understand what you're buying and manage the rental price risk.

s190
00:15:36.840 --> 00:15:40.280
For an intelligence producer, rent is one part of the cost.

s191
00:15:40.280 --> 00:15:46.080
The operator still has to turn that compute into work that the customer can actually use.

s192
00:15:46.480 --> 00:15:49.520
This brings us full circle.

s193
00:15:54.400 --> 00:15:57.320
So, a GPU falls to the level of the operator.

s194
00:15:57.320 --> 00:16:00.960
So, who can produce the intelligence you need at the lower cost?

s195
00:16:00.960 --> 00:16:02.080
At the lowest cost?

s196
00:16:02.080 --> 00:16:09.160
The producer who can repeatedly meet your requirements at the lowest full cost per successfully completed task.

s197
00:16:09.160 --> 00:16:15.000
Count everything spent to deliver the service, then divide by the output that the customer can actually use.

s198
00:16:15.000 --> 00:16:22.440
Operating capability affects both sides of that equation, what you spend and how much more useful work you can deliver.

s199
00:16:22.440 --> 00:16:25.320
Also, your position depends on that of your competitors.

s200
00:16:25.320 --> 00:16:30.760
You can improve and still fall behind if your competitors improve faster than you.

s201
00:16:30.760 --> 00:16:35.800
That's why understanding the operator matters through the cycle, not just at one point in time.

s202
00:16:35.800 --> 00:16:41.560
Uh especially when customers have more choices and prices come under more pressure.

s203
00:16:41.560 --> 00:16:45.280
In conclusion, if you just walked in and want the TLDR, this is it.

s204
00:16:45.280 --> 00:16:49.280
A GPU falls to the level of the operator.

s205
00:16:50.560 --> 00:16:52.120
Help build a better informed compute market.

s206
00:16:52.120 --> 00:16:54.640
Ladies and gentlemen, thank you so much for your time today.

s207
00:16:54.640 --> 00:16:59.280
At SemiAnalysis, we know that markets do not become informed and efficient on their own.

s208
00:16:59.280 --> 00:17:02.600
We seek to be an active engaged partner in that work.

s209
00:17:02.600 --> 00:17:04.199
It takes two to tango though.

s210
00:17:04.199 --> 00:17:06.439
Uh ideally more than two actually.

s211
00:17:06.439 --> 00:17:12.360
Um If you'd like to partner with us, uh contribute, or join the team, we would love to hear from you.

s212
00:17:12.360 --> 00:17:14.560
We are always hiring great people.

s213
00:17:14.560 --> 00:17:16.120
Please reach out.

s214
00:17:16.120 --> 00:17:24.760
Um If this resonated, uh please go ahead and email sales@semianalysis.com uh for partnership opportunities.

s215
00:17:24.760 --> 00:17:37.280
Uh and if you're interested in potentially joining the team, my email is kennon@semianalysis.com, k e n n o n. Um thank you all for listening and have a fantastic rest of your day.

s216
00:17:37.837 --> 00:17:39.837
[applause]