Same GPU, 50% More Output: Why the Operator Matters — Oliver Kennon, SemiAnalysis

AI Engineer · 17 min · 216 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Thank you everyone for taking the time to be here today.
00:14My name is Oliver Cannon.
00:16I'm a research analyst at SemiAnalysis.
00:19Um time and attention are the most valuable things any of us have.
00:23Uh unless of course you're an early stage investor in OpenAI or Anthropic.
00:27Um with that, uh let's get into it.
00:30Uh not all GPUs are deployed the same.
00:32A GPU falls to the level of the operator.
00:35Um buying hardware gives you potential.
00:38Uh operating capability is what determines how much useful work you actually are able to deliver.
00:43Uh that leads us to the question I will attempt to answer with all of you today.
00:47Who can produce the intelligence you need at the lowest cost?
00:51Cost matters.
00:52Uh who uh in today's presentation I want to show you guys why and how operating capability changes that cost and why it matters in the commodity market that is compute.
01:05What matters in a commodity market?
01:07Um a lot of tech investors or technologists may not be familiar with this framework.
01:11Um so a little crash course.
01:14Uh in a commodity market producers compete to deliver comparable output.
01:18Uh a lower production cost gives more room to compete when prices fall.
01:22Uh that's what a cost curve shows.
01:25Producers lined up by what they spend to deliver the same usable work.
01:30If two producers earn the same price, the one spending less to deliver the work gets to keep more of it.
01:36Pretty simple, right?
01:39For us, intelligence is the economic output we care about.
01:43Eventually, our North Star metric will be price per successfully completed task at the required quality.
01:49Uh unfortunately, we're not quite there yet.
01:51That's a pretty hard thing to standardize.
01:53Um today price per token is a useful practical proxy, especially for comparable workloads and quality requirements, and gets us closer to delivered output.
02:04Uh much closer than the often quoted uh all you finance bros out here will have heard this uh GPU uh dollar per hour price.
02:13Um or as we view it, a cost.
02:16Um We view GPU hours as a measure of input, not output, the cost versus price distinction.
02:23Customer prices are different from the producer's cost of doing the work.
02:31GPU hours are the input, intelligence is the output.
02:34Uh on this slide, we can visualize the inputs needed to create useful intelligence.
02:39Uh we start with hardware and electricity, GPUs uh for the the this presentation.
02:45Software and operations can be thought of as a multiplier on those first two inputs, with the end product being useful intelligence.
02:56The operator is who chooses and implements the software that serves the model and keeps requests moving through the system.
03:03The machines might be available, but you still need someone to make them deliver responses customers can actually use.
03:10What matters is how much useful work you get and whether it meets your requirements for the specified task.
03:16The conversion can differ with the same kind of chip.
03:20Let's look at that in the measured example.
03:23Here's a concrete example from our InferenceX public InferenceX calculator.
03:28You guys can all check it out, semi-analysis.com.
03:31Uh both estimates use B200 GPUs serving the same model with roughly 8,000 input tokens and 1,000 output tokens, a pretty standard ratio.
03:42The P50 target is 100 output tokens per second for each user.
03:46Uh the serving stack is the software handling the model's responses.
03:50In the P90 scenario, one of these stacks gives about 200 generated output tokens per second, while the other one gives about 300.
03:59That's a big difference and comes down to the software, settings, and number of requests running.
04:05All choices made by the operator.
04:11Same chip, same target interactivity, about 50% more output.
04:16That's a lot.
04:17So, what does that mean for the buyer's bill?
04:23What does that intelligence actually cost the customer?
04:26For this this example, let's assume the hourly bill is the same for both deployments and both meet your requirements.
04:33At roughly 1 and 1/2 times the output, your rental spend per acceptable token is proportionally lower.
04:39That is to say, the hour buys you more usable and useful output.
04:45What counts as useful intelligence?
04:48Useful intelligence has to do the job you hired it to do.
04:51Pretty simple, right?
04:52The answer needs to be good enough for the task, arrive quickly enough, and be available when you need it.
04:58Your information also needs to stay within the access boundaries you you've agreed upon.
05:03AKA, it needs to be secure.
05:06This is an overlooked requirement that is not captured in the chip's performance metrics, but it is essential for a chip to be usable and useful.
05:15In our published security work, we found monitoring access that exposed another customer's logs and metrics.
05:21This is a case, and an unfortunately common one at that, where the GPUs were fast, and in the case I'm referencing, really, really fast, which makes the whole thing sadder.
05:31Uh and the service still failed horrifically.
05:34Um ensuring cluster security takes ongoing work, just as delivering the promised performance does.
05:42In most cases, it is the operator that owns that responsibility.
05:48Someone has to make the system work in production.
05:51Suppose a faster serving update becomes available.
05:54Someone has to test it on the customer's workload, check whether the output still meets the requirements, and then decide when, where, and how to deploy it.
06:03That means checking the actual responses, testing it under load, and keeping a way to return to previous version if the update causes problems once it's deployed.
06:14Once it is deployed, someone has to maintain it and respond when issues come up.
06:20That work may sit with your provider, your own team, or both.
06:25Irregardless, you need to know who's responsible and whether they can actually do it.
06:33So, how do we assess the operator?
06:35Uh SemiAnalysis ClusterMax, we're in version 3.0 at this point, uh assesses operating cap- capability on managed GPU clusters, couple couple hundred, uh growing.
06:46Please reach out if you want to be on ClusterMax 4.0. Um we look at provisioning, reliability, monitoring, and security.
06:54Uh hands-on testing, documentation, and customer's feedback help us understand how the service runs, not just on a good day, but also when a machine fails.
07:05I want to take the chance to reiterate that ClusterMax ratings apply to managed clusters, uh different from bare metal, very different.
07:13Um with bare metal, more operating work usually sits with the customer.
07:17Uh applying a managed service rating there does not automatically cover every product a provider sells.
07:24The tiers are relative to peers.
07:27Little bit of a tongue twister.
07:28Uh I nailed it, pretty happy with that.
07:30Uh
07:30[laughter]
07:31and pricing and availability are also important factors in this assessment.
07:35Uh these are metrics that do not directly measure uh token output, uh but are incredibly important.
07:42Obviously, very important.
07:43Uh assessment helps us understand who can deliver.
07:46Now we can move on to the next question.
07:48What does it actually cost to produce useful intelligence?
07:51Let's start with a compute bill and other costs that producers still has to cover.
07:57Rental price and production cost are different.
08:00In our example, GPU provider the GPU provider rents out hours, the inference a producer buys those hours and sells responses.
08:08The rental bill is revenue to the provider and an input cost to the inference producer.
08:13The GPU owner pays for the equipment, power, space, and support.
08:17The inference producer's full cost includes rent plus whatever software, staff, and other services it pays for beyond that rental.
08:25If the inference producer owns its own it equipment instead, we spread the equipment investment over its working life and add running costs.
08:32What the producer charges and what it spends are separate numbers.
08:36To understand the cost of useful intelligence, we need to know how much useful work the spending actually produces.
08:42Uh better operations lowers that cost of useful intelligence.
08:46The B200 comparison showed why the amount of output matters.
08:50More acceptable responses from the same paid hours can spread the inference producer's rental expense over more useful work.
08:57For a producer that owns its GPUs, the same logic spreads the equipment investment and running costs over more output.
09:03Count all the spending needed to deliver the service over the same period, then divide by the output customers can actually use.
09:13That gives us cost for acceptable token for a comparable workload.
09:19Better operations also require investment.
09:22Testing an update, deploying it safely, uh and keeping it working take people and money.
09:30The extra output has to justify the extra spending.
09:33Uh this advantage can persist when a team repeats that work as software and workloads change, but one good benchmark doesn't establish it.
09:42You have to earn it every single day.
09:45Uh each new workload still needs checking, but a repeatable operating process, from what we've observed, can support a lasting cost advantage and a price advantage, too.
09:55Uh even if you do keep improving, though, your competitors are not sitting still.
10:01Uh your position on the cost curve is relative.
10:04This is This slide, out of all the slides, should make clear the importance of the commodity framework in all of this.
10:10Um your position on the cost curve depends on what your competitors can deliver, not just yourself.
10:15It's a relative measure.
10:16Here's a simple hypothetical.
10:18We're comparing the same task, the same required quality, using made-up cost units.
10:24Uh I like to imagine bars of gold.
10:26Uh your full cost per successful task falls from 10 to 8. Uh in the same period, your competitor's falls from 6 to 12.
10:37You improved.
10:38That's awesome.
10:39But both of you did.
10:40And in this example, you started as the cheaper producer and ended as the more expensive one.
10:45That gives the competitor more room to lower its selling price, provide provided customers find the two services equally acceptable.
10:52Uh getting better does not automatically move you ahead.
10:56Applying the commodity framework makes you realize that you have to compare your your cost per useful work on a relative basis.
11:02Comparing your own product with what your competitors can deliver.
11:07That comparison can be easy to overlook in moments like today, where demand far outstrips supply and will for the foreseeable future.
11:15Uh two to three years visibility.
11:19Good.
11:21Scarcity can obscure those differences.
11:24When demand is strong and supply is constrained, producers with very different costs can can find customers.
11:30Strong revenue can make those differences feel less urgent.
11:33If you're a CEO, why would you focus on your precious time and energy and money shoring up your spot on the cost curve when you can be building a massive new data center?
11:43Uh in moments like this, it's important to step back and have a little perspective.
11:47Uh take a little historical perspective.
11:49Uh every commodity in the history of humanity has had swings in supply and demand.
11:53Just because we don't see when that will be yet does not mean that this cycle will be any different.
11:57Uh unless you're a total AI maxi, which some of us are.
12:01So, I'm very open to being wrong.
12:03I'm wrong every day.
12:05Um but I would say if you're a business with leverage, you need to be especially conscientious of this.
12:11Being proactive with your built with building your cost advantage uh might be not be as sexy as making your top line bigger, but it matters deeply if you want to survive through a business cycle.
12:22Uh that's why you want to build your operating capability and cost advantage while you have the opportunity uh when the sun is shining, when you can raise capital.
12:31Uh they matter even more when customers have more choices to make and prices come under pressure.
12:36So, how do we make clear what uh the buyer actually gets?
12:41Comparable compute needs clear grades.
12:44Now, let's look at the compute uh that an intelligence producer is buying.
12:48We've already seen that the same chip can deliver different results depending on how it's operated.
12:54Uh in two offers for B200 GPU hours can include different services and leave different operating jobs with the customer.
13:01A commodity framework needs a way to describe these differences so buyers know which services they can treat as comparable.
13:08That's the job of a grade.
13:12The grade describes a particular compute service and what its buyers can expect.
13:16It doesn't make intelligence interchangeable across every task.
13:20Once we've defined that service, we can ask what its rental price tells us.
13:27What does a rental price benchmark measure?
13:30Uh a rental price benchmark summarizes the price of GPU hours within its defined market.
13:35It isn't the producer's production cost or a measure of intelligence delivered.
13:40We need to know where those prices come from.
13:42An advertised offer and a completed trade are not the same thing.
13:47Service scope, contract terms, uh whose prices are sampled and how they're combined all shape that reference point.
13:56Uh operating quality can influence those prices without the index measuring it separately.
14:01Running a rental reference doesn't tell you how much useful work your deployment will produce.
14:06To use it as a hedge, you need to understand how it tracks the rent you actually pay or earn.
14:13Uh what can compute futures hedge?
14:16There's been a lot of excitement lately about compute futures.
14:19For all of you that don't know, a futures contract sets a standardized term for future delivery or cash settlement.
14:25Cash settlement depends on a price reference.
14:27Buyers and sellers both face uncertainty about rental prices.
14:31Think back to the serving stacks and monitor in the monitoring failure.
14:35Assessing who runs the system helps describe the service buyers get.
14:39That helps us define grade so buyers know which compute services are comparable.
14:44The benchmark then needs to track the rent they actually pay or earn.
14:48That's why I make this argument.
14:51Rating the operator is an essential building block for building a credible compute futures market.
14:57A rating informs that work.
14:58It doesn't by itself define a futures grade, establish a sound benchmark, or create enough trading to buy and sell easily.
15:06It also doesn't necessarily make a hedge effective.
15:09A rental price hedge can help offset rental price changes.
15:12It does not fix a failed deployment or ensure you that your customer pays.
15:18Charging more than the index does not by itself make it a bad hedge.
15:23What matters is whether changes in the reference track changes in your rent.
15:27Unexpected changes in that relationship leave real price risk.
15:31Those tool Those tools can help you understand what you're buying and manage the rental price risk.
15:36For an intelligence producer, rent is one part of the cost.
15:40The operator still has to turn that compute into work that the customer can actually use.
15:46This brings us full circle.
15:54So, a GPU falls to the level of the operator.
15:57So, who can produce the intelligence you need at the lower cost?
16:00At the lowest cost?
16:02The producer who can repeatedly meet your requirements at the lowest full cost per successfully completed task.
16:09Count everything spent to deliver the service, then divide by the output that the customer can actually use.
16:15Operating capability affects both sides of that equation, what you spend and how much more useful work you can deliver.
16:22Also, your position depends on that of your competitors.
16:25You can improve and still fall behind if your competitors improve faster than you.
16:30That's why understanding the operator matters through the cycle, not just at one point in time.
16:35Uh especially when customers have more choices and prices come under more pressure.
16:41In conclusion, if you just walked in and want the TLDR, this is it.
16:45A GPU falls to the level of the operator.
16:50Help build a better informed compute market.
16:52Ladies and gentlemen, thank you so much for your time today.
16:54At SemiAnalysis, we know that markets do not become informed and efficient on their own.
16:59We seek to be an active engaged partner in that work.
17:02It takes two to tango though.
17:04Uh ideally more than two actually.
17:06Um If you'd like to partner with us, uh contribute, or join the team, we would love to hear from you.
17:12We are always hiring great people.
17:14Please reach out.
17:16Um If this resonated, uh please go ahead and email sales@semianalysis.com uh for partnership opportunities.
17:24Uh and if you're interested in potentially joining the team, my email is kennon@semianalysis.com, k e n n o n. Um thank you all for listening and have a fantastic rest of your day.
17:37[applause]