WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=prGP0huECPo

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/prGP0huECPo.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.760 --> 00:00:13.600
Hey everyone.

s3
00:00:13.600 --> 00:00:14.600
Thanks for coming.

s4
00:00:14.600 --> 00:00:15.400
I'm George.

s5
00:00:15.400 --> 00:00:19.480
I'm one of the co-founders of artificial analysis.

s6
00:00:19.480 --> 00:00:24.560
Today I'll be talking through optimizing the cost of intelligence.

s7
00:00:24.560 --> 00:00:31.440
To give a quick background to who we are at artificial analysis, we're a independent AI benchmarking company.

s8
00:00:31.440 --> 00:00:33.640
We benchmark across the AI stack.

s9
00:00:33.640 --> 00:00:47.680
So the agent layer without coding agent benchmarks and other agent benchmarks, language models for their intelligence, image, video, speech, and music models, as well as the hardware and inference layers.

s10
00:00:47.680 --> 00:00:52.280
So that we can benchmark the tradeoffs between all these layers.

s11
00:00:52.280 --> 00:00:58.520
When you're making decisions, you're not just choosing a model, you're also choosing you're making an inferences decision.

s12
00:00:58.520 --> 00:00:59.760
How do I access the model?

s13
00:00:59.760 --> 00:01:01.040
How much am I paying for it?

s14
00:01:01.040 --> 00:01:03.200
What's the speed that I'm that I'm getting?

s15
00:01:03.200 --> 00:01:04.480
What's the cash hit rate?

s16
00:01:04.480 --> 00:01:08.560
We believe strongly that all of the kind of layers of the stack are interrelated.

s17
00:01:08.560 --> 00:01:10.400
We're all making decisions across them.

s18
00:01:10.400 --> 00:01:14.840
And so that's why we benchmark the full stack of AI.

s19
00:01:15.800 --> 00:01:21.200
Kind of leading companies kind of reference artificial analysis and our results.

s20
00:01:21.200 --> 00:01:34.720
I think but what we're most proud of here, when we take a look at these references from OpenAI, from Anthropic, from Google, and and Nvidia, they're not just referencing

s21
00:01:34.720 --> 00:01:39.320
our raw intelligence benchmark numbers.

s22
00:01:39.320 --> 00:01:41.000
It's not just a bar chart.

s23
00:01:41.000 --> 00:01:51.280
We We're big fans of our intelligence index, but we're kind of most interested in talking through the tradeoffs that we're making between these technologies.

s24
00:01:51.320 --> 00:02:02.372
And so that's why we're proud that all of these are kind of two by two charts showing tradeoffs between uh, choices developers need to make.

s25
00:02:02.372 --> 00:02:10.132
Um, sometimes not when they're when they're getting things working, uh, but really when they're rolling it out in production.

s26
00:02:11.772 --> 00:02:18.932
We're also quite widely referenced, um, by kind of traditional and, uh, and new media.

s27
00:02:18.932 --> 00:02:21.572
We are based in San Francisco.

s28
00:02:21.572 --> 00:02:25.372
We've got a team of 50, um, and we are hiring.

s29
00:02:25.372 --> 00:02:36.372
Um, so we're hiring across uh, four deployed engineers, um, AI research staff as part of our member of technical staff team, and ML engineers.

s30
00:02:36.372 --> 00:02:45.332
Um, and so, you know, catch me after or please reach out if you if you might be interested, um, in joining the the world of AI benchmarking.

s31
00:02:47.332 --> 00:02:57.572
I think before we understand cost, we need to understand the kind of landscape of AI and the metrics to to understand and to optimize.

s32
00:02:57.572 --> 00:03:05.212
And so, starting with understanding the AI landscape, I'll first introduce our intelligence index.

s33
00:03:05.212 --> 00:03:14.052
It's we view it as the best single number to understand the general general intelligence, uh, of language models.

s34
00:03:14.052 --> 00:03:21.292
It's a synthesis metric of 10 different evaluations covering different capability areas of AI.

s35
00:03:21.292 --> 00:03:24.292
That includes knowledge work.

s36
00:03:24.292 --> 00:03:27.252
It includes scientific reasoning.

s37
00:03:27.252 --> 00:03:47.012
Uh, it includes coding and terminal use, uh, and it includes other capabilities including kind of knowledge, uh, hallucination and other relevant capabilities and considerations, um, regarding intelligence that people use when when choosing between language models.

s38
00:03:47.012 --> 00:03:51.052
What we say is for the current use cases being built and the next use cases.

s39
00:03:51.052 --> 00:03:53.837
It's not necessarily to measure and

s40
00:03:53.837 --> 00:03:54.052
[clears throat]

s41
00:03:54.052 --> 00:03:56.972
and benchmark progress towards AGI.

s42
00:03:56.972 --> 00:04:05.214
It's really to help developers make decisions for the next use cases that they're building uh, using AI.

s43
00:04:05.214 --> 00:04:11.054
And so of course Opus 5.5 from Anthropic is currently leading the index.

s44
00:04:11.054 --> 00:04:14.694
I think to many in this room that won't be a big surprise.

s45
00:04:14.694 --> 00:04:18.694
But also to discuss today is that there's as you can see here many other models.

s46
00:04:18.694 --> 00:04:22.774
We have over 500 models on the artificial analysis website.

s47
00:04:22.774 --> 00:04:27.654
Um and many kind of current and and and presented on this chart.

s48
00:04:27.654 --> 00:04:38.174
And I think what we'll be talking through today is a bit that that kind of takeaway that there's also reason to use models other than the most intelligent model in the world.

s49
00:04:39.294 --> 00:04:41.974
We also publish individual eval results.

s50
00:04:41.974 --> 00:04:49.774
So we released AA analyst agent which is a finance and accounting focused focused benchmark.

s51
00:04:49.774 --> 00:04:52.094
Thought I'd share that this being New York.

s52
00:04:52.094 --> 00:05:04.734
And we share the individual results for those wanting to go to the next layer of specificity when thinking about their model selection and and selection of other technologies because

s53
00:05:04.734 --> 00:05:12.174
it's not always the case that models exactly follow the generalist view of the intelligence index.

s54
00:05:12.174 --> 00:05:19.574
And so here we kind of highlight that the Gemini series of models is actually very good at navigating complex complex spreadsheets.

s55
00:05:19.574 --> 00:05:24.334
Of course Anthropic series of models is is of course as well.

s56
00:05:25.534 --> 00:05:34.094
To step back a bit looking at our intelligence index and the leading model over time from the top 12 or so labs here.

s57
00:05:34.094 --> 00:05:37.294
Um firstly I think big takeaway it's kind of hectic.

s58
00:05:37.294 --> 00:05:46.494
There's many dots and I think probably that resonates for a lot of us in the room who have kind of felt this over the past few years where there've been

s59
00:05:46.494 --> 00:05:51.854
huge number of model releases and the number of model releases in is increasing.

s60
00:05:51.854 --> 00:06:00.494
And in a sense kind of the the pace of progress in our view is also accelerating and I think that is shown in this chart here.

s61
00:06:00.494 --> 00:06:12.615
And so I think the takeaway here is that AA is moving quickly, progress is moving quickly, and so kind of we might feel exhausted trying to keep up with it, but

s62
00:06:12.615 --> 00:06:22.495
um it makes sense to to it to invest that time and energy because if you want to stay on on top of AI and what AI's capable of, um then staying on top of releases

s63
00:06:22.495 --> 00:06:25.695
um is a a big part of that.

s64
00:06:25.695 --> 00:06:35.495
The next view won't deep dive, this could be a whole talk, but open weights models continue to keep within 3 to 9 months of the leading proprietary models.

s65
00:06:35.495 --> 00:06:38.855
And so open weights models remain an option.

s66
00:06:38.855 --> 00:06:49.495
Uh and so I think that's that's the takeaway for the talk today and and it flows through to a lot of um kind of cost implications uh which sometimes open weight models can have favorable

s67
00:06:49.495 --> 00:06:51.575
uh characteristics.

s68
00:06:53.015 --> 00:06:56.135
Next, let's dive into cost and how to think about cost.

s69
00:06:56.135 --> 00:07:10.615
And I want to discuss think talking about cost from the perspective of being kind of mid-late 2026 because I think if we were doing the if I was doing this talk even six or nine months ago, I think it would be a different talk.

s70
00:07:10.615 --> 00:07:12.495
AI has moved a long way.

s71
00:07:12.495 --> 00:07:23.975
And so we'll go into the drivers, but to kind of start with a little bit kind of the answer of why we're so conscious of cost is because these models

s72
00:07:23.975 --> 00:07:29.135
are costing many dollars um to complete individual tasks.

s73
00:07:29.135 --> 00:07:32.135
That wasn't the case uh 12 months ago.

s74
00:07:32.135 --> 00:07:39.895
Um people were spending more than ever on AI and can spend more than ever on a single task um on AI.

s75
00:07:39.895 --> 00:07:46.975
And so here we present Opus and Fable costing above $5 to complete a task.

s76
00:07:46.975 --> 00:07:53.495
But on the other side, um there's also models that cost 5 cents and less to complete a task.

s77
00:07:53.495 --> 00:08:02.815
And so what we're seeing is there's orders of magnitude, multiple orders of magnitude uh difference in cost uh between the between the models.

s78
00:08:02.815 --> 00:08:08.095
And that means that the cost decision and the model decision uh matters.

s79
00:08:08.095 --> 00:08:16.737
And what I would say is that yes, you might say oh the uh the models on the left here are less intelligent and that's totally true.

s80
00:08:16.737 --> 00:08:18.217
But they're still very good.

s81
00:08:18.217 --> 00:08:30.457
Uh uh Luna here, GBD5 um sorry, GBD6 Luna is 7 cents uh per task um and it's much better than than GBD4 was um originally, right?

s82
00:08:30.457 --> 00:08:38.297
Which kind of and and and way better than GBD3.5 Turbo, which really kind of like opened up um th- this wave of kind of language models.

s83
00:08:38.297 --> 00:08:52.057
And so, I would say that there's two orders of magnitude difference between the leading proprietary models and much cheaper, but still very good, very capable models.

s84
00:08:53.537 --> 00:08:59.777
A next way to kind of consider looking at cost is uh from a time perspective.

s85
00:08:59.777 --> 00:09:15.297
And what we've done is we've bucketed models by their intelligence and looked at when that intelligence tier was first reached and then how did the kind of cost of trying to access the intelligence with the cheapest model develop over time.

s86
00:09:15.297 --> 00:09:26.577
And what we see here is that within 1 year, accessing that level of intelligence, a 30 on the intelligence index or a 50 on the intelligence index, essentially reduces

s87
00:09:26.577 --> 00:09:30.617
by an order of magnitude, kind of 10x, within around a year.

s88
00:09:30.617 --> 00:09:37.817
So, picking the the light purple here, um once we reached a 30, cost a bit over a dollar.

s89
00:09:37.817 --> 00:09:42.337
A year later, it fell to around 10 cents.

s90
00:09:42.337 --> 00:09:45.097
And then now we're well below uh the 10 cent.

s91
00:09:45.097 --> 00:09:48.617
We're in the kind of um 1 cent to 10 cents territory.

s92
00:09:48.617 --> 00:10:00.177
And so, I think it's there's also a time dimension here whereby we should be um you know, making choices for now, but also conscious that that is going to change very quickly

s93
00:10:00.177 --> 00:10:11.538
in terms of the intelligence that we can achieve and and that maybe $1 million um spent for a big gigantic can you know, is likely we can access that level of intelligence at 100k

s94
00:10:11.538 --> 00:10:15.498
and then in 2 years potentially around 10k.

s95
00:10:18.058 --> 00:10:21.458
Now, here we have a build up of the cost per task.

s96
00:10:21.458 --> 00:10:28.218
This is a lot more complex than where we were 12 months and definitely 24 months ago.

s97
00:10:28.218 --> 00:10:30.698
I think 12 months ago we're talking about reasoning models.

s98
00:10:30.698 --> 00:10:41.258
We're saying that we have to pay for reasoning tokens and um you know, the the labs were just talking about the output token price um and so we needed to do an analysis

s99
00:10:41.258 --> 00:10:45.578
um as a as a community to really understand the cost because we have to count reasoning tokens.

s100
00:10:45.578 --> 00:10:48.458
Now with agents it's become way more complex.

s101
00:10:48.458 --> 00:10:49.218
Way more complex.

s102
00:10:49.218 --> 00:10:56.258
So, to understand the build up of a cost per task, here we use our AA briefcase agent technology workbench mark.

s103
00:10:56.258 --> 00:11:05.618
It looks like agents in an environment of thousands of files completing knowledge work tasks like creating reports, creating PowerPoints, creating marketing videos.

s104
00:11:05.618 --> 00:11:12.778
And what we can see here is on average Opus is costing $21 to complete a task.

s105
00:11:12.778 --> 00:11:13.858
And it does a great job.

s106
00:11:13.858 --> 00:11:15.178
No arguments there.

s107
00:11:15.178 --> 00:11:22.778
Focusing on cost though, what we can see here is that uh answer tokens, you know, cost around 16 cents.

s108
00:11:22.778 --> 00:11:27.757
Reasoning tokens, um oh sorry, uh it's like 2 cents.

s109
00:11:27.757 --> 00:11:28.138
[laughter]

s110
00:11:28.138 --> 00:11:39.298
Um and then we have then we have reasoning tokens a bit more, but then really where the you know, building up to the single language model call, the first kind of column that hits the ground and I'll talk about the second there.

s111
00:11:39.298 --> 00:11:42.938
Um so each call is 13 cents, right?

s112
00:11:42.938 --> 00:11:47.498
But with agentic trajectories we're having to think about the number of steps, of course.

s113
00:11:47.498 --> 00:11:51.658
And so it took 165 steps to complete the task.

s114
00:11:51.658 --> 00:11:56.378
And so that's where that's a new dimension that we need to think about when it comes to agents.

s115
00:11:56.378 --> 00:12:09.018
And then thinking to kind of what goes into a single language model call, you can see there's answer tokens, reasoning tokens, input tokens and then with input tokens you need to think about how many of those input tokens uh kind of cash hits

s116
00:12:09.018 --> 00:12:15.499
um and so get that roughly 10x uh discount on an input token price.

s117
00:12:15.499 --> 00:12:26.819
And what we can see with the build up is the biggest driver of the kind of input into what a single language model calls is the number of input tokens

s118
00:12:26.819 --> 00:12:32.699
and then secondly the amount of or proportion of those input tokens that are cache hits.

s119
00:12:32.699 --> 00:12:38.939
And then the third biggest driver after that is of course or it's actually the biggest but it is the number of turns per task.

s120
00:12:38.939 --> 00:12:45.339
And so we were talking a year ago around the number of output tokens, how much reasoning, the reasoning verbosity.

s121
00:12:45.339 --> 00:12:57.019
I think today we really need to be talking about input tokens, uh cache hit percentage, um and then the number of turns per task is the key metrics that we should be thinking about when thinking about the cost

s122
00:12:57.019 --> 00:12:59.459
um of running of running agents.

s123
00:12:59.459 --> 00:13:09.139
And I think just to really emphasize the number of cache hits, if we don't have that then and and we don't have any cache hits, right?

s124
00:13:09.139 --> 00:13:21.019
Cache hit percentage is zero, caching isn't working, maybe we've got our own setup with VLMs, we haven't enabled prompt caching, um then instead of $21 per task, this costs over $200.

s125
00:13:21.019 --> 00:13:30.459
So that's just to really emphasize how much caching matters here when we're talking about agents and also when talking about inference providers, how much you really need to to care about that.

s126
00:13:30.459 --> 00:13:34.899
We started recently publishing benchmarks around cache hit percentage on our website.

s127
00:13:34.899 --> 00:13:38.939
And next consideration is um you know different reasoning efforts.

s128
00:13:38.939 --> 00:13:51.499
And so here we have um kind of column um column charts showing the different reasoning efforts um for Opus 5.5 and the implications on these metrics.

s129
00:13:51.499 --> 00:14:02.539
And so I think pretty much the key takeaway here is when we're talking about reasoning efforts, we're not just talking about um uh kind of reasoning the number of reasoning tokens

s130
00:14:02.539 --> 00:14:07.979
um for an agent single agentic call, it does impact the number of turns.

s131
00:14:07.979 --> 00:14:21.621
Um And of course then you have reasoning tokens and then those reasoning tokens feed throughout the agentic trajectory into kind of much more input tokens um because of the way that like message histories work

s132
00:14:21.621 --> 00:14:24.221
uh when it comes to comes to using agents.

s133
00:14:24.221 --> 00:14:34.221
And so even with it when using the same model, very important to think about the reasoning effort because here we can see that at low Opus 5 can cost 5.5

s134
00:14:34.221 --> 00:14:38.861
can cost a dollar per task and at max it can cost $21.

s135
00:14:39.675 --> 00:14:41.381
[clears throat]

s136
00:14:41.381 --> 00:14:55.741
And here's kind of um a breakdown going from Opus 5.5 to GLM 5.3 Flash, a much cheaper but still extremely good, very capable model um that's that's open weights.

s137
00:14:55.741 --> 00:15:08.541
And I think when we look at this when we look at this breakdown, this is really just to show that one, there's big differences between as we saw between the leading models and the and much cheaper models and two is that all of the different

s138
00:15:08.541 --> 00:15:16.661
drivers that I was talking about play a role in driving cost differences um between models.

s139
00:15:16.661 --> 00:15:28.901
And then thirdly, of course, that input token prices and your cache hit percentage um are the biggest driver um of cost when doing long uh horizon agent tasks.

s140
00:15:30.381 --> 00:15:44.461
So we've talked about the cost and I think there's no points for knowing that GLM 5.3 Flash is cheaper than Opus 5.5. I think a lot of people in this room know that Lunar is cheaper than than Astra.

s141
00:15:44.461 --> 00:15:45.221
Great.

s142
00:15:45.221 --> 00:15:49.981
So the challenge for us as builders is making the trade-off decision.

s143
00:15:49.981 --> 00:15:51.261
That's the hard thing.

s144
00:15:51.261 --> 00:15:56.461
How do we understand our trade-off curves for our tasks?

s145
00:15:56.461 --> 00:16:07.661
And I think a starting point that we think is useful to use is the generalist um view of intelligence versus cost per task and starting with the Pareto.

s146
00:16:07.661 --> 00:16:20.622
So starting with okay, my task um where does it sit across this uh this intelligence Pareto in terms of the model that's needed.

s147
00:16:20.622 --> 00:16:26.302
If I'm able to spend more or wanting leading intelligence, I want to look to the right of the Pareto.

s148
00:16:26.302 --> 00:16:30.342
If I want cheaper, then I should let look to the left of the Pareto.

s149
00:16:30.342 --> 00:16:35.742
The Pareto being this this line depicted in this in this chart here.

s150
00:16:35.742 --> 00:16:50.382
And when choosing and wanting a cheaper model, let's take a look at the at the Pareto and seeing, "Okay, I'm willing to my budget is less and so what's the model that at that point offers the best intelligence

s151
00:16:50.382 --> 00:16:51.862
for that that dollar?"

s152
00:16:51.862 --> 00:16:57.942
Or from the other perspective of I want this level of intelligence, what's the cheapest way I can get it?

s153
00:16:57.942 --> 00:17:02.822
How to think about navigating that decision and asking yourself those questions?

s154
00:17:02.822 --> 00:17:07.302
A way that we think's helpful is to bucket tasks into two categories.

s155
00:17:07.302 --> 00:17:14.502
So first is unlimited intelligence tasks, whereby more intelligence can always deliver a better result.

s156
00:17:14.502 --> 00:17:20.502
An example being, "Analyze drivers of the costs in my business and how do I optimize them?"

s157
00:17:20.502 --> 00:17:24.342
Kind of smarter will always deliver a better result there.

s158
00:17:24.342 --> 00:17:32.182
Next, the second bucket is limited intelligence tasks and here is when more intelligence won't give you a better result.

s159
00:17:32.182 --> 00:17:35.782
So the question, "How much did I spend on subscriptions last month?"

s160
00:17:35.782 --> 00:17:41.622
I think a cheaper like Luna will answer that question as long as it's set up with the right MCPs.

s161
00:17:41.622 --> 00:17:43.462
Currently, it'll tell me $80.

s162
00:17:43.462 --> 00:17:45.942
Using Astra will also tell me $80.

s163
00:17:45.942 --> 00:17:46.862
$80.

s164
00:17:46.862 --> 00:17:50.062
I'm not getting any benefit from more intelligence.

s165
00:17:50.062 --> 00:18:01.622
Um and so then I should focus on what's the cheapest way what's the level of intelligence needed, the ceiling, and what's the cheapest way that I can get that level of intelligence.

s166
00:18:01.622 --> 00:18:02.662
So that's limited intelligence.

s167
00:18:02.662 --> 00:18:05.342
Unlimited intelligence, the question is, "What's my budget?

s168
00:18:05.342 --> 00:18:10.142
How much am I willing to pay for increased intelligence?"

s169
00:18:12.582 --> 00:18:19.182
In terms of how we suggest using artificial intelligence, we suggest to kind of and how to navigate these decisions.

s170
00:18:19.182 --> 00:18:34.903
I think it's understand the landscape, the metrics, then compare models based on priorities, shortlist models, and then for kind of smaller prototypy workloads, use the shortlist, test on your own,

s171
00:18:34.903 --> 00:18:37.983
uh you know, in your own context and your own agents.

s172
00:18:37.983 --> 00:18:50.023
For more advice users, particularly when rolling out in production, we recommend creating your own benchmark to take that shortlist to five models and try and make um the decision optimize for the decision

s173
00:18:50.023 --> 00:18:53.903
um because of the kind of magnitudes of cost that we're dealing with now.

s174
00:18:53.903 --> 00:19:08.463
Um we recently released a product called Optima um that allows users to come to Artificial Analysis, upload their context, and we will um together with you using a lot of our um knowledge that we've distilled into this kind of Optima agent,

s175
00:19:08.463 --> 00:19:21.823
we generate you a benchmark um and give you your own trade-off curves between your view of intelligence for your task um and the cost to help you help you make these trade-offs.

s176
00:19:22.183 --> 00:19:24.623
So, that's the cost of intelligence.

s177
00:19:24.623 --> 00:19:26.743
Um thanks very much for for listening.

s178
00:19:26.743 --> 00:19:34.103
Come check us out our website at artificialanalysis.ai, and um as mentioned at the start, we're uh we're hiring quite broadly across roles.

s179
00:19:34.103 --> 00:19:36.393
So, thank you very much.

s180
00:19:36.393 --> 00:19:38.393
[applause]