The Cheaper AI Model That Costs You More - Rayan Krishnan, Vals AI

AI Engineer · 15 min · 168 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12I'm going to talk a little bit about uh going from beyond the model card, how evaluations can be used to measure this jagged frontier.
00:19Uh special focus on finance for this audience as well.
00:23Um yeah, so first off talking about model cards themselves, we've built uh some of the most important benchmarks over the last couple of years that the major foundation model labs have included in their model releases.
00:32So people I think are familiar with artifacts like this.
00:35Uh last week we have Sunder talking about our Val's index on stage and then uh even just yesterday Muse models uh reciting our our Finagint benchmark.
00:44You know, it keeps our lives interesting to have to keep re-updating these slides as new models come out.
00:48I had to double-check new models didn't come out this morning, but um these are say these these kind of model cards reflect a general understanding of what's newly capable with the models.
00:57Um and so uh they they can reveal from a single number how intelligent is a model.
01:02In particular, our Val's index is compiling results to try and measure the economic impact of models uh in domains like finance, legal, but also coding, health care, and things like that.
01:12So from these high-level numbers, you can get a sense of how capable a model is for uh valuable work.
01:18Um but that doesn't give the whole picture.
01:20And so when we dial in a little bit more closely and look at the constituent benchmarks that comprise the Val's index, we can see a more nuanced picture.
01:27And this may be something you all are familiar with, the jagged frontier.
01:30And the idea that within a particular domain or set of tasks, you may see different performance uh from when you aggregate results to produce that single number.
01:37And so you can see here across the different models, uh we actually see an inconsistent performance for who's number one.
01:43Uh but that's made even more so when you look at particular organizations or companies.
01:47So in a lot of ways, coding is indicative of what's happening across every major vertical.
01:52Um when we look at coding and actually create custom benchmarks for particular repositories, we see that that jaggedness just increases even more.
02:01And so on on the x-axis you've got different programming languages and libraries.
02:04And on the y-axis you see the relative performance of models and see that it's even more fragmented.
02:09But this isn't even to take into account that there are different harnesses in which these models can can be used or the idea that you might be optimizing for cost and latency as well as performance.
02:20And so this all feeds into the idea that the the granularity of of the performance you see in a general model card benchmark is not necessarily what's most applicable for people who are using these models or or products in practice.
02:33In parallel, AI adoption has gone up considerably.
02:36So year over year there's roughly a 2x increase in enterprise AI spend.
02:40But with all that spend it's still really obtuse to understand where the ROI is coming from.
02:47You can see that people regularly using AI is now 90% but if you look at the smallest bar now we have established ROI just at 7%.
02:55So for all this token spend and increased adoption it's still very unclear where AI is actually driving value for a lot of companies.
03:02And I think two interesting stats that jump out at me, the groups that are able to actually track AI cost and spend are much much more likely to actually understand where ROI is going to come from.
03:11And this is also a key blocker for adoption itself.
03:15The groups that are unable to actually quantify the cost to the return on their investment often see this as a big bottleneck to actually widely roll out AI.
03:23And so I think evals is the key story here.
03:24If people are able to construct key evaluations that measure what our model's capable of and what will be the most efficient utilization of those dollars, they'll be able to adopt AI in their firm more broadly.
03:40I think one particular interesting finding that that I want to share with you all is an indication of why the intuition that are intuition of model performance as presented by labs
03:49doesn't often align with experience.
03:51So if you look at Opus 5.5 that's the kind of second tier model from Anthropic.
03:56And on paper it's actually cheaper.
03:57$4 and $20 out compared to Fable 5.1's $10 and $50 out.
04:02But in practice, and particularly for this tax benchmark, you see that it's actually more expensive, 14 more expensive.
04:08And that's because the less intelligent model is trying to expend more tokens to try and get its work done.
04:13But this actually bears out as a 3% lower score.
04:16Um and so you are actually would have been better off using the frontier intelligence.
04:20You would have spent less and got a better performance.
04:23Um but that kind of intuition isn't isn't directly available for people who are using the models in practice.
04:27And so they may try and say, "Actually, I should use a Sonnet or an Opus."
04:30But end up overpaying for uh inferior performance.
04:34Uh and so I think this, you know, reveals that we should actually take a more diligent effort to figuring out where return is coming from.
04:39And and that will inform a more realistic picture as opposed to the intuition that labs are feeding directly.
04:45If we look at look at finance in particular, so that this kind of jagged frontier concept, uh I've highlighted the Vals Index, but we also have a finance agent benchmark
04:54looking at research tasks for finance, and then Excel modeling, uh and then tax agent.
04:58Uh and even across these benchmarks again, it's not immediately obvious which is number one or will be consistently two or three.
05:06This picture is made even more nuanced when you look at the Pareto curve.
05:09So on the x-axis, you've got cost.
05:11In particular, it's cost per task, not just token in token out cost.
05:15And on the y-axis, you have performance.
05:16And and here what we're looking at is that Excel modeling benchmark.
05:19So seeing whether these AI models can create and work with Excel models that are core to work that's happening in finance.
05:27Uh in the green, you have the overall performance across all of the questions.
05:31And then in red, what we've done is is segmented out the LBO tasks.
05:35So this is a particular subset of Excel model tasks that you might you might be working with in the in uh um using any of these models.
05:42Um and I want to call out a couple things.
05:44In particular, first what you see is that the entire frontier for LBO is lower than the overall task score.
05:50So the intuition you would have from just the the high-level EMB results would not match what you see for the LBO categories of tasks.
05:57And so if your firm is especially doing, you know, one type of Excel models over another, it's important to have a clearer picture of what's going to be actually relevant to you.
06:05But the other is that the frontier is also changing when you go between the overall scores to just the LBO subcategory.
06:11So, whereas you might have thought you would have get the best possible performance from using Feable 5.1, when you go to the LBO it actually drops off the Pareto frontier.
06:18You're going to be massively overpaying for what is no longer the state of the art model for that category.
06:24Uh and so again, this kind of reiterates this idea that it's it's less important what you see at the top, but actually the subset of tasks that are most relevant for you and the cost performance optimization that you want to see within your company.
06:36So, I want to share uh some some lessons from actually building the benchmarks.
06:39We talked a little bit about the results and and why the results may be intuitive unintuitive.
06:43Um when we actually build benchmarks, one of the most important things that we need to look at is the metric we use to report accuracy.
06:50Uh and so you you you have a different takeaway if you pick a different metric.
06:53A lot of benchmarks deal with partial credit.
06:55And so what this is answering is in expectation, if I were to give a question or task to a model, what percentage of the points would I expect to uh see satisfied by the the the work the work output of that model,
07:07especially compared to humans?
07:09And then the fully corrected answers metric is more looking at of the percentage of questions that I give to models, what percentage of them will come back perfect?
07:16And that's that's a very different metric.
07:18It may mean uh that a model is very good at producing uh an incremental response that you can work with to build a better response.
07:25Um but if you were to pass this directly to a client, it would be a coin toss of whether it was meeting all the criteria that you have.
07:32Uh and so picking the right metric is very important to accurately representing the performance of a model and actually forming a good intuition of what it can do.
07:40There's another three components that we think a lot about.
07:43So, validity, are you actually evaluating what's going on in the real world?
07:46That means are the questions or tasks reflective of actual work going on or are they contrived?
07:50Um and then are the rubrics actually describing what we're looking for or the standard of excellence we want to see?
07:55Reliability, this is usually a function of the number of tasks or the sample count.
08:00Um a lot of companies, they have maybe five or six samples and they're kind of vibe checking new models or products as they get access.
08:06Um but it's actually really important to have a rigorous approach and have enough tasks and samples to make sure your repeated runs actually show consistent results.
08:14And then lastly, discriminability.
08:15So, we we don't think it's that valuable to have benchmarks where all models score 100% or all models score 0%.
08:21Ideally, we can we can actually discriminate between models and see what the frontier looks like.
08:26In particular, we also want to make sure we're building benchmarks forward-looking.
08:29So, what are the new set of tasks that are even more ambitious than what models can do today uh to make sure that there's room at the frontier for us to uh you could uh to continue to see model progress as they get better.
08:41Okay.
08:41So, and then some of the high-level problems that we've seen and I think others uh you know, can and and do run into.
08:48The first is that agents solve the task, not the problem.
08:50So, this is talked a lot about as reward hacking.
08:53Um, and we've seen a lot of instances where we'll have a sandbox and models will be run with our model proxy.
08:59And sometimes they may try and actually call other models uh within that proxy to escape the sandbox or make use of the internet.
09:06Uh, so it's really important to to build the the containers in a way that you're actually testing for what matters.
09:11Can they can they do the task at hand um or are they actually leveraging other tools or maybe even other models to get there?
09:16Um, we've also seen this issue in finance in particular where a lot of the times uh evals are looking at a set of rubrics or or items that we want to see included in the response.
09:25And so, models can hedge and give three possible answers uh which has a higher chance of satisfying all the criteria um but in practice, that's way way too much.
09:34A person does not want to consider three different answers.
09:37They just want to get the the first answer and and have it be right.
09:40Um, and so, the goal here is to try and build evals that are reflective of the actual work people are doing and the work output they want to see
09:47um not to build these contrived evals that that the models can actually gain.
09:51The second is I think a lot of people are starting to see that the cost of software is coming down and it's it's actually very very easy to produce code.
09:59But unfortunately, we still see in practice that running evals is incredibly challenging, especially from an infrastructure side of things.
10:06Um, and in particular, as we get to longer-running agents, it means that it's really important to have infrastructure that can stay consistent over a long period of time and then if interrupted,
10:15can be resumed and respawned back up.
10:17Um, we have I think just recently crossed 6 trillion tokens just in evaluation.
10:22Um, so we run a considerable asset of evals and we run everything in as highest degree parallel as possible.
10:29Um, so I think this has really been load tested, but I think other people are going to run into this even on a much smaller scale.
10:35Uh, and the last is that I think a lot of the existing benchmarks, they deal with a single prompt, single response, but in practice everything is multi-turn.
10:43Uh, you have an initial spec which you think covers everything, but then you get questions back or you have additional feedback once something's built and then it takes more of this iteration to actually drive to the end output.
10:52And so I think benchmarks and evals need to actually cover this multi-turn component.
10:57That's not going to go away.
10:59So to look at a case study in particular, when we looked at our team, we did a little token maxing experiment uh, where we we got free access to some tools and just gave it to all the engineers.
11:08Um, and they they really made use of these unlimited tokens.
11:11Um, and and if if we actually look back for that month, we spent 1.5 million dollars worth of tokens.
11:17This is like a team of I think 10 or 12.
11:18So it's roughly 10 times the amount we were spending on salary for tokens in that month.
11:23Uh, and so that was pretty insane, but we saw great output.
11:26And so our goal was to try and maintain the same level of output across all these engineers, but massively cut cut down on cost.
11:33And and really our solution to do that was of course to build an eval.
11:36Um, we built a bunch of tools internally and then have now exposed this as a product called Val Smith.
11:41Uh, and so this is you know, what we're doing in coding and I think it's again a sign of what's going to what's going to come in other parts of knowledge work.
11:47Um, so what we were able to do is look at the GitHub repo um, as a essentialized way of seeing what work has already been done and what work is continuing to happen.
11:55Um, and in particular GitHub repos are comprised of PRs or pull requests which have a GitHub issue and a resolution.
12:01And so what we can do is unwind the clock and give those same tasks to agents to see if they could have equivalently done the work of humans.
12:08Um, and in doing that we found a a a much clearer understanding of which models were overkill, were way too expensive, and did not need to be used, and then which ones were actually capable of getting the task done.
12:18Uh and I think again, this is another place where the intuitions that a lot of people have may not be what's actually born out in practice.
12:23So, I think you may see, "Oh, we're using a lot of close-source models.
12:27We should try and cut costs and use our own self-hosted open-source model."
12:30Uh you can see the comparison here we have between GPT-5.6 Soul and Kimiko 3, and we see that 5.6 Soul is better.
12:37It's It's going 2% better on this internal eval.
12:40It's also considerably cheaper than than hosting it through a very, you know, inference-optimized uh stack.
12:45So, it's $2.50 instead of $11, and it's faster, half the time.
12:49And so, there's no real reason why um this kind of intuitive sense of using open source would be a legitimate way to drive down cost.
12:56Um and so, this has resulted in two main differences.
12:58One, we can actually figure out which tools we're going to procure, what the usage limits of those things should be across the organization, but then also, we can give recommendations for engineers
13:08and say, "We think that for this GitHub issue, you should actually use this particular coding tool with this model."
13:15Uh and this is able to drive the cost down considerably.
13:17I think we're spending 90% less, but still able to get the same output.
13:23I think the question I would have for you all We're We're continuing to expand this approach and and take this model of coding to other domains, knowledge work, legal, finance, things like that.
13:31But the key question is, where are you tracking the history of work that has already been done by that firm?
13:37Some of these law firms have been around for like 100 years.
13:40And so, where Where is that I think quality of the organization stored?
13:43Where are you tracking the standard of excellence?
13:45And so, that's today may exist in places like email from the history of responses that people have had with their clients, may exist in SharePoint, may exist in Databricks dumps,
13:55but I think this is the key treasure trove that a lot of companies have and don't realize will be the major unlock for evals, and then the unlock for AI adoption more systematically.
14:10Once these evals are constructed, they can enable adoption to happen, but we've also seen that there's more advantageous ways that they can be used.
14:18So, first is right-sizing.
14:20So, rather than just token maximizing, how do you figure out what the right usage limit should be across the organization?
14:26Negotiation.
14:27Once you realize uh you have actual leverage, you can take that to the conversation you have in bulk pricing.
14:33Then routing.
14:34This is becoming a core primitive in uh in a lot of AI products, um but can also be done by users directly.
14:41And so, they can build intelligent routers that actually figure out the right model to use.
14:46And then fine-tuning or post-training.
14:48Uh this this can be the most reliable way for you to actually mint the tasks and then have an external eval set to make sure that this approach is working.
14:57Last thought I'd leave you with is I think for a lot of companies, they have a competitive advantage that has existed in the heads of employees through things like institutional knowledge.
15:06And what evals allow you to do is actually codify the competitive advantage and put that down in words or in scripts in language.
15:13Uh and that will actually be the the biggest unlock for uh making sure that a company succeeds through the agentic era.
15:19Okay.
15:20That's all.
15:20Thanks, everyone.
15:22[applause]