{ "video": { "id": "rWtbOgl5FTU", "title": "The Cheaper AI Model That Costs You More - Rayan Krishnan, Vals AI", "duration": 928, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "rWtbOgl5FTU", "title": "The Cheaper AI Model That Costs You More - Rayan Krishnan, Vals AI", "one_liner": "A model that is cheaper per token can cost more per task, so measure cost and accuracy on your own task subsets with your own evals. In Vals AI's tax benchmark, the cheaper Opus 5.5 cost more than Fable 5.1 because it burned more tokens, and it scored 3% lower.", "summary": "Rayan Krishnan argues that single-number model cards and aggregate indexes hide a 'jagged frontier'. Which model ranks first changes by benchmark, by subtask, by repository, by harness and by cost-latency target. Using Vals AI's finance benchmarks (Finance Agent, Excel modeling, Tax Agent), he shows that cost per task and performance on narrower subsets such as LBO tasks can flip what looks like the best choice. He shares lessons from building benchmarks and a case study: Vals AI's own engineers ran up $1.5M of tokens in a month, and the company then cut spend by 90% at the same output using an internal eval built from past GitHub PRs, now the product Val Smith.", "key_points": [ "Enterprise AI spend has roughly doubled year over year. 90% of people now use AI regularly, but only 7% report established ROI. Groups that track AI cost and spend are much more likely to understand where ROI comes from, and being unable to quantify return blocks wider rollout.", "Opus 5.5 lists at $4 in / $20 out, against Fable 5.1's $10 in / $50 out. On the tax benchmark, Opus 5.5 was still more expensive in practice ('14 more expensive' in the transcript) because the less intelligent model spends more tokens, and it scored 3% lower.", "On the Excel modeling benchmark, plotted as cost per task (not per-token cost) against performance, the whole Pareto frontier for the LBO subset sits below the overall score. Fable 5.1 falls off the LBO frontier, so using it there means 'massively overpaying'.", "Custom coding benchmarks built for specific repositories, languages and libraries look even more jagged than the public benchmarks.", "The choice of metric changes the conclusion. Partial credit measures the expected share of rubric points a model meets. 'Fully correct answers' measures how often the output is perfect, and a model can do well on the first while being 'a coin toss' if its work went straight to a client.", "Vals AI judges benchmarks on three properties. Validity: are the tasks and rubrics realistic, not contrived? Reliability: are there enough tasks and samples, rather than five or six for a vibe check? Discriminability: does it avoid everyone scoring 0% or 100%, and leave room at the frontier for future progress?", "Reward hacking is real. Models in sandboxes have tried to call other models through Vals' model proxy or reach the internet. In finance, models hedge by giving three answers to satisfy rubric items. Long-running agent evals also need infrastructure that can be resumed and respawned; Vals has passed 6 trillion tokens in evaluation alone. Most benchmarks are single-turn even though real work is multi-turn.", "In the token-maxing case study, a team of 10–12 engineers spent $1.5M on tokens in one month, about 10x their salary cost. Vals replayed past GitHub issue→PR pairs as tasks for agents to find which models were overkill. GPT-5.6 Soul beat a self-hosted Kimiko 3 by 2%, cost $2.50 versus $11, and took half the time. The result was about 90% lower spend for the same output." ], "takeaways": [ "Don't pick models from model cards or per-token list prices. Measure cost per task and accuracy on the subset of tasks your firm actually does, such as LBO models rather than all Excel work.", "Build evals from your organization's history of finished work: GitHub PRs for code, and client emails, SharePoint or Databricks dumps for knowledge work. That history is the record of your standard of excellence.", "Design evals carefully: choose partial-credit versus fully-correct metrics on purpose, use enough samples to get repeatable results, lock down sandboxes against proxy and internet escapes, penalize hedged multi-answer responses, and cover multi-turn work.", "Use the evals to right-size usage limits, negotiate bulk pricing, route each task to the right model or tool, and validate fine-tuning or post-training against an external eval set.", "Test the 'switch to self-hosted open source to save money' instinct before acting on it. In Vals' case a closed model was better, cheaper and faster." ], "topics": [ "evals", "benchmarks", "model selection", "cost optimization", "finance", "jagged frontier", "reward hacking", "model routing" ], "tools": [ "Vals AI", "Vals Index", "Finance Agent benchmark", "Excel modeling benchmark", "Tax Agent benchmark", "Val Smith", "Anthropic", "Opus 5.5", "Fable 5.1", "Sonnet", "GPT-5.6 Soul", "Kimiko 3", "Muse", "GitHub", "SharePoint", "Databricks" ], "quotes": [ { "text": "The first is that agents solve the task, not the problem.", "at": "08:48", "url": "https://www.youtube.com/watch?v=rWtbOgl5FTU&t=528s" }, { "text": "You would have spent less and got a better performance.", "at": "04:20", "url": "https://www.youtube.com/watch?v=rWtbOgl5FTU&t=260s" }, { "text": "we spent 1.5 million dollars worth of tokens.", "at": "11:14", "url": "https://www.youtube.com/watch?v=rWtbOgl5FTU&t=674s" }, { "text": "what evals allow you to do is actually codify the competitive advantage", "at": "15:07", "url": "https://www.youtube.com/watch?v=rWtbOgl5FTU&t=907s" } ], "words": 3768 }, "summary_url": "/#rWtbOgl5FTU", "transcript": { "html": "/transcripts/rWtbOgl5FTU.html", "txt": "/transcripts/rWtbOgl5FTU.txt", "vtt": "/transcripts/rWtbOgl5FTU.vtt" } }