If Your AI Scores 100%, Your Eval Is Wrong - Ramakrishnan Lokanathan, eslams.com

AI Engineer · 18 min · 188 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12I think why all of us are here primarily is because evals are currently very broken, right?
00:17And like that is what we're trying to explore and understand how to go about it.
00:21More often than not, I think evals are broken because we overcomplicate it and we don't really understand, right?
00:28We hear things like the eval is more important than the model and the eval is more important than the hardness or whatnot, but what even is an eval, right?
00:35It's a question that we all sort of come to and like this is something that I consistently hit when I speak with people building high-stake applications to even understand
00:43to the first principle level of what an eval is, right?
00:46And what is an eval is just a abbreviation of the term evaluation, right?
00:51And how we understand evaluation is like we know exams, which are examinations and this is a concept that we understand in human beings, right?
01:00And if you think about it, as people we write a variety of exams through our life, like we write the SATs, the GMAT, the bar exam and like regardless of what we do.
01:09That is the same sort of idea you can apply as a parallel into evals and then you can think about evals as those examinations for AI systems, right?
01:17You have the Sweet Bench, the humanities law exam and all of this.
01:21In multiple ways, this is trying to evaluate and try to understand how a model or an agent will be working in something, right?
01:30At the end of the day, examinations and evals are just tasks, standards, and evidences to sort of evaluate and understand how something is to perform on something as you expect, right?
01:40And I think that brings us to a very key distinction here as well.
01:44An exam must answer a bounded question and let's take the example of the SATs.
01:49Um it's a college entrance exam and it tells you how good a student is on language and math, but it's just one signal of a college readiness, right?
02:00I come from India and even today in India your college admissions are done with a single examination.
02:05However, let's say you look at the US exam, SAT is one signal of college readiness, right?
02:11And I think that's extremely key.
02:12You should not index on one eval or one test and it should just be one signal of your readiness to the platform rather than trying to sort of over index on
02:22your model being really good at humanities class exam or at W bench, right?
02:26So, no eval or no test is a measure of someone's complete abilities and that should always be clear going into that.
02:36So, at the end of the day, what we try to answer with evals is is our performance matching the promise that we're making to the consumer or the user at any given point in time, right?
02:47And that's the job of any eval to make sure that we go ahead and ensure that the promise of the product of the agent that we're providing is meeting its performance to the consumer.
02:58Before we move ahead, I wanted to sort of give a small brief on who I am.
03:01I'm Ramakrishnan.
03:02I come from the city called Cochin in in South of India.
03:06I started coding very early on and started building machine learning classifiers in high school and whatnot.
03:11And I started my career with an Indian EdTech unicorn which gave me the opportunity in '21 to get a demo of a GPT-3 based product on how to better our content flow
03:21which changed my mind being someone who really loved machine learning from an early age.
03:25And I was like, this is going to really change the world, right?
03:28And my claim to fame or like what I was right in doing was in September 2022, we decided that there is a new wave of applications that are going to be possible that are going to be built on the GPT-3 API,
03:40right?
03:40And about 2 months after we went full-time, OpenAI released released their research preview for GPT-3.5 called ChatGPT and like that literally changed life for us and it validated us and killed us as well, right?
03:51So, went through multiple pivots and then we eventually landed on a pet care AI assistant and we monetized it using e-commerce and recommendation and engines on top of it and we got acquired in India and that was the end of that journey, right?
04:05Uh after that, I've been super interested in understanding how these systems evolve and because we were super early on building on GPT-3, evals were critical for us to build.
04:15So, that took me to being a red teamer at for the Nematron 3 models.
04:19I work at the Lawrence Berkeley Lab right now on a AI safety research and also with the London Business School data science and AI initiative.
04:26Broadly to explore how to sort of look at the new technology that we are working on and then how we see and explore, right?
04:33So, I am not aligned or like, you know, with anybody here, so I'm not going to sponsor a vendor.
04:37It's just to understand how do you think about eval and like build these high-stake applications, right?
04:42And the consolidation of the conversations that I've had over time with multiple implementations that we have done with millions of users using our AI product, sorry.
04:51So, this is something I strongly believe.
04:53So, evals are not going to be relevant in a decade's time.
04:56Because if we are to believe that model capability is to keep improving, we will not have to have as much eval as we do.
05:03However, in the midterm, like production cannot wait.
05:08The problem is if you don't have the right amount of eval structure, you get reports like the MIT GenAI divide where 95% of organizations are not able to get any returns because you are not delivering the promise in production
05:19that you're trying to make with the LLM, right?
05:22And that is what we have to solve with evals today and in 10 years time, evals is just going to be a validation of this.
05:28But today, it is to make sure that in production you are delivering the promise that you're going ahead for, right?
05:34So, the first thing that you have to think about as you build evals for high-stake applications is to define the most ambitious success, right?
05:42Before designing any test.
05:44Like, assume in in of AGI, what is your AI agent going to do?
05:48Or what is the best outcome you can get out of it, right?
05:51And setting this goal really gives you the scope of what you can go ahead and design further, right?
05:57And you start with the job.
05:58And simple the first question that you ask is what must happen.
06:02And what must never happen?
06:04And when should it stop?
06:06And you can extend it into other WH questions and first principles to go ahead and think about what could be the most ambitious outcome that you can get out of it, right?
06:15And aim at the job you need and not the score you can get.
06:19If you have an eval that is scoring a 100 of a 100, your eval is wrong, right?
06:23And you're not ambitious enough and you're not testing enough, right?
06:25If you look at any model eval today, people are at like 85, 86 max.
06:29Nobody gets to 100.
06:31So, you have to make sure that the evaluation and your ambition is beyond what the model ability today is for you to make sure that you're putting it to the stress test and not making it easy, right?
06:41And always test the whole workflow, right?
06:45And not just the final answer.
06:47And this is the original picture, and my request would be only change the background, right?
06:52And I am just looking at the output.
06:55And the output has changed the background to a cream background, but it has two handles now, right?
07:01So, it has to be very clear that you have to always test the work.
07:04And this is a very simplified example of this.
07:06Let's say you have a 100 steps in your workflow.
07:08Each step has to be evaluated for what goes into that step and what comes out of that step and what is the expected outcome.
07:15You can't look at the start point and the end point and be like, "Okay, if you look at the start point and the end point, the background has changed."
07:20But is that the objective that you're trying to achieve, right?
07:22So, always for look at the whole workflow when you're approaching things, right?
07:28And the third thing, and I'm I'm going to talk a little bit about something I've been working on in this space as well, is one score is not the system, right?
07:37And the idea is to build a holistic evaluation system.
07:41When you started off with models in 2022 and 2023, you were measuring for coding ability or let's say you were measuring for writing ability, etc. But today you have agentic persistent systems that have to do multiple things, right?
07:54So you have to make sure that you're designing a holistic evaluation system at all points in time.
07:58And the first element of that is that automated checks and eval being part of your CI pipeline, right?
08:04And your eval questions have to be asked to your agent every time they are going through a production deployment process, right?
08:11So that being part of the automated checks makes sure that it's not like one eval score you are taking at the time you're deciding the model.
08:19Continuously human review and make sure that the results make sense.
08:23Don't rely on your agents to be the reviewer of your eval at any point always, right?
08:28You can fast-track your eval process with it, but make sure you're doing human review.
08:32And the third part is adversarial test, right?
08:36Where can you break the workflow and like put it into challenging situations like where you can have adversarial testing?
08:43And one thing I've been recently working on and would love to talk about is um the idea of how you can choose the test for the failures it can reveal, right?
08:53And the idea of a test that can fight back.
08:56Today you have a task on SW bench, the agent or the model is asked to go ahead and perform one task and give an output.
09:02But there is no continuous thinking or evaluation, right?
09:06And getting one right answer more and more is not the same as making a good sequence of decisions and you need agentic action and long-chain reasoning and all of these things.
09:17For the longest time the best measure of in human beings have been board games like chess and go.
09:23And even when you go ahead to um like like evaluation of like a model, AlphaGo is a very important step to AI development, but it never very specifically trained model for playing Go.
09:36Today's general-purpose LLMs we don't test on Go or chess, and that is what I'm trying to solve with an open-source repo called E-Slams, which is electronic slams, and we make
09:45AI play games, right?
09:46So, you can just install the package.
09:48It's a Python PyPi package, and you can make your agent and your model play games and see how it works.
09:55And how can you put it to test, right?
09:57You have You're replacing an agent with an improved model.
10:00You might want your old agent to play with the new agent on on a chess game of chess.
10:05And let's say I don't I'm not saying the winner of that match is a better model, but how they enter that match and play that match will tell you a lot more about how to sort of go ahead with that model.
10:15Because playing chess, you are changing your strategy, you have to get out of blunders, and move that you make can become liabilities for you in the past.
10:23So, how is your agent reasoning through that is an additional paradigm to the holistic evaluation you can have.
10:30And the idea here is do a coding evaluation, do a writing evaluation, do humanities last exam, and try to have as many paradigms as possible, and that's what I'm trying to build with E-Slams, right?
10:40So, we support about 50 games and 90 different like providers.
10:45We have chess and Go, which looks at planning invisible state.
10:48We have poker, which is decision-based on hidden information.
10:51And we have cooperation games based on shared goals and cooperation.
10:54And we also have trials for arcade games, right?
10:58And all of this tells you a lot about models and how they think.
11:01And on our platform, we have preview on eslams.com of how you can use the repo.
11:07And this is an example of Fable 5.1 playing tic-tac-toe against GPT-6 Astra, right?
11:12And you can look at how Fable and Astra are looking at each other's turn, and how they go ahead and play this.
11:18And this match, interestingly, ends in a tie.
11:20And most cases, a model wins, but Fable and Astra ended up knocking their heads to each other, right?
11:27And let's say you are deploying an agent in production.
11:30You can You want to go ahead and play against these AI models and see how they are.
11:34You can come on Islam.com and like play against these AI models on any different game that you want to.
11:40And you can all do all of this using the open source repo and make it play with your own agents and like evaluate your systems, right?
11:47And we are going to host the Grand Slams of AI on in February and the idea is to put an Olympics of for AI models to go ahead and compete and know who will stand out of all of this, right?
11:59So, please do check out Islam to know where we're going with it and this could be one paradigm of model evaluation you can add into your stack.
12:07So, now coming back, the main key part after all of this, right?
12:11Like you keep evolving, you keep adding paradigms, is to make sure that you keep testing the evaluation workflow, right?
12:17And the most important part of that is keeping what works in the flow.
12:21And my best example of this is Google Assistant moving to Gemini.
12:26I used to use Google Assistant for all of my reminders.
12:29And one day it moved to Gemini and it stopped giving me reminders and it said it's a language model that cannot set up reminders, right?
12:36So, it was excellent in the old Google Assistant model they had and before they moved it to Gemini and it stopped working and this is an example of Google
12:43not doing eval on what used to work, right?
12:46When they went ahead and launched the Gemini product.
12:48And that's extremely important to go ahead and make sure that you keep testing like the regression cases, right?
12:53And known task and past failures because a model that is extremely better than the old one could very well be incapable of something the old one did very well, for example, setting a reminder,
13:04right?
13:05And um fresh challenge cases at all points goes back to what I was saying in terms of setting the most ambitious target.
13:13The idea is that you have to keep pushing your model.
13:16If you feel that your model your is getting to an 85 or a 90 on your eval test, it's time that you better your eval and like push the frontier, right?
13:25And make sure that your model is challenged and has cases where it is not able to perform to expectations.
13:32And at all points, right?
13:35And I think this has been true in software throughout history is that you have to turn real-world failures into future tests, right?
13:42Your customer and your user is the most important evaluator of your agent and your model.
13:48So, every failure should have a proper workflow for it to enter into your system and make sure that like be that, right?
13:57So, one is to keep observing and there are multiple ways of making sure that you can observe.
14:02There are multiple observability tools that give you automatic data, give option for your users to report it to you with your simple thumbs up or a thumbs down, and collect as much data where your customers are telling you where there is a failure in the
14:16pipeline, right?
14:17Classic QA on software, go ahead and reproduce that issue, and try to understand where is the model failing?
14:24Is it a tool call?
14:25Is it Is it And what part of your eval suit failed in going ahead and doing that, right?
14:30And review it, add it to the pipeline, and add that to the eval pipeline.
14:35This becomes extremely important to keep this feedback loop going so that you go ahead and access new frontiers of things that you are missing out in every point in time.
14:45And I think the two ways to go ahead and get to eval that is as holistic as possible is one to consistently human review a process, and then ensure that your customer feedback is part of your eval pipeline,
15:00right?
15:00Because these are the last 5% edge cases that really come out which a model or an existing eval framework might not catch.
15:08So, to conclude, test the job and not just the answer, right?
15:12And that's extremely key to the whole process.
15:15And I'll just wrap up with all the things that we've discussed is one is always define your most ambitious success.
15:22Think about what life would be in AGI, right?
15:25And think about life in a model that has no inabilities and can absolutely perform like the human that you're trying to sort of emulate and try to define in this process, right?
15:39Second is continuously test the workflow at every part of the workflow.
15:44Don't assume things in the workflow or like assume for an input and output to be the outcome of anything, right?
15:51And when you test the workflow, I think something I slightly missed on here is to make sure that your your tests are designed for what that element of the workflow does.
16:01For example, let's say in that in your workflow there's a specific part that is doing tool calls consistently to get the information, your eval has to be designed for tool calling and not for any different process
16:14separately, right?
16:15Because the objective in that workflow process is to know what is your tool call ability and the tool tool call accuracy, right?
16:22At that at that point.
16:24So, every time make sure that when you're testing the workflow, you design the test to that specific part of the workflow and measure what is the success in that element of the workflow as well.
16:35And continuously gather evidence holistically.
16:38In an agentic world, what evidence for agentic action eval is something that keeps evolving.
16:45Uh and games is something I'm exploring to see how it works out, but there are so many different frontiers for you to go ahead and look at, right?
16:53Something super interesting that I came across while here I'm here is the parallel search model index, right?
16:58And how models are good at fetching search from parallels like AI search index.
17:03And that becomes very important for you to evaluate when you are designing a model that needs real-time data, right?
17:09So, make sure that you as a company or a person building these products is consistently looking for more evidence and more arenas and more paradigms for you to evaluate your models
17:20and that is one sure shot way to making sure that you cover most edge cases.
17:26The fourth one is to evolve the test as time flow and making sure that you cover all aspects from the past and the future into the test and the simplest thing is if your test is scoring high is it 85 90 that is the time you change your test and you have to make it harder.
17:42Right?
17:42The simplest thing is if you think everyone is scoring like in the 99th percentile for the SATs, it is time that the test have to change in the internet of itself, right?
17:52And always learn from reality.
17:54Reality is going to be your best teacher and the idea of eval that I started off with is to make sure that we deliver the promise that we are making to the consumer and the user in production
18:06and that is most defined in the customer feedback and the reality that really happens, right?
18:13So, consistently think about how you can incorporate real user feedback and real world test and how you can have human review and human testing into the workflow as you go ahead and get started.
18:24I think with these five thought processes and the idea that you go ahead and test the job and not just the answer, you will have a good mental framework
18:33to go ahead and build eval for high stake applications.
18:35Thank you for coming.
18:36I'm happy to catch up for questions afterwards.