How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada

AI Engineer · 19 min · 256 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Thank you for joining this station.
00:14It's a bit of an emotional one.
00:18My name is Nathaniel.
00:20And I'm a non-evaluator.
00:23It's been a lot of time since the last agent I've been shipping without any evaluation.
00:30And today I want to tell you the story how I quit this addiction of shipping an agent without any evaluation.
00:39So, I'm working at Air Canada.
00:40So, compared to the previous talk, it's more like focus on specific business.
00:46I joined Air Canada in June of this year.
00:49And Air Canada is a huge company.
00:52We work in operation, corporate, commercial, and loyalty.
00:55Each of those segments have their own challenges.
00:58So, my team is in charge of delivering AI solution for all of those.
01:04And the problem when we tackle task we create an agent for a task that is so specific, we don't have any public benchmark we can rely on.
01:15And that's the main main problem we I'm trying to address in this presentation.
01:21More importantly, it's not that we don't have a benchmark, is we are data scientists that can build agents or AI engineers that can build agents, but we are not domain experts.
01:33And those domain experts sit in different business units.
01:36And they are very busy in their own task.
01:39And we have a limited access to that.
01:40So, both of those things went together in a situation where it's very hard for us to build any evaluation or any any benchmark for our task.
01:52So, before going to Air Canada, I was working for a different company.
01:56It was a company that and the task was to build a a Q&A assistant for telecommunication engineer.
02:05So, I don't know if you're familiar with the domain.
02:07The telecommunication is a huge domain.
02:09There is about 10,000 documents.
02:12A lot of vendor documents on top of that.
02:15There's a logic to know which document you should you should find in order to answer the question.
02:19So, it's it's a bit of a complex task.
02:22And then 2024, everybody wanted to build RAGs.
02:25So, we started to do that.
02:27Yeah, on LinkedIn it's super easy.
02:31It's not.
02:31So, um The thing is when we tackle this project, the same problem as at Air Canada, we don't have access to expert.
02:40We have the limited access to experts.
02:42And we we could not actually grade the answer ourselves.
02:47So, that's the the problem I I want to talk today.
02:50Funny enough, uh we were in competition with other teams.
02:54Um, two external, one internal.
02:56It's not like in parallel, but in segments.
02:59And our solution actually worked.
03:02What mean worked is actually delivering the value.
03:05The people liked the solution, actually used it.
03:09Uh, but to be honest, I don't know why we actually made it work compared to the other the other team.
03:15So, today I'm just going to show what I actually did or my team actually did to make it work.
03:22Um, so the the all have the same document, same task, everything is the same, but our solution somehow worked.
03:29So, for those that are familiar with RAGs, um the principle is very is is to ask a question to the system, the system found the the relevant documents, based on this relevant document,
03:42formulate an answer.
03:43It's something that exists for a long time now.
03:46And for those that actually tried it, if you found the right page or the right documents, 19 90% of the time you have actually the right answer.
03:54So, the challenge to find the right documents, at least in this task.
04:00So, we created a very complex pipeline that get complexified over time if you publication if you found publication back in the time this everybody has an amazing technique to improve the the solution.
04:13We try them.
04:15It's actually didn't improve the So, on paper it worked for them for the publication but for us it didn't really help.
04:22And more importantly when we add complexity, we found that like we break something else.
04:27So, something that used to work now is not working anymore.
04:29And that's actually the the real problem.
04:32So, we can add and more we can get more ideas on into solving the different problems but we we keep continue we keep breaking things over and over.
04:44So, complexity doesn't always help.
04:47And the other problem is this one.
04:49Uh we can change an agent in few minutes, add a certain layer of query writing or whatever reasoning step.
04:58But to grade that, we had those crazy meeting once a week with experts.
05:03We put a giant giant screen with five different chatbot, ask the same question to all of them and ask the the expert to grade them.
05:11I prefer this one, I prefer this one.
05:14And funny enough they're not always agreeing.
05:17It's very hard when you when you put expert to explain the room you get three opinions.
05:22So, that's the the the problem we were facing.
05:25And so, our budget was we can try one solution a week and this was not enough to go fast.
05:33So, the before uh going further so, the the the the process was the following.
05:38We ask we ask a question the agent answer.
05:40We ask the expert to read.
05:42Sometimes they don't know the answer.
05:43They have to look up.
05:44It takes a lot of time.
05:46And we had the budget of one trial a week.
05:51Um, so there is nothing new here.
05:53This abiding that is well known is we reverse the problem.
05:58So we start with a page.
06:00We ask an LLM to write a question about this page, which is uh an easy task.
06:06We ask the agent to search for this answer and compare to what the agent found, compared to the to the page that has been used for to generate the question.
06:17We can actually use code to grade that.
06:19And no longer opinions, no longer anything.
06:22We can automate this task.
06:24And that allow us to go much much faster.
06:28So for those that work in rags, there is plenty of implementation of this thing.
06:33I didn't invented it.
06:34I just found it very very uh very smart.
06:38And it stick to me.
06:40Like why this principle cannot be applied to other task?
06:45So the impact of that is we went from one evaluation a week to dozens a day.
06:49And we start to see the the accuracy of the the the grading from the expert going up and up.
06:55And actually removing a lot of complexity help.
06:58Like we didn't need a lot of those.
07:00And focusing on the right uh embedding model, the right chunking, the right alpha in the BM25 uh is the right thing to do.
07:09And we we saw the we saw the the accuracy increasing over over weeks.
07:14[snorts]
07:15So I don't know why we succeeded, but I'm pretty sure the other team didn't had any test set.
07:20So they were using this kind of long loop with expert maybe a much more expert than we had access to.
07:26But we had it uh we had this test set.
07:30And my learning from this uh back in 2024 is if you cannot grade the agent, uh you can't make it better.
07:39So that stick to me until this day has been 2 years now.
07:44And I want to go back to Air Canada now.
07:47Where we have a different project.
07:50So in 2026 the project is about analyzing supplier contract.
07:58So you can imagine Air Canada has a lot of suppliers.
08:01We get a lot of those contract.
08:03There's expert looking at them one by one.
08:06Checking each of those clauses.
08:09And they have to analyze it and provide a feedback.
08:12Like this clause is not compliant.
08:13This clause needs to be rewritten.
08:15Like there's a lot of work to do that.
08:18It's a very long task and very costly because if we need the if we need some supplier to deliver right now and we don't, then it cause a lot of delay in the in the chain.
08:29And if we miss a clause, it's also very costly because at at the end of the day it can backfire to us.
08:35So it's a tricky problem.
08:37Um and they came to me I mean to the to the data science team to build an agent that can actually do that for them.
08:46So can we automatize this task?
08:49And obviously there's a lot of documents, a lot of rules, a lot of things to consider when you do that.
08:56Uh expert are super busy.
08:58You can imagine the price of of a lawyer by hour.
09:02And there's no benchmark for that.
09:03We cannot go on the web and found documents I found an existing benchmark we can apply.
09:10So go back to the same problem as I was in 2024.
09:15Uh cycle were super slow and we knew that we want to be there in a in a in in a in a in a shape where we can go fast and try and try and try.
09:26Um and I knew that if we if we learn to this situation, we can have a better trajectory in term of accuracy.
09:34And that What the challenge for this uh for this project?
09:38So, the the problem is creating the answer from documents is a well-known and defined thing in the rags, but for this specific project, there's no guy in the world that say, "Hey, you should use this technique.
09:49You should use this technique."
09:51So, we try to think how we can reapply the same mechanism from what I learned on rag to this specific project.
09:59And we have plenty of project at Air Canada.
10:01So, we're using the same concept of getting the answer first and then try the agent to find it back.
10:08So, going into the details, uh it's a bit simplified, but actually the um the agent has four steps.
10:15Uh we need to spot the clause in the contract.
10:18We need to find the relevant documents to support an amendment.
10:23Uh we need to write the the amendment.
10:26And then at the end, there's a classification system that tells you should we keep it or not uh this amendment.
10:32So, the first step, for example, spot is classifying all the section of the contract into specific domain domains.
10:39And then found if there's a defect or there's a change to be to be done in this section.
10:46So, we did have this trick from the rag.
10:48So, we know how to find the documents.
10:49So, this is covered.
10:51Uh but for the other other section of the um of the agent, we didn't have anything.
10:59So, I going to present five different techniques that we use in this project or we try to use in this project.
11:05That is trying to address this specific problem.
11:09My goal is not to show off.
11:12My goal is to get you inspired in how to reapply those techniques in your own uh challenges.
11:20So, the first one is to use past decision and answer.
11:22So, we we you can retrieve past contract, compare with the one that has been changed.
11:29The second one is to choose something you're looking for, write the contract around or generate the contract.
11:35The third one is uh take a good contract that has been signed and break it.
11:41Um the fourth one is about things that should not change no matter what you do, you should not get more flags.
11:49And the fifth one is actually the most powerful.
11:52It's very hard.
11:53It's to turn an example into many using the concept of templates.
12:00So, immediately we jump into this approach.
12:03Uh we ask the expert to provide us the original contract, the one that has been signed.
12:09We're going to take that, do a diff like you do in the code, compare what has been changed, and the agent should should provide the same exact flags.
12:17That would be perfect.
12:19However, for many reason, our our team do not keep all the contract.
12:25Um they don't keep the original one, they just keep the last one, they don't have all the version of the contracts.
12:31It sometimes goes into negotiation, there is a lot of things.
12:34So, we could not have any of those pairs of contracts, so this thing doesn't work.
12:42Okay, second one.
12:44We took a rule.
12:45And so, that's changing a bit the topic.
12:48Um instead of creating um flags or creating an agent that can provide flags, we start from rules.
12:56And the rule is, for example, we have the the cap above $1 million.
13:01So, that's every contract that Arcada should follow this rule or many of those in specific domains.
13:07So, you know the rule, you create uh a clause without breaking this rule and generate the contract around, and you should found um the rule back the the flag,
13:19which is breaking the rule.
13:20So, that's simple.
13:22It doesn't require a lot of uh of work.
13:25A a of things can be generated.
13:26The problem is you need to know the rules.
13:29So, luckily enough, we could actually get some of those from either the expert or the documentation they had, uh but cannot guarantee the coverage of everything.
13:39So, we came up we also with our own rules, etc. So, we generate at the end uh 275 tests that we know can run and we know the answer.
13:51So, similar to the rug, we inverse it we inverted the problem, we planted the problem, and we try to find it back.
13:58Another one which is a very hard, actually, uh we took an example of a contract that has been signed, we remove a clause, and the agent is supposed to say, "Hey, there's something missing here."
14:12Um it's a very hard task because some supplier has different templates, have different kind of content.
14:20Knowing what should be in what contract require a bit of thinking.
14:24Uh and we mix those with existing contract that should not show any flag.
14:29So, that's the most powerful approach what that we had today.
14:32Um using this approach, we can generate a lot of contract, and at the end of the day, we found that our agents are missing uh most of the clause that has been removed.
14:42So, that's a very very strong benchmark.
14:46Okay.
14:47Uh fourth technique, um you don't change you change something that should not affect the agent or should should not change anything in the result.
14:58For example, the name of the supplier, the address of the supplier, something that should not have any impact on the number of flags.
15:05Um you can reorder the clause, so some randomness should also lead to the same results.
15:13Uh and you can run the same uh contract again and again.
15:16All of those are things that we check to make sure that the the output is consistent over all of those.
15:23In this example two, everything can be checked using code.
15:27You don't need to do any LLM as a judge.
15:29The previous uh speaker showed like it's very hard task.
15:33So, we stay away from uh LLM as a judge as much as possible.
15:38The fifth one, um it's the concept of template.
15:44Our expert have different kind of templates with blank.
15:47So, I don't know if you ever write a contract.
15:50Those blanks should be filled by the expert.
15:52And instead of trying to fill them, we generate um an array for each of those blanks to be filled with.
16:02So, that's allow us to generate plenty of contracts.
16:04Some tuples break some rules, some tuples doesn't.
16:07We know that.
16:09And so, we know what actually should be flagged as a as a as a breach or not.
16:14Again, this is a code check.
16:16We don't need to have any LLM involved.
16:18Um What What The good thing is with this one is our expert already use templates, so we can reuse them to generate some contracts.
16:29So, the cycle was a bit slower in this case, so 4 weeks to get a a feedback, and we can get a script to run and get the the evaluation back in 5 minutes.
16:41So, I just want to highlight one point, and we went through all of those um kind of categories of grading at the at the bottom.
16:48Exact match or code that are very fast.
16:51As soon as we step into LLM as a judge, uh it's can be very fancy, it can be very useful.
16:57It's so hard.
16:58Like, we got so much noise from the LLM as a judge compared to the noise with the with the agent that we building.
17:05Solving this problem is actually almost as hard as solving the the the right problem.
17:10So, finding the right prompt of the right harness for the LLM as a judge is a very hard task.
17:15There's some padding you can do.
17:17task.
17:18You can do fact checking.
17:20You can do a bunch of stuff, but actually it require so much work and we could not found the right one for this project.
17:29There's plenty of other techniques.
17:30I won't go over all of them.
17:32For example, trajectory.
17:33We have another project that require the agent to eat some hand points.
17:38So, trajectory you can you you know what the end point should be in some order.
17:43You generate a story about it.
17:45You ask the LM to fill to use the agent and look at the trace to to see if the the trajectory is correct.
17:52Test to SQL, you might already know it.
17:55Doesn't matter what your agent is.
17:57If you know how to pick the answer, if you know what you're looking for, and I showed a couple of examples.
18:04Generate the input and then try try to find it.
18:07Doesn't apply to everything.
18:09And sometime we had to re-engineer our solution to fit this paradigm, even if we move away slightly from the original design of the solution.
18:18We could with the original design we could not evaluate it.
18:21We switched to something that we can actually evaluate because of that.
18:27Um one thing to highlight is not the end of the day.
18:32You generate the test.
18:34You get a better agent.
18:35You get some real feedback.
18:37So, expert can provide some feedback and we don't use this feedback to fix the agent.
18:42We use this feedback to generate better test.
18:46Better prompt.
18:47Better ideas how to remove clauses, etc. etc. And one loop at a time.
18:53Uh that can be done by Rise.
18:56There's a lot of tools there.
18:57Thanks for the invitation.
18:59[snorts]
19:00Um and that's my conclusion slide.
19:02Why the answer first?
19:04Um I think I I explained that.
19:10When you generate those tests, it's not enough to know if you can ship the product.
19:14Just a starting point.
19:16It's a tool to help anticipate the developer.
19:18It's not the go no go for the for the solution.
19:23So I hope you like this this presentation.
19:27I know that the 24 hour chip is the hardest one.
19:30If you want to quit releasing agent without evaluation and I would be happy to support you and have your own feedback.
19:40Thank you very much.