How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada
https://www.youtube.com/watch?v=c_SPW__jF9w
AI Engineer · 19 min · 256 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:12] Thank you for joining this station.
[00:14] It's a bit of an emotional one.
[00:18] My name is Nathaniel.
[00:20] And I'm a non-evaluator.
[00:23] It's been a lot of time since the last agent I've been shipping without any evaluation.
[00:30] And today I want to tell you the story how I quit this addiction of shipping an agent without any evaluation.
[00:39] So, I'm working at Air Canada.
[00:40] So, compared to the previous talk, it's more like focus on specific business.
[00:46] I joined Air Canada in June of this year.
[00:49] And Air Canada is a huge company.
[00:52] We work in operation, corporate, commercial, and loyalty.
[00:55] Each of those segments have their own challenges.
[00:58] So, my team is in charge of delivering AI solution for all of those.
[01:04] And the problem when we tackle task we create an agent for a task that is so specific, we don't have any public benchmark we can rely on.
[01:15] And that's the main main problem we I'm trying to address in this presentation.
[01:21] More importantly, it's not that we don't have a benchmark, is we are data scientists that can build agents or AI engineers that can build agents, but we are not domain experts.
[01:33] And those domain experts sit in different business units.
[01:36] And they are very busy in their own task.
[01:39] And we have a limited access to that.
[01:40] So, both of those things went together in a situation where it's very hard for us to build any evaluation or any any benchmark for our task.
[01:52] So, before going to Air Canada, I was working for a different company.
[01:56] It was a company that and the task was to build a a Q&A assistant for telecommunication engineer.
[02:05] So, I don't know if you're familiar with the domain.
[02:07] The telecommunication is a huge domain.
[02:09] There is about 10,000 documents.
[02:12] A lot of vendor documents on top of that.
[02:15] There's a logic to know which document you should you should find in order to answer the question.
[02:19] So, it's it's a bit of a complex task.
[02:22] And then 2024, everybody wanted to build RAGs.
[02:25] So, we started to do that.
[02:27] Yeah, on LinkedIn it's super easy.
[02:31] It's not.
[02:31] So, um The thing is when we tackle this project, the same problem as at Air Canada, we don't have access to expert.
[02:40] We have the limited access to experts.
[02:42] And we we could not actually grade the answer ourselves.
[02:47] So, that's the the problem I I want to talk today.
[02:50] Funny enough, uh we were in competition with other teams.
[02:54] Um, two external, one internal.
[02:56] It's not like in parallel, but in segments.
[02:59] And our solution actually worked.
[03:02] What mean worked is actually delivering the value.
[03:05] The people liked the solution, actually used it.
[03:09] Uh, but to be honest, I don't know why we actually made it work compared to the other the other team.
[03:15] So, today I'm just going to show what I actually did or my team actually did to make it work.
[03:22] Um, so the the all have the same document, same task, everything is the same, but our solution somehow worked.
[03:29] So, for those that are familiar with RAGs, um the principle is very is is to ask a question to the system, the system found the the relevant documents, based on this relevant document,
[03:42] formulate an answer.
[03:43] It's something that exists for a long time now.
[03:46] And for those that actually tried it, if you found the right page or the right documents, 19 90% of the time you have actually the right answer.
[03:54] So, the challenge to find the right documents, at least in this task.
[04:00] So, we created a very complex pipeline that get complexified over time if you publication if you found publication back in the time this everybody has an amazing technique to improve the the solution.
[04:13] We try them.
[04:15] It's actually didn't improve the So, on paper it worked for them for the publication but for us it didn't really help.
[04:22] And more importantly when we add complexity, we found that like we break something else.
[04:27] So, something that used to work now is not working anymore.
[04:29] And that's actually the the real problem.
[04:32] So, we can add and more we can get more ideas on into solving the different problems but we we keep continue we keep breaking things over and over.
[04:44] So, complexity doesn't always help.
[04:47] And the other problem is this one.
[04:49] Uh we can change an agent in few minutes, add a certain layer of query writing or whatever reasoning step.
[04:58] But to grade that, we had those crazy meeting once a week with experts.
[05:03] We put a giant giant screen with five different chatbot, ask the same question to all of them and ask the the expert to grade them.
[05:11] I prefer this one, I prefer this one.
[05:14] And funny enough they're not always agreeing.
[05:17] It's very hard when you when you put expert to explain the room you get three opinions.
[05:22] So, that's the the the problem we were facing.
[05:25] And so, our budget was we can try one solution a week and this was not enough to go fast.
[05:33] So, the before uh going further so, the the the the process was the following.
[05:38] We ask we ask a question the agent answer.
[05:40] We ask the expert to read.
[05:42] Sometimes they don't know the answer.
[05:43] They have to look up.
[05:44] It takes a lot of time.
[05:46] And we had the budget of one trial a week.
[05:51] Um, so there is nothing new here.
[05:53] This abiding that is well known is we reverse the problem.
[05:58] So we start with a page.
[06:00] We ask an LLM to write a question about this page, which is uh an easy task.
[06:06] We ask the agent to search for this answer and compare to what the agent found, compared to the to the page that has been used for to generate the question.
[06:17] We can actually use code to grade that.
[06:19] And no longer opinions, no longer anything.
[06:22] We can automate this task.
[06:24] And that allow us to go much much faster.
[06:28] So for those that work in rags, there is plenty of implementation of this thing.
[06:33] I didn't invented it.
[06:34] I just found it very very uh very smart.
[06:38] And it stick to me.
[06:40] Like why this principle cannot be applied to other task?
[06:45] So the impact of that is we went from one evaluation a week to dozens a day.
[06:49] And we start to see the the accuracy of the the the grading from the expert going up and up.
[06:55] And actually removing a lot of complexity help.
[06:58] Like we didn't need a lot of those.
[07:00] And focusing on the right uh embedding model, the right chunking, the right alpha in the BM25 uh is the right thing to do.
[07:09] And we we saw the we saw the the accuracy increasing over over weeks.
[07:14] [snorts]
[07:15] So I don't know why we succeeded, but I'm pretty sure the other team didn't had any test set.
[07:20] So they were using this kind of long loop with expert maybe a much more expert than we had access to.
[07:26] But we had it uh we had this test set.
[07:30] And my learning from this uh back in 2024 is if you cannot grade the agent, uh you can't make it better.
[07:39] So that stick to me until this day has been 2 years now.
[07:44] And I want to go back to Air Canada now.
[07:47] Where we have a different project.
[07:50] So in 2026 the project is about analyzing supplier contract.
[07:58] So you can imagine Air Canada has a lot of suppliers.
[08:01] We get a lot of those contract.
[08:03] There's expert looking at them one by one.
[08:06] Checking each of those clauses.
[08:09] And they have to analyze it and provide a feedback.
[08:12] Like this clause is not compliant.
[08:13] This clause needs to be rewritten.
[08:15] Like there's a lot of work to do that.
[08:18] It's a very long task and very costly because if we need the if we need some supplier to deliver right now and we don't, then it cause a lot of delay in the in the chain.
[08:29] And if we miss a clause, it's also very costly because at at the end of the day it can backfire to us.
[08:35] So it's a tricky problem.
[08:37] Um and they came to me I mean to the to the data science team to build an agent that can actually do that for them.
[08:46] So can we automatize this task?
[08:49] And obviously there's a lot of documents, a lot of rules, a lot of things to consider when you do that.
[08:56] Uh expert are super busy.
[08:58] You can imagine the price of of a lawyer by hour.
[09:02] And there's no benchmark for that.
[09:03] We cannot go on the web and found documents I found an existing benchmark we can apply.
[09:10] So go back to the same problem as I was in 2024.
[09:15] Uh cycle were super slow and we knew that we want to be there in a in a in in a in a in a shape where we can go fast and try and try and try.
[09:26] Um and I knew that if we if we learn to this situation, we can have a better trajectory in term of accuracy.
[09:34] And that What the challenge for this uh for this project?
[09:38] So, the the problem is creating the answer from documents is a well-known and defined thing in the rags, but for this specific project, there's no guy in the world that say, "Hey, you should use this technique.
[09:49] You should use this technique."
[09:51] So, we try to think how we can reapply the same mechanism from what I learned on rag to this specific project.
[09:59] And we have plenty of project at Air Canada.
[10:01] So, we're using the same concept of getting the answer first and then try the agent to find it back.
[10:08] So, going into the details, uh it's a bit simplified, but actually the um the agent has four steps.
[10:15] Uh we need to spot the clause in the contract.
[10:18] We need to find the relevant documents to support an amendment.
[10:23] Uh we need to write the the amendment.
[10:26] And then at the end, there's a classification system that tells you should we keep it or not uh this amendment.
[10:32] So, the first step, for example, spot is classifying all the section of the contract into specific domain domains.
[10:39] And then found if there's a defect or there's a change to be to be done in this section.
[10:46] So, we did have this trick from the rag.
[10:48] So, we know how to find the documents.
[10:49] So, this is covered.
[10:51] Uh but for the other other section of the um of the agent, we didn't have anything.
[10:59] So, I going to present five different techniques that we use in this project or we try to use in this project.
[11:05] That is trying to address this specific problem.
[11:09] My goal is not to show off.
[11:12] My goal is to get you inspired in how to reapply those techniques in your own uh challenges.
[11:20] So, the first one is to use past decision and answer.
[11:22] So, we we you can retrieve past contract, compare with the one that has been changed.
[11:29] The second one is to choose something you're looking for, write the contract around or generate the contract.
[11:35] The third one is uh take a good contract that has been signed and break it.
[11:41] Um the fourth one is about things that should not change no matter what you do, you should not get more flags.
[11:49] And the fifth one is actually the most powerful.
[11:52] It's very hard.
[11:53] It's to turn an example into many using the concept of templates.
[12:00] So, immediately we jump into this approach.
[12:03] Uh we ask the expert to provide us the original contract, the one that has been signed.
[12:09] We're going to take that, do a diff like you do in the code, compare what has been changed, and the agent should should provide the same exact flags.
[12:17] That would be perfect.
[12:19] However, for many reason, our our team do not keep all the contract.
[12:25] Um they don't keep the original one, they just keep the last one, they don't have all the version of the contracts.
[12:31] It sometimes goes into negotiation, there is a lot of things.
[12:34] So, we could not have any of those pairs of contracts, so this thing doesn't work.
[12:42] Okay, second one.
[12:44] We took a rule.
[12:45] And so, that's changing a bit the topic.
[12:48] Um instead of creating um flags or creating an agent that can provide flags, we start from rules.
[12:56] And the rule is, for example, we have the the cap above $1 million.
[13:01] So, that's every contract that Arcada should follow this rule or many of those in specific domains.
[13:07] So, you know the rule, you create uh a clause without breaking this rule and generate the contract around, and you should found um the rule back the the flag,
[13:19] which is breaking the rule.
[13:20] So, that's simple.
[13:22] It doesn't require a lot of uh of work.
[13:25] A a of things can be generated.
[13:26] The problem is you need to know the rules.
[13:29] So, luckily enough, we could actually get some of those from either the expert or the documentation they had, uh but cannot guarantee the coverage of everything.
[13:39] So, we came up we also with our own rules, etc. So, we generate at the end uh 275 tests that we know can run and we know the answer.
[13:51] So, similar to the rug, we inverse it we inverted the problem, we planted the problem, and we try to find it back.
[13:58] Another one which is a very hard, actually, uh we took an example of a contract that has been signed, we remove a clause, and the agent is supposed to say, "Hey, there's something missing here."
[14:12] Um it's a very hard task because some supplier has different templates, have different kind of content.
[14:20] Knowing what should be in what contract require a bit of thinking.
[14:24] Uh and we mix those with existing contract that should not show any flag.
[14:29] So, that's the most powerful approach what that we had today.
[14:32] Um using this approach, we can generate a lot of contract, and at the end of the day, we found that our agents are missing uh most of the clause that has been removed.
[14:42] So, that's a very very strong benchmark.
[14:46] Okay.
[14:47] Uh fourth technique, um you don't change you change something that should not affect the agent or should should not change anything in the result.
[14:58] For example, the name of the supplier, the address of the supplier, something that should not have any impact on the number of flags.
[15:05] Um you can reorder the clause, so some randomness should also lead to the same results.
[15:13] Uh and you can run the same uh contract again and again.
[15:16] All of those are things that we check to make sure that the the output is consistent over all of those.
[15:23] In this example two, everything can be checked using code.
[15:27] You don't need to do any LLM as a judge.
[15:29] The previous uh speaker showed like it's very hard task.
[15:33] So, we stay away from uh LLM as a judge as much as possible.
[15:38] The fifth one, um it's the concept of template.
[15:44] Our expert have different kind of templates with blank.
[15:47] So, I don't know if you ever write a contract.
[15:50] Those blanks should be filled by the expert.
[15:52] And instead of trying to fill them, we generate um an array for each of those blanks to be filled with.
[16:02] So, that's allow us to generate plenty of contracts.
[16:04] Some tuples break some rules, some tuples doesn't.
[16:07] We know that.
[16:09] And so, we know what actually should be flagged as a as a as a breach or not.
[16:14] Again, this is a code check.
[16:16] We don't need to have any LLM involved.
[16:18] Um What What The good thing is with this one is our expert already use templates, so we can reuse them to generate some contracts.
[16:29] So, the cycle was a bit slower in this case, so 4 weeks to get a a feedback, and we can get a script to run and get the the evaluation back in 5 minutes.
[16:41] So, I just want to highlight one point, and we went through all of those um kind of categories of grading at the at the bottom.
[16:48] Exact match or code that are very fast.
[16:51] As soon as we step into LLM as a judge, uh it's can be very fancy, it can be very useful.
[16:57] It's so hard.
[16:58] Like, we got so much noise from the LLM as a judge compared to the noise with the with the agent that we building.
[17:05] Solving this problem is actually almost as hard as solving the the the right problem.
[17:10] So, finding the right prompt of the right harness for the LLM as a judge is a very hard task.
[17:15] There's some padding you can do.
[17:17] task.
[17:18] You can do fact checking.
[17:20] You can do a bunch of stuff, but actually it require so much work and we could not found the right one for this project.
[17:29] There's plenty of other techniques.
[17:30] I won't go over all of them.
[17:32] For example, trajectory.
[17:33] We have another project that require the agent to eat some hand points.
[17:38] So, trajectory you can you you know what the end point should be in some order.
[17:43] You generate a story about it.
[17:45] You ask the LM to fill to use the agent and look at the trace to to see if the the trajectory is correct.
[17:52] Test to SQL, you might already know it.
[17:55] Doesn't matter what your agent is.
[17:57] If you know how to pick the answer, if you know what you're looking for, and I showed a couple of examples.
[18:04] Generate the input and then try try to find it.
[18:07] Doesn't apply to everything.
[18:09] And sometime we had to re-engineer our solution to fit this paradigm, even if we move away slightly from the original design of the solution.
[18:18] We could with the original design we could not evaluate it.
[18:21] We switched to something that we can actually evaluate because of that.
[18:27] Um one thing to highlight is not the end of the day.
[18:32] You generate the test.
[18:34] You get a better agent.
[18:35] You get some real feedback.
[18:37] So, expert can provide some feedback and we don't use this feedback to fix the agent.
[18:42] We use this feedback to generate better test.
[18:46] Better prompt.
[18:47] Better ideas how to remove clauses, etc. etc. And one loop at a time.
[18:53] Uh that can be done by Rise.
[18:56] There's a lot of tools there.
[18:57] Thanks for the invitation.
[18:59] [snorts]
[19:00] Um and that's my conclusion slide.
[19:02] Why the answer first?
[19:04] Um I think I I explained that.
[19:10] When you generate those tests, it's not enough to know if you can ship the product.
[19:14] Just a starting point.
[19:16] It's a tool to help anticipate the developer.
[19:18] It's not the go no go for the for the solution.
[19:23] So I hope you like this this presentation.
[19:27] I know that the 24 hour chip is the hardest one.
[19:30] If you want to quit releasing agent without evaluation and I would be happy to support you and have your own feedback.
[19:40] Thank you very much.