Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber

AI Engineer · 21 min · 261 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:17My name is Jay and I'm here with Sonya.
00:19We are part of the computer vision team at Aruba.
00:23We're going to talk to you about a real world production use.
00:27Oh, my son done.
00:28Okay.
00:30Try again.
00:32Okay.
00:33Don't worry.
00:33I'll I'll manage.
00:34You hear me now?
00:36Okay, so we're going to talk to you today about a real world production use case and specifically we're going to dive into how we design the e-bows and the e-bow loops.
00:47So All right, cool.
00:55So just before we get into the agent design, we're going to talk about a little bit about the use case.
01:01So our delivery marketplace Uber Eats, we do about 90 billion run rate per year at the moment.
01:08We were adding millions of items to the marketplace each and every year.
01:14Sorry, every every month.
01:15We're growing at 20% year-on-year and and we operate in 10,000 cities globally.
01:21So not many people actually know this but our delivery marketplace is just as big as the mobility side on Uber today.
01:30Visual content actually plays a really important role for the user experience.
01:35So a photo is quite often the first signal that a customer gets that gives them that initial impression about a merchant.
01:44So a good photo can make the difference between someone scrolling through the feed and actually clicking on an item and adding to the cart.
01:52And more and more we're seeing different modalities on Uber Eats uh especially video content.
01:59But this is a problem.
02:01So, our smaller independent merchants simply just don't have the level of quality for their photos that reflect what the eater is actually going to get.
02:11And when we speak to our merchants, there are three themes that kind of emerge.
02:15Lack of time, lack of know-how, and costs cuz these professional um photo shoots actually cost a lot of money.
02:24And this can be especially
02:25[snorts]
02:25problematic if the merchant is updating their menu over time.
02:32So, this problem is actually pretty challenging to solve for at scale, right?
02:37Because our consumers, they want authentic, real-looking photos, um but a meaningful fraction of uh consumers actually distrust anything that is AI-generated.
02:48So, if you open up the Uber Eats app, the last thing that you want is to be scrolling through uh you know, food photography that looks like AI's lock.
02:56So, we're threading the needle here.
02:57We need to be able to stay faithful to the original image, preserve the brand of the merchant, and avoid everything looking the same.
03:06If we have the same prompt for every photo that we're editing, the diversity of the marketplace is going to collapse.
03:15We also, because we operate globally, we also have this long-tail distribution of different quality that we see it across the marketplace.
03:24Um so, we've got some examples here.
03:25You might see food photography that, you know, has poor sharpness, poor composition, not centered, uh or or poor colors as well.
03:34We also have a wide range of spectrum of user-generated content on the platform as well.
03:42So, what are our goals when we're designing these agents?
03:45When you think through these goals, you might actually be thinking through, you know, your own agents that you're building yourself.
03:50But for us, it's about one, preserving authenticity and trust.
03:54Two, improving the quality when we need to.
03:56So, we want to be able to improve quality selectively.
04:00We want to optimize globally for for the entire marketplace.
04:03We don't cannibalize certain merchants.
04:07We want to ship safely, and this is going to be an important theme throughout the talk.
04:11We want to learn continuously, and we want to operate at scale in a cost-efficient manner.
04:18So, agents are actually really well-suited to solve this problem.
04:23So, if you imagine a spectrum, on the one side, you've got something that's more deterministic.
04:28It's more rules-based.
04:30Um and uh you you have more control over it, but it's fairly it's a brittle system.
04:36It's not actually going to be able to scale for the entire marketplace.
04:40Imagine the other side, you provide an agent with obviously a lot of creativity, it has a lot of agency.
04:46Um and that's actually what we want to lean into, but we can't leave that unconstrained, right?
04:51Cuz we have certain safety and certain guardrails in place that we need to adhere to.
04:56So, we want to find a balancing act.
04:59Uh and that's kind of set the principle for the way that we think and design around agents and evals.
05:05So, now we're going to actually like dive a little bit deeper into a simplified but representative example of what we have in production.
05:14And we're going to go through each stage and how we eval it, and then talk through some continuous learning loops as well.
05:21So, first up, we have what we call an image understanding and routing agents.
05:26So, this is where multimodality is is pretty important.
05:29We actually ask the LLM to describe what it sees in the photo.
05:33Um and then we we create a structured output from that, and we send it to a router.
05:39The router will then determine, do we enhance it, or do we skip it?
05:42We skip it, we will keep the original.
05:45If we enhance it, we send it to our next agent, which is an image editing agent.
05:50And this can actually run in a loop.
05:52So, it gets feedback from a QA agent.
05:55Um it can edit uh in the in this loop and self-correct and fix things um as it goes.
06:04If it goes through a number of loops and it still fails, we we don't publish it.
06:09Then we actually send it to a final post-processing and QA step.
06:14If that's all good, we'll publish it to the menu.
06:17And the last thing that's really critical is we log everything.
06:22Just a quick note about logging.
06:25Don't know if you can actually read the JSON here, but you might notice that all of the agents in this end-to-end orchestration is within one It's It's basically a flat structure in this JSON.
06:37Um and so, this is actually incredibly useful for the entire team because anyone, be it non-technical uh technical um folks on engineering product, can actually dive in um and look at specific cases to diagnose
06:51and also roll up things to look in aggregates.
06:54Um and it's important to note here that, you know, we think this is important to start with.
07:00You want to start with your logging cuz if you don't start with it, you have nothing to optimize for, let alone set up a self-learning loop.
07:07And at Uber, we um we use our eyes.
07:11Cool.
07:11We're going to dive um a bit deeper into the router.
07:16So, the router's actually pretty straightforward.
07:18If you remember, we you know, we have this multimodality input.
07:21We look at certain text description metadata, the image itself.
07:24We ask it to to to to under um describe what it's seeing.
07:29We create structured output from that.
07:31With that structured output, we can then grade against a rubric.
07:34So, we have these pass and fail criteria.
07:37The last step is we want to decide whether or not we should enhance or skip.
07:43How do we actually eval this?
07:45This is you could think of this as a more sort of traditional classifier.
07:48So, here we we have a confusion matrix.
07:51You know, many of you are probably pretty familiar with this.
07:54Um but we can look at things like the true positive cases, the false negative negative cases, and so on and so forth.
07:59Essentially, what we're doing is we're measuring the precision recall.
08:03In practice, your routers might actually be much more sophisticated.
08:07So, for example, we might want to route an image to a lower latency smaller model to be able to save on cost and improve the user experience at the trade-off of quality.
08:17And if that's the case, instead of having a 2 by 2 matrix for your confusion matrix, you might actually have an n by n matrix.
08:25Where each grid is actually telling you whether or not you're correctly routing to that specific branch.
08:32So, I'm going to now hand over to Somya who's going to dive a little bit deeper into how we handle drift and human alignment.
08:48So, now that we spoke about how we eval the routing, I want to talk about how do you get the first version of the model out.
08:54For our use case, we consider human labels as the golden source of truth.
08:58And this is what we want to align our models to.
09:01The way we do about this is we go collect a dataset which is representative.
09:05So, you know, different cuts, geographies, dish type, image quality type.
09:09Send it to our human labelers and give them a very objective guideline to label on.
09:14This is to remove any subjective biases or any noise coming in from human labelers.
09:18Once we've got that system set up is when we start tuning our model.
09:22We take our agent, we go ahead get output from the agent, compare it to your golden dataset, evaluate if it's good enough to ship, if it meets your guardrail metrics, you go ahead and ship it.
09:31If not, then you go tune and you keep doing this until you meet your guardrail metrics.
09:36For routing, our guardrail metric is recall.
09:38We don't want any bad image to slip through our system.
09:44Here are some examples of the failures we've seen.
09:47Uh on your left you see a very good image of cheeseburger.
09:50Uh on the right you notice that the routing agent actually failed this.
09:53It said the technical is low ball and it will go send this image for enhancement.
09:57Now there's two challenges when you send this image for enhancement.
10:00Firstly, you pay the compute cost for a zero quality lift from this image.
10:04And secondly, uh there is a risk of degrading this image given it's already such a high quality image.
10:13And on the other end of the spectrum, you have a recall miss.
10:16So on your left you have an image with six chicken wings and on your right if you notice the dish name, it says eight pieces chicken wings.
10:23And your routing agent approved this image.
10:25That means So now there's a risk here if you send up send this image for enhancement and you only see six chicken wings, there's a chance your model's going to hallucinate these two extra wings to to match the description.
10:36And that's also an that's a the cut we take at our faithfulness metric that Jay earlier showed us.
10:43So the meta point I'm trying to get here is you've trained your offline model, but there will be long cases where your model is going to continue to fail and the static model will not work in the real system.
10:52You need a way such that your prompts, agents, system itself is evolving over time.
10:57And that's what we've done uh for our system as well.
11:01And I'm talking more from the routing perspective, but every component in our system is able to tune itself uh for any drift online.
11:08So what we do is we sample production data at regular cadence, uh send this to the human labelers with the same guidelines that we have seen before.
11:16Once you've got that data, we compare our agents' output with the output we got from the labelers and see if there's a mismatch.
11:22If there's a mismatch, we have an umbrella diagnosis agent which takes in the feedback, localizes where this issue is happening, and and and triggers our auto-tuning pipeline.
11:33Once we tune this agent, we go and benchmark it against our golden data set that we saw earlier, and if we pass our golden data set on the metrics that we had designed,
11:41we go ahead and ship this model.
11:43Uh if not, then you kind of keep iterating.
11:44And this happens on a regular basis on production data set.
11:48Um the beauty of this is this is completely config driven and doesn't require human in the loop.
11:53Your diagnoser agent can write your config and trigger the auto-tuning pipeline here.
11:58And this is what will keep your model sharp over time.
12:01You will have one static model with the offline, but this is what is going to keep your system alive.
12:08Um so Jay is going to spend more time on the diagnosis side of it.
12:11What I want to do is zoom into the auto-tuning bit.
12:14And again, we're looking at routing, but this is how we tune every agent in our system.
12:19Uh so we start with a target agent, and we've already got these uh unseen eval samples from our humans.
12:25We go find out the mismatch and matches and call a prompt optimizer agent.
12:30Now, this itself is two sub-agents.
12:32There's the reflect agent and the up synthesize agent.
12:35What reflect does is it it just looks at the mismatches, tries to find remove any noise, find any systemic issues that might be in your data set, and reflect on it and send that feedback to the synthesize agent.
12:47Now, the synthesize agent takes this feedback.
12:50It has your agent config.
12:51It goes and updates your agent with the new config based on the feedback it's getting.
12:55And goes and benchmarks again.
12:57If this benchmark is passed, you actually register this new agent in the new agent store.
13:02And next time your production runs, you pick up the new version of the agent.
13:07And this is a closed-loop system as I mentioned, no human in the loop.
13:10We definitely have observability on the guardrails, quick rollback built in in case of any issues with the system itself.
13:20Moving on to the next step of our orchestration flow.
13:22So we spoke about routing, moving on to the enhancement bit of it.
13:26It's a three-step process.
13:28What we do is the first step, we generate a prompt specific to this image.
13:32We take in the description, we take in the directives we were getting from our routing agent, and we go ahead and generate a prompt for this image.
13:38What needs improvement in this image specifically?
13:40And we go ahead and enhance this image.
13:42Then you've got the QA gate, which is a multi-dimensional gate, looks at multiple things like plating, faithfulness, colors.
13:49And if it passes is when you actually go ahead and publish this.
13:51If it doesn't pass, you take the feedback back from the QA gate, push it back to your generate prompt along with the initial inputs you sent it, and go ahead and enhance it again.
14:00So, there's two end results here.
14:01You either keep enhancing for K iterations and you pass your QA gate and you publish, or you take a coverage hit and you never enhance this image.
14:11Here's an example.
14:12On your left, you see a bowl of sweet potato fries.
14:14We send it up for the first iteration and our QA agent rejects it because the portion size is incorrect, the plating is very unrealistic.
14:22We take that feedback in, go for the second iteration, and we're actually able to pass it the second iteration.
14:27So, the metric we are measuring here is pass at K. Pass at K is essentially the pass rate at Kth iteration.
14:33And ideally with the more the iterations, your pass rate will increase because you're getting more feedback in.
14:40Now, I'll pass it on back to Jay to cover the rest of this.
14:47Thanks Thanks, Somya.
14:49Um So, yeah, just before we end here on the um on on the generation of the vowels, we use what's called pairwise comparison, right, for our pass at K. So, it's looking at the input image and the the output image,
15:04and it's assessing whether or not it's better.
15:07But how do we actually find what's better?
15:10So, um we're not going to dive into too much of the details here cuz this is kind of like proprietary stuff, and so we'll just mention it at a higher level
15:18that this is where you sort of For least for us at Rue Ba, we have to make sure that we're aligning with product, design, policy, legal.
15:26And this is where we're baking in what we define as a better image on the platform into our Evals.
15:33Um so, examples here, is it faithful?
15:35Is it complete?
15:36Is it natural?
15:37Is it realistic?
15:38And there's a bunch of other things as well.
15:40The output of this is then uh a yes, no, or unsure.
15:45So, here are some examples of failure modes.
15:48So, input and output on the right.
15:50The inputs on the left, outputs on the right-hand side.
15:53This might be a little bit uh difficult to to see at the first pass.
15:56We actually added shrimp here and we shouldn't be.
15:59So, we failed faithfulness.
16:03This is where we go the other way.
16:05So, the input um has some source at the bottom of the sushi.
16:09We actually remove it.
16:11So, we failed completeness.
16:15Here's actually a pretty interesting example where the agent actually attempted a more creative edit in the first iteration.
16:22Um and then the QA said, "Nope, it's not good enough."
16:26Uh and then it actually oversteers the other other way.
16:29And it becomes overly conservative.
16:32Sort of falls back to this generic ceramic plate uh ceramic bowl, sorry.
16:37So, this is an example of a reward hacking actually.
16:40And and this is a nugatory change, but something that we don't think is a meaningful or influential change despite the actual raw pixels of the input and output being pretty different.
16:50Here's another example where in the output the plate is covering the sauce.
16:55This is an example where for some some of the frontier models that we're using for the actual image editing, some of their um some of their problems will actually sort of leak up into our applied use case.
17:06Um and so, so object coherence and physics plausibility of the Evals that sometimes will coordinate with the frontier teams and and let them know about these problems and work together with them.
17:18Here's uh an example of why multimodality is is pretty important.
17:22In the input and the output, we we can't actually see that there are eight pieces here of of the wontons.
17:28So, we're not confident, actually.
17:30We're not sure.
17:31And so, this is an example where we would actually reject it in production and and it wouldn't it wouldn't go through.
17:39So, the last step after all of that is a post-processing and what we refer to as the publish-ready QA.
17:46This is the final gate before we decide we want to publish something to production.
17:53Here, we do some policy checks.
17:56We also do some more quality checks.
17:58Um and you might be wondering, like, we've already done some QA.
18:01Like, why are we going to do another step of QA?
18:04The reason is because we think of this like a Swiss cheese model.
18:09So, we want to try and optimize for reducing the chance of a failure getting into production.
18:15And so, there is some redundancy here or there.
18:18And that's okay.
18:20Um and so, this QA gate is is a little bit more holistic.
18:24It captures more things.
18:25But, it also will will try and flag things that we should have caught upstream, as well.
18:32All right.
18:33So, we've talked about a couple of uh feedback loops here.
18:37So, to summarize, we talked about predominantly this first one here, which is the model loop.
18:44And this is accounting for drifts and aligning with human labeled data set that we have and we've established offline.
18:52But, we actually have more feedback loops.
18:55So, we we have that Uber what we we have is a is a great sort of dog dog feeding culture.
19:00Um we will test apps before they go live.
19:03Um but, we also have when it goes live in production, how do we get that feedback back into our agent to be able to steer it appropriately?
19:11So, as we're adding more of these feedback loops, we want to be able to generalize the system.
19:17So, this is where we've actually created um a higher level of abstraction on top, which we call the diagnoser.
19:24So, the diagnoser can take in any input from these different feedback loops that we're we're capturing.
19:29It can reflect on what actual agent within the overall system needs to be optimized, and it can route that agent to be able to fix that configuration specifically.
19:40It could be one agent, it could be multiple agents.
19:45So, here's an example of internal dog fooding.
19:47You might see these in sort of different apps that you've got where you got the thumbs down and the thumbs up.
19:52We also take some free form feedback as well.
19:55Uh and this is actually great cuz we'll get feedback from merchants directly.
19:59We'll get feedback from, you know, design teams, other product teams uh at Uber.
20:03And we'll incorporate that feedback back into our diagnosis step and tune the system over time.
20:11Again, similar sort of workflow pattern here.
20:13We'll replay the examples that we know are those ones that have been flagged, be it good examples, be it bad examples, uh and then we'll benchmark the metrics before we push the latest config version.
20:27The last step is is actually getting this into production.
20:31And and this is where we're looking for a whole heap of different metrics we track for for the marketplace quality and health.
20:38Uh I've just called out one here, which is conversion.
20:40So, we're looking for improvements in people adding to cart, converting, completing their orders.
20:46Um I think this one's actually an interesting one to call out because now at I mean, at least at Uber, but especially in production um settings at scale, you have a wide
20:55um uh you have a lot of data that you can actually slice and dice.
21:00So, in this area as opposed to the others, what we can do is sort of slice by geos, by device type, by dish type, etc. And we can look at where things are improving
21:10in different segments and actually tune on certain segments as well.
21:17Cool, and that's it for our presentation.
21:19Appreciate it.
21:20[applause]
21:36[music]