WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=31GUkCBD-Uc

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/31GUkCBD-Uc.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:17.040 --> 00:00:19.760
My name is Jay and I'm here with Sonya.

s3
00:00:19.760 --> 00:00:23.560
We are part of the computer vision team at Aruba.

s4
00:00:23.560 --> 00:00:27.560
We're going to talk to you about a real world production use.

s5
00:00:27.560 --> 00:00:28.480
Oh, my son done.

s6
00:00:28.480 --> 00:00:30.480
Okay.

s7
00:00:30.840 --> 00:00:32.880
Try again.

s8
00:00:32.880 --> 00:00:33.440
Okay.

s9
00:00:33.440 --> 00:00:33.720
Don't worry.

s10
00:00:33.720 --> 00:00:34.400
I'll I'll manage.

s11
00:00:34.400 --> 00:00:36.000
You hear me now?

s12
00:00:36.000 --> 00:00:47.160
Okay, so we're going to talk to you today about a real world production use case and specifically we're going to dive into how we design the e-bows and the e-bow loops.

s13
00:00:47.160 --> 00:00:55.360
So All right, cool.

s14
00:00:55.360 --> 00:01:01.520
So just before we get into the agent design, we're going to talk about a little bit about the use case.

s15
00:01:01.520 --> 00:01:08.920
So our delivery marketplace Uber Eats, we do about 90 billion run rate per year at the moment.

s16
00:01:08.920 --> 00:01:14.120
We were adding millions of items to the marketplace each and every year.

s17
00:01:14.120 --> 00:01:15.960
Sorry, every every month.

s18
00:01:15.960 --> 00:01:21.280
We're growing at 20% year-on-year and and we operate in 10,000 cities globally.

s19
00:01:21.280 --> 00:01:30.040
So not many people actually know this but our delivery marketplace is just as big as the mobility side on Uber today.

s20
00:01:30.320 --> 00:01:35.680
Visual content actually plays a really important role for the user experience.

s21
00:01:35.680 --> 00:01:44.480
So a photo is quite often the first signal that a customer gets that gives them that initial impression about a merchant.

s22
00:01:44.480 --> 00:01:52.360
So a good photo can make the difference between someone scrolling through the feed and actually clicking on an item and adding to the cart.

s23
00:01:52.360 --> 00:01:59.120
And more and more we're seeing different modalities on Uber Eats uh especially video content.

s24
00:01:59.480 --> 00:02:01.240
But this is a problem.

s25
00:02:01.240 --> 00:02:11.200
So, our smaller independent merchants simply just don't have the level of quality for their photos that reflect what the eater is actually going to get.

s26
00:02:11.200 --> 00:02:15.920
And when we speak to our merchants, there are three themes that kind of emerge.

s27
00:02:15.920 --> 00:02:24.480
Lack of time, lack of know-how, and costs cuz these professional um photo shoots actually cost a lot of money.

s28
00:02:24.480 --> 00:02:25.411
And this can be especially

s29
00:02:25.411 --> 00:02:25.600
[snorts]

s30
00:02:25.600 --> 00:02:31.120
problematic if the merchant is updating their menu over time.

s31
00:02:32.320 --> 00:02:37.280
So, this problem is actually pretty challenging to solve for at scale, right?

s32
00:02:37.280 --> 00:02:48.360
Because our consumers, they want authentic, real-looking photos, um but a meaningful fraction of uh consumers actually distrust anything that is AI-generated.

s33
00:02:48.360 --> 00:02:56.280
So, if you open up the Uber Eats app, the last thing that you want is to be scrolling through uh you know, food photography that looks like AI's lock.

s34
00:02:56.280 --> 00:02:57.920
So, we're threading the needle here.

s35
00:02:57.920 --> 00:03:06.360
We need to be able to stay faithful to the original image, preserve the brand of the merchant, and avoid everything looking the same.

s36
00:03:06.360 --> 00:03:14.080
If we have the same prompt for every photo that we're editing, the diversity of the marketplace is going to collapse.

s37
00:03:15.920 --> 00:03:24.120
We also, because we operate globally, we also have this long-tail distribution of different quality that we see it across the marketplace.

s38
00:03:24.120 --> 00:03:25.959
Um so, we've got some examples here.

s39
00:03:25.959 --> 00:03:34.959
You might see food photography that, you know, has poor sharpness, poor composition, not centered, uh or or poor colors as well.

s40
00:03:34.959 --> 00:03:41.320
We also have a wide range of spectrum of user-generated content on the platform as well.

s41
00:03:42.440 --> 00:03:45.080
So, what are our goals when we're designing these agents?

s42
00:03:45.080 --> 00:03:50.200
When you think through these goals, you might actually be thinking through, you know, your own agents that you're building yourself.

s43
00:03:50.200 --> 00:03:54.560
But for us, it's about one, preserving authenticity and trust.

s44
00:03:54.560 --> 00:03:56.840
Two, improving the quality when we need to.

s45
00:03:56.840 --> 00:04:00.200
So, we want to be able to improve quality selectively.

s46
00:04:00.200 --> 00:04:03.760
We want to optimize globally for for the entire marketplace.

s47
00:04:03.760 --> 00:04:07.080
We don't cannibalize certain merchants.

s48
00:04:07.080 --> 00:04:11.280
We want to ship safely, and this is going to be an important theme throughout the talk.

s49
00:04:11.280 --> 00:04:18.320
We want to learn continuously, and we want to operate at scale in a cost-efficient manner.

s50
00:04:18.440 --> 00:04:23.080
So, agents are actually really well-suited to solve this problem.

s51
00:04:23.080 --> 00:04:28.320
So, if you imagine a spectrum, on the one side, you've got something that's more deterministic.

s52
00:04:28.320 --> 00:04:30.040
It's more rules-based.

s53
00:04:30.040 --> 00:04:36.000
Um and uh you you have more control over it, but it's fairly it's a brittle system.

s54
00:04:36.000 --> 00:04:40.200
It's not actually going to be able to scale for the entire marketplace.

s55
00:04:40.200 --> 00:04:46.520
Imagine the other side, you provide an agent with obviously a lot of creativity, it has a lot of agency.

s56
00:04:46.520 --> 00:04:51.960
Um and that's actually what we want to lean into, but we can't leave that unconstrained, right?

s57
00:04:51.960 --> 00:04:56.960
Cuz we have certain safety and certain guardrails in place that we need to adhere to.

s58
00:04:56.960 --> 00:04:59.280
So, we want to find a balancing act.

s59
00:04:59.280 --> 00:05:05.720
Uh and that's kind of set the principle for the way that we think and design around agents and evals.

s60
00:05:05.720 --> 00:05:14.320
So, now we're going to actually like dive a little bit deeper into a simplified but representative example of what we have in production.

s61
00:05:14.320 --> 00:05:21.480
And we're going to go through each stage and how we eval it, and then talk through some continuous learning loops as well.

s62
00:05:21.480 --> 00:05:26.760
So, first up, we have what we call an image understanding and routing agents.

s63
00:05:26.760 --> 00:05:29.240
So, this is where multimodality is is pretty important.

s64
00:05:29.240 --> 00:05:33.280
We actually ask the LLM to describe what it sees in the photo.

s65
00:05:33.280 --> 00:05:39.120
Um and then we we create a structured output from that, and we send it to a router.

s66
00:05:39.120 --> 00:05:42.440
The router will then determine, do we enhance it, or do we skip it?

s67
00:05:42.440 --> 00:05:45.560
We skip it, we will keep the original.

s68
00:05:45.560 --> 00:05:50.640
If we enhance it, we send it to our next agent, which is an image editing agent.

s69
00:05:50.640 --> 00:05:52.840
And this can actually run in a loop.

s70
00:05:52.840 --> 00:05:55.880
So, it gets feedback from a QA agent.

s71
00:05:55.880 --> 00:06:04.760
Um it can edit uh in the in this loop and self-correct and fix things um as it goes.

s72
00:06:04.760 --> 00:06:09.080
If it goes through a number of loops and it still fails, we we don't publish it.

s73
00:06:09.080 --> 00:06:14.200
Then we actually send it to a final post-processing and QA step.

s74
00:06:14.200 --> 00:06:17.680
If that's all good, we'll publish it to the menu.

s75
00:06:17.680 --> 00:06:22.880
And the last thing that's really critical is we log everything.

s76
00:06:22.919 --> 00:06:25.560
Just a quick note about logging.

s77
00:06:25.560 --> 00:06:37.720
Don't know if you can actually read the JSON here, but you might notice that all of the agents in this end-to-end orchestration is within one It's It's basically a flat structure in this JSON.

s78
00:06:37.720 --> 00:06:51.520
Um and so, this is actually incredibly useful for the entire team because anyone, be it non-technical uh technical um folks on engineering product, can actually dive in um and look at specific cases to diagnose

s79
00:06:51.520 --> 00:06:54.600
and also roll up things to look in aggregates.

s80
00:06:54.600 --> 00:07:00.040
Um and it's important to note here that, you know, we think this is important to start with.

s81
00:07:00.040 --> 00:07:07.040
You want to start with your logging cuz if you don't start with it, you have nothing to optimize for, let alone set up a self-learning loop.

s82
00:07:07.040 --> 00:07:11.080
And at Uber, we um we use our eyes.

s83
00:07:11.120 --> 00:07:11.600
Cool.

s84
00:07:11.600 --> 00:07:16.000
We're going to dive um a bit deeper into the router.

s85
00:07:16.919 --> 00:07:18.720
So, the router's actually pretty straightforward.

s86
00:07:18.720 --> 00:07:21.480
If you remember, we you know, we have this multimodality input.

s87
00:07:21.480 --> 00:07:24.760
We look at certain text description metadata, the image itself.

s88
00:07:24.760 --> 00:07:29.160
We ask it to to to to under um describe what it's seeing.

s89
00:07:29.160 --> 00:07:31.160
We create structured output from that.

s90
00:07:31.160 --> 00:07:34.919
With that structured output, we can then grade against a rubric.

s91
00:07:34.919 --> 00:07:37.720
So, we have these pass and fail criteria.

s92
00:07:37.720 --> 00:07:43.120
The last step is we want to decide whether or not we should enhance or skip.

s93
00:07:43.200 --> 00:07:45.280
How do we actually eval this?

s94
00:07:45.280 --> 00:07:48.760
This is you could think of this as a more sort of traditional classifier.

s95
00:07:48.760 --> 00:07:51.280
So, here we we have a confusion matrix.

s96
00:07:51.280 --> 00:07:54.440
You know, many of you are probably pretty familiar with this.

s97
00:07:54.440 --> 00:07:59.920
Um but we can look at things like the true positive cases, the false negative negative cases, and so on and so forth.

s98
00:07:59.920 --> 00:08:03.280
Essentially, what we're doing is we're measuring the precision recall.

s99
00:08:03.280 --> 00:08:07.320
In practice, your routers might actually be much more sophisticated.

s100
00:08:07.320 --> 00:08:17.360
So, for example, we might want to route an image to a lower latency smaller model to be able to save on cost and improve the user experience at the trade-off of quality.

s101
00:08:17.360 --> 00:08:25.160
And if that's the case, instead of having a 2 by 2 matrix for your confusion matrix, you might actually have an n by n matrix.

s102
00:08:25.160 --> 00:08:32.840
Where each grid is actually telling you whether or not you're correctly routing to that specific branch.

s103
00:08:32.840 --> 00:08:40.560
So, I'm going to now hand over to Somya who's going to dive a little bit deeper into how we handle drift and human alignment.

s104
00:08:48.360 --> 00:08:54.839
So, now that we spoke about how we eval the routing, I want to talk about how do you get the first version of the model out.

s105
00:08:54.839 --> 00:08:58.800
For our use case, we consider human labels as the golden source of truth.

s106
00:08:58.800 --> 00:09:01.360
And this is what we want to align our models to.

s107
00:09:01.360 --> 00:09:05.360
The way we do about this is we go collect a dataset which is representative.

s108
00:09:05.360 --> 00:09:09.480
So, you know, different cuts, geographies, dish type, image quality type.

s109
00:09:09.480 --> 00:09:14.240
Send it to our human labelers and give them a very objective guideline to label on.

s110
00:09:14.240 --> 00:09:18.640
This is to remove any subjective biases or any noise coming in from human labelers.

s111
00:09:18.640 --> 00:09:22.480
Once we've got that system set up is when we start tuning our model.

s112
00:09:22.480 --> 00:09:31.640
We take our agent, we go ahead get output from the agent, compare it to your golden dataset, evaluate if it's good enough to ship, if it meets your guardrail metrics, you go ahead and ship it.

s113
00:09:31.640 --> 00:09:36.160
If not, then you go tune and you keep doing this until you meet your guardrail metrics.

s114
00:09:36.160 --> 00:09:38.680
For routing, our guardrail metric is recall.

s115
00:09:38.680 --> 00:09:42.640
We don't want any bad image to slip through our system.

s116
00:09:44.240 --> 00:09:47.080
Here are some examples of the failures we've seen.

s117
00:09:47.080 --> 00:09:50.280
Uh on your left you see a very good image of cheeseburger.

s118
00:09:50.280 --> 00:09:53.280
Uh on the right you notice that the routing agent actually failed this.

s119
00:09:53.280 --> 00:09:57.720
It said the technical is low ball and it will go send this image for enhancement.

s120
00:09:57.720 --> 00:10:00.520
Now there's two challenges when you send this image for enhancement.

s121
00:10:00.520 --> 00:10:04.880
Firstly, you pay the compute cost for a zero quality lift from this image.

s122
00:10:04.880 --> 00:10:11.440
And secondly, uh there is a risk of degrading this image given it's already such a high quality image.

s123
00:10:13.040 --> 00:10:16.120
And on the other end of the spectrum, you have a recall miss.

s124
00:10:16.120 --> 00:10:23.280
So on your left you have an image with six chicken wings and on your right if you notice the dish name, it says eight pieces chicken wings.

s125
00:10:23.280 --> 00:10:25.600
And your routing agent approved this image.

s126
00:10:25.600 --> 00:10:36.760
That means So now there's a risk here if you send up send this image for enhancement and you only see six chicken wings, there's a chance your model's going to hallucinate these two extra wings to to match the description.

s127
00:10:36.760 --> 00:10:43.520
And that's also an that's a the cut we take at our faithfulness metric that Jay earlier showed us.

s128
00:10:43.520 --> 00:10:52.280
So the meta point I'm trying to get here is you've trained your offline model, but there will be long cases where your model is going to continue to fail and the static model will not work in the real system.

s129
00:10:52.280 --> 00:10:57.480
You need a way such that your prompts, agents, system itself is evolving over time.

s130
00:10:57.480 --> 00:11:01.120
And that's what we've done uh for our system as well.

s131
00:11:01.120 --> 00:11:08.640
And I'm talking more from the routing perspective, but every component in our system is able to tune itself uh for any drift online.

s132
00:11:08.640 --> 00:11:16.160
So what we do is we sample production data at regular cadence, uh send this to the human labelers with the same guidelines that we have seen before.

s133
00:11:16.160 --> 00:11:22.720
Once you've got that data, we compare our agents' output with the output we got from the labelers and see if there's a mismatch.

s134
00:11:22.720 --> 00:11:33.280
If there's a mismatch, we have an umbrella diagnosis agent which takes in the feedback, localizes where this issue is happening, and and and triggers our auto-tuning pipeline.

s135
00:11:33.280 --> 00:11:41.160
Once we tune this agent, we go and benchmark it against our golden data set that we saw earlier, and if we pass our golden data set on the metrics that we had designed,

s136
00:11:41.160 --> 00:11:43.080
we go ahead and ship this model.

s137
00:11:43.080 --> 00:11:44.840
Uh if not, then you kind of keep iterating.

s138
00:11:44.840 --> 00:11:48.720
And this happens on a regular basis on production data set.

s139
00:11:48.720 --> 00:11:53.680
Um the beauty of this is this is completely config driven and doesn't require human in the loop.

s140
00:11:53.680 --> 00:11:58.360
Your diagnoser agent can write your config and trigger the auto-tuning pipeline here.

s141
00:11:58.360 --> 00:12:01.000
And this is what will keep your model sharp over time.

s142
00:12:01.000 --> 00:12:06.520
You will have one static model with the offline, but this is what is going to keep your system alive.

s143
00:12:08.720 --> 00:12:11.960
Um so Jay is going to spend more time on the diagnosis side of it.

s144
00:12:11.960 --> 00:12:14.800
What I want to do is zoom into the auto-tuning bit.

s145
00:12:14.800 --> 00:12:19.960
And again, we're looking at routing, but this is how we tune every agent in our system.

s146
00:12:19.960 --> 00:12:25.880
Uh so we start with a target agent, and we've already got these uh unseen eval samples from our humans.

s147
00:12:25.880 --> 00:12:30.120
We go find out the mismatch and matches and call a prompt optimizer agent.

s148
00:12:30.120 --> 00:12:32.120
Now, this itself is two sub-agents.

s149
00:12:32.120 --> 00:12:35.600
There's the reflect agent and the up synthesize agent.

s150
00:12:35.600 --> 00:12:47.920
What reflect does is it it just looks at the mismatches, tries to find remove any noise, find any systemic issues that might be in your data set, and reflect on it and send that feedback to the synthesize agent.

s151
00:12:47.920 --> 00:12:50.320
Now, the synthesize agent takes this feedback.

s152
00:12:50.320 --> 00:12:51.800
It has your agent config.

s153
00:12:51.800 --> 00:12:55.880
It goes and updates your agent with the new config based on the feedback it's getting.

s154
00:12:55.880 --> 00:12:57.560
And goes and benchmarks again.

s155
00:12:57.560 --> 00:13:02.320
If this benchmark is passed, you actually register this new agent in the new agent store.

s156
00:13:02.320 --> 00:13:07.000
And next time your production runs, you pick up the new version of the agent.

s157
00:13:07.000 --> 00:13:10.240
And this is a closed-loop system as I mentioned, no human in the loop.

s158
00:13:10.240 --> 00:13:18.640
We definitely have observability on the guardrails, quick rollback built in in case of any issues with the system itself.

s159
00:13:20.320 --> 00:13:22.920
Moving on to the next step of our orchestration flow.

s160
00:13:22.920 --> 00:13:26.240
So we spoke about routing, moving on to the enhancement bit of it.

s161
00:13:26.240 --> 00:13:28.080
It's a three-step process.

s162
00:13:28.080 --> 00:13:32.080
What we do is the first step, we generate a prompt specific to this image.

s163
00:13:32.080 --> 00:13:38.320
We take in the description, we take in the directives we were getting from our routing agent, and we go ahead and generate a prompt for this image.

s164
00:13:38.320 --> 00:13:40.560
What needs improvement in this image specifically?

s165
00:13:40.560 --> 00:13:42.560
And we go ahead and enhance this image.

s166
00:13:42.560 --> 00:13:49.080
Then you've got the QA gate, which is a multi-dimensional gate, looks at multiple things like plating, faithfulness, colors.

s167
00:13:49.080 --> 00:13:51.960
And if it passes is when you actually go ahead and publish this.

s168
00:13:51.960 --> 00:14:00.240
If it doesn't pass, you take the feedback back from the QA gate, push it back to your generate prompt along with the initial inputs you sent it, and go ahead and enhance it again.

s169
00:14:00.240 --> 00:14:01.640
So, there's two end results here.

s170
00:14:01.640 --> 00:14:09.840
You either keep enhancing for K iterations and you pass your QA gate and you publish, or you take a coverage hit and you never enhance this image.

s171
00:14:11.080 --> 00:14:12.160
Here's an example.

s172
00:14:12.160 --> 00:14:14.800
On your left, you see a bowl of sweet potato fries.

s173
00:14:14.800 --> 00:14:22.160
We send it up for the first iteration and our QA agent rejects it because the portion size is incorrect, the plating is very unrealistic.

s174
00:14:22.160 --> 00:14:27.160
We take that feedback in, go for the second iteration, and we're actually able to pass it the second iteration.

s175
00:14:27.160 --> 00:14:33.480
So, the metric we are measuring here is pass at K. Pass at K is essentially the pass rate at Kth iteration.

s176
00:14:33.480 --> 00:14:40.360
And ideally with the more the iterations, your pass rate will increase because you're getting more feedback in.

s177
00:14:40.520 --> 00:14:44.880
Now, I'll pass it on back to Jay to cover the rest of this.

s178
00:14:47.880 --> 00:14:49.640
Thanks Thanks, Somya.

s179
00:14:49.640 --> 00:15:04.680
Um So, yeah, just before we end here on the um on on the generation of the vowels, we use what's called pairwise comparison, right, for our pass at K. So, it's looking at the input image and the the output image,

s180
00:15:04.680 --> 00:15:07.560
and it's assessing whether or not it's better.

s181
00:15:07.560 --> 00:15:10.120
But how do we actually find what's better?

s182
00:15:10.120 --> 00:15:18.600
So, um we're not going to dive into too much of the details here cuz this is kind of like proprietary stuff, and so we'll just mention it at a higher level

s183
00:15:18.600 --> 00:15:26.480
that this is where you sort of For least for us at Rue Ba, we have to make sure that we're aligning with product, design, policy, legal.

s184
00:15:26.480 --> 00:15:33.160
And this is where we're baking in what we define as a better image on the platform into our Evals.

s185
00:15:33.160 --> 00:15:35.600
Um so, examples here, is it faithful?

s186
00:15:35.600 --> 00:15:36.600
Is it complete?

s187
00:15:36.600 --> 00:15:37.440
Is it natural?

s188
00:15:37.440 --> 00:15:38.360
Is it realistic?

s189
00:15:38.360 --> 00:15:40.040
And there's a bunch of other things as well.

s190
00:15:40.040 --> 00:15:45.120
The output of this is then uh a yes, no, or unsure.

s191
00:15:45.120 --> 00:15:48.160
So, here are some examples of failure modes.

s192
00:15:48.160 --> 00:15:50.600
So, input and output on the right.

s193
00:15:50.600 --> 00:15:53.560
The inputs on the left, outputs on the right-hand side.

s194
00:15:53.560 --> 00:15:56.680
This might be a little bit uh difficult to to see at the first pass.

s195
00:15:56.680 --> 00:15:59.560
We actually added shrimp here and we shouldn't be.

s196
00:15:59.560 --> 00:16:02.680
So, we failed faithfulness.

s197
00:16:03.280 --> 00:16:05.280
This is where we go the other way.

s198
00:16:05.280 --> 00:16:09.480
So, the input um has some source at the bottom of the sushi.

s199
00:16:09.480 --> 00:16:11.280
We actually remove it.

s200
00:16:11.280 --> 00:16:14.440
So, we failed completeness.

s201
00:16:15.160 --> 00:16:22.520
Here's actually a pretty interesting example where the agent actually attempted a more creative edit in the first iteration.

s202
00:16:22.520 --> 00:16:26.040
Um and then the QA said, "Nope, it's not good enough."

s203
00:16:26.040 --> 00:16:29.839
Uh and then it actually oversteers the other other way.

s204
00:16:29.839 --> 00:16:32.360
And it becomes overly conservative.

s205
00:16:32.360 --> 00:16:37.160
Sort of falls back to this generic ceramic plate uh ceramic bowl, sorry.

s206
00:16:37.160 --> 00:16:40.040
So, this is an example of a reward hacking actually.

s207
00:16:40.040 --> 00:16:50.520
And and this is a nugatory change, but something that we don't think is a meaningful or influential change despite the actual raw pixels of the input and output being pretty different.

s208
00:16:50.520 --> 00:16:55.839
Here's another example where in the output the plate is covering the sauce.

s209
00:16:55.839 --> 00:17:06.880
This is an example where for some some of the frontier models that we're using for the actual image editing, some of their um some of their problems will actually sort of leak up into our applied use case.

s210
00:17:06.880 --> 00:17:17.880
Um and so, so object coherence and physics plausibility of the Evals that sometimes will coordinate with the frontier teams and and let them know about these problems and work together with them.

s211
00:17:18.120 --> 00:17:22.240
Here's uh an example of why multimodality is is pretty important.

s212
00:17:22.240 --> 00:17:28.840
In the input and the output, we we can't actually see that there are eight pieces here of of the wontons.

s213
00:17:28.840 --> 00:17:30.440
So, we're not confident, actually.

s214
00:17:30.440 --> 00:17:31.240
We're not sure.

s215
00:17:31.240 --> 00:17:37.800
And so, this is an example where we would actually reject it in production and and it wouldn't it wouldn't go through.

s216
00:17:39.120 --> 00:17:46.880
So, the last step after all of that is a post-processing and what we refer to as the publish-ready QA.

s217
00:17:46.880 --> 00:17:52.640
This is the final gate before we decide we want to publish something to production.

s218
00:17:53.680 --> 00:17:56.120
Here, we do some policy checks.

s219
00:17:56.120 --> 00:17:58.760
We also do some more quality checks.

s220
00:17:58.760 --> 00:18:01.720
Um and you might be wondering, like, we've already done some QA.

s221
00:18:01.720 --> 00:18:04.640
Like, why are we going to do another step of QA?

s222
00:18:04.640 --> 00:18:09.320
The reason is because we think of this like a Swiss cheese model.

s223
00:18:09.320 --> 00:18:15.440
So, we want to try and optimize for reducing the chance of a failure getting into production.

s224
00:18:15.440 --> 00:18:18.400
And so, there is some redundancy here or there.

s225
00:18:18.400 --> 00:18:20.280
And that's okay.

s226
00:18:20.280 --> 00:18:24.000
Um and so, this QA gate is is a little bit more holistic.

s227
00:18:24.000 --> 00:18:25.600
It captures more things.

s228
00:18:25.600 --> 00:18:31.480
But, it also will will try and flag things that we should have caught upstream, as well.

s229
00:18:32.960 --> 00:18:33.480
All right.

s230
00:18:33.480 --> 00:18:37.920
So, we've talked about a couple of uh feedback loops here.

s231
00:18:37.920 --> 00:18:44.240
So, to summarize, we talked about predominantly this first one here, which is the model loop.

s232
00:18:44.240 --> 00:18:52.480
And this is accounting for drifts and aligning with human labeled data set that we have and we've established offline.

s233
00:18:52.480 --> 00:18:55.000
But, we actually have more feedback loops.

s234
00:18:55.000 --> 00:19:00.360
So, we we have that Uber what we we have is a is a great sort of dog dog feeding culture.

s235
00:19:00.360 --> 00:19:03.240
Um we will test apps before they go live.

s236
00:19:03.240 --> 00:19:11.200
Um but, we also have when it goes live in production, how do we get that feedback back into our agent to be able to steer it appropriately?

s237
00:19:11.200 --> 00:19:17.240
So, as we're adding more of these feedback loops, we want to be able to generalize the system.

s238
00:19:17.240 --> 00:19:24.040
So, this is where we've actually created um a higher level of abstraction on top, which we call the diagnoser.

s239
00:19:24.040 --> 00:19:29.840
So, the diagnoser can take in any input from these different feedback loops that we're we're capturing.

s240
00:19:29.840 --> 00:19:40.440
It can reflect on what actual agent within the overall system needs to be optimized, and it can route that agent to be able to fix that configuration specifically.

s241
00:19:40.440 --> 00:19:44.440
It could be one agent, it could be multiple agents.

s242
00:19:45.320 --> 00:19:47.960
So, here's an example of internal dog fooding.

s243
00:19:47.960 --> 00:19:52.600
You might see these in sort of different apps that you've got where you got the thumbs down and the thumbs up.

s244
00:19:52.600 --> 00:19:55.720
We also take some free form feedback as well.

s245
00:19:55.720 --> 00:19:59.400
Uh and this is actually great cuz we'll get feedback from merchants directly.

s246
00:19:59.400 --> 00:20:03.960
We'll get feedback from, you know, design teams, other product teams uh at Uber.

s247
00:20:03.960 --> 00:20:11.160
And we'll incorporate that feedback back into our diagnosis step and tune the system over time.

s248
00:20:11.160 --> 00:20:13.960
Again, similar sort of workflow pattern here.

s249
00:20:13.960 --> 00:20:26.920
We'll replay the examples that we know are those ones that have been flagged, be it good examples, be it bad examples, uh and then we'll benchmark the metrics before we push the latest config version.

s250
00:20:27.480 --> 00:20:31.600
The last step is is actually getting this into production.

s251
00:20:31.600 --> 00:20:38.080
And and this is where we're looking for a whole heap of different metrics we track for for the marketplace quality and health.

s252
00:20:38.080 --> 00:20:40.480
Uh I've just called out one here, which is conversion.

s253
00:20:40.480 --> 00:20:46.520
So, we're looking for improvements in people adding to cart, converting, completing their orders.

s254
00:20:46.520 --> 00:20:55.680
Um I think this one's actually an interesting one to call out because now at I mean, at least at Uber, but especially in production um settings at scale, you have a wide

s255
00:20:55.680 --> 00:21:00.840
um uh you have a lot of data that you can actually slice and dice.

s256
00:21:00.840 --> 00:21:10.240
So, in this area as opposed to the others, what we can do is sort of slice by geos, by device type, by dish type, etc. And we can look at where things are improving

s257
00:21:10.240 --> 00:21:15.680
in different segments and actually tune on certain segments as well.

s258
00:21:17.080 --> 00:21:19.320
Cool, and that's it for our presentation.

s259
00:21:19.320 --> 00:21:20.539
Appreciate it.

s260
00:21:20.539 --> 00:21:22.539
[applause]

s261
00:21:36.259 --> 00:21:38.259
[music]