{ "video": { "id": "31GUkCBD-Uc", "title": "Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber", "duration": 1299, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "31GUkCBD-Uc", "title": "Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber", "one_liner": "Uber's computer-vision team explains the eval and auto-tuning loops behind the multimodal agent that enhances Uber Eats food photos — routing graded by recall against human golden labels, a pass@K QA loop on the image editor, and a config-driven diagnoser that retunes agents in production with no human in the loop.", "summary": "Jai Chopra and Soumya Gupta walk through a production agent pipeline that decides whether an Uber Eats merchant photo should be enhanced, edits it, and gates it before publishing — and argue the interesting engineering is the eval loop, not the agent. Each stage has its own eval: the router is a classifier measured with a confusion matrix and recall as the guardrail; the editing agent runs a QA-feedback loop scored with pass@K on pairwise before/after comparison. Because a static offline model drifts, they sample production data, send it to human labelers under the same objective guideline, and feed mismatches into a diagnoser agent that localizes the fault and triggers an auto-tuning pipeline (reflect + synthesize sub-agents) that rewrites an agent's config, benchmarks it against the golden set, and registers a new version — closed loop, no human in it.", "key_points": [ "Scale framing: Uber Eats runs at ~90 billion run rate per year, adds millions of items to the marketplace every month, grows 20% year-on-year, operates in 10,000 cities, and the delivery marketplace is now as big as Uber's mobility side.", "The constraint is trust, not quality: a meaningful fraction of consumers distrust anything AI-generated, so edits must stay faithful to the original and preserve merchant brand — and 'if we have the same prompt for every photo that we're editing, the diversity of the marketplace is going to collapse.'", "Pipeline: a multimodal image-understanding agent describes the photo into structured output → a router grades it against a pass/fail rubric and decides enhance vs skip → an editing agent loops with a QA agent, self-correcting → a final post-processing 'publish-ready QA' gate → publish to the menu. Everything is logged.", "Logging is designed as one flat JSON per end-to-end orchestration so non-technical and technical staff can both diagnose single cases and roll up aggregates; they treat logging as the thing you build first because without it there is nothing to optimize against.", "Router eval is a classifier confusion matrix (precision/recall), with recall as the guardrail so bad images don't slip through; a more sophisticated router routing to smaller/cheaper/lower-latency models becomes an n-by-n matrix, one cell per branch.", "Concrete router failures: a good cheeseburger wrongly sent for enhancement costs compute for zero quality lift and risks degrading it; a recall miss passed an image showing six chicken wings whose dish name said 'eight pieces', creating the risk the editor hallucinates two extra wings to match the description.", "Drift loop: sample production at a regular cadence → human labelers using the same objective guideline → diff against agent output → an umbrella diagnosis agent localizes the issue and triggers auto-tuning → benchmark against the golden set → ship. Config-driven, no human in the loop, with guardrail observability and quick rollback.", "Auto-tuning uses a prompt-optimizer agent made of two sub-agents: a reflect agent that strips label noise and finds systemic issues in the mismatches, and a synthesize agent that rewrites the target agent's config from that feedback; a passing benchmark registers a new agent version in an agent store that the next production run picks up.", "Enhancement is three steps — generate a per-image prompt from the description plus the router's directives, enhance, then a multi-dimensional QA gate (plating, faithfulness, colors); failures feed back into prompt generation. Outcome is pass@K or a deliberate coverage hit where the image is never enhanced.", "'Better' for the pairwise input-vs-output comparison is defined jointly with product, design, policy and legal — faithful, complete, natural, realistic — and the judge can answer yes, no, or unsure (an unsure, e.g. not being able to count eight wontons, is rejected in production).", "Named failure modes: adding shrimp (faithfulness), removing sauce from sushi (completeness), an agent that over-steers after QA rejection into a generic ceramic bowl — reward hacking producing a 'nugatory change' despite large pixel differences — and frontier image-editing model defects like a plate covering sauce, which leak into the applied use case and get reported back to the frontier teams as object-coherence and physics-plausibility evals.", "The final publish-ready QA gate deliberately duplicates upstream checks: a Swiss cheese model where redundancy is acceptable and the last gate also flags what should have been caught earlier.", "Beyond the model loop there are dogfooding loops (thumbs up/down plus free-form feedback from merchants, design and product teams) feeding the same diagnoser, and a production loop tracking marketplace quality and conversion (add-to-cart, order completion) sliced by geo, device type and dish type so tuning can target specific segments." ], "takeaways": [ "Build the flat, end-to-end trace log before anything else — without it you have nothing to optimize against and no self-learning loop is possible.", "Establish a human-labeled golden dataset with deliberately objective labeling guidelines (to strip subjective bias), sampled across geographies, dish types and quality tiers, and pick one guardrail metric per stage — recall for routing, so nothing bad slips through.", "Don't ship a static offline model and stop: sample production on a cadence, re-label it against the same guideline, and make every agent's prompt/config auto-tunable so it can be rewritten, benchmarked against the golden set, versioned and rolled back without a human.", "Score iterative agent loops with pass@K on pairwise before/after comparison, and accept a coverage hit (never publish) rather than shipping an output that can't clear the gate after K iterations.", "Watch for reward hacking in QA-feedback loops — an agent that over-corrects into a safe, generic output passes the gate while delivering no real value — and let the 'unsure' verdict be a rejection.", "Layer redundant gates on purpose (Swiss cheese), and define 'better' with product, design, policy and legal rather than letting the eval author decide alone." ], "topics": [ "agents", "evals", "multimodal", "llm-as-judge", "image-generation", "human-in-the-loop", "drift", "production-ml" ], "tools": [ "Uber", "Uber Eats" ], "quotes": [ { "text": "If we have the same prompt for every photo that we're editing, the diversity of the marketplace is going to collapse.", "at": "03:07", "url": "https://www.youtube.com/watch?v=31GUkCBD-Uc&t=187s" }, { "text": "You want to start with your logging cuz if you don't start with it, you have nothing to optimize for, let alone set up a self-learning loop.", "at": "07:01", "url": "https://www.youtube.com/watch?v=31GUkCBD-Uc&t=421s" }, { "text": "For routing, our guardrail metric is recall. We don't want any bad image to slip through our system.", "at": "09:37", "url": "https://www.youtube.com/watch?v=31GUkCBD-Uc&t=577s" }, { "text": "You will have one static model with the offline, but this is what is going to keep your system alive.", "at": "12:01", "url": "https://www.youtube.com/watch?v=31GUkCBD-Uc&t=721s" } ], "words": 4380 }, "summary_url": "/#31GUkCBD-Uc", "transcript": { "html": "/transcripts/31GUkCBD-Uc.html", "txt": "/transcripts/31GUkCBD-Uc.txt", "vtt": "/transcripts/31GUkCBD-Uc.vtt" } }