Agent Output Is Not UX: Rendering Layer Your LLM Pipeline Is Missing - Bala Ramdoss, Amazon Lens
https://www.youtube.com/watch?v=maTp79FD9gI
AI Engineer · 14 min · 186 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] I asked my AI assistant to help me reserve a table at a popular restaurant.
[00:06] Here's what it gave me.
[00:09] Now, it's not wrong.
[00:12] The phone number is there.
[00:13] The hours are right.
[00:15] It even knows the walk-in oyster bar at the front.
[00:19] The model did the real work, but look at the outcome.
[00:24] I have to do my research and work towards actually booking the table.
[00:28] Like, we've been here for a while.
[00:30] Now, imagine if you were to build something like this for your customers.
[00:35] What would you want it to do?
[00:39] Same answer, rendered like this instead.
[00:42] A date, a time, a couple of taps, and then you're done.
[00:47] The agentic product you're building already has all the tools to support your customer's needs.
[00:53] The only thing that you need to focus on is the layer between the model and something that a human can interact with.
[01:00] That layer is what I'm going to talk about.
[01:02] The models and agents are here to stay.
[01:05] And we should learn how to make it friendly to the humans.
[01:09] Hello.
[01:10] Welcome to my talk.
[01:11] I'm Balaram Das.
[01:13] I have been building customer-facing apps for over a decade.
[01:17] And the past 6 years I've spent building Amazon Lens.
[01:22] Amazon Lens is our suite of camera features powered by AI.
[01:26] It enables you to shop using images, screenshots, and barcodes to discover visually similar products.
[01:32] If you have Amazon app installed, I encourage you to try.
[01:37] This talk comes from my experience building customer-facing products that are in millions and millions of mobile devices.
[01:45] Um quick dis- disclaimer before I go further.
[01:48] I'm giving this talk on my own.
[01:50] The opinions in this talk are mine and not my employer's.
[01:56] When you think of building UX for an agentic AI, obviously ChatGPT comes into mind.
[02:02] How do you draw these cards?
[02:03] How do you choose carousel versus list?
[02:06] How do you craft this so the human can understand and interact better?
[02:12] When you attempt to build something like this, you run into a wall.
[02:16] There are problems that need to be solved from the very early stages of your system.
[02:22] Is that experience going to be snappy or slow?
[02:26] Do we get all the information at first or one thing at a time?
[02:32] And how do you scale this for mobile apps where versions and device capabilities are fragmented?
[02:38] Like if you notice, none of these problems are due to the model itself.
[02:42] The model does its job well.
[02:44] Um these are delivery problems and they live in between the model output and what's on the screen.
[02:50] That is the layer that decides whether your product succeeds or not.
[02:55] So uh for this delivery problem, if if you had asked me about uh about this layer 2 years ago, I wouldn't have had a name for it.
[03:06] I'd built a couple of AI features by then and every single time I solved this part from scratch.
[03:12] A bespoke pattern shaped around whatever system I was in.
[03:18] Uh there was no shared vocabulary for any of it.
[03:22] What gets me is that this thing now has a name.
[03:25] Generative UI.
[03:26] And there's an open spec for it.
[03:29] ATUI from Google.
[03:31] Instead of the agent handing you a raw text or HTML, it describes the UI as data.
[03:37] A list of components and the client renders them with its own native widgets.
[03:43] A problem I used to solve alone is becoming something teams get to start from.
[03:48] And that's exactly what I'm excited to dig into.
[03:52] Um normal API returns data and the client decides how to draw it.
[03:57] Like your model is good at tool use, coding, and even other complex tasks.
[04:04] It can do UI, too.
[04:05] And that is what the generator UI is about.
[04:08] And it's it's a spectrum.
[04:10] Uh Co-pilot Kit, a company that does this primarily, lays out in three rounds.
[04:17] At the bottom, we have control.
[04:20] The model picks a pre-built component like a product card.
[04:25] It never invents anything.
[04:27] In the middle, um declarative.
[04:30] The model composes UI from a catalog.
[04:34] Uh date field, a time field, or a submit button.
[04:38] Um that's where A2UI sits.
[04:41] And at the top, it's fully open-ended.
[04:44] The model generates a novel UI on the fly like MC PA apps.
[04:50] The higher you go, the more your client has to trust whatever the model hands it.
[04:56] Um most production mobile apps live in the bottom two rounds because, you know, that's where you stay safe.
[05:02] And that's what we'll focus on.
[05:06] Um adding to the complexity of UX building, um mobile apps play an important part.
[05:12] Um on the web, if uh render breaks, you know, you ship a fix and it's live in minutes.
[05:18] Uh mobile apps cannot do that.
[05:20] Like you're looking at hundreds of millions of installs and you don't control uh when any of them update.
[05:28] So, when a client meets a content type it's never seen, it doesn't gracefully degrade, it crashes.
[05:36] And it keeps crashing for days or weeks um in hands of people who haven't updated.
[05:41] So, one rule holds everything that follows for mobile clients.
[05:45] You cannot meaningfully patch the client.
[05:48] Um how does this system look when you zoom out?
[05:53] Um here's the whole simplified pipeline.
[05:56] Um version aware context face that feeds the model.
[06:00] Uh yes, that is context engineering.
[06:03] The model outputs typed UI intent.
[06:06] Uh that flows through a BFF backend for frontend.
[06:10] And to a client render that draws it and uh falls back safely when it cannot.
[06:17] We'll break into three patterns.
[06:19] Pattern one, the rendering contract.
[06:22] Pattern two, streaming.
[06:24] Um that's the flow in between them.
[06:26] Pattern three, uh the BFF that sits in the middle.
[06:30] Uh we'll take them one at a time starting with the contract.
[06:34] Um the contract is all about making the model be aware of what the client capabilities are.
[06:42] You ensure it it stays true to the client that the model is trying to draw the UI for.
[06:48] Um what it ensures is that that the onus is not on the client or the rendering layer to infer what to show by looking at the token output.
[06:59] Um you also maintain a repository of version map with the capabilities.
[07:04] For example, you introduced a new flight uh card UI in version 2.0. Uh make sure to surface it to the model only from version 2.0 onwards.
[07:15] Um when you build the context.
[07:20] The takeaway here is that the model is going to choose the CX and you provide the right information in its con- context.
[07:29] Um let's take a look at this example.
[07:31] And this this is usually the hard part of building the system.
[07:36] This is the model facing half of the contract.
[07:40] And you don't want the model to send back text.
[07:44] Like, you want it to stream blocks of UI components.
[07:48] Um a conversation block for what it says in text and a UI block for what what it wants the client to render.
[07:58] The contract even encodes layout rules.
[08:02] In this example, one to three flights a swipable carousel.
[08:08] Four or more, a vertical list.
[08:11] The model picks the intent.
[08:13] And notice what the model never does.
[08:16] It never invents a component.
[08:18] It chooses from a fixed menu that you provide to it.
[08:22] Obviously, it won't be in the millions.
[08:25] It would be a handful.
[08:27] It turns out that it gets significantly harder as you scale your features to more surfaces.
[08:33] I'll let you think about how to engineer that context to make sure the model succeeds in picking one.
[08:42] So, that's the contract.
[08:44] Now, let's move on to the streaming part.
[08:48] This was eye-opening to me when I first encountered it.
[08:52] Traditionally, apps make an API call and wait for something to happen.
[08:57] When LLMs are involved, this pattern becomes ineffective as often the latency is higher due to the model themselves.
[09:06] And on top of it, there are tons of additional checks to ensure safety and whatnot.
[09:12] And we also introduce a new layer that it's going to be a little bit complex to do the dynamic UI.
[09:20] Um streaming helps here uh by not making the client wait for the whole response.
[09:26] It renders things in chunks at a time.
[09:31] Take a look at the illustrative example when you have to surface information one chunk at a time.
[09:37] First, you show a skeleton, then partially fill it, then complete it.
[09:43] It may take 3 to 4 seconds to get there, but the wait is bearable.
[09:50] That kind of changes what you measure for an app.
[09:53] Like you stop chasing the total latency, which we have done for over a decade.
[09:59] You start chasing time to first chunk.
[10:03] You know, the first useful thing that your user sees.
[10:07] For this reason, the traditional loading spinner won't work for AI features.
[10:13] Sometimes, you have to get creative and design your feature around it.
[10:18] This may not work for all, but here's a product example.
[10:21] And when you know it's going to take time to render something, keep the user engaged.
[10:26] The Lens live allows the user to focus different things, um tap on an object that they're interested in while they wait for the results.
[10:36] Uh if you don't have the full screen in your control, you can show the thinking CX.
[10:42] But please use it sparingly.
[10:43] The overall AI users have moved out of the forgiving phase, and now they expect to know what is happening.
[10:52] Now that you have a way to stream back different states, you know how to show what your agent is doing.
[10:59] Um here's an example I copied from Gemini.
[11:02] When when it starts to work on a task, it gives a glimpse into what the agent is doing.
[11:08] Even though it takes 10 seconds, I'm okay if I know what's happening and be able to trust the agent's final output.
[11:17] We have covered streaming.
[11:20] Let's move on to the final and most important part.
[11:24] Your new BFF.
[11:26] Pattern one was about how UI gets picked.
[11:29] Pattern two was about how the chunks delivered.
[11:33] Pattern three is all about how the chunks become meaningful UI elements.
[11:39] This is a concept of server-driven UI.
[11:42] And it's a spectrum of server control.
[11:44] And the more the server controls, the more you build.
[11:49] The BFF is a subset of it and decides how to render.
[11:53] It owns the platform specific rules like Android versus iOS, that kind of stuff.
[11:59] Uh it also helps the client to make less decisions and draw what's handed to it.
[12:04] That's the whole idea.
[12:08] The BF BFF doesn't just ship layout.
[12:13] It does a little more.
[12:15] In addition to the format and context, it does the hydration and adds actions.
[12:23] Every rendered element carries an action payload to handle what what a tap does, what the deep link hit opens.
[12:30] It could even name the impression metric to log.
[12:34] It carries this conversational context across turns.
[12:38] So, the next response know what came before.
[12:42] And the good thing about this is that you can actually do this to your existing apps.
[12:47] You can reuse the existing CX units you already have.
[12:51] The flight row, the product card, the components your app already shipped in production.
[12:58] You're not building a new agentic look into your app.
[13:02] And that's a good way to build for humans.
[13:05] Same brand, same density, same familiar feel.
[13:09] It looks and feels native.
[13:13] That brings to our takeaways.
[13:18] One, the models are highly capable already.
[13:22] You feed a properly typed, versioned contract so it can pick and choose the right CX.
[13:29] Two, stream into typed components, not text.
[13:34] Let the user know what your agent is doing.
[13:38] Three, let the BFF absorb the model output so the client can stay dumb and safe.
[13:47] No, none of these are about the model.
[13:50] The model was fine.
[13:52] This layer is what ships the product.
[13:56] Bringing back to where we started, your agent output is not the CX.
[14:01] You build on it.
[14:03] Thanks for watching.
[14:04] If you'd like to learn more or connect with me, here's a QR code.