Agent Output Is Not UX: Rendering Layer Your LLM Pipeline Is Missing - Bala Ramdoss, Amazon Lens

AI Engineer · 14 min · 186 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01I asked my AI assistant to help me reserve a table at a popular restaurant.
2. 00:06Here's what it gave me.
3. 00:09Now, it's not wrong.
4. 00:12The phone number is there.
5. 00:13The hours are right.
6. 00:15It even knows the walk-in oyster bar at the front.
7. 00:19The model did the real work, but look at the outcome.
8. 00:24I have to do my research and work towards actually booking the table.
9. 00:28Like, we've been here for a while.
10. 00:30Now, imagine if you were to build something like this for your customers.
11. 00:35What would you want it to do?
12. 00:39Same answer, rendered like this instead.
13. 00:42A date, a time, a couple of taps, and then you're done.
14. 00:47The agentic product you're building already has all the tools to support your customer's needs.
15. 00:53The only thing that you need to focus on is the layer between the model and something that a human can interact with.
16. 01:00That layer is what I'm going to talk about.
17. 01:02The models and agents are here to stay.
18. 01:05And we should learn how to make it friendly to the humans.
19. 01:09Hello.
20. 01:10Welcome to my talk.
21. 01:11I'm Balaram Das.
22. 01:13I have been building customer-facing apps for over a decade.
23. 01:17And the past 6 years I've spent building Amazon Lens.
24. 01:22Amazon Lens is our suite of camera features powered by AI.
25. 01:26It enables you to shop using images, screenshots, and barcodes to discover visually similar products.
26. 01:32If you have Amazon app installed, I encourage you to try.
27. 01:37This talk comes from my experience building customer-facing products that are in millions and millions of mobile devices.
28. 01:45Um quick dis- disclaimer before I go further.
29. 01:48I'm giving this talk on my own.
30. 01:50The opinions in this talk are mine and not my employer's.
31. 01:56When you think of building UX for an agentic AI, obviously ChatGPT comes into mind.
32. 02:02How do you draw these cards?
33. 02:03How do you choose carousel versus list?
34. 02:06How do you craft this so the human can understand and interact better?
35. 02:12When you attempt to build something like this, you run into a wall.
36. 02:16There are problems that need to be solved from the very early stages of your system.
37. 02:22Is that experience going to be snappy or slow?
38. 02:26Do we get all the information at first or one thing at a time?
39. 02:32And how do you scale this for mobile apps where versions and device capabilities are fragmented?
40. 02:38Like if you notice, none of these problems are due to the model itself.
41. 02:42The model does its job well.
42. 02:44Um these are delivery problems and they live in between the model output and what's on the screen.
43. 02:50That is the layer that decides whether your product succeeds or not.
44. 02:55So uh for this delivery problem, if if you had asked me about uh about this layer 2 years ago, I wouldn't have had a name for it.
45. 03:06I'd built a couple of AI features by then and every single time I solved this part from scratch.
46. 03:12A bespoke pattern shaped around whatever system I was in.
47. 03:18Uh there was no shared vocabulary for any of it.
48. 03:22What gets me is that this thing now has a name.
49. 03:25Generative UI.
50. 03:26And there's an open spec for it.
51. 03:29ATUI from Google.
52. 03:31Instead of the agent handing you a raw text or HTML, it describes the UI as data.
53. 03:37A list of components and the client renders them with its own native widgets.
54. 03:43A problem I used to solve alone is becoming something teams get to start from.
55. 03:48And that's exactly what I'm excited to dig into.
56. 03:52Um normal API returns data and the client decides how to draw it.
57. 03:57Like your model is good at tool use, coding, and even other complex tasks.
58. 04:04It can do UI, too.
59. 04:05And that is what the generator UI is about.
60. 04:08And it's it's a spectrum.
61. 04:10Uh Co-pilot Kit, a company that does this primarily, lays out in three rounds.
62. 04:17At the bottom, we have control.
63. 04:20The model picks a pre-built component like a product card.
64. 04:25It never invents anything.
65. 04:27In the middle, um declarative.
66. 04:30The model composes UI from a catalog.
67. 04:34Uh date field, a time field, or a submit button.
68. 04:38Um that's where A2UI sits.
69. 04:41And at the top, it's fully open-ended.
70. 04:44The model generates a novel UI on the fly like MC PA apps.
71. 04:50The higher you go, the more your client has to trust whatever the model hands it.
72. 04:56Um most production mobile apps live in the bottom two rounds because, you know, that's where you stay safe.
73. 05:02And that's what we'll focus on.
74. 05:06Um adding to the complexity of UX building, um mobile apps play an important part.
75. 05:12Um on the web, if uh render breaks, you know, you ship a fix and it's live in minutes.
76. 05:18Uh mobile apps cannot do that.
77. 05:20Like you're looking at hundreds of millions of installs and you don't control uh when any of them update.
78. 05:28So, when a client meets a content type it's never seen, it doesn't gracefully degrade, it crashes.
79. 05:36And it keeps crashing for days or weeks um in hands of people who haven't updated.
80. 05:41So, one rule holds everything that follows for mobile clients.
81. 05:45You cannot meaningfully patch the client.
82. 05:48Um how does this system look when you zoom out?
83. 05:53Um here's the whole simplified pipeline.
84. 05:56Um version aware context face that feeds the model.
85. 06:00Uh yes, that is context engineering.
86. 06:03The model outputs typed UI intent.
87. 06:06Uh that flows through a BFF backend for frontend.
88. 06:10And to a client render that draws it and uh falls back safely when it cannot.
89. 06:17We'll break into three patterns.
90. 06:19Pattern one, the rendering contract.
91. 06:22Pattern two, streaming.
92. 06:24Um that's the flow in between them.
93. 06:26Pattern three, uh the BFF that sits in the middle.
94. 06:30Uh we'll take them one at a time starting with the contract.
95. 06:34Um the contract is all about making the model be aware of what the client capabilities are.
96. 06:42You ensure it it stays true to the client that the model is trying to draw the UI for.
97. 06:48Um what it ensures is that that the onus is not on the client or the rendering layer to infer what to show by looking at the token output.
98. 06:59Um you also maintain a repository of version map with the capabilities.
99. 07:04For example, you introduced a new flight uh card UI in version 2.0. Uh make sure to surface it to the model only from version 2.0 onwards.
100. 07:15Um when you build the context.
101. 07:20The takeaway here is that the model is going to choose the CX and you provide the right information in its con- context.
102. 07:29Um let's take a look at this example.
103. 07:31And this this is usually the hard part of building the system.
104. 07:36This is the model facing half of the contract.
105. 07:40And you don't want the model to send back text.
106. 07:44Like, you want it to stream blocks of UI components.
107. 07:48Um a conversation block for what it says in text and a UI block for what what it wants the client to render.
108. 07:58The contract even encodes layout rules.
109. 08:02In this example, one to three flights a swipable carousel.
110. 08:08Four or more, a vertical list.
111. 08:11The model picks the intent.
112. 08:13And notice what the model never does.
113. 08:16It never invents a component.
114. 08:18It chooses from a fixed menu that you provide to it.
115. 08:22Obviously, it won't be in the millions.
116. 08:25It would be a handful.
117. 08:27It turns out that it gets significantly harder as you scale your features to more surfaces.
118. 08:33I'll let you think about how to engineer that context to make sure the model succeeds in picking one.
119. 08:42So, that's the contract.
120. 08:44Now, let's move on to the streaming part.
121. 08:48This was eye-opening to me when I first encountered it.
122. 08:52Traditionally, apps make an API call and wait for something to happen.
123. 08:57When LLMs are involved, this pattern becomes ineffective as often the latency is higher due to the model themselves.
124. 09:06And on top of it, there are tons of additional checks to ensure safety and whatnot.
125. 09:12And we also introduce a new layer that it's going to be a little bit complex to do the dynamic UI.
126. 09:20Um streaming helps here uh by not making the client wait for the whole response.
127. 09:26It renders things in chunks at a time.
128. 09:31Take a look at the illustrative example when you have to surface information one chunk at a time.
129. 09:37First, you show a skeleton, then partially fill it, then complete it.
130. 09:43It may take 3 to 4 seconds to get there, but the wait is bearable.
131. 09:50That kind of changes what you measure for an app.
132. 09:53Like you stop chasing the total latency, which we have done for over a decade.
133. 09:59You start chasing time to first chunk.
134. 10:03You know, the first useful thing that your user sees.
135. 10:07For this reason, the traditional loading spinner won't work for AI features.
136. 10:13Sometimes, you have to get creative and design your feature around it.
137. 10:18This may not work for all, but here's a product example.
138. 10:21And when you know it's going to take time to render something, keep the user engaged.
139. 10:26The Lens live allows the user to focus different things, um tap on an object that they're interested in while they wait for the results.
140. 10:36Uh if you don't have the full screen in your control, you can show the thinking CX.
141. 10:42But please use it sparingly.
142. 10:43The overall AI users have moved out of the forgiving phase, and now they expect to know what is happening.
143. 10:52Now that you have a way to stream back different states, you know how to show what your agent is doing.
144. 10:59Um here's an example I copied from Gemini.
145. 11:02When when it starts to work on a task, it gives a glimpse into what the agent is doing.
146. 11:08Even though it takes 10 seconds, I'm okay if I know what's happening and be able to trust the agent's final output.
147. 11:17We have covered streaming.
148. 11:20Let's move on to the final and most important part.
149. 11:24Your new BFF.
150. 11:26Pattern one was about how UI gets picked.
151. 11:29Pattern two was about how the chunks delivered.
152. 11:33Pattern three is all about how the chunks become meaningful UI elements.
153. 11:39This is a concept of server-driven UI.
154. 11:42And it's a spectrum of server control.
155. 11:44And the more the server controls, the more you build.
156. 11:49The BFF is a subset of it and decides how to render.
157. 11:53It owns the platform specific rules like Android versus iOS, that kind of stuff.
158. 11:59Uh it also helps the client to make less decisions and draw what's handed to it.
159. 12:04That's the whole idea.
160. 12:08The BF BFF doesn't just ship layout.
161. 12:13It does a little more.
162. 12:15In addition to the format and context, it does the hydration and adds actions.
163. 12:23Every rendered element carries an action payload to handle what what a tap does, what the deep link hit opens.
164. 12:30It could even name the impression metric to log.
165. 12:34It carries this conversational context across turns.
166. 12:38So, the next response know what came before.
167. 12:42And the good thing about this is that you can actually do this to your existing apps.
168. 12:47You can reuse the existing CX units you already have.
169. 12:51The flight row, the product card, the components your app already shipped in production.
170. 12:58You're not building a new agentic look into your app.
171. 13:02And that's a good way to build for humans.
172. 13:05Same brand, same density, same familiar feel.
173. 13:09It looks and feels native.
174. 13:13That brings to our takeaways.
175. 13:18One, the models are highly capable already.
176. 13:22You feed a properly typed, versioned contract so it can pick and choose the right CX.
177. 13:29Two, stream into typed components, not text.
178. 13:34Let the user know what your agent is doing.
179. 13:38Three, let the BFF absorb the model output so the client can stay dumb and safe.
180. 13:47No, none of these are about the model.
181. 13:50The model was fine.
182. 13:52This layer is what ships the product.
183. 13:56Bringing back to where we started, your agent output is not the CX.
184. 14:01You build on it.
185. 14:03Thanks for watching.
186. 14:04If you'd like to learn more or connect with me, here's a QR code.