WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=maTp79FD9gI NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/maTp79FD9gI.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.480 --> 00:00:06.760 I asked my AI assistant to help me reserve a table at a popular restaurant. s2 00:00:06.760 --> 00:00:09.440 Here's what it gave me. s3 00:00:09.800 --> 00:00:12.040 Now, it's not wrong. s4 00:00:12.040 --> 00:00:13.960 The phone number is there. s5 00:00:13.960 --> 00:00:15.920 The hours are right. s6 00:00:15.920 --> 00:00:19.600 It even knows the walk-in oyster bar at the front. s7 00:00:19.600 --> 00:00:24.080 The model did the real work, but look at the outcome. s8 00:00:24.080 --> 00:00:28.560 I have to do my research and work towards actually booking the table. s9 00:00:28.560 --> 00:00:30.840 Like, we've been here for a while. s10 00:00:30.840 --> 00:00:35.800 Now, imagine if you were to build something like this for your customers. s11 00:00:35.800 --> 00:00:38.920 What would you want it to do? s12 00:00:39.120 --> 00:00:42.400 Same answer, rendered like this instead. s13 00:00:42.400 --> 00:00:47.080 A date, a time, a couple of taps, and then you're done. s14 00:00:47.080 --> 00:00:53.000 The agentic product you're building already has all the tools to support your customer's needs. s15 00:00:53.000 --> 00:01:00.640 The only thing that you need to focus on is the layer between the model and something that a human can interact with. s16 00:01:00.640 --> 00:01:02.840 That layer is what I'm going to talk about. s17 00:01:02.840 --> 00:01:05.560 The models and agents are here to stay. s18 00:01:05.560 --> 00:01:09.440 And we should learn how to make it friendly to the humans. s19 00:01:09.720 --> 00:01:10.280 Hello. s20 00:01:10.280 --> 00:01:11.680 Welcome to my talk. s21 00:01:11.680 --> 00:01:13.520 I'm Balaram Das. s22 00:01:13.520 --> 00:01:17.600 I have been building customer-facing apps for over a decade. s23 00:01:17.600 --> 00:01:22.520 And the past 6 years I've spent building Amazon Lens. s24 00:01:22.520 --> 00:01:26.560 Amazon Lens is our suite of camera features powered by AI. s25 00:01:26.560 --> 00:01:32.960 It enables you to shop using images, screenshots, and barcodes to discover visually similar products. s26 00:01:32.960 --> 00:01:37.520 If you have Amazon app installed, I encourage you to try. s27 00:01:37.520 --> 00:01:45.440 This talk comes from my experience building customer-facing products that are in millions and millions of mobile devices. s28 00:01:45.440 --> 00:01:48.440 Um quick dis- disclaimer before I go further. s29 00:01:48.440 --> 00:01:50.480 I'm giving this talk on my own. s30 00:01:50.480 --> 00:01:55.240 The opinions in this talk are mine and not my employer's. s31 00:01:56.080 --> 00:02:02.000 When you think of building UX for an agentic AI, obviously ChatGPT comes into mind. s32 00:02:02.000 --> 00:02:03.800 How do you draw these cards? s33 00:02:03.800 --> 00:02:06.640 How do you choose carousel versus list? s34 00:02:06.640 --> 00:02:12.119 How do you craft this so the human can understand and interact better? s35 00:02:12.520 --> 00:02:16.600 When you attempt to build something like this, you run into a wall. s36 00:02:16.600 --> 00:02:22.120 There are problems that need to be solved from the very early stages of your system. s37 00:02:22.120 --> 00:02:26.560 Is that experience going to be snappy or slow? s38 00:02:26.560 --> 00:02:32.120 Do we get all the information at first or one thing at a time? s39 00:02:32.200 --> 00:02:38.640 And how do you scale this for mobile apps where versions and device capabilities are fragmented? s40 00:02:38.640 --> 00:02:42.160 Like if you notice, none of these problems are due to the model itself. s41 00:02:42.160 --> 00:02:44.440 The model does its job well. s42 00:02:44.440 --> 00:02:50.440 Um these are delivery problems and they live in between the model output and what's on the screen. s43 00:02:50.440 --> 00:02:55.640 That is the layer that decides whether your product succeeds or not. s44 00:02:55.959 --> 00:03:06.000 So uh for this delivery problem, if if you had asked me about uh about this layer 2 years ago, I wouldn't have had a name for it. s45 00:03:06.000 --> 00:03:12.840 I'd built a couple of AI features by then and every single time I solved this part from scratch. s46 00:03:12.840 --> 00:03:18.120 A bespoke pattern shaped around whatever system I was in. s47 00:03:18.120 --> 00:03:22.480 Uh there was no shared vocabulary for any of it. s48 00:03:22.480 --> 00:03:25.400 What gets me is that this thing now has a name. s49 00:03:25.400 --> 00:03:26.920 Generative UI. s50 00:03:26.920 --> 00:03:29.040 And there's an open spec for it. s51 00:03:29.040 --> 00:03:31.120 ATUI from Google. s52 00:03:31.120 --> 00:03:37.920 Instead of the agent handing you a raw text or HTML, it describes the UI as data. s53 00:03:37.920 --> 00:03:43.040 A list of components and the client renders them with its own native widgets. s54 00:03:43.040 --> 00:03:48.840 A problem I used to solve alone is becoming something teams get to start from. s55 00:03:48.840 --> 00:03:52.400 And that's exactly what I'm excited to dig into. s56 00:03:52.400 --> 00:03:57.880 Um normal API returns data and the client decides how to draw it. s57 00:03:57.880 --> 00:04:04.040 Like your model is good at tool use, coding, and even other complex tasks. s58 00:04:04.040 --> 00:04:05.840 It can do UI, too. s59 00:04:05.840 --> 00:04:08.680 And that is what the generator UI is about. s60 00:04:08.680 --> 00:04:10.840 And it's it's a spectrum. s61 00:04:10.840 --> 00:04:17.880 Uh Co-pilot Kit, a company that does this primarily, lays out in three rounds. s62 00:04:17.880 --> 00:04:20.600 At the bottom, we have control. s63 00:04:20.600 --> 00:04:25.520 The model picks a pre-built component like a product card. s64 00:04:25.520 --> 00:04:27.800 It never invents anything. s65 00:04:27.800 --> 00:04:30.760 In the middle, um declarative. s66 00:04:30.760 --> 00:04:34.080 The model composes UI from a catalog. s67 00:04:34.080 --> 00:04:38.240 Uh date field, a time field, or a submit button. s68 00:04:38.240 --> 00:04:41.520 Um that's where A2UI sits. s69 00:04:41.520 --> 00:04:44.560 And at the top, it's fully open-ended. s70 00:04:44.560 --> 00:04:50.200 The model generates a novel UI on the fly like MC PA apps. s71 00:04:50.200 --> 00:04:56.440 The higher you go, the more your client has to trust whatever the model hands it. s72 00:04:56.440 --> 00:05:02.800 Um most production mobile apps live in the bottom two rounds because, you know, that's where you stay safe. s73 00:05:02.800 --> 00:05:05.680 And that's what we'll focus on. s74 00:05:06.240 --> 00:05:12.120 Um adding to the complexity of UX building, um mobile apps play an important part. s75 00:05:12.120 --> 00:05:18.560 Um on the web, if uh render breaks, you know, you ship a fix and it's live in minutes. s76 00:05:18.560 --> 00:05:20.600 Uh mobile apps cannot do that. s77 00:05:20.600 --> 00:05:28.040 Like you're looking at hundreds of millions of installs and you don't control uh when any of them update. s78 00:05:28.160 --> 00:05:36.000 So, when a client meets a content type it's never seen, it doesn't gracefully degrade, it crashes. s79 00:05:36.000 --> 00:05:41.080 And it keeps crashing for days or weeks um in hands of people who haven't updated. s80 00:05:41.080 --> 00:05:45.640 So, one rule holds everything that follows for mobile clients. s81 00:05:45.640 --> 00:05:48.680 You cannot meaningfully patch the client. s82 00:05:48.680 --> 00:05:53.360 Um how does this system look when you zoom out? s83 00:05:53.360 --> 00:05:56.080 Um here's the whole simplified pipeline. s84 00:05:56.080 --> 00:06:00.360 Um version aware context face that feeds the model. s85 00:06:00.360 --> 00:06:03.040 Uh yes, that is context engineering. s86 00:06:03.040 --> 00:06:06.480 The model outputs typed UI intent. s87 00:06:06.480 --> 00:06:10.680 Uh that flows through a BFF backend for frontend. s88 00:06:10.680 --> 00:06:17.080 And to a client render that draws it and uh falls back safely when it cannot. s89 00:06:17.080 --> 00:06:19.240 We'll break into three patterns. s90 00:06:19.240 --> 00:06:22.280 Pattern one, the rendering contract. s91 00:06:22.280 --> 00:06:24.720 Pattern two, streaming. s92 00:06:24.720 --> 00:06:26.880 Um that's the flow in between them. s93 00:06:26.880 --> 00:06:30.680 Pattern three, uh the BFF that sits in the middle. s94 00:06:30.680 --> 00:06:34.800 Uh we'll take them one at a time starting with the contract. s95 00:06:34.800 --> 00:06:42.000 Um the contract is all about making the model be aware of what the client capabilities are. s96 00:06:42.000 --> 00:06:48.320 You ensure it it stays true to the client that the model is trying to draw the UI for. s97 00:06:48.320 --> 00:06:59.120 Um what it ensures is that that the onus is not on the client or the rendering layer to infer what to show by looking at the token output. s98 00:06:59.120 --> 00:07:04.400 Um you also maintain a repository of version map with the capabilities. s99 00:07:04.400 --> 00:07:15.480 For example, you introduced a new flight uh card UI in version 2.0. Uh make sure to surface it to the model only from version 2.0 onwards. s100 00:07:15.480 --> 00:07:18.840 Um when you build the context. s101 00:07:20.120 --> 00:07:29.080 The takeaway here is that the model is going to choose the CX and you provide the right information in its con- context. s102 00:07:29.080 --> 00:07:31.160 Um let's take a look at this example. s103 00:07:31.160 --> 00:07:36.080 And this this is usually the hard part of building the system. s104 00:07:36.320 --> 00:07:40.640 This is the model facing half of the contract. s105 00:07:40.640 --> 00:07:44.160 And you don't want the model to send back text. s106 00:07:44.160 --> 00:07:48.280 Like, you want it to stream blocks of UI components. s107 00:07:48.280 --> 00:07:58.440 Um a conversation block for what it says in text and a UI block for what what it wants the client to render. s108 00:07:58.440 --> 00:08:02.480 The contract even encodes layout rules. s109 00:08:02.480 --> 00:08:08.360 In this example, one to three flights a swipable carousel. s110 00:08:08.360 --> 00:08:11.240 Four or more, a vertical list. s111 00:08:11.240 --> 00:08:13.840 The model picks the intent. s112 00:08:13.840 --> 00:08:16.320 And notice what the model never does. s113 00:08:16.320 --> 00:08:18.960 It never invents a component. s114 00:08:18.960 --> 00:08:22.680 It chooses from a fixed menu that you provide to it. s115 00:08:22.680 --> 00:08:25.120 Obviously, it won't be in the millions. s116 00:08:25.120 --> 00:08:27.400 It would be a handful. s117 00:08:27.400 --> 00:08:33.919 It turns out that it gets significantly harder as you scale your features to more surfaces. s118 00:08:33.919 --> 00:08:41.599 I'll let you think about how to engineer that context to make sure the model succeeds in picking one. s119 00:08:42.120 --> 00:08:44.880 So, that's the contract. s120 00:08:44.880 --> 00:08:48.240 Now, let's move on to the streaming part. s121 00:08:48.240 --> 00:08:52.200 This was eye-opening to me when I first encountered it. s122 00:08:52.200 --> 00:08:57.880 Traditionally, apps make an API call and wait for something to happen. s123 00:08:57.880 --> 00:09:06.920 When LLMs are involved, this pattern becomes ineffective as often the latency is higher due to the model themselves. s124 00:09:06.920 --> 00:09:12.680 And on top of it, there are tons of additional checks to ensure safety and whatnot. s125 00:09:12.680 --> 00:09:20.160 And we also introduce a new layer that it's going to be a little bit complex to do the dynamic UI. s126 00:09:20.160 --> 00:09:26.120 Um streaming helps here uh by not making the client wait for the whole response. s127 00:09:26.120 --> 00:09:30.800 It renders things in chunks at a time. s128 00:09:31.160 --> 00:09:37.680 Take a look at the illustrative example when you have to surface information one chunk at a time. s129 00:09:37.680 --> 00:09:43.320 First, you show a skeleton, then partially fill it, then complete it. s130 00:09:43.320 --> 00:09:49.960 It may take 3 to 4 seconds to get there, but the wait is bearable. s131 00:09:50.400 --> 00:09:53.800 That kind of changes what you measure for an app. s132 00:09:53.800 --> 00:09:59.760 Like you stop chasing the total latency, which we have done for over a decade. s133 00:09:59.760 --> 00:10:03.160 You start chasing time to first chunk. s134 00:10:03.160 --> 00:10:07.200 You know, the first useful thing that your user sees. s135 00:10:07.200 --> 00:10:13.040 For this reason, the traditional loading spinner won't work for AI features. s136 00:10:13.360 --> 00:10:18.080 Sometimes, you have to get creative and design your feature around it. s137 00:10:18.080 --> 00:10:21.800 This may not work for all, but here's a product example. s138 00:10:21.800 --> 00:10:26.520 And when you know it's going to take time to render something, keep the user engaged. s139 00:10:26.520 --> 00:10:36.520 The Lens live allows the user to focus different things, um tap on an object that they're interested in while they wait for the results. s140 00:10:36.600 --> 00:10:42.200 Uh if you don't have the full screen in your control, you can show the thinking CX. s141 00:10:42.200 --> 00:10:43.960 But please use it sparingly. s142 00:10:43.960 --> 00:10:52.920 The overall AI users have moved out of the forgiving phase, and now they expect to know what is happening. s143 00:10:52.920 --> 00:10:59.720 Now that you have a way to stream back different states, you know how to show what your agent is doing. s144 00:10:59.720 --> 00:11:02.560 Um here's an example I copied from Gemini. s145 00:11:02.560 --> 00:11:08.720 When when it starts to work on a task, it gives a glimpse into what the agent is doing. s146 00:11:08.720 --> 00:11:17.080 Even though it takes 10 seconds, I'm okay if I know what's happening and be able to trust the agent's final output. s147 00:11:17.880 --> 00:11:20.520 We have covered streaming. s148 00:11:20.839 --> 00:11:24.520 Let's move on to the final and most important part. s149 00:11:24.520 --> 00:11:26.520 Your new BFF. s150 00:11:26.520 --> 00:11:29.920 Pattern one was about how UI gets picked. s151 00:11:29.920 --> 00:11:33.520 Pattern two was about how the chunks delivered. s152 00:11:33.520 --> 00:11:39.320 Pattern three is all about how the chunks become meaningful UI elements. s153 00:11:39.320 --> 00:11:42.640 This is a concept of server-driven UI. s154 00:11:42.640 --> 00:11:44.880 And it's a spectrum of server control. s155 00:11:44.880 --> 00:11:49.120 And the more the server controls, the more you build. s156 00:11:49.120 --> 00:11:53.760 The BFF is a subset of it and decides how to render. s157 00:11:53.760 --> 00:11:59.480 It owns the platform specific rules like Android versus iOS, that kind of stuff. s158 00:11:59.480 --> 00:12:04.920 Uh it also helps the client to make less decisions and draw what's handed to it. s159 00:12:04.920 --> 00:12:07.880 That's the whole idea. s160 00:12:08.800 --> 00:12:13.800 The BF BFF doesn't just ship layout. s161 00:12:13.800 --> 00:12:15.560 It does a little more. s162 00:12:15.560 --> 00:12:23.160 In addition to the format and context, it does the hydration and adds actions. s163 00:12:23.160 --> 00:12:30.680 Every rendered element carries an action payload to handle what what a tap does, what the deep link hit opens. s164 00:12:30.680 --> 00:12:34.560 It could even name the impression metric to log. s165 00:12:34.560 --> 00:12:38.440 It carries this conversational context across turns. s166 00:12:38.440 --> 00:12:42.360 So, the next response know what came before. s167 00:12:42.720 --> 00:12:47.440 And the good thing about this is that you can actually do this to your existing apps. s168 00:12:47.440 --> 00:12:51.320 You can reuse the existing CX units you already have. s169 00:12:51.320 --> 00:12:58.240 The flight row, the product card, the components your app already shipped in production. s170 00:12:58.240 --> 00:13:02.800 You're not building a new agentic look into your app. s171 00:13:02.800 --> 00:13:05.280 And that's a good way to build for humans. s172 00:13:05.280 --> 00:13:09.000 Same brand, same density, same familiar feel. s173 00:13:09.000 --> 00:13:12.280 It looks and feels native. s174 00:13:13.760 --> 00:13:17.280 That brings to our takeaways. s175 00:13:18.800 --> 00:13:22.760 One, the models are highly capable already. s176 00:13:22.760 --> 00:13:29.600 You feed a properly typed, versioned contract so it can pick and choose the right CX. s177 00:13:29.600 --> 00:13:34.960 Two, stream into typed components, not text. s178 00:13:34.960 --> 00:13:38.360 Let the user know what your agent is doing. s179 00:13:38.360 --> 00:13:46.120 Three, let the BFF absorb the model output so the client can stay dumb and safe. s180 00:13:47.520 --> 00:13:50.240 No, none of these are about the model. s181 00:13:50.240 --> 00:13:52.480 The model was fine. s182 00:13:52.480 --> 00:13:55.960 This layer is what ships the product. s183 00:13:56.520 --> 00:14:01.360 Bringing back to where we started, your agent output is not the CX. s184 00:14:01.360 --> 00:14:03.160 You build on it. s185 00:14:03.160 --> 00:14:04.840 Thanks for watching. s186 00:14:04.840 --> 00:14:09.720 If you'd like to learn more or connect with me, here's a QR code.