The End of the Static Screen: Architecting Intent-Driven UX — Gus Iwanaga, commercetools
https://www.youtube.com/watch?v=QrMcNe2jjt8
AI Engineer · 23 min · 325 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:12] How's it going?
[00:14] The end of uh the end of the conference, how's everybody feeling?
[00:18] Tired, drinking from the fire hose as well?
[00:21] Are you guys a little bit tired of hearing loop engineering, harness engineering, software factory, evolves, pre-training, post-training data, and whatnot?
[00:31] Uh but anyways, those topics were more than valid, right?
[00:34] Uh hi.
[00:35] Some uh really uh common faces here.
[00:38] By the way, I'm super excited to be here and talk to you guys about this title.
[00:43] And And I'm sorry, when I was when I submitted the application, I was thinking of potentially a catchy title, but I don't like this at all.
[00:52] So, with all due respect to the organizers, I'll have to make a change.
[00:56] Uh and the actual topic that I want to focus here today is lessons learned uh from a team that is building proper generative uh UX and UI.
[01:08] And I was going to uh touch on agentic orchestration, but come on.
[01:13] Uh over the last 3 days, this is what we heard all the time.
[01:17] So, I'd rather focus on what I didn't hear enough about here in the conference, and hopefully you walk away, if not with something very tangible, but with a new mental model that can spark meaningful discussions down the road.
[01:29] Sound good?
[01:31] All right.
[01:32] Very good.
[01:33] So, hi everybody.
[01:34] Uh I'm Gus.
[01:35] I'm a general manager at commercetools.
[01:37] I lead product, UX, and engineering for zero-to-one products.
[01:41] I tell people I'm in a very privileged position because we get to build really cool stuff.
[01:46] So, we cook really interesting stuff at commercetools.
[01:49] Fantastic.
[01:50] And this is what you guys can expect uh at least during the presentation.
[01:54] Uh I'd like to make sure that we're on the same page with respect to the problem uh the problem space uh followed by a quick demo of the product because I'm not sure if you guys agree, an image speaks more than 1,000 words.
[02:05] So, it would just uh make it more tangible for uh everybody here.
[02:10] Followed by a rapid discussion on the emergence of UI protocols, and I'm not sure if you guys joined maybe some of the talks here.
[02:17] Even the founder of some of these protocols were here uh this week, and that was really cool.
[02:22] Followed by and last but not least, the challenges that uh my team uh and I faced and we still face.
[02:28] And some of the mitigation tactics uh that we put in place to overcome some of these challenges.
[02:33] So, with that let's continue.
[02:36] The problem space.
[02:37] This is This is more like a a a statement um and we're still adapt to the software that we ship, not the other way around.
[02:46] Right?
[02:47] And then uh although you could argue when GPT came out, this was November uh 2022, we had a really good glimpse of uh real personalization, but everything else uh remained static.
[03:00] And even with AI, we keep shipping a lot of stuff much faster, but to a significant extent, it is still static.
[03:08] And my question is, why?
[03:11] So, for the uh over the last 40 years, uh we kept shipping a static experiences.
[03:18] And if I put it myself in the shoes of uh some of uh my customers, they need several SaaS applications for uh the day-to-day work, and each with its own mental model and its own way to get anything done.
[03:35] And over time, it just kept getting worse, just accruing uh debt.
[03:40] The cognitive load that we wanted to remove and that we wanted to transfer to the machine, uh it's on us.
[03:47] And now with AI, things can be different.
[03:52] I don't know if you guys agree, but this is uh what we what we think.
[03:55] And I do have a couple of examples.
[03:57] I'm not going to say out loud the name of these apps, but let's take a look.
[04:03] Here.
[04:03] First one.
[04:05] You know, that icon is very well known.
[04:07] This is probably the most famous CRM of all times.
[04:11] But when I look at this screen, there's a lot going on.
[04:15] I don't even know where to start.
[04:16] Right?
[04:17] Not only the information overload, how many features how many teams do you think are somewhat involved just to ship this?
[04:24] Many, probably.
[04:25] Right?
[04:26] And this is just one.
[04:27] Let's have a look.
[04:28] I have a couple of other examples.
[04:30] So, this here.
[04:33] Look at that.
[04:34] Oh my god, this is a fancy table.
[04:36] I don't even know where to start.
[04:38] But anyways, one more.
[04:40] Does everybody know this one here?
[04:43] Beautiful.
[04:44] Beautiful UI.
[04:46] Right?
[04:46] It's a fantastic, super intuitive, and something else just to highlight.
[04:52] Do you guys know how much time these companies need to invest in onboarding people?
[04:58] So, that was the the trade-off.
[05:00] Right?
[05:00] So, you got to allocate a lot of time for a lot of people just to onboard newcomers.
[05:06] As a result of this complexity that has been introduced over time.
[05:12] So, this is just at least the hardcore evidence.
[05:16] So, different apps, different logic every single time.
[05:20] And then more apps are coming out.
[05:22] So, imagine just put yourself in the shoes of average user, and then oh, now I have five five apps.
[05:28] And then every single one I need to learn how to navigate, how to browse, and so on and so forth.
[05:33] And then this has been the history up until now.
[05:37] But then this was August last year, I sat down with my boss.
[05:41] It happens to be the founder of the company, so big shout out to my boss.
[05:45] And then we asked this uh question because at Commerce Tools, we are an API first company, 300 plus still counting.
[05:51] And we ask this posing question, through the lens of artificial intelligence, what are the foundational shifts that could be made if we could change drastically the way that we interact with software?
[06:03] Not in a uh static uh fashion.
[06:05] And the answer to that question led to the product that I'm going to demo right now.
[06:10] Let's go.
[06:11] How about quick demo?
[06:12] Guys like the idea?
[06:14] Give me a thumbs up.
[06:15] Bye.
[06:15] I know everybody's tired.
[06:16] Let's go.
[06:17] Yeah.
[06:18] All right.
[06:19] Cool.
[06:20] Look at this.
[06:21] I know, I'm going to zoom in.
[06:22] No worries.
[06:23] So, I have this query here.
[06:25] Create a sales report for Q1.
[06:28] And the UX side of me, when I look at uh what was generated, and by the way, everything here on the right side has been auto-generated, guided by us, but this is AI, right?
[06:41] Deciding on the placement, on the information architecture, deciding which uh components had to be actually retrieved from the catalog, but I don't like it at all.
[06:52] Right?
[06:52] It's a Even if you don't know a lot uh about UX, just let's, you know, I have at least four different variations because those were four different terms for the same query.
[07:03] Let's check it out.
[07:05] First one here, it was Q1 uh Urban Thread, whatever that is.
[07:09] Uh Q1, I see all of these KPI cards.
[07:12] There's a lot going on here.
[07:14] And then my intuition tells me, man, this doesn't add up.
[07:18] Okay.
[07:19] Second one, it's not Q1 anymore.
[07:22] Now this is January and March.
[07:24] There's no consistency.
[07:26] Does that help?
[07:27] Yes or no?
[07:29] No, right?
[07:29] This is This is You only create confusion.
[07:32] If this is a heavy personalized experience for the user, imagine if every single time you need to prompt, and then uh at least the model will output something different.
[07:42] This is not good, and this is just the second turn.
[07:44] Let's have a look at the third one.
[07:47] Oh my god, now there is even more stuff here on the right side.
[07:51] But so, this is I have this component, the KPI cards, a bunch of text, a bunch of charts, and then this this was the beginning of a journey.
[08:02] One more?
[08:04] Yes.
[08:04] Okay, now it's still Q1, but still for me this still it's confusing.
[08:11] And because this has been a very experimental journey, at least okay, let's let's move on here.
[08:18] My feedback to the team and to myself was no, no, and no. There's no way that I would ship this to to prod whatsoever, right?
[08:28] And then I have my colleagues here just to confirm what I just said.
[08:32] But then things evolved, and I'll like to demo the current state of the product.
[08:37] It's much more sophisticated, and let's have a look.
[08:40] I I'll like to plan a campaign, and for what it's worth, I'm going to save you from all the nitty-gritty details for everything that is domain specific, but I'm going to pick this query here,
[08:53] and then let's see what happens.
[08:55] And this is the the agentic orchestration part that I was going to highlight.
[08:58] Underneath the hood we have the orchestrator, and the orchestrator can then just extract the intent of the query.
[09:04] Based off of the intent of the query, it can locate the tools, right?
[09:07] Those can be first party, third party tools, and the outputs of this different it could be agents on MCP servers, combined will give enough what I call ammunition and context
[09:19] for the UX agent to eventually render something that we call meaningful.
[09:24] So, compared to the previous turns, this is this is decent, right?
[09:30] I want to just remove my bias, but the overall aesthetics, the look and feel of this query, uh it it resonates with me.
[09:38] Would you agree?
[09:39] Give me a thumbs up if you like if you agree.
[09:42] Okay, at least um the vast majority here.
[09:45] And then see, it is decent.
[09:47] And let me just continue here.
[09:48] And then once again, right?
[09:50] This was decided by AI guided by us.
[09:54] I would just want to make make that clear here.
[09:57] And then I'm going to touch on the on the UI protocols and then how you can make this happen.
[10:02] But okay, let's see.
[10:03] If I approve here, and then this is already live.
[10:06] And this is pre-prod, right?
[10:07] So, uh this is great.
[10:09] Let me go back to my presentation.
[10:12] Perfect.
[10:12] I have one more question.
[10:14] Are you guys as skeptical that this is possible?
[10:17] Because I can tell you this it is possible.
[10:19] If you're still skeptical, don't worry.
[10:20] I have all of these guys uh here uh also uh every day just looking at me and then challenging whether uh this uh can be made possible at scale, right?
[10:30] And then uh okay.
[10:31] And then this is the part that I'll like just to touch uh touch base on the uh on the three ways that you can render what you just saw.
[10:39] And they're different UI protocols.
[10:41] Are you guys familiar with uh generative uh UI?
[10:44] Have you guys played with it?
[10:46] Let me see here.
[10:47] Okay, well, that's really cool.
[10:49] Uh okay.
[10:50] So, uh what I would like just to share with you guys it's all about how much control you want to exercise over the experience.
[10:58] And this matters a lot because uh you know, as a non-deterministic solution, uh you can decide if you want something I really um like this.
[11:07] So, let's have a look.
[11:08] Here, this is uh ChatGPT.
[11:11] And my query was help me find a Japanese restaurant uh in SF today.
[11:18] If you guys see here, this component this component is very opinionated.
[11:23] Would you agree with that?
[11:24] Here?
[11:25] Right?
[11:26] So, uh you can have complete control over this uh component.
[11:31] So, depending upon the nature of your business, this works really well, right?
[11:36] And for that, let me just go back here.
[11:39] And then this is what I call a control.
[11:41] Essentially, you ship the component as it is.
[11:45] The agent will pick and it will display exactly the way that you describe.
[11:49] However, depending upon the nature of your business, at least for us, right, at my company, where a B2B SaaS, the there's so much configuration that we don't want to be over uh prescriptive
[12:01] because uh the feedback that I keep getting from my customers, "Oh, the flows are so confusing.
[12:06] There's so much configuration.
[12:07] How can you remove the cognitive load uh for me?"
[12:11] But uh if you're like Booking, for example, this approach uh works uh really well.
[12:16] And then let's see uh how it works.
[12:18] So, essentially, you have the agent.
[12:22] The agent will just pick the component from your catalog and then it will render uh as it is, right?
[12:28] And uh in the interest uh of time, I'm not going to touch base on the code snippets uh that I have for this uh three different types.
[12:36] But afterwards, if you guys are interested, uh I can share the presentation and then you can have a look, all right?
[12:41] Very good.
[12:42] This is uh at least on the left side.
[12:45] Let's discuss a little bit on the right side because this is when you give full autonomy to the LLM.
[12:50] If you guys remember uh at least the previous attempts for my product, this exactly what we did.
[12:55] So, we just said, "So, hey LLM, how uh how would you compose this experience knowing that you have these components?"
[13:03] Right?
[13:03] But uh that was uh that was a little bit uh a little bit of our opinion because uh it has uh it had uh some of the components available.
[13:12] But it could happen that uh you can just delegate fully to the LLM.
[13:18] And then right now, I'm here on Claude and I asked Claude, "So, hey, create an org chart with three levels."
[13:23] That was it.
[13:24] And then Claude just rendered this diagram and it works really well.
[13:30] Right?
[13:31] But here, if I put myself in the shoes of a company, I'm not sure I would delegate fully to the LLM.
[13:37] Because I cannot control at least the app the output and the outcome.
[13:42] And me personally, me guys, as a UX leader, the UX side of me will always say no. You got to be in control.
[13:49] There's been a couple of talks here at least this week on design, on taste, and judgment.
[13:56] And this matters a lot.
[13:57] If you guys want to embark on this journey of leveraging these protocols, you don't want to delegate too much of the actual experience to the LLM.
[14:05] You got to find alternatives and I'm going to touch on that in just a little bit.
[14:10] Okay, cool.
[14:11] And then this is how the open-ended approach works.
[14:15] So essentially there there's going to be an MCP tool and then this will literally ship the HTML and then in a sandbox I frame environment, this will be rendered in the host of your choice.
[14:30] But it can be a chat.
[14:31] It can be this cloud.
[14:33] It could be a chat GPT.
[14:34] It could be perplexity or it could be any other chat.
[14:37] If you're willing just to to give full control to the LLM, good luck.
[14:42] But the one that I would like to highlight is this here and this was our choice that we call the declarative.
[14:50] Declarative is in the middle.
[14:52] If you guys heard some of the protocols and I don't want to get into the specifics of each because they have different characteristics.
[15:00] But HTMX from Google, JSON render from Vercel, OpenUI by Thesis, those are some of the protocols that will give you this in between here.
[15:11] And let let's have a look, right?
[15:13] So at least for my product, the one that I just showed, the orchestrator agent will eventually, if you think of the whole traversal, the user will enter the query
[15:25] and then there's going to be the intent classification.
[15:27] Based off of the intent classification, then the tools will be invoked, the data will be retrieved, and somewhat somewhat in between Alice, there will be the mapping of the eligible components from your catalog
[15:41] to the entities of of the tools, right?
[15:45] And then the orchestrator will just broadcast this UI description.
[15:49] It's like a UI spec.
[15:50] This UI spec will be also we have this component catalog here and we use the Zod schema and then you got to be compliant with this protocols.
[15:58] This is just one of the requirements and then you just render that.
[16:02] Right?
[16:03] And then their final output will be the native UI, in this case the React components.
[16:09] This is it.
[16:10] The good thing about the declarative approach is it will be compliant with your design system everywhere.
[16:18] This matters a lot.
[16:19] So, in our case, we did not want to delegate to the LLM because you guys saw over there you can't change the copy, right?
[16:28] So, it's not key one.
[16:29] Sometimes it's going to be March January to March.
[16:33] It matters a lot.
[16:34] So, within UX we have different factors, right?
[16:37] We have the actual UX's if you think of the overall experience.
[16:41] There is UI, there's also copy, UX writing, and so on and so forth.
[16:46] But this approach gives us this in between.
[16:49] It is less deterministic and I think this is a really good segue to some of the challenges because imagine, I'm going to use my example once again, the orchestrator
[16:58] will just fetch the eligible components for that query, but then it's up to the LLM how to place in the UI.
[17:07] And this can get really really messy, but and those are the challenges that I'll like to share with you guys here.
[17:14] I got to be careful because I've got only 30 minutes, but let's go.
[17:18] So, the first challenge is if the agent picks the components, who's in charge or what entity is in charge of arranging them?
[17:26] And this is information architecture, right?
[17:29] This is a critical aspect of UX.
[17:32] And then once again, if you put yourself in the shoes of the average customer, it matters a lot.
[17:37] So, if you're just left alone, the placement can be totally random.
[17:41] So, at least in my team, we borrow this concept of atomic design.
[17:47] And atomic design, it goes like this.
[17:50] Let me just change here.
[17:51] Uh yeah.
[17:52] So, as you can see, atomic design, and then I have the definition, is a methodology composed of five distinct stages working together to create interface design systems in a more deliberate
[18:05] and hierarchical manner.
[18:06] This helps a lot.
[18:07] Why?
[18:07] Because those are the individual elements, and if you think of the overall structure of the page, it gives me the ability to steer it as I see fit.
[18:16] And what we've done in my team, so this UX agent, we harnessed this UX agent.
[18:22] So, we eventually taught this this UX agent what good looks like, what is the optimal layout for a given situation, and we have a catalog of different templates.
[18:35] But I'll like to show you at least this because this detail is very important.
[18:41] Here, it is the overall hierarchy hierarchy.
[18:45] So, if you remember part of the orchestrator, the orchestrator will eventually just fetch the eligible components to accomplish the query of the user, but then the next big question is, how do we arrange that?
[18:58] The approach that we used was think of this hierarchy.
[19:01] So, you have the overall page, the layout.
[19:05] The layout will contain different slots.
[19:08] So, think of this one here, the header.
[19:10] You can have the main, and then you can have sub sub slots.
[19:14] Sub slots can have sub slots.
[19:16] And within the sub slots, you can have eligible component categories.
[19:20] And then this will allow us to steer eventually the the the optimal placement of the components that have been retrieved by the orchestrator.
[19:31] So this is literally us codifying our UX knowledge into this agent.
[19:38] So the next time, you know, it doesn't matter.
[19:40] We'll just follow the same approach.
[19:42] And then this is the hierarchy that we're using.
[19:44] So layout to slot to sub slot to components.
[19:48] However, because of the orchestrator, we we flipped the order.
[19:52] So from components, components will map to sub slot, sub slots to slots, and slots to templates.
[19:58] And then we can just arrange as needed.
[20:02] So this was a hell of a challenge.
[20:04] It is still a challenge, by the way, and it's a really good segue to the second one, the second challenge, which is the actual design, your design system and your catalog.
[20:16] This becomes the heartbeat of the whole thing, right?
[20:19] I cannot stress enough, if you guys see the potential of leveraging this UI protocols for your product, this is going to be a big deal, right?
[20:27] So we're pushing the boundaries and then we're testing and we keep on testing these different protocols that I mentioned, ATUI, JSON Render, Open UI, and so on and so forth.
[20:38] But this has been a quite challenging because the the catalog is the contract between the agent and the UI.
[20:45] So every property mat- matters.
[20:48] And not only for the catalog, but for the layout as well.
[20:51] So if you remember, the layout has its own components, the slots and the sub slots.
[20:57] Each of those components will have its own attributes.
[21:01] And all of that, this curation, let me just encapsulate into curation.
[21:05] This curation is absolutely needed so that you can deliver something meaningful.
[21:10] Not as some demo that you will see out there for the sake of demo, right?
[21:14] So, this this this will give you a control will allow you to steer from UX perspective.
[21:20] Very good.
[21:21] And last but not least, one significant challenge that we had is that my teams do not design the pixel anymore.
[21:29] I don't know if you could see that, right?
[21:31] So, we're not here designing at the entire flow.
[21:34] Now, AI can dictate that to a significant extent, but the nature of the work shifted quite a bit and it's been an interesting journey to say the least, a really good one, but even for the non-technical
[21:47] PMs and UX designers, it it was a big hit because right now we talk about the schema.
[21:54] Let's talk about this curation of the catalog.
[21:57] Let's talk about the rules.
[21:58] Let's talk about the synthetic data that we can generate.
[22:02] How can we generate the queries that will map to a given component as part of this mapping logic?
[22:08] Let's talk about interaction patterns.
[22:11] So, this if once again, if you guys are going to embark on this journey, be aware that the people element is very important.
[22:18] When I talk to other leaders, I I talk about the three P's: people, product, and process, right?
[22:24] And then a very lightweight process, but this matters a lot.
[22:28] And with that, I'm going to leave a couple of resources here.
[22:33] So, and by the way, those are talks from AIE for what it's worth.
[22:38] So, yeah, people that have been talking about this protocols over and over and over.
[22:43] So, please just take advantage, take a screenshot.
[22:46] And if you guys want to connect with me here, yeah, my LinkedIn or just take a screenshot.
[22:52] I'll love to talk more about the topic.
[22:56] I can tell this is just a matter of time, right?
[22:58] So, this is coming.
[22:58] So, thank you so much.
[23:02] I know.
[23:15] [music]