The End of the Static Screen: Architecting Intent-Driven UX — Gus Iwanaga, commercetools

AI Engineer · 23 min · 325 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12How's it going?
00:14The end of uh the end of the conference, how's everybody feeling?
00:18Tired, drinking from the fire hose as well?
00:21Are you guys a little bit tired of hearing loop engineering, harness engineering, software factory, evolves, pre-training, post-training data, and whatnot?
00:31Uh but anyways, those topics were more than valid, right?
00:34Uh hi.
00:35Some uh really uh common faces here.
00:38By the way, I'm super excited to be here and talk to you guys about this title.
00:43And And I'm sorry, when I was when I submitted the application, I was thinking of potentially a catchy title, but I don't like this at all.
00:52So, with all due respect to the organizers, I'll have to make a change.
00:56Uh and the actual topic that I want to focus here today is lessons learned uh from a team that is building proper generative uh UX and UI.
01:08And I was going to uh touch on agentic orchestration, but come on.
01:13Uh over the last 3 days, this is what we heard all the time.
01:17So, I'd rather focus on what I didn't hear enough about here in the conference, and hopefully you walk away, if not with something very tangible, but with a new mental model that can spark meaningful discussions down the road.
01:29Sound good?
01:31All right.
01:32Very good.
01:33So, hi everybody.
01:34Uh I'm Gus.
01:35I'm a general manager at commercetools.
01:37I lead product, UX, and engineering for zero-to-one products.
01:41I tell people I'm in a very privileged position because we get to build really cool stuff.
01:46So, we cook really interesting stuff at commercetools.
01:49Fantastic.
01:50And this is what you guys can expect uh at least during the presentation.
01:54Uh I'd like to make sure that we're on the same page with respect to the problem uh the problem space uh followed by a quick demo of the product because I'm not sure if you guys agree, an image speaks more than 1,000 words.
02:05So, it would just uh make it more tangible for uh everybody here.
02:10Followed by a rapid discussion on the emergence of UI protocols, and I'm not sure if you guys joined maybe some of the talks here.
02:17Even the founder of some of these protocols were here uh this week, and that was really cool.
02:22Followed by and last but not least, the challenges that uh my team uh and I faced and we still face.
02:28And some of the mitigation tactics uh that we put in place to overcome some of these challenges.
02:33So, with that let's continue.
02:36The problem space.
02:37This is This is more like a a a statement um and we're still adapt to the software that we ship, not the other way around.
02:46Right?
02:47And then uh although you could argue when GPT came out, this was November uh 2022, we had a really good glimpse of uh real personalization, but everything else uh remained static.
03:00And even with AI, we keep shipping a lot of stuff much faster, but to a significant extent, it is still static.
03:08And my question is, why?
03:11So, for the uh over the last 40 years, uh we kept shipping a static experiences.
03:18And if I put it myself in the shoes of uh some of uh my customers, they need several SaaS applications for uh the day-to-day work, and each with its own mental model and its own way to get anything done.
03:35And over time, it just kept getting worse, just accruing uh debt.
03:40The cognitive load that we wanted to remove and that we wanted to transfer to the machine, uh it's on us.
03:47And now with AI, things can be different.
03:52I don't know if you guys agree, but this is uh what we what we think.
03:55And I do have a couple of examples.
03:57I'm not going to say out loud the name of these apps, but let's take a look.
04:03Here.
04:03First one.
04:05You know, that icon is very well known.
04:07This is probably the most famous CRM of all times.
04:11But when I look at this screen, there's a lot going on.
04:15I don't even know where to start.
04:16Right?
04:17Not only the information overload, how many features how many teams do you think are somewhat involved just to ship this?
04:24Many, probably.
04:25Right?
04:26And this is just one.
04:27Let's have a look.
04:28I have a couple of other examples.
04:30So, this here.
04:33Look at that.
04:34Oh my god, this is a fancy table.
04:36I don't even know where to start.
04:38But anyways, one more.
04:40Does everybody know this one here?
04:43Beautiful.
04:44Beautiful UI.
04:46Right?
04:46It's a fantastic, super intuitive, and something else just to highlight.
04:52Do you guys know how much time these companies need to invest in onboarding people?
04:58So, that was the the trade-off.
05:00Right?
05:00So, you got to allocate a lot of time for a lot of people just to onboard newcomers.
05:06As a result of this complexity that has been introduced over time.
05:12So, this is just at least the hardcore evidence.
05:16So, different apps, different logic every single time.
05:20And then more apps are coming out.
05:22So, imagine just put yourself in the shoes of average user, and then oh, now I have five five apps.
05:28And then every single one I need to learn how to navigate, how to browse, and so on and so forth.
05:33And then this has been the history up until now.
05:37But then this was August last year, I sat down with my boss.
05:41It happens to be the founder of the company, so big shout out to my boss.
05:45And then we asked this uh question because at Commerce Tools, we are an API first company, 300 plus still counting.
05:51And we ask this posing question, through the lens of artificial intelligence, what are the foundational shifts that could be made if we could change drastically the way that we interact with software?
06:03Not in a uh static uh fashion.
06:05And the answer to that question led to the product that I'm going to demo right now.
06:10Let's go.
06:11How about quick demo?
06:12Guys like the idea?
06:14Give me a thumbs up.
06:15Bye.
06:15I know everybody's tired.
06:16Let's go.
06:17Yeah.
06:18All right.
06:19Cool.
06:20Look at this.
06:21I know, I'm going to zoom in.
06:22No worries.
06:23So, I have this query here.
06:25Create a sales report for Q1.
06:28And the UX side of me, when I look at uh what was generated, and by the way, everything here on the right side has been auto-generated, guided by us, but this is AI, right?
06:41Deciding on the placement, on the information architecture, deciding which uh components had to be actually retrieved from the catalog, but I don't like it at all.
06:52Right?
06:52It's a Even if you don't know a lot uh about UX, just let's, you know, I have at least four different variations because those were four different terms for the same query.
07:03Let's check it out.
07:05First one here, it was Q1 uh Urban Thread, whatever that is.
07:09Uh Q1, I see all of these KPI cards.
07:12There's a lot going on here.
07:14And then my intuition tells me, man, this doesn't add up.
07:18Okay.
07:19Second one, it's not Q1 anymore.
07:22Now this is January and March.
07:24There's no consistency.
07:26Does that help?
07:27Yes or no?
07:29No, right?
07:29This is This is You only create confusion.
07:32If this is a heavy personalized experience for the user, imagine if every single time you need to prompt, and then uh at least the model will output something different.
07:42This is not good, and this is just the second turn.
07:44Let's have a look at the third one.
07:47Oh my god, now there is even more stuff here on the right side.
07:51But so, this is I have this component, the KPI cards, a bunch of text, a bunch of charts, and then this this was the beginning of a journey.
08:02One more?
08:04Yes.
08:04Okay, now it's still Q1, but still for me this still it's confusing.
08:11And because this has been a very experimental journey, at least okay, let's let's move on here.
08:18My feedback to the team and to myself was no, no, and no. There's no way that I would ship this to to prod whatsoever, right?
08:28And then I have my colleagues here just to confirm what I just said.
08:32But then things evolved, and I'll like to demo the current state of the product.
08:37It's much more sophisticated, and let's have a look.
08:40I I'll like to plan a campaign, and for what it's worth, I'm going to save you from all the nitty-gritty details for everything that is domain specific, but I'm going to pick this query here,
08:53and then let's see what happens.
08:55And this is the the agentic orchestration part that I was going to highlight.
08:58Underneath the hood we have the orchestrator, and the orchestrator can then just extract the intent of the query.
09:04Based off of the intent of the query, it can locate the tools, right?
09:07Those can be first party, third party tools, and the outputs of this different it could be agents on MCP servers, combined will give enough what I call ammunition and context
09:19for the UX agent to eventually render something that we call meaningful.
09:24So, compared to the previous turns, this is this is decent, right?
09:30I want to just remove my bias, but the overall aesthetics, the look and feel of this query, uh it it resonates with me.
09:38Would you agree?
09:39Give me a thumbs up if you like if you agree.
09:42Okay, at least um the vast majority here.
09:45And then see, it is decent.
09:47And let me just continue here.
09:48And then once again, right?
09:50This was decided by AI guided by us.
09:54I would just want to make make that clear here.
09:57And then I'm going to touch on the on the UI protocols and then how you can make this happen.
10:02But okay, let's see.
10:03If I approve here, and then this is already live.
10:06And this is pre-prod, right?
10:07So, uh this is great.
10:09Let me go back to my presentation.
10:12Perfect.
10:12I have one more question.
10:14Are you guys as skeptical that this is possible?
10:17Because I can tell you this it is possible.
10:19If you're still skeptical, don't worry.
10:20I have all of these guys uh here uh also uh every day just looking at me and then challenging whether uh this uh can be made possible at scale, right?
10:30And then uh okay.
10:31And then this is the part that I'll like just to touch uh touch base on the uh on the three ways that you can render what you just saw.
10:39And they're different UI protocols.
10:41Are you guys familiar with uh generative uh UI?
10:44Have you guys played with it?
10:46Let me see here.
10:47Okay, well, that's really cool.
10:49Uh okay.
10:50So, uh what I would like just to share with you guys it's all about how much control you want to exercise over the experience.
10:58And this matters a lot because uh you know, as a non-deterministic solution, uh you can decide if you want something I really um like this.
11:07So, let's have a look.
11:08Here, this is uh ChatGPT.
11:11And my query was help me find a Japanese restaurant uh in SF today.
11:18If you guys see here, this component this component is very opinionated.
11:23Would you agree with that?
11:24Here?
11:25Right?
11:26So, uh you can have complete control over this uh component.
11:31So, depending upon the nature of your business, this works really well, right?
11:36And for that, let me just go back here.
11:39And then this is what I call a control.
11:41Essentially, you ship the component as it is.
11:45The agent will pick and it will display exactly the way that you describe.
11:49However, depending upon the nature of your business, at least for us, right, at my company, where a B2B SaaS, the there's so much configuration that we don't want to be over uh prescriptive
12:01because uh the feedback that I keep getting from my customers, "Oh, the flows are so confusing.
12:06There's so much configuration.
12:07How can you remove the cognitive load uh for me?"
12:11But uh if you're like Booking, for example, this approach uh works uh really well.
12:16And then let's see uh how it works.
12:18So, essentially, you have the agent.
12:22The agent will just pick the component from your catalog and then it will render uh as it is, right?
12:28And uh in the interest uh of time, I'm not going to touch base on the code snippets uh that I have for this uh three different types.
12:36But afterwards, if you guys are interested, uh I can share the presentation and then you can have a look, all right?
12:41Very good.
12:42This is uh at least on the left side.
12:45Let's discuss a little bit on the right side because this is when you give full autonomy to the LLM.
12:50If you guys remember uh at least the previous attempts for my product, this exactly what we did.
12:55So, we just said, "So, hey LLM, how uh how would you compose this experience knowing that you have these components?"
13:03Right?
13:03But uh that was uh that was a little bit uh a little bit of our opinion because uh it has uh it had uh some of the components available.
13:12But it could happen that uh you can just delegate fully to the LLM.
13:18And then right now, I'm here on Claude and I asked Claude, "So, hey, create an org chart with three levels."
13:23That was it.
13:24And then Claude just rendered this diagram and it works really well.
13:30Right?
13:31But here, if I put myself in the shoes of a company, I'm not sure I would delegate fully to the LLM.
13:37Because I cannot control at least the app the output and the outcome.
13:42And me personally, me guys, as a UX leader, the UX side of me will always say no. You got to be in control.
13:49There's been a couple of talks here at least this week on design, on taste, and judgment.
13:56And this matters a lot.
13:57If you guys want to embark on this journey of leveraging these protocols, you don't want to delegate too much of the actual experience to the LLM.
14:05You got to find alternatives and I'm going to touch on that in just a little bit.
14:10Okay, cool.
14:11And then this is how the open-ended approach works.
14:15So essentially there there's going to be an MCP tool and then this will literally ship the HTML and then in a sandbox I frame environment, this will be rendered in the host of your choice.
14:30But it can be a chat.
14:31It can be this cloud.
14:33It could be a chat GPT.
14:34It could be perplexity or it could be any other chat.
14:37If you're willing just to to give full control to the LLM, good luck.
14:42But the one that I would like to highlight is this here and this was our choice that we call the declarative.
14:50Declarative is in the middle.
14:52If you guys heard some of the protocols and I don't want to get into the specifics of each because they have different characteristics.
15:00But HTMX from Google, JSON render from Vercel, OpenUI by Thesis, those are some of the protocols that will give you this in between here.
15:11And let let's have a look, right?
15:13So at least for my product, the one that I just showed, the orchestrator agent will eventually, if you think of the whole traversal, the user will enter the query
15:25and then there's going to be the intent classification.
15:27Based off of the intent classification, then the tools will be invoked, the data will be retrieved, and somewhat somewhat in between Alice, there will be the mapping of the eligible components from your catalog
15:41to the entities of of the tools, right?
15:45And then the orchestrator will just broadcast this UI description.
15:49It's like a UI spec.
15:50This UI spec will be also we have this component catalog here and we use the Zod schema and then you got to be compliant with this protocols.
15:58This is just one of the requirements and then you just render that.
16:02Right?
16:03And then their final output will be the native UI, in this case the React components.
16:09This is it.
16:10The good thing about the declarative approach is it will be compliant with your design system everywhere.
16:18This matters a lot.
16:19So, in our case, we did not want to delegate to the LLM because you guys saw over there you can't change the copy, right?
16:28So, it's not key one.
16:29Sometimes it's going to be March January to March.
16:33It matters a lot.
16:34So, within UX we have different factors, right?
16:37We have the actual UX's if you think of the overall experience.
16:41There is UI, there's also copy, UX writing, and so on and so forth.
16:46But this approach gives us this in between.
16:49It is less deterministic and I think this is a really good segue to some of the challenges because imagine, I'm going to use my example once again, the orchestrator
16:58will just fetch the eligible components for that query, but then it's up to the LLM how to place in the UI.
17:07And this can get really really messy, but and those are the challenges that I'll like to share with you guys here.
17:14I got to be careful because I've got only 30 minutes, but let's go.
17:18So, the first challenge is if the agent picks the components, who's in charge or what entity is in charge of arranging them?
17:26And this is information architecture, right?
17:29This is a critical aspect of UX.
17:32And then once again, if you put yourself in the shoes of the average customer, it matters a lot.
17:37So, if you're just left alone, the placement can be totally random.
17:41So, at least in my team, we borrow this concept of atomic design.
17:47And atomic design, it goes like this.
17:50Let me just change here.
17:51Uh yeah.
17:52So, as you can see, atomic design, and then I have the definition, is a methodology composed of five distinct stages working together to create interface design systems in a more deliberate
18:05and hierarchical manner.
18:06This helps a lot.
18:07Why?
18:07Because those are the individual elements, and if you think of the overall structure of the page, it gives me the ability to steer it as I see fit.
18:16And what we've done in my team, so this UX agent, we harnessed this UX agent.
18:22So, we eventually taught this this UX agent what good looks like, what is the optimal layout for a given situation, and we have a catalog of different templates.
18:35But I'll like to show you at least this because this detail is very important.
18:41Here, it is the overall hierarchy hierarchy.
18:45So, if you remember part of the orchestrator, the orchestrator will eventually just fetch the eligible components to accomplish the query of the user, but then the next big question is, how do we arrange that?
18:58The approach that we used was think of this hierarchy.
19:01So, you have the overall page, the layout.
19:05The layout will contain different slots.
19:08So, think of this one here, the header.
19:10You can have the main, and then you can have sub sub slots.
19:14Sub slots can have sub slots.
19:16And within the sub slots, you can have eligible component categories.
19:20And then this will allow us to steer eventually the the the optimal placement of the components that have been retrieved by the orchestrator.
19:31So this is literally us codifying our UX knowledge into this agent.
19:38So the next time, you know, it doesn't matter.
19:40We'll just follow the same approach.
19:42And then this is the hierarchy that we're using.
19:44So layout to slot to sub slot to components.
19:48However, because of the orchestrator, we we flipped the order.
19:52So from components, components will map to sub slot, sub slots to slots, and slots to templates.
19:58And then we can just arrange as needed.
20:02So this was a hell of a challenge.
20:04It is still a challenge, by the way, and it's a really good segue to the second one, the second challenge, which is the actual design, your design system and your catalog.
20:16This becomes the heartbeat of the whole thing, right?
20:19I cannot stress enough, if you guys see the potential of leveraging this UI protocols for your product, this is going to be a big deal, right?
20:27So we're pushing the boundaries and then we're testing and we keep on testing these different protocols that I mentioned, ATUI, JSON Render, Open UI, and so on and so forth.
20:38But this has been a quite challenging because the the catalog is the contract between the agent and the UI.
20:45So every property mat- matters.
20:48And not only for the catalog, but for the layout as well.
20:51So if you remember, the layout has its own components, the slots and the sub slots.
20:57Each of those components will have its own attributes.
21:01And all of that, this curation, let me just encapsulate into curation.
21:05This curation is absolutely needed so that you can deliver something meaningful.
21:10Not as some demo that you will see out there for the sake of demo, right?
21:14So, this this this will give you a control will allow you to steer from UX perspective.
21:20Very good.
21:21And last but not least, one significant challenge that we had is that my teams do not design the pixel anymore.
21:29I don't know if you could see that, right?
21:31So, we're not here designing at the entire flow.
21:34Now, AI can dictate that to a significant extent, but the nature of the work shifted quite a bit and it's been an interesting journey to say the least, a really good one, but even for the non-technical
21:47PMs and UX designers, it it was a big hit because right now we talk about the schema.
21:54Let's talk about this curation of the catalog.
21:57Let's talk about the rules.
21:58Let's talk about the synthetic data that we can generate.
22:02How can we generate the queries that will map to a given component as part of this mapping logic?
22:08Let's talk about interaction patterns.
22:11So, this if once again, if you guys are going to embark on this journey, be aware that the people element is very important.
22:18When I talk to other leaders, I I talk about the three P's: people, product, and process, right?
22:24And then a very lightweight process, but this matters a lot.
22:28And with that, I'm going to leave a couple of resources here.
22:33So, and by the way, those are talks from AIE for what it's worth.
22:38So, yeah, people that have been talking about this protocols over and over and over.
22:43So, please just take advantage, take a screenshot.
22:46And if you guys want to connect with me here, yeah, my LinkedIn or just take a screenshot.
22:52I'll love to talk more about the topic.
22:56I can tell this is just a matter of time, right?
22:58So, this is coming.
22:58So, thank you so much.
23:02I know.
23:15[music]