Keyword Search Is Dying. Is Your Catalog Ready for AI Agents? — PayPal

AI Engineer · 16 min · 133 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hi everyone.
00:13Um, welcome.
00:14I I'm Nixon Den.
00:15I'm the director of product for Aenticcommerce at PayPal.
00:19And what that really means is I lead a lot of our efforts on how AI agents are reshaping shopping and payments.
00:27Um, and you know, most of my career has been on commerce and payments.
00:31And right now, I'm really excited about helping merchants navigate this shift uh towards like agent-driven commerce.
00:37And specifically, what we're going to talk about today is um the catalog piece and it being agentic ready and sharing some of our learnings on this on this journey.
00:50So I think it's very important we just kind of level set a little bit when we look at the evolution of commerce.
00:56We've seen it transform from you install brick and mortar transactions to the advent of e-commerce where this started off quite linear.
01:05Um a user would simply search and buy from e-commerce sites and then through like internet 2.0 O we've seen that transform further into more complex uh channels that involve email, SMS and like now social commerce
01:19for us like we see AI as the next evolution in that journey and with any evolution we know two things to be true.
01:28The first like all change it can seem like a threat to a business.
01:32Um but on the flip side of that can also be seen as a big opportunity.
01:37And the second thing that we know is that whenever we have seen consumer behavior change because of a new technology, commerce tends to evolve alongside with it.
01:48And so we are starting to see that now with Aentic where not only is the new this is a new channel that's enabling uh merchants to engage with customers in a very deeply personalized
01:59way, but also how customers are searching and finding products is changing.
02:06And uh now is really the time that merchants are starting to look to invest on how to to change with with their customers and and uh succeed in the agentic era.
02:18So earlier I touched on how the consumer behavior is starting to change.
02:26Um but I just want to dive deeper a little bit on this.
02:29So we, you know, when we first uh when we think about consumer shopping, that behavior up to today has been now what we're calling the search era.
02:40Um it goes through a phase where like it's keywords, you're navigating a catalog, there's SEO strategies, and this is what most businesses have optimized towards over the last decade.
02:53And while for many businesses this worked, this model put the burden on the shopper to really find what they needed.
03:02Um, now what we're seeing is that we're moving into this phase called the intent error.
03:06So instead of typing keywords like I want school supplies, um, I might say something like, you know, my daughter's starting school in September in fifth grade and I need uh to to shop back to school.
03:18And that's a very completely different interaction.
03:22Um, so you're now expressing your needs in your own words and AI is now asked to meet you where you are.
03:32And so the brands are coming to you instead of you having to go find them.
03:36And that's a very very different interaction model.
03:40Um, and most businesses now are starting to find that they need to set themselves up for this interaction model.
03:47Um so what we believe will happen next is after we go through the intent error what will have to happen is customers need to start trusting the recommendations that they're getting from the from the agents
04:03and once that trust starts to build we do start to believe that they will start delegating more and more actions to their agents and that's also where as PayPal we're building towards is
04:13um continue to build on that trust and moving to a world where humans start to delegate more and more of their task in commerce and shopping to their agents.
04:26So although we talk about this delegation era and we think it sounds like it's so far into the future, you might be asking like why this matters now.
04:35And so we have couple pretty big data points that we want to share to really underscore why this matters.
04:41The first is uh through a study at Bay & Co.
04:46um they project that by 2030 15 to 25% of all commerce will be executed by an agent and furthermore within a decade agents are predicted to outnumber humans online
04:58and so these numbers don't just indicate that this is like not meaningful now it is a very marginal shift that businesses can't ignore and that this is a very structural transformation
05:12happening entire entire e-commerce ecosystem So to go further I think it's very good to understand why PayPal feels like we are bullish on this opportunity.
05:27Um as a industry we expect uh to be some very tangible opportunities coming forward.
05:35We expect 1.1 trillion dollars of uh commerce to happen uh in US retail to be agentic and then we expect that to be 3 to 5 trillion globally and what we've have seen is that there are over two billion users
05:49um projected to use like AI to start their shopping journey over the next few years.
05:57And so when we think through this, what does this actually mean for businesses and merchants?
06:07When we talk to our customers and what we talk to our partners, what we've seen is six they what they've reported is that over 693% uh referral traffic from AI engines.
06:18And when we did consumer studies, we found that 39% of US shoppers are already using AI for shopping today.
06:25Um, and I'm sure many of you have probably defaulted to that pattern today.
06:30And what we have found also is that when you do start and interact with an agent first, businesses have tend to see up to four times higher conversion on a completion of an order because now they can get more rich and in-depth data
06:43on a product.
06:50And so with that all happening and the traffic moving, we want to break this down further onto like what this really means ac deeper for merchants.
07:03In the search era, this was very like SEO based and you might optimize for like a sum dress.
07:11In the intent era, you have to think through how do I be discoverable for where an agent is representing my my business.
07:20And then in the delegation era, we need to think through uh how I can convert that demand safely and trusted in a trusted way.
07:31And so the challenge becomes to think through is like as COVID moves to AI agents, how do you as a business be visible where the consumers interact?
07:45The issue with this is that cataloges weren't really built for agents.
07:49Um most merchants today maybe advertise on Google.
07:53Um they have a catalog they syndicate to there or maybe to Meta or Facebook or other advertising platforms.
08:00But the spec wasn't built for agents.
08:02And so agents now um you know if you it's not optimized for agents to discover products.
08:10Um it's not as readable for agents as well.
08:13And they're really just built for humans search.
08:18And then to compound that even more, the models that agents use are still for the most part a black box.
08:25Agents can use different varying models to make their decisions.
08:30Um, different models use different ranking and recommendations and and and uh algorithms to to determine their recommendation.
08:38Uh, vector database retrieval is also not always deterministic.
08:42And because of all of this, how do you decide as a business what model do you do to you optimize for?
08:49In the old SEO world, there was a clear strategy.
08:51In this new world, the strategies are still remain largely unclear.
08:58And so what do you do when you don't know?
09:01You run experiments.
09:03And so we we ran some experiments.
09:05We enriched uh some of our merchant product data.
09:08And we ran experiments to see how agents uh uh searched uh and and recommended products.
09:18And before I get into the experiments that we ran, um I think very good to level set with everyone how like the differences between keyword and semantic search and it it trickles through to the themes I talked through earlier
09:31with keyword search being you know very matching the the search that the user is entering in.
09:38So, in a classic search, it might match literal words.
09:42Um, for example, I'm searching for blue running shoes, and you only get results that contain those words.
09:49The good part about this, it's very precise, it's fast, it's predictable.
09:53But the downside is it misses intent.
09:57And so, if you ever search footwear for jogging, it may not even return those shoes.
10:03Uh, and that's only simply because the words don't match.
10:06Whereas with semantic search, it matches based off meaning.
10:10Um, what it does is it converts the query into vectors that create uh meanings of intent.
10:17And so you might type something, oh, I'm searching for something comfortable for marathon training.
10:22And you might find your running shoes even though none of those words exist in the search query.
10:28Um so the good part about the that is it broadens and how it can search but the downside is if you inject too much meaning it can lose focus and hallucinate.
10:41So there is a balance of how much you want to enrich and how much you want to inject into your enrichment.
10:46So to sum it up keyword search is quite precise.
10:49It's it's very literal.
10:50Semantic search is just it's smart but it's very fuzzy.
10:55And realistically, the real answer isn't an either or.
10:58It's it's combining both, which is exactly what our hybrid approach does.
11:03Um, and that's kind of where we're headed next.
11:08So, now that we understand what semantic search is, it's important to highlight that there is no singular way to really enrich product data for semantic search.
11:17I think it's really important for you as a business to understand the kind kind of catalog data you already have.
11:25um do you have a thin catalog?
11:27Do you have a catalog of products that are more spec heavy?
11:30Depending on where your business is today, you should think through what you might want to enrich.
11:36And so for the experiment though, we thought through different patterns of enrichment.
11:40Um and I won't go through all of them in detail, but things around like filling attributes, making sure there's description depth, um make sure there's buyer context and trust signals, and even some product identity.
11:51I think the main take away from this slide is that enrichment isn't one sizefits-all.
11:56The right approach really depends on the catalog you're starting with and that's the framing we'll build on next.
12:03And so here's just an example of enrichment in action for us.
12:08Um example on we're starting with we have men's uh men's shoe model 8543 that's blue and then the description will be I have blue running shoes.
12:20um available multiple sizes.
12:22there's an attribute that's blue keyword matches maybe blue running or shoe and then um the semant for semantic there's you know very little context for that when you enrich
12:34there now we might have the title as a men's lightweight running shoe um the description is now more enriched like it's cushion it's breathable uh there are further attributes we've pumped into that as well so there's like true to size it's it's the the material is mesh
12:48um we also put in some reviews here as well so enrich that with some review data and now what the search sees is um it matches running, it matches training, it matches meshes.
12:59There's a lot more that it matches.
13:01So with semantic there's a lot of rich meaning that's put there uh to kind of enhance the search.
13:08And so here the the some of the examples we use was attribute filling or bio context or even uh the trust signals around the reviews as well.
13:20And so ultimately at the end of the experiment um you know our thesis was generally speaking does enrichment help boost uh recommendations and at its core yes it does.
13:33Um what we saw was an unenriched baseline like when it went headto-head um it just completely underperformed.
13:41We found that merchants with the thinnest cataloges tend to gain the most.
13:45Uh merchants with weak product data also had the most room to improve.
13:50Um but I think what we needed to also understand is that it's all about content quality that drives the results and it's not the schema that actually and actually piling on you unstructured data can actually dilute the signal as well.
14:06And so what do we mean by like unstructured data?
14:09like we just don't want to stuff the the context of anything any words we want.
14:14We don't want to put any boilerplate messaging around um potential like you know welcome to our store.
14:20We're familyowned business.
14:21Those things are irrelevant to an agent at times.
14:23And so how you enrich what you structure into the data really really matters as well.
14:31And so just generally speaking a sum up of our learnings was enrichment helps keyword search.
14:39Semantic retrieval requires a very different set of data structures.
14:44But there is a balance to how much you want to enrich uh to improve semantic retrieval.
14:50And while this was only one set of experiments, we did find that enriching just yeah head-to-head always improved recommendations.
14:59Um what we had found in some cases when we overenriched the agent tended to hallucinate more and it underperformed.
15:05Um and we don't have a concrete recommendation for every business but a good place to start is just assess your catalog and then figure out the right strategy for that.
15:16Um and so just to wrap up you know enrichment works assess your catalog structure and more text isn't better.
15:24um structured quality content, structured quality content is the one thing that wins out and AI will reward content quality and not brands.
15:33Um but it does optimize for other trust signals as well.
15:41So thank you everyone.
15:42Uh if you want to learn more about PayPal's Agent Commerce Services, please visit our booth at P11.