The Missing Layer in Agentic AI — Giedrius Šteimantas, Oxylabs

AI Engineer · 15 min · 138 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:13What a beautiful voice.
00:16All right, thank you for coming.
00:18Um, today I'm going to talk a lot about about missing layer of Aentic AI and explain a little bit about how web scraping infrastructure can actually help you.
00:27But first, let me talk uh a little bit about my friend's idea.
00:32So, my friend had this idea.
00:34Uh he built this AI chatbot that, you know, chatted with people about their style and it was supposed to help them pick out new items uh as you know, some sort of a personal shopper.
00:48And once those items were picked out, you know, this uh this this this chatbot would uh produce prompts that a shopping agent would then take and attempt to find them online
01:00and purchase them for uh you know for for for the customers.
01:04Um this idea I know is not new and uh it could be applicable to many scenarios but my friend was kind of you know uh he was u he was good at building agents uh but u he ran into
01:15different problems and asked me for advice and when he ran it he he would usually you know instead of you know product pages or whatever he would get things like that
01:27it's uh you know he would get captured you know and uh you know of course you So he was uh he was doing it very very quickly.
01:35So he wipe coded the whole thing while having a you know a thought about you know infrastructure and underlying layers and how it should work.
01:43I didn't at all.
01:45Uh he was using a browser automation framework for everything and it was slow, expensive and unreliable.
01:55So at the end he made a product that uh uh that does not work and is expensive to run.
02:03So he asked me for help and you know I was a little bit reluctant at first because uh you know I don't like giving out professional advice you know for free
02:12but uh I took a look at it and uh you know I got a little curious I have to be honest.
02:18I noticed that he was missing something.
02:22Um he was missing a layer an infrastructural layer that would allow this agent to operate freely on the open web.
02:32My name is Gedrus.
02:33I I work for Oxyabs uh where in the past 10 years we've helped you know companies that trained large language models uh get their data and now we use this infrastructure to help AI agents
02:49to access uh web on scale and at low cost.
02:56And uh before we go into this agent and see how we can build it, I wanted to talk a little bit about the scraping industry and how we operate.
03:04And uh the principles that we operate on can be summed up by one uh sentence.
03:12You know, cost matters.
03:15And the first principle is use a browser when you absolutely have to validate content.
03:23HTTP response 200 does not mean that we are good to go.
03:28Lighter content is preferred.
03:30Websites are full of JavaScript, CSS, HTML, and there's a lot of bites that do not deliver any value whatsoever.
03:40And today I will demonstrate how these principles are also applicable when building agents that interact with the web.
03:49So coming back to my friend's agent, right?
03:51Let's uh let's take a look and see how uh we could do a better job and uh making this agent run more reliably.
03:58So here's how my friends set it all up, you know?
04:01So four different stages.
04:03Discovery, the agent was supposed to find products pages on websites where these items can be bought.
04:11Then a decision stage, right?
04:12and uh where an agent can decide uh what products to buy based on you know uh the the content of these pages.
04:21So the agent has to visit them verify that the the stock is there the price is right the the description fits uh you know the prompt and once that
04:31decision is made user is given with a choice you know whether to go ahead with the purchase or you know reject it altogether.
04:40The problem was that sometimes and of course we go to execution right away then execution just making the purchase but the problem was that sometimes it worked and sometimes it did not
04:53that was a little problematic.
04:55So let's dissect it step by step and see how we could build this differently while improving performance and reducing the cost dramatically by using the same principles from the scraping industry.
05:09So the first stage discovery.
05:12So my friend uh you know he chose to go with a predefined list of websites major retailers uh and query their search pages in order to find these products.
05:23He used the browser automation tool for that.
05:26It kind of worked but you know it did have challenges.
05:29So their browser automation tool lacked what we call stealth.
05:34So they could so they would get captures and sometimes fail access to access the sites.
05:38all together.
05:38This would break down the flow.
05:41So a retry mechanism would have to be put in place making the whole process very long.
05:46Uh you know costly um and sometimes the size would not be uh accessed at all and also you know as a result also became very difficult to predict the final cost per transaction.
06:01The list of websites that my friend was checking was also deterministic.
06:05So selection of items would only be limited to the few choices he put in.
06:12Websites themselves were heavy on JavaScript, making the whole process very slow and costly.
06:18And finally, even if it worked, items ended up being unavailable at checkout because in the discovery phase, the he was not able to use energy location capabilities and a lot of e-commerce
06:32websites are uh you know uh they take your users location into account when displaying stock options sizes and soever.
06:45So now we solve these problems at Oxabs every day.
06:48So when scraping you always want the results to appear on the first try and to not to use browser unless absolutely necessary.
06:58However, for this specific discovery phase, you also want to use to allow your agent to search the web.
07:04Doing so with a browser is very cumbersome.
07:07That is why I chose to use a product that we built especially for agents fast search API.
07:14It returns a compact JSON which is less than 2,000 tokens per response.
07:19Has fast response times less than 700 milliseconds on average.
07:24And it's uh has a a high success rate at a predictable low price.
07:30And most importantly, it gives your agent access to the mo to you know to many popular search engines that all of these websites have been instant indexed already a long time ago.
07:42So in the discovery phase instead of predefined list and the browser we give agent a tool to search the web fast search API agent formulates fan out queries and selects the relevant URLs from search results.
07:55Since the responses are quite small and there's no need for complicated models we can have the agent run quite quickly in this stage.
08:05Um, yeah.
08:07So, so now the agent has searched the web and selected some relevant URLs.
08:13It is time for those for for the agent to visit those pages to see what they're all about in order to confirm price, stock level, description, and product details and so on.
08:25With this, we can go to in the decision phase.
08:28This is where agent selects the items we will purchase.
08:32For this, my friend also used the browser.
08:34He ran many browsers on parallel so it could uh you know so the whole process could happen faster and that is not a bad thing.
08:42He managed to get some results however many of the results would end up like this and the result the agent would be left with very few choices with the majority of popular retailers
08:58being left out.
09:00It's a good thing he did well with observability.
09:03So he actually noticed when it happened.
09:05But what we see when working with these types of customers is that they often fail to detect the failure.
09:13They end up checking only the content size and HTTP response code and then feeding this large HTML to an LLM.
09:21Now an a large language model of course can distinguish between valid esop content and a capture.
09:27But we need to spend tokens in order to do that.
09:30And when we attempt to open 10 websites, but only three return valid content but feed all of the 10 to the to the model, it is a problem.
09:44It means that we waste 70% of the tokens and that is a little crazy in my in my opinion.
09:53So I noticed this problem as well.
09:55Uh my initial hunch was compression was to compress the output.
10:00But then I thought wait the problem is not the compression.
10:04The problem is that the content is not valid.
10:07We need to make sure that the content is valid before even attempting any compression.
10:12This will lead to more options for the agent to choose from and fewer wasted tokens.
10:18And then I remember rule number one of scraping.
10:21Use the browser when you absolutely need it.
10:26Otherwise look for other solutions.
10:29So I I tried to rebuild the stage without a browser and I uh only by using ox web scraper API and this gave me many benefits.
10:39Uh but firstly only valid content was returned.
10:43In case of captures or other blocks the request would fail with an explicit error message.
10:48So I know not to include it when sending to a large language model.
10:51But the success rates are quite high and even for protected websites.
10:56So that wasn't that much of you know much of a problem.
11:00So no browser was needed and uh everything is a lightweight rest API.
11:04I can run hundreds of requests in parallel and receive content at the same time.
11:11Also the API supports markdown.
11:13So no need to submit raw HTML uh to LLMs.
11:18If a website is dynamic, it runs a full browser under the hood to render the content correctly.
11:25And finally, it supports geoloccation options.
11:28So I can localize my results and get relevant content.
11:33The best part, customers only pay for successful results.
11:39So actually, yeah, that's uh that's what's uh that's what that's what the best thing about it.
11:45No cure or no pay.
11:47If if the scraper fails, there's no cost and it fails loudly.
11:54So now we have all of the information to make a decision.
11:58We present a decision to the user and the user makes the final call.
12:03Once it's affirmative, we move to the last stage of the workflow, the purchase.
12:09So I remember what I said a couple of times about browsers.
12:13This time, but this time is different.
12:15you this time you absolutely need to use a browser.
12:20We need to process inputs and the content is highly dynamic.
12:24Now this time my implementation, my friend's implementation does not differ much.
12:29We both use playright MCP with a browser and a large language model.
12:37The main problem my friend faced however just like in in the previous stages while using browser was access.
12:45Just like in the beginning as he was using the browser he was getting captured into oblivion making it impossible to automate the flow.
12:55Well the fix was quite easy.
12:58I just connected Oxab's headless browser since it supports playright MCP is just a drop in replacement.
13:04With this replacement, I hardened this agent with years of scraping experience and got proper stealth done at the browser source code level, a residential proxy attached to it out of the box,
13:18and most importantly in this in this case, a geoloccation capability.
13:24So my results are localized the same way as in the verification stage.
13:31So if we run it, we actually have a a a a browser that that access the content and can actually automate the flow by, you know, selecting the right size from the prompt,
13:45add it to cart and complete the purchase.
13:49And boom, we have an agent that commands a powerful infrastructure hardened by years of web scraping experience.
14:00Not only does it open the up the web, but also saves the time on implementation and token cost.
14:08And if I can leave you with a few lessons we learned today was that you know when building agents use the same principles from the scraping industry.
14:20Use the browser when you absolutely need to.
14:24You have to validate content before feeding it to the large language models and most importantly fill the missing layer with the proper infrastructure so you can focus on building stuff.
14:38But remember cost matters.
14:42Thank you very much.
14:44[applause]