Building AI Agents with Real-Time Web Data — Yohan Raju, Bright Data

AI Engineer · 27 min · 226 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:13All right, let's begin.
00:14So, nice to meet everyone.
00:15My name is Johan Raju.
00:17I'm a solutions architect here at Bright Data.
00:19I've been with the company for about six years, and I've been working in this space of data collection and web scraping.
00:26We initially started off as a proxy infrastructure company uh with the largest residential IP network in the world and then we moved into the data collection platform.
00:37Now we're the world's largest web data collection platform collecting pabytes of data per month only second to Google.
00:44So just a little information for you guys on you know where we started and and how how we moved over to today.
00:51So I'm going to be covering two topics.
00:53First is I'm going to be speaking about building AI agents with real-time web data.
00:58Some tools that people are generally using when it comes to you know AI agents versus depending on your rag model with pre-index data and what are the advantages what we see in the industry and you know what changes
01:11uh are to come and what it takes to uh run these agents.
01:14And then finally, I'm going to have my hands-on exercise that you guys can follow along with me.
01:19Uh, where I have a product intelligence agent that will essentially use a product from Amazon, get sentiment on Reddit, and then provide you analysis with, you know, should I buy that product?
01:30Yes or no. So working in the scraping space for a long period of time, one thing I've noticed is, you know, when you think about building scraping or collecting data,
01:47what does it really take?
01:48Right now, the early customers in the scraping space that I've worked with, you know, when when you first start, it's always about, hey, I want these are my target sites and this is what I want to scrape, right?
01:58You know, people usually start off with low volume.
02:01you know, that's what comprises a general traditional scraping archite architecture.
02:07Now, let's say I'm an e-commerce uh user and I want to collect e-commerce data.
02:11You know, my biggest targets might be Amazon and Walmart because, you know, I can find a lot of products on there and I can get price intelligence and data on reviews and, you know, all the general information for a vast majority of products.
02:24Now, when does scraping become challenging?
02:28And this is a change that I've actually seen with customers that I work with today um occurring in the industry when it's not just about Amazon and Walmart anymore.
02:38Now you want Shopppee.
02:39Now you want Shopify stores.
02:41Now you want um Etsy and you know a bunch of other sites.
02:45Now if we take a step back and I mentioned you know you know your target sites in order to build scrapers around your target sites it takes generally about you know 3 to 5 days to develop a scraper if you're a developer and that's not only developing it if it's QAing it running it in development testing it
03:02out and you know the whole endto-end pipeline now if we think about you know 10 sites okay might take a bit of time now you think okay I want to add a 100 sites a thousand sites this is where things start getting complicated.
03:16Now before AI even came into the picture, we would developers would have to manually understand the site, you know, understand where the selectors are located.
03:25Can I target these data points in the API and actually build crawlers around that knowledge and I've worked with many people, you know, that's that have been scraping for
03:3510 plus years that have been managing and maintaining these crawlers.
03:39But when it comes to scaling up, then you're thinking, okay, like now I'm adding more sites to what I already am scraping, but I also need to manage the scrapers that I already have.
03:49So let's say Amazon makes a change tomorrow and two other sites make a change tomorrow, but you're managing a thousand sites.
03:56This requires a lot more engineering resources and a lot more time for you to actually figure out, okay, what change was made?
04:02How can I count counteract it?
04:04How can I fix it?
04:05And is this actually impacting my customers, my SLA?
04:08and my data quality.
04:10Now before we had this bottleneck of having to scale up your resources and when at those times when you would have to manage the thousand sites you'd need you know two times more engineers, four times more engineers etc. But now in this AI space,
04:25none of that is needed because we have things like agents and agents can actually go out and discover and build sites, build scrapers for these sites a lot easier than, you know, team of developers and engineers.
04:39Now, this is one classic example of an e-commerce site of targeting selectors on the page.
04:45So, you can see here we're just in this example, it's just targeting price and title.
04:50Now, what happens if the price has changed?
04:52the selector for the prices changed, you know, tomorrow.
04:55That will require some downtime on your end because, you know, if you're the one scraping the site because, you know, now you're not getting price and that can affect the rest of your pipeline that you have to maybe analyze products, you know, etc.
05:08Now, usually that require like a team to go in, fix it, then maybe back fill on the data that you know you've lost.
05:15But now with AI agents that can be automated and done automatically.
05:20So when you also build scrapers, one thing to always know is every website is different.
05:28Now sites in the example I use like Amazon and Walmart, they have two different layouts even though they both might have things like price, title, description.
05:37So each site requires a unique scraper to build build for and the even more important than that is each site wants to protect their data.
05:49Now sites usually like to ensure that no one else is uh you know has enough like the same product intelligence they have or you know depending on what type of data that they're you know trying to keep secure
06:02they will always implement different types of anti-bot and different types of capture to throw you know scrapers and bots off from collecting that data.
06:11Now I've seen all different types of blocks.
06:14I've seen, you know, the most basic sites that you can get through with data center.
06:18And then I've seen sites that, you know, are protected by Cloudflare, Okami, you know, you name it, that you actually need to have a residential IP address.
06:26You need to have a real browser.
06:27And you need to seem like a real user.
06:29You know, if you click too fast on the page, if you scroll too fast, that could all trigger anti-bot if you're blocking images, things like that.
06:36Now, like I mentioned before, engineering has been a bottleneck until the AI space came here when we we had the AI agents ready like you know able to build crawlers
06:46and you'd have to actually have a deep understanding of the site you know um a lot of sites like I mentioned because the layout is very different it means that
06:56they'll try to make sure the data is as hard as possible to collect for you and you would have to have a deep level of domain knowledge expertise to get to that data.
07:05data.
07:07Now this is a diagram of how the traditional scraping architecture look like.
07:12So just to briefly go over this we have on the left hand side the you know the a the actor essentially what's triggering this crawl.
07:22So some people like using like schedulers you know um cron jobs you name it where they want the data fed to them daily weekly etc. And that kicks off the job directly over to the next layer.
07:34Some people like doing, you know, specific config, specific URLs.
07:38Same uh same idea right there.
07:40It goes and that goes right over to the proxy layer.
07:43Now, the proxy is essentially how you mask yourself from being detected.
07:48There's different types of proxies that you can use.
07:51you know, some static IP addresses like data center ISP, some rotating residential IP addresses that come from like laptops, tablets, TVs that are more authentic to sites because they can say, "Okay, this is registered to an actual device, we'll let them in."
08:06And then some tools that have anti-bot capabilities built in like what bright data offers web unlocker or browser API which is in the browser automation side of of of it
08:16where the the actual product itself manages okay is my request getting blocked am I going to retry am I using the right IP that's all in a part of the layer when you think about the anti-bot side of things
08:29now from there then the next layer is you know parsing the data and actually validating if the data is, you know, authentic.
08:37That's a big part of, you know, after you actually get to the site and collect the data, am I getting price back 90% of the time and do I want to validate that I'm getting it back 90% of the time?
08:48Do I need review data?
08:50Do I need ratings?
08:51You know, things like that.
08:52You have to think about, okay, like how do I ensure that I'm not going to be setting up when the data goes through my pipeline that it's going to be checked accurately by validation rules.
09:04Then we go into the post-processing side of like the ingestion layer.
09:07So you know dduping records enriching your data pro processing your batches that is once you you know some people like delivering it to cloud stoages some people like delivering it to their internal systems once it gets there you know what are you doing with the data how do you ensure that you're not providing the same record twice
09:24or you know unclean data.
09:26So you know moving it from in the medallion architecture bronze silver and then finally moving it over to your database and if anything fails in that process then you need to start the whole
09:39uh process over and this is where I was mentioning you know if something breaks in the antibbots layer or if it breaks and you know selector has changed you need to start the whole process over sometimes backfilling
09:51you know maybe you might miss SLAs's and things like that And so here I'm going to be talking about now specifically AI.
10:00Now how many of you guys today are using things like you know claw GPT any of the uh actual like AI brains?
10:09Okay, cool.
10:10How many of you guys have entered a query where it's it's you see it says thinking or looking for an answer or searching the web?
10:20Okay, cool.
10:20Nice.
10:21So this is where you see and I've split it up into two where the left hand side is where you see okay when the AI is actually thinking or searching the web
10:31that's where it decides okay the query that you gave me I actually need to go out and find the answer to this I can't use my pre-trained data so some prompts like I'll give a basic example
10:42when's the best time to travel to San Francisco so if I say that sometimes the the uh AI brain will usually go over to the rag side, the async where it'll
10:54be used to pre-train data that you know, hey, maybe the best time to travel is during the summer, and it'll actually provide that back to you.
11:01Now, if I change my query and say, when's the best time to travel this year to San Francisco, then the AI the AI has to actually go out into the web and see, okay, like do I need to look at what's going on?
11:15You know, is my user asking me for events that are going on?
11:18you know, how do I like I need to make a decision this year real time to find out when I can recommend for them to travel.
11:24So, these are two different paths that um each a the each AI can actually make and provide you an output with.
11:33Some some things I noticed is like responses sometimes can be cached.
11:38Like you know if someone's asking what's the weather today in in New York City like a few times a day that might be a good response to cash because
11:46you only need one person to really ask that and you can you know what the weather is going to be for the day and you can just provide it, you know, so you don't have to burn through tokens or
11:56you know um uh like money to actually go out in the web and and actually find that data.
12:05Now like I mentioned there's like two paths to uh gather this intelligence.
12:10The sync path and then the async path.
12:13Both are advantageous for different cases.
12:16The sync path is you know if you really actually want to go out on the web and actually find what you're looking for and async is if you know it's okay to rely on pre-trained data.
12:26And both require different sort of setups.
12:30And we also do offer a tool which I'm going to cover in the next slide um where you can actually use leverage Bright Data's tools to actually go out into the web and collect data.
12:40Now one thing I actually noticed is if you go into any website and if any of you guys are familiar with like robots.txt, a lot of sites usually block things like Claudebot
12:51and GPTbot.
12:52So if you ask these engines these questions that are actually going out to the web and trying to find the data, they might not be able to get those data from those specific sites because they're actually blocking those you know GPT or claude from actually accessing the data.
13:06Now that's where we sort of come in and because what our advantageous is advantage is we're able to actually unlock the web.
13:15You know whether it's if you have a scraper or if you have an agent or you're using um you know something like cloud or GPT we have tools that will allow it to actually go out into the web without actually getting blocked and scraping data.
13:30And we've been masters at this for for a long time now.
13:34I'm going to be talking about the the web model context protocol.
13:38That's what MCP means.
13:40So it usually integrates with something like claude code, lang chain, you know, name it like you could think of it as like a brain that it needs to speak to
13:50and that brain essentially connects to our MCP endpoint which is essentially 60 plus APIs that we control in our library.
14:00And some examples you can see here are you know search engine.
14:03So, you know, Google results, scraping pages as markdown files, and scraping pages as HTML.
14:09Now, if I want to target specific sites like, you know, Amazon, Reddit, LinkedIn, Instagram, that's all included as well.
14:18I actually, as an example, I I worked with someone two weeks ago that, you know, was trying to build a social media listener with the MCP, and one use case they said was, "Hey,
14:30um, this is my name.
14:32I want to go out into the web and collect me all of my social media profiles and return me how many followers I have.
14:38So what this did was he was using it on cloud code.
14:41It made a call to our MCP using the search engine first found the results his pages on Instagram, Tik Tok, LinkedIn, brought that back, made the API request to our
14:54data pipeline and then brought back the results on how many followers you have and gave them the context back.
15:00So that's the power of you know what this can do and how much of this the web that the MCP can unlock.
15:06And also this works for browser automations.
15:08Um so you know creating things that you want to use a browser for like clicking scrolling you know maybe door dash like automating an order or um automating you know on a travel site trying to find you know what's the best times to
15:23buy a fight a flight book a flight.
15:26Um, so things like that it's very advantageous for.
15:30And I'm going to show you a little video on how the MCP works.
15:35Let me bring.
15:38So on the left hand side you can see this is clawed with bright data's MCP setup and on the right side this is without bright data's MCP setup on claude.
15:48So you can see it's the same prompt being asked and it's use your browser to get the contents of this page which is a glass door URL and on the left side
15:59you can see with bright data as MCP it actually went over to glass door and got the results but on the right hand side it actually got blocked from going into glass door.
16:10So this is one perfect example of you know sites like Glaster I've worked with on a on a data collection level that's like a very protected site and you know this is one great example on why the MCP is advantageous you know pairing it up with something like claw or you know GPT or any sort of brains
16:27that you know you'd want to run it on.
16:35Okay.
16:35So now I'm going to be speaking about an example use case.
16:40So working with AI companies, these are examples of you know customers that I speak with that use bright data in any form of way.
16:51So you know training models um fine-tune enriching their uh enriching their models or fine-tuning their models um providing AI agents real-time access like like we mentioned or enriching rags.
17:04So in this example, I'm going to be speaking about specifically jobs, right?
17:09So let's say someone came to me and said, "Hey, listen, I I need job data and I want to train a model with jobs data like maybe resume matching, you know, as an example."
17:20So one thing we need to know is when you're thinking about jobs data, it's not like other, you know, collecting other forms of data, you know, where it's I can pull in the data and then I can just forget about it or train a model.
17:32You need continuous updates about jobs.
17:35You know, which companies are posting what, you know, how many jobs are available, what's the salary, you know, location, etc. So, this is why it's important that the data is actually structured and has a consistent output.
17:47The actual performance is very key as well because you know if you're building a model to analyze like job patterns or you know anything in the job market you need that data flowing to you consistently
17:59and you need also need this to be very reliable because if something breaks or you know if there's downtime then that can actually affect the actual in the entire pipeline that you're you want to run
18:10and the timeline for this you know it's not only about getting data starting from today about jobs throughout.
18:18Some people need jobs data previously, you know, like the past few months or past few years.
18:24So that's why it's very important that you know, you're covering all of these because if you want to build a big picture on, you know, in this example for jobs
18:32on, you know, how do I find train my model on jobs data so I can find the best matches for this position.
18:39You need to understand how the trends changed.
18:41you know maybe salary has increased for software engineers over time and you need to understand why what the demand is what's the current you know offering and you know you can you build your model around that so
18:53this is like the example use case like an AI agent for resume matching so it's exactly what we discussed here um so immediately you need real-time access to actually
19:04go through and um you know scrape the data and be able to train this model along with it.
19:14Okay.
19:15So, I'm going to be covering a hands-on exercise here.
19:19So, I have a Google notebook here.
19:24And if any of you guys are following along, there's two things that you require to actually run this script.
19:31One is a bright data API token and one is a Gemini API key.
19:37And I have the list of instructions in the Google Drive on how to set up both.
19:41Um, and essentially I have everything set up here for you to run.
19:45So I'll speak a bit about what we're doing here.
19:47So first we're calling Bright Data's API endpoint to collect Amazon data.
19:53We're collecting the response from Amazon.
19:56Then calling the API for Reddit itself, capturing the raw Reddit data response and then performing AI analysis over that.
20:06Now before I run this, I want to show you the exact data that we're getting from each of these APIs.
20:13So Amazon, we're getting product level data.
20:17So I picked a Tumblr cup as an example and essentially we have every single data that about the product.
20:27So you know price, images, URLs, etc. And I I'll show you this once it runs as well.
20:34My screen's a little small, so it might not show the entire thing, but and we also have a Reddit API where what I'm going to do in this uh script that I built is I'll actually run a keyword
20:48search for that specific Tumblr cup.
20:53And I'm going to be outputting posts that, you know, talk about what what what's exactly, you know, is should I buy this or not?
21:02Is this worth it?
21:03Is this not worth it?
21:04What are people saying?
21:05So, let's run this live.
21:32All right.
21:33Right.
21:33So, we got our keys loaded here.
21:41All right.
21:41So, we got our Amazon data that came back.
21:43You can see it has price, rating, you know, other sellers that are selling it, all of the actual data from the product itself, features, you know, you name it.
22:00Now, we'll go ahead and trigger a Reddit search for the Stanley Quenter um Tumblr.
22:08Just having, you know, reviews about the actual product itself.
22:34And while this is running, I will show you a bit about what I spoke about the model context protocol, the MCP server.
22:41So we have different layers.
22:44I know I talked about things like you know browser automation and search engine but we also do cover things like you know e-commerce, social media, financial intelligence, research, business intelligence.
22:58So these are can all be plugged into an MCP endpoint you know app store travels and we are still adding to it.
23:07Currently we have about like 60 plus tools available to use for AI agents.
23:12Um, some people prefer actually using scrapers themselves, but we do this will provide allow your AI agent to choose which tools to use.
23:19So, it'll be a lot more streamlined and easier While this is running, I can take any questions if anyone has.
24:05Yeah.
24:05So the question was how come you guys can scrape resume level data but you know bots can't do it right.
24:12So what what gives bright data the advantage and this is why we're able to do it whereas like you know things like bots aren't able to is because we have the world's largest infrastructure like proxy infrastructure
24:24and where you're talking about you know a few requests maybe like you know maybe a few days worth of scraping pages like you know glass door as I use as an example
24:33might work right but let's say after a while glass door catches on like hey you know he's using a data center IP he's accessing the page you nothing's actually being done on the page and then they start adding all these layers of anti-bot
24:47into the picture.
24:48That's when companies like us bright data would come into the play because you need to actually not only own like a scraper or a bot, but you need to think about how do I bypass you know what the site is protected with the capture
25:01and how do I do it consistently without triggering the block.
25:06So, we've been doing it for years and we have tools that bake into this anti-bot logic where we're changing our approach to a site every time at scale, but
25:15sometimes um comp like people that are starting off scraping might be doing it in-house and managing it.
25:21You have to also think that's also engineering time and resource time because trying to figure out what change did that site make also takes time, even probably more time than managing the scraper.
25:35Okay, cool.
25:36Looks like it came back with data here.
25:39So, I can show you.
25:40So, it h it c captured a few Reddit posts about the actual Tumblr itself.
25:46So, some comments about, you know, what people were saying about the actual Tumblr.
25:54It looks like people were saying it was embarrassing and people were obsessed, but and then it gives the actual response here.
26:02So you could see actually buy it.
26:06It's on a decent discount.
26:08You know, there's overwhelmingly positive reviews, but a few minor mixed opinions.
26:13There's high demand, but low stock on Amazon because it returned that from the data.
26:18And you know, a few other analysis points about the actual Tumblr and the amount of reviews it has.
26:23It actually dissected all the data points that we returned from our API and gave you a decision on yes, should I buy it?
26:30Should I not buy it?
26:31Now this is just one example of you know using data from starting from Amazon running it on Reddit and then making a decision but you can think of it it think of this on like a bigger picture you know you can
26:42it doesn't have to be e-commerce it can be any sort of use case you know as long as we have like we have the tools that can allow you to go to the page and access the data.
26:50Now it's up to you what logic and how you want to build that pipeline and build it at scale as well because what one thing we do best is actually provide the data at scale.
27:02Cool.
27:02I think we're uh able to wrap up and if you guys um want to c if you have questions after I'll be available or you could stop by at the Bright Data booth.
27:10But thank you for your time guys.
27:26[music]