{ "video": { "id": "IK8cC4FScps", "title": "Building AI Agents with Real-Time Web Data — Yohan Raju, Bright Data", "duration": 1649, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "IK8cC4FScps", "title": "Building AI Agents with Real-Time Web Data — Yohan Raju, Bright Data", "one_liner": "Hand-built scrapers take roughly 3–5 days per site and break whenever selectors or anti-bot defences change, so Bright Data argues agents should reach the live web through its MCP server: 60+ tools that unlock sites like Glassdoor, which plain Claude gets blocked from.", "summary": "Bright Data solutions architect Yohan Raju argues that traditional scraping does not scale. Every site needs its own scraper, which takes about 3–5 days to build and QA. Adding hundreds or thousands of sites means more engineers to maintain them as selectors and anti-bot measures change. He describes the traditional pipeline (trigger, proxy/anti-bot, parsing and validation, ingestion via a bronze/silver medallion architecture). He contrasts the 'sync' path, where an AI searches the live web, with the 'async' path, where it relies on RAG or pre-trained data. He then pitches Bright Data's MCP server as the way to give agents unblocked real-time access. The talk ends with a live notebook demo: Amazon product data plus Reddit sentiment, analysed by Gemini, produces a buy/don't-buy verdict on a Stanley Quencher tumbler.", "key_points": [ "Building one scraper, including QA, development runs and testing the end-to-end pipeline, takes about 3–5 days per site. Scaling from 10 sites to 100 or 1,000 used to need 2x or 4x more engineers, because each site has a different layout and existing scrapers break when sites change.", "A selector change (for example on price) causes downtime, a team fix and backfilling of lost data. Raju claims AI agents can now discover sites, build scrapers and handle these fixes automatically.", "Anti-bot protection ranges from sites that a data-center IP gets through to Cloudflare- or Akamai-protected sites. Those need a residential IP, a real browser and human-like behaviour; clicking or scrolling too fast, or blocking images, can trigger blocks.", "The traditional architecture has four layers. Scheduler/cron or config triggers feed a proxy layer (data center, ISP, rotating residential, or managed unblockers like Web Unlocker and Browser API). Parsing and validation follow, such as checking that price comes back about 90% of the time. Last is ingestion (dedupe, enrich, batch, bronze/silver medallion, database). A failure anywhere means restarting the whole process.", "AI answers take a 'sync' path (live web search) or an 'async' path (RAG or pre-trained data). Adding 'this year' to 'when's the best time to travel to San Francisco' pushes the model onto the live web. Answers asked many times a day, like today's weather in New York, can be cached to save tokens and money.", "Many sites block ClaudeBot and GPTBot in robots.txt, so native AI web search can't reach them. In a demo video, Claude without Bright Data's MCP was blocked from a Glassdoor URL, while Claude with it returned the page.", "Bright Data's MCP server exposes 60+ tools: search engine, scrape as markdown or HTML, site-specific APIs (Amazon, Reddit, LinkedIn, Instagram), browser automation, and e-commerce, social, financial, research, business-intelligence, app store and travel categories. In one example, a customer used Claude Code with the MCP to find their Instagram, TikTok and LinkedIn profiles and return follower counts.", "For a resume-matching agent trained on jobs data, Raju says the data must be structured and consistent, continuously updated, reliable and historical (past months or years), so the model can learn trends like salary changes." ], "takeaways": [ "Budget for maintenance, not just the build. Raju says working out what a site changed can take more time than running the scraper.", "Validate scraped output against field-level expectations (for example, price present about 90% of the time) before data enters your pipeline. Dedupe and stage it through bronze/silver layers.", "Decide per query whether the agent needs live web data (the sync path) or can rely on pre-indexed or RAG data (async). Cache frequently repeated real-time answers to save tokens and cost.", "Assume protected sites (Glassdoor, sites behind Cloudflare or Akamai) and robots.txt blocks on ClaudeBot and GPTBot will defeat naive agent browsing. Plan for residential IPs, real browsers and human-like interaction, or a managed unblocking layer.", "Expose data sources as MCP tools so the agent picks which tool to call, instead of hard-wiring a scraper per site." ], "topics": [ "agents", "web scraping", "mcp", "real-time data", "rag", "anti-bot", "data pipelines", "proxies" ], "tools": [ "Bright Data", "Bright Data MCP server", "Web Unlocker", "Browser API", "Claude", "Claude Code", "GPT", "LangChain", "Gemini", "Google notebook (Colab)", "Amazon", "Walmart", "Shopee", "Shopify", "Etsy", "Reddit", "Glassdoor", "LinkedIn", "Instagram", "TikTok", "Cloudflare", "Akamai", "DoorDash", "Google" ], "quotes": [ { "text": "it takes generally about you know 3 to 5 days to develop a scraper if you're a developer", "at": "02:53", "url": "https://www.youtube.com/watch?v=IK8cC4FScps&t=173s" }, { "text": "a lot of sites usually block things like Claudebot and GPTbot.", "at": "12:48", "url": "https://www.youtube.com/watch?v=IK8cC4FScps&t=768s" }, { "text": "on the right hand side it actually got blocked from going into glass door.", "at": "16:05", "url": "https://www.youtube.com/watch?v=IK8cC4FScps&t=965s" }, { "text": "trying to figure out what change did that site make also takes time, even probably more time than managing the scraper.", "at": "25:25", "url": "https://www.youtube.com/watch?v=IK8cC4FScps&t=1525s" } ], "words": 5205 }, "summary_url": "/#IK8cC4FScps", "transcript": { "html": "/transcripts/IK8cC4FScps.html", "txt": "/transcripts/IK8cC4FScps.txt", "vtt": "/transcripts/IK8cC4FScps.vtt" } }