How Web Data Infrastructure Powers the Next Generation of AI — Patricija Žemaitytė, Oxylabs

AI Engineer · 19 min · 198 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:13Okay, hello everyone.
00:15So, mostly I talk today starts with models.
00:19This one starts somewhere less glamorous with infrastructure that decides whether those models get fresh, usable, real-time data at all.
00:28So, I work at Oxylabs and Oxylabs was established in 2015 and describes itself as a web intelligence platform and a premium proxy provider.
00:39In simple terms, we built infrastructure that allows companies to extract public web data at scale.
00:47And as we all know, public web data theoretically is available for everyone.
00:52But when you But in practice, if you want to connect your AI models, agents, databases, you need infrastructure layer.
01:00Uh so, this is what we do and this is where what matters now more than ever.
01:06Uh because the industry is shifting away from static knowledge and training itself still matters, of course.
01:13But training alone is no longer enough and to stay useful models needs to get access to fresh information, live search, real external data.
01:24And without that, even the smartest model is limited by what it knows.
01:30And this is where my story begins.
01:32So, my name is Patricia and as I mentioned, I worked as uh in Oxylabs as a product manager now.
01:38But I started actually closer to engineering.
01:40I was leading teams dealing with service, core services.
01:45But the first squad that actually taught me one thing was what we called UX.
01:53Uh and what is UX?
01:54UX is usually means user experience.
01:56That is completely correct.
01:58But for us, that often meant closer to this, that client needs something really unusual.
02:05There is no ready-to-made product.
02:07The timeline is extremely painful, and somehow we need to build everything fast and make it work beautiful.
02:15So, the lesson that I learned with that team, that innovation never comes as a neat road map.
02:22It comes as a pressure, as a deadline, and sometimes, and quite often, as a trip report from San Francisco.
02:31And this is how the first story started.
02:33One day, our sales team came back from San Francisco and said, "There is a demand for video API for AI training."
02:42Um there is one question that you actually really scared to ask the sales team.
02:47What's the deadline?
02:49Two weeks.
02:50What's the What's the scale?
02:52At least 5 petabytes per month.
02:55At that point, we have never built nothing like that, and it seems a lot.
03:01And actually, this is also a moment when the feature stops sounding less as a product feature.
03:07It sounds like infrastructure, because what client actually is asking to build is not just to download some videos.
03:14They are asking for a pipeline, collection, transfer, storage, delivery.
03:20And do it with enough reliability that would be compatible with AI training workloads.
03:26So, that story actually aged surprisingly well, because the market has moved exactly into that direction.
03:33And AI infrastructure is becoming increasingly more multimodal.
03:37It's no longer about the text, and companies now need pipelines for video, metadata, transcripts, subtitles, and another structural context around the content itself.
03:49So, what with that?
03:50So, in two weeks, we had to build a new dedicated scraper with a brand new logic, new storage integrations, and with delivery flow of something that we actually never built before.
04:04And we actually made it, and somehow we made it even on time.
04:07But, this is not where the actually story ended.
04:11That was only the version one.
04:14So, client asked, "Great that you have a downloader, but what about transcripts?"
04:21So, we built a transcript support.
04:25Uh client tested out, and we see that all of the requests are failing.
04:30Then we start talking with the client, and we see that there is nothing that we did something wrong.
04:36That client actually didn't need a transcript, they needed the subtitles.
04:40So, we adapt again.
04:41We build a subtitle support.
04:44Um then another request comes.
04:47"We are struggling to find videos in languages that we actually need.
04:51Can you build a search that we could gather those ideas?"
04:55So, we do it again.
04:56"What about metadata?"
04:58Of course, we do it once again.
05:02And this is the part of the story that I really loved, because once it started as a one product feature request, it actually became um became the whole product suite, because we started thinking that we're building just a downloader.
05:16Then we realized that we're building a transcript support, subtitle support, uh adding metadata, channel information, and ended up building our own internal library that glues everything together.
05:28And after enough iterations, uh as a one as I mentioned that started as a one-off time request, uh it became the product family.
05:39And in roughly 3 months, we actually ended up having the whole video API suite that supported downloaders, transcripts, subtitles, channel information.
05:49And yeah, after all of this, the final twist came.
05:54So, it's 2026.
05:56Client already gathered 30 petabytes of data, and we're still waiting for a payment.
06:02So, yes, the first lesson is really technical, but also very human.
06:08Uh that innovation is actually a repeated adaptation under high pressure.
06:14Because once you learn that the client actually doesn't buy the first product iteration, they buy your ability to adapt.
06:21The next question becomes, can you actually make it under extreme latency constraints, too?
06:28And this is a part where I tell you a little about SERP data.
06:32And search data has always mattered.
06:35But the AI changed the role it plays.
06:38Um before, SERP was often used for analytics, SEO, monitoring, market intelligence.
06:45But now, it's a huge part of AI systems.
06:48It feeds retrieval pipelines.
06:51It grounds uh it powers assistance.
06:54It grounds answers.
06:55It helps agents interact with live information instead of stale training memory.
07:02And that shift is not hypothetical.
07:05Google's grounding documentation explicitly positions Google Search as a way to connect models to current public knowledge.
07:14In simple terms, the model layer is increasingly expected to work with live retrieval layer around it.
07:22And that's why the next request mattered so much.
07:26So, back in 2024, client came and asked for SERP delivery with sub- sub-second SERP delivery.
07:35At that time, our traditional regular search scraper was around 4 seconds average latency.
07:41So, the gap was huge, but we still decided to go for it just to see if it's possible and we actually did it.
07:49But, the story doesn't have happy ending here because client did it not did not test it out.
07:54And to be honest, the market wasn't ready for that.
07:57So, we just put it on a shelf.
08:00But, what became clay clear later on that was never about making uh the old scraper faster because the regular scraper, what he does here is built to retrieve as much information as possible.
08:13So, we're talking ads, widgets, rich results, AI-generated results, different layouts.
08:20And when we think about fast search API, it takes a different approach.
08:24It focuses on the things that actually matters only for AI systems.
08:28So, it's mostly organic results, top stories, news, and it cuts away all the heavy layout.
08:35So, even this small scope, it's already something to start thinking about lower latency.
08:42So, fast forward, it's 2025.
08:46Another client comes in and their request was simple: zero data retention, sub-second latency, and 2 weeks.
08:55For us, that meant to support different geolocation and query parameters, to have a system that is capable to deliver results under 800 milliseconds, and to have a solution that is ready
09:08uh to be tested out in less than 2 weeks.
09:12So, when your baseline is at 4 seconds, we are not talking about optimization.
09:17We are talking about redesign.
09:20Uh so, we started from the scratch.
09:23And actually, the first version worked.
09:26In less than 2 weeks, we got around 650 milliseconds P90.
09:33So, that alone would be a great story, but the real story happened on the next call.
09:39So, we're sitting on a call with the client getting ready to test it out our new product.
09:45And while we were on the call, we got blocked.
09:48And we got blocked really bad.
09:51And to to be honest, this is really honest moment about when you think about infrastructure and systems, because this is a kind reminder that there is a difference between system
10:03that works in development, system that works in a test, and system that actually survives reality.
10:10So, we had to start over, because nothing worked.
10:14And at this second iteration was the hardest one, because we actually had to rely a lot on browsers.
10:21And don't get me wrong, browsers are amazing.
10:23They are extremely useful, but browsers also are slow, expensive, complex, and deeply incompatible with dreams about low latency.
10:35So, there is So, we had a contradiction.
10:40The reality and the client wanted sub-second, the reality needed browsers, and browsers really wanted us to give us 4 seconds.
10:49So, at this point, there is no magic trick.
10:53You just go hunting for a time.
10:55So, you you review everything, layouts, parsers, sessions, proxies, every place when you can cut off a second, a two, a three, or four.
11:05And this is how systems becomes fast, not by giant breakthroughs as we thought at first, but by small decision that adds up.
11:14And that work paid off and actually evolved into something new.
11:19So, today we have fast search API that delivers results and fresh data directly into AI workflows with 550 milliseconds average latency.
11:32And our scale move from 400 million daily requests to almost 6 billion daily requests.
11:39Uh so, that number matters.
11:41Because going from 400 million daily requests to 6 billion daily requests is not just a change, not just a growth.
11:49It's a change in operating model.
11:50It changes how you think about costs, observability, and failure of domains.
11:57So, the lesson of this part uh that in AI era, speed is not just performance.
12:04Speed actually defines what product can exist.
12:08Because in 4 seconds, you have a slow pipeline.
12:11In sub-second delivery, you have something that can sit and interact in your AI workflows.
12:17So, when a speed becomes product, what's next?
12:23Next is then scale actually becomes the real test.
12:27So, the first story was about mm adapting product scope.
12:32The second was adapting architecture for latency.
12:36The third one is going to be adapting systems for scale.
12:41And the scale is where infrastructure becomes really humbling.
12:45At one point, another demand has forced us to scale our web and blocker quite aggressively.
12:51Uh I added just slides just to see how it works.
12:54Uh it's simple terms, it's similar to scraper, but it has proxy integration.
12:59So, we were working our way around 10,000 requests per second.
13:03Demand has forced us to scale to 60,000 requests per second and in less than 2 months.
13:11So, now that number alone sounds impressive, but it might be also misleading misleading if if you are thinking about as as a simple HTTP request.
13:21In our world, that means the end-to-end scraping job.
13:26Part uh it it will be routing, rendering, proxy handling, browsers execution, parsing, retries, normalization, and delivery itself.
13:36So, when you kind of scale to that workload, even adding up additional 2,000 servers doesn't solve the problem.
13:43You need an architecture.
13:45You need a central components that actually are reliable.
13:48You need observability that still tells you the truth.
13:52And you need testing that resembles ev- uh reality enough to matter.
13:59And this is where our main bottleneck showed up, not in dramatic outage, in load testing.
14:06Uh the hardest part was not generating synthetic traffic.
14:10Synthetic traffic is relatively easy comparing to reality.
14:14Uh but the hardest part, organic data testing.
14:17That means processing traffic that behave enough like real client usage to tell us something useful.
14:23And during one of those load tests, we hit the wall around 20,000 requests per second.
14:29At that point, there is no question if the system is actually working.
14:33It is working.
14:34The question becomes, do we actually know that it can go further?
14:39And that uncertainty was the real bottleneck.
14:42So, uh so are all the pain points, metrics, logs, and generating and processing everything at scale.
14:50So, everybody loves observability in theory, but observability at scale becomes a true work because collecting logs is hard, processing logs is harder.
15:02Um and same applies to metrics.
15:06Uh they're essential, but when you scale up to that kind of a load, the telemetry itself becomes a part of the load and a part of the complexity.
15:15So, what we did?
15:16We scaled gradually.
15:18And eventually, we had to accept one unavoidable truth that the real testing is going to be with production traffic.
15:25And thankfully, that part actually went completely fine.
15:30Uh but the story doesn't end up here uh because the drama is still happening right now.
15:37Uh internally, we call this project 60 because we had to scale up to 60,000 requests per second.
15:45Now, it's already becoming project 150.
15:49So, while we were scaling our infrastructure to 60,000 requests per second, now we are talking and seeing results and scale up to about to about 1 100,000 requests per second.
16:02So, the lesson from this part is also simple that the scale is never a finish line.
16:08Well, at least not for us.
16:10And probably when you reach one target number, the next one will appear.
16:16So, anyways, what does Oxylabs do in this whole thing?
16:20The I guess the stories make one thing quite clear that we are not just a proxy provider.
16:25Proxies are essential, they are important, but the hardest part and the larger job is building the infrastructure layer that allows companies to extract public web data up and and operate it at scale.
16:40That means reaching the open web, collecting data reliably, dealing with antibot systems, handling browsers when they are needed, instruction and delivering data, and doing in that manner that AI companies can actually plug into their systems.
16:58And this is exactly why it matters because the best thing we can offer is not just data access, it is this.
17:06That you build the intelligence and we take the messy maintenance underneath because the messy part is is is real.
17:14The targets change, layouts change, detection changes, market itself changes, client needs changes.
17:21So, this is not a build-once business.
17:23This is an adapt-forever business.
17:26And honestly, that may be the most useful definition of innovation that I know.
17:31That innovation is the ability to keep adapting fast enough that a changing requirements becomes a new infrastructure.
17:40So, if I need you to leave with one thought today, I will probably get back where it started.
17:47That the next generation of AI will not be powered by better models.
17:52It will be powered by better infrastructure around it.
17:56Infrastructure that can connect models to reality, infrastructure that can push the web data directly into your pipelines, databases, agents, AI tools, infrastructure that can scale from 400 million daily requests to 6 billion daily requests.
18:14Because this is really the story.
18:16Not just scale, not just scraping, not just speed, adaptation.
18:20Adapting products, adapting architecture, adapting systems.
18:25And doing it fast enough that the AI companies and you can keep on building while the maintenance burden stays with us.
18:33So, and this is what it actually means for me in AI world.
18:37It means that the model is not alone anymore.
18:40It already has bridge to it.
18:42Thank you.
19:01[music]