WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=1UmZHb_E_SM

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/1UmZHb_E_SM.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.200 --> 00:00:15.760
Okay, hello everyone.

s3
00:00:15.760 --> 00:00:19.200
So, mostly I talk today starts with models.

s4
00:00:19.200 --> 00:00:28.880
This one starts somewhere less glamorous with infrastructure that decides whether those models get fresh, usable, real-time data at all.

s5
00:00:28.880 --> 00:00:39.440
So, I work at Oxylabs and Oxylabs was established in 2015 and describes itself as a web intelligence platform and a premium proxy provider.

s6
00:00:39.440 --> 00:00:47.080
In simple terms, we built infrastructure that allows companies to extract public web data at scale.

s7
00:00:47.080 --> 00:00:52.400
And as we all know, public web data theoretically is available for everyone.

s8
00:00:52.400 --> 00:01:00.800
But when you But in practice, if you want to connect your AI models, agents, databases, you need infrastructure layer.

s9
00:01:00.800 --> 00:01:06.120
Uh so, this is what we do and this is where what matters now more than ever.

s10
00:01:06.120 --> 00:01:13.880
Uh because the industry is shifting away from static knowledge and training itself still matters, of course.

s11
00:01:13.880 --> 00:01:24.920
But training alone is no longer enough and to stay useful models needs to get access to fresh information, live search, real external data.

s12
00:01:24.920 --> 00:01:30.600
And without that, even the smartest model is limited by what it knows.

s13
00:01:30.600 --> 00:01:32.720
And this is where my story begins.

s14
00:01:32.720 --> 00:01:38.320
So, my name is Patricia and as I mentioned, I worked as uh in Oxylabs as a product manager now.

s15
00:01:38.320 --> 00:01:40.960
But I started actually closer to engineering.

s16
00:01:40.960 --> 00:01:45.560
I was leading teams dealing with service, core services.

s17
00:01:45.560 --> 00:01:53.040
But the first squad that actually taught me one thing was what we called UX.

s18
00:01:53.040 --> 00:01:54.360
Uh and what is UX?

s19
00:01:54.360 --> 00:01:56.920
UX is usually means user experience.

s20
00:01:56.920 --> 00:01:58.920
That is completely correct.

s21
00:01:58.920 --> 00:02:05.120
But for us, that often meant closer to this, that client needs something really unusual.

s22
00:02:05.120 --> 00:02:07.640
There is no ready-to-made product.

s23
00:02:07.640 --> 00:02:15.640
The timeline is extremely painful, and somehow we need to build everything fast and make it work beautiful.

s24
00:02:15.640 --> 00:02:22.600
So, the lesson that I learned with that team, that innovation never comes as a neat road map.

s25
00:02:22.600 --> 00:02:31.040
It comes as a pressure, as a deadline, and sometimes, and quite often, as a trip report from San Francisco.

s26
00:02:31.040 --> 00:02:33.280
And this is how the first story started.

s27
00:02:33.280 --> 00:02:42.400
One day, our sales team came back from San Francisco and said, "There is a demand for video API for AI training."

s28
00:02:42.400 --> 00:02:47.520
Um there is one question that you actually really scared to ask the sales team.

s29
00:02:47.520 --> 00:02:49.520
What's the deadline?

s30
00:02:49.520 --> 00:02:50.680
Two weeks.

s31
00:02:50.680 --> 00:02:52.880
What's the What's the scale?

s32
00:02:52.880 --> 00:02:55.440
At least 5 petabytes per month.

s33
00:02:55.440 --> 00:03:01.600
At that point, we have never built nothing like that, and it seems a lot.

s34
00:03:01.600 --> 00:03:07.320
And actually, this is also a moment when the feature stops sounding less as a product feature.

s35
00:03:07.320 --> 00:03:14.920
It sounds like infrastructure, because what client actually is asking to build is not just to download some videos.

s36
00:03:14.920 --> 00:03:20.880
They are asking for a pipeline, collection, transfer, storage, delivery.

s37
00:03:20.880 --> 00:03:26.920
And do it with enough reliability that would be compatible with AI training workloads.

s38
00:03:26.920 --> 00:03:33.920
So, that story actually aged surprisingly well, because the market has moved exactly into that direction.

s39
00:03:33.920 --> 00:03:37.800
And AI infrastructure is becoming increasingly more multimodal.

s40
00:03:37.800 --> 00:03:49.840
It's no longer about the text, and companies now need pipelines for video, metadata, transcripts, subtitles, and another structural context around the content itself.

s41
00:03:49.840 --> 00:03:50.920
So, what with that?

s42
00:03:50.920 --> 00:04:04.360
So, in two weeks, we had to build a new dedicated scraper with a brand new logic, new storage integrations, and with delivery flow of something that we actually never built before.

s43
00:04:04.360 --> 00:04:07.680
And we actually made it, and somehow we made it even on time.

s44
00:04:07.680 --> 00:04:11.880
But, this is not where the actually story ended.

s45
00:04:11.880 --> 00:04:14.600
That was only the version one.

s46
00:04:14.600 --> 00:04:21.799
So, client asked, "Great that you have a downloader, but what about transcripts?"

s47
00:04:21.799 --> 00:04:25.280
So, we built a transcript support.

s48
00:04:25.280 --> 00:04:30.919
Uh client tested out, and we see that all of the requests are failing.

s49
00:04:30.919 --> 00:04:36.160
Then we start talking with the client, and we see that there is nothing that we did something wrong.

s50
00:04:36.160 --> 00:04:40.280
That client actually didn't need a transcript, they needed the subtitles.

s51
00:04:40.280 --> 00:04:41.480
So, we adapt again.

s52
00:04:41.480 --> 00:04:44.400
We build a subtitle support.

s53
00:04:44.400 --> 00:04:47.320
Um then another request comes.

s54
00:04:47.320 --> 00:04:51.480
"We are struggling to find videos in languages that we actually need.

s55
00:04:51.480 --> 00:04:55.120
Can you build a search that we could gather those ideas?"

s56
00:04:55.120 --> 00:04:56.680
So, we do it again.

s57
00:04:56.680 --> 00:04:58.640
"What about metadata?"

s58
00:04:58.640 --> 00:05:02.000
Of course, we do it once again.

s59
00:05:02.000 --> 00:05:16.640
And this is the part of the story that I really loved, because once it started as a one product feature request, it actually became um became the whole product suite, because we started thinking that we're building just a downloader.

s60
00:05:16.640 --> 00:05:28.760
Then we realized that we're building a transcript support, subtitle support, uh adding metadata, channel information, and ended up building our own internal library that glues everything together.

s61
00:05:28.760 --> 00:05:39.160
And after enough iterations, uh as a one as I mentioned that started as a one-off time request, uh it became the product family.

s62
00:05:39.160 --> 00:05:49.480
And in roughly 3 months, we actually ended up having the whole video API suite that supported downloaders, transcripts, subtitles, channel information.

s63
00:05:49.480 --> 00:05:54.120
And yeah, after all of this, the final twist came.

s64
00:05:54.120 --> 00:05:56.640
So, it's 2026.

s65
00:05:56.640 --> 00:06:02.600
Client already gathered 30 petabytes of data, and we're still waiting for a payment.

s66
00:06:02.600 --> 00:06:08.120
So, yes, the first lesson is really technical, but also very human.

s67
00:06:08.120 --> 00:06:14.360
Uh that innovation is actually a repeated adaptation under high pressure.

s68
00:06:14.360 --> 00:06:21.720
Because once you learn that the client actually doesn't buy the first product iteration, they buy your ability to adapt.

s69
00:06:21.720 --> 00:06:28.800
The next question becomes, can you actually make it under extreme latency constraints, too?

s70
00:06:28.800 --> 00:06:32.960
And this is a part where I tell you a little about SERP data.

s71
00:06:32.960 --> 00:06:35.919
And search data has always mattered.

s72
00:06:35.919 --> 00:06:38.800
But the AI changed the role it plays.

s73
00:06:38.800 --> 00:06:45.640
Um before, SERP was often used for analytics, SEO, monitoring, market intelligence.

s74
00:06:45.640 --> 00:06:48.919
But now, it's a huge part of AI systems.

s75
00:06:48.919 --> 00:06:51.200
It feeds retrieval pipelines.

s76
00:06:51.200 --> 00:06:54.000
It grounds uh it powers assistance.

s77
00:06:54.000 --> 00:06:55.160
It grounds answers.

s78
00:06:55.160 --> 00:07:02.600
It helps agents interact with live information instead of stale training memory.

s79
00:07:02.600 --> 00:07:05.560
And that shift is not hypothetical.

s80
00:07:05.560 --> 00:07:14.240
Google's grounding documentation explicitly positions Google Search as a way to connect models to current public knowledge.

s81
00:07:14.240 --> 00:07:22.000
In simple terms, the model layer is increasingly expected to work with live retrieval layer around it.

s82
00:07:22.000 --> 00:07:26.680
And that's why the next request mattered so much.

s83
00:07:26.680 --> 00:07:35.120
So, back in 2024, client came and asked for SERP delivery with sub- sub-second SERP delivery.

s84
00:07:35.120 --> 00:07:41.720
At that time, our traditional regular search scraper was around 4 seconds average latency.

s85
00:07:41.720 --> 00:07:49.160
So, the gap was huge, but we still decided to go for it just to see if it's possible and we actually did it.

s86
00:07:49.160 --> 00:07:54.760
But, the story doesn't have happy ending here because client did it not did not test it out.

s87
00:07:54.760 --> 00:07:57.280
And to be honest, the market wasn't ready for that.

s88
00:07:57.280 --> 00:08:00.080
So, we just put it on a shelf.

s89
00:08:00.080 --> 00:08:13.720
But, what became clay clear later on that was never about making uh the old scraper faster because the regular scraper, what he does here is built to retrieve as much information as possible.

s90
00:08:13.720 --> 00:08:20.000
So, we're talking ads, widgets, rich results, AI-generated results, different layouts.

s91
00:08:20.000 --> 00:08:24.280
And when we think about fast search API, it takes a different approach.

s92
00:08:24.280 --> 00:08:28.560
It focuses on the things that actually matters only for AI systems.

s93
00:08:28.560 --> 00:08:35.200
So, it's mostly organic results, top stories, news, and it cuts away all the heavy layout.

s94
00:08:35.200 --> 00:08:42.880
So, even this small scope, it's already something to start thinking about lower latency.

s95
00:08:42.880 --> 00:08:46.120
So, fast forward, it's 2025.

s96
00:08:46.120 --> 00:08:55.800
Another client comes in and their request was simple: zero data retention, sub-second latency, and 2 weeks.

s97
00:08:55.800 --> 00:09:08.880
For us, that meant to support different geolocation and query parameters, to have a system that is capable to deliver results under 800 milliseconds, and to have a solution that is ready

s98
00:09:08.880 --> 00:09:12.160
uh to be tested out in less than 2 weeks.

s99
00:09:12.160 --> 00:09:17.800
So, when your baseline is at 4 seconds, we are not talking about optimization.

s100
00:09:17.800 --> 00:09:20.080
We are talking about redesign.

s101
00:09:20.080 --> 00:09:23.400
Uh so, we started from the scratch.

s102
00:09:23.400 --> 00:09:26.800
And actually, the first version worked.

s103
00:09:26.800 --> 00:09:33.360
In less than 2 weeks, we got around 650 milliseconds P90.

s104
00:09:33.360 --> 00:09:39.480
So, that alone would be a great story, but the real story happened on the next call.

s105
00:09:39.480 --> 00:09:45.200
So, we're sitting on a call with the client getting ready to test it out our new product.

s106
00:09:45.200 --> 00:09:48.720
And while we were on the call, we got blocked.

s107
00:09:48.720 --> 00:09:51.280
And we got blocked really bad.

s108
00:09:51.280 --> 00:10:03.160
And to to be honest, this is really honest moment about when you think about infrastructure and systems, because this is a kind reminder that there is a difference between system

s109
00:10:03.160 --> 00:10:10.920
that works in development, system that works in a test, and system that actually survives reality.

s110
00:10:10.920 --> 00:10:14.760
So, we had to start over, because nothing worked.

s111
00:10:14.760 --> 00:10:21.120
And at this second iteration was the hardest one, because we actually had to rely a lot on browsers.

s112
00:10:21.120 --> 00:10:23.920
And don't get me wrong, browsers are amazing.

s113
00:10:23.920 --> 00:10:35.880
They are extremely useful, but browsers also are slow, expensive, complex, and deeply incompatible with dreams about low latency.

s114
00:10:35.880 --> 00:10:40.839
So, there is So, we had a contradiction.

s115
00:10:40.839 --> 00:10:49.920
The reality and the client wanted sub-second, the reality needed browsers, and browsers really wanted us to give us 4 seconds.

s116
00:10:49.920 --> 00:10:53.320
So, at this point, there is no magic trick.

s117
00:10:53.320 --> 00:10:55.320
You just go hunting for a time.

s118
00:10:55.320 --> 00:11:05.120
So, you you review everything, layouts, parsers, sessions, proxies, every place when you can cut off a second, a two, a three, or four.

s119
00:11:05.120 --> 00:11:14.480
And this is how systems becomes fast, not by giant breakthroughs as we thought at first, but by small decision that adds up.

s120
00:11:14.480 --> 00:11:19.000
And that work paid off and actually evolved into something new.

s121
00:11:19.000 --> 00:11:32.720
So, today we have fast search API that delivers results and fresh data directly into AI workflows with 550 milliseconds average latency.

s122
00:11:32.720 --> 00:11:39.160
And our scale move from 400 million daily requests to almost 6 billion daily requests.

s123
00:11:39.160 --> 00:11:41.760
Uh so, that number matters.

s124
00:11:41.760 --> 00:11:49.040
Because going from 400 million daily requests to 6 billion daily requests is not just a change, not just a growth.

s125
00:11:49.040 --> 00:11:50.960
It's a change in operating model.

s126
00:11:50.960 --> 00:11:57.520
It changes how you think about costs, observability, and failure of domains.

s127
00:11:57.520 --> 00:12:04.720
So, the lesson of this part uh that in AI era, speed is not just performance.

s128
00:12:04.720 --> 00:12:08.240
Speed actually defines what product can exist.

s129
00:12:08.240 --> 00:12:11.120
Because in 4 seconds, you have a slow pipeline.

s130
00:12:11.120 --> 00:12:17.800
In sub-second delivery, you have something that can sit and interact in your AI workflows.

s131
00:12:17.840 --> 00:12:23.680
So, when a speed becomes product, what's next?

s132
00:12:23.680 --> 00:12:27.840
Next is then scale actually becomes the real test.

s133
00:12:27.840 --> 00:12:32.720
So, the first story was about mm adapting product scope.

s134
00:12:32.720 --> 00:12:36.400
The second was adapting architecture for latency.

s135
00:12:36.400 --> 00:12:41.040
The third one is going to be adapting systems for scale.

s136
00:12:41.040 --> 00:12:45.120
And the scale is where infrastructure becomes really humbling.

s137
00:12:45.120 --> 00:12:51.240
At one point, another demand has forced us to scale our web and blocker quite aggressively.

s138
00:12:51.240 --> 00:12:54.560
Uh I added just slides just to see how it works.

s139
00:12:54.560 --> 00:12:59.080
Uh it's simple terms, it's similar to scraper, but it has proxy integration.

s140
00:12:59.080 --> 00:13:03.720
So, we were working our way around 10,000 requests per second.

s141
00:13:03.720 --> 00:13:11.839
Demand has forced us to scale to 60,000 requests per second and in less than 2 months.

s142
00:13:11.839 --> 00:13:21.960
So, now that number alone sounds impressive, but it might be also misleading misleading if if you are thinking about as as a simple HTTP request.

s143
00:13:21.960 --> 00:13:26.440
In our world, that means the end-to-end scraping job.

s144
00:13:26.440 --> 00:13:36.360
Part uh it it will be routing, rendering, proxy handling, browsers execution, parsing, retries, normalization, and delivery itself.

s145
00:13:36.360 --> 00:13:43.680
So, when you kind of scale to that workload, even adding up additional 2,000 servers doesn't solve the problem.

s146
00:13:43.680 --> 00:13:45.040
You need an architecture.

s147
00:13:45.040 --> 00:13:48.760
You need a central components that actually are reliable.

s148
00:13:48.760 --> 00:13:52.880
You need observability that still tells you the truth.

s149
00:13:52.880 --> 00:13:59.480
And you need testing that resembles ev- uh reality enough to matter.

s150
00:13:59.480 --> 00:14:06.120
And this is where our main bottleneck showed up, not in dramatic outage, in load testing.

s151
00:14:06.280 --> 00:14:10.080
Uh the hardest part was not generating synthetic traffic.

s152
00:14:10.080 --> 00:14:14.280
Synthetic traffic is relatively easy comparing to reality.

s153
00:14:14.280 --> 00:14:17.360
Uh but the hardest part, organic data testing.

s154
00:14:17.360 --> 00:14:23.840
That means processing traffic that behave enough like real client usage to tell us something useful.

s155
00:14:23.840 --> 00:14:29.040
And during one of those load tests, we hit the wall around 20,000 requests per second.

s156
00:14:29.040 --> 00:14:33.120
At that point, there is no question if the system is actually working.

s157
00:14:33.120 --> 00:14:34.320
It is working.

s158
00:14:34.320 --> 00:14:39.560
The question becomes, do we actually know that it can go further?

s159
00:14:39.560 --> 00:14:42.000
And that uncertainty was the real bottleneck.

s160
00:14:42.000 --> 00:14:50.200
So, uh so are all the pain points, metrics, logs, and generating and processing everything at scale.

s161
00:14:50.200 --> 00:15:02.360
So, everybody loves observability in theory, but observability at scale becomes a true work because collecting logs is hard, processing logs is harder.

s162
00:15:02.360 --> 00:15:06.920
Um and same applies to metrics.

s163
00:15:06.920 --> 00:15:15.680
Uh they're essential, but when you scale up to that kind of a load, the telemetry itself becomes a part of the load and a part of the complexity.

s164
00:15:15.680 --> 00:15:16.440
So, what we did?

s165
00:15:16.440 --> 00:15:18.200
We scaled gradually.

s166
00:15:18.200 --> 00:15:25.920
And eventually, we had to accept one unavoidable truth that the real testing is going to be with production traffic.

s167
00:15:25.920 --> 00:15:30.320
And thankfully, that part actually went completely fine.

s168
00:15:30.320 --> 00:15:37.760
Uh but the story doesn't end up here uh because the drama is still happening right now.

s169
00:15:37.760 --> 00:15:45.040
Uh internally, we call this project 60 because we had to scale up to 60,000 requests per second.

s170
00:15:45.040 --> 00:15:49.080
Now, it's already becoming project 150.

s171
00:15:49.080 --> 00:16:02.800
So, while we were scaling our infrastructure to 60,000 requests per second, now we are talking and seeing results and scale up to about to about 1 100,000 requests per second.

s172
00:16:02.800 --> 00:16:08.040
So, the lesson from this part is also simple that the scale is never a finish line.

s173
00:16:08.040 --> 00:16:10.000
Well, at least not for us.

s174
00:16:10.000 --> 00:16:16.160
And probably when you reach one target number, the next one will appear.

s175
00:16:16.400 --> 00:16:20.480
So, anyways, what does Oxylabs do in this whole thing?

s176
00:16:20.480 --> 00:16:25.880
The I guess the stories make one thing quite clear that we are not just a proxy provider.

s177
00:16:25.880 --> 00:16:40.080
Proxies are essential, they are important, but the hardest part and the larger job is building the infrastructure layer that allows companies to extract public web data up and and operate it at scale.

s178
00:16:40.080 --> 00:16:57.960
That means reaching the open web, collecting data reliably, dealing with antibot systems, handling browsers when they are needed, instruction and delivering data, and doing in that manner that AI companies can actually plug into their systems.

s179
00:16:58.280 --> 00:17:06.640
And this is exactly why it matters because the best thing we can offer is not just data access, it is this.

s180
00:17:06.640 --> 00:17:14.680
That you build the intelligence and we take the messy maintenance underneath because the messy part is is is real.

s181
00:17:14.680 --> 00:17:21.160
The targets change, layouts change, detection changes, market itself changes, client needs changes.

s182
00:17:21.160 --> 00:17:23.560
So, this is not a build-once business.

s183
00:17:23.560 --> 00:17:26.240
This is an adapt-forever business.

s184
00:17:26.240 --> 00:17:31.920
And honestly, that may be the most useful definition of innovation that I know.

s185
00:17:31.920 --> 00:17:40.560
That innovation is the ability to keep adapting fast enough that a changing requirements becomes a new infrastructure.

s186
00:17:40.560 --> 00:17:47.400
So, if I need you to leave with one thought today, I will probably get back where it started.

s187
00:17:47.400 --> 00:17:52.480
That the next generation of AI will not be powered by better models.

s188
00:17:52.480 --> 00:17:56.000
It will be powered by better infrastructure around it.

s189
00:17:56.000 --> 00:18:14.360
Infrastructure that can connect models to reality, infrastructure that can push the web data directly into your pipelines, databases, agents, AI tools, infrastructure that can scale from 400 million daily requests to 6 billion daily requests.

s190
00:18:14.360 --> 00:18:16.360
Because this is really the story.

s191
00:18:16.360 --> 00:18:20.920
Not just scale, not just scraping, not just speed, adaptation.

s192
00:18:20.920 --> 00:18:25.320
Adapting products, adapting architecture, adapting systems.

s193
00:18:25.320 --> 00:18:33.240
And doing it fast enough that the AI companies and you can keep on building while the maintenance burden stays with us.

s194
00:18:33.240 --> 00:18:37.960
So, and this is what it actually means for me in AI world.

s195
00:18:37.960 --> 00:18:40.760
It means that the model is not alone anymore.

s196
00:18:40.760 --> 00:18:42.800
It already has bridge to it.

s197
00:18:42.800 --> 00:18:45.080
Thank you.

s198
00:19:01.023 --> 00:19:03.023
[music]