The Rise of CaaS: Context-as-a-Service for Agentic AI — Omer Primor, Bright Data

AI Engineer · 22 min · 257 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12So hi everyone.
00:14Thank you so much for taking the time to join this session.
00:16I hope I'll or at least I can guarantee I'll do whatever it takes uh to make it worth your time.
00:22My name is Om.
00:23I lead the product marketing team over at Bright Data.
00:26Just by maybe a quick show of hands, who here is familiar with Bright Data?
00:31Okay, we can do better.
00:32I'll pass it on to our brand team.
00:34Bright Data is a web data company.
00:37Basically, we help more than 20,000 teams around the world, including more than 70% of the world's biggest AI labs to extract data from the web.
00:45Just to put this in perspective of what scale we're talking about, we're talking well over 50 billion pages, HTMLs every day.
00:52More than 20 pabytes of video, audio, and other media data.
00:56So that's just a perspective of the type of work that we do at Bright Data.
01:01But enough about us.
01:02Personally, I joined Bright Data about three years ago, which essentially gave me front row seats at everything around AI and the web and how they started to actually connect.
01:14It sounds very old but if you think about it only maybe less than two years ago right we were able to start using access the web and search the web through
01:26cloud or through uh chpt that option didn't even exist in the earlier versions right so that connection that way in which both AI and the web are starting to converge is something that is still evolving and evolving rapidly and that's part of what I want to try and shed some light on today
01:42and talk about a new emerging breed of companies.
01:44That's coming out of this connection.
01:48I think we can basically agree that the web is by far the world's greatest source of data.
01:54At least historically when it comes to bright air, that's all we cared about, right?
01:57It's helping our customers extract data from the web.
02:00But with the emergence of AI and the emergence more recently of of AI agents that need to do knowledge work, right?
02:06The web is no longer just a source of data.
02:07We can actually start looking at it as a source of context.
02:11context in the sense that if I do knowledge work and I have knowledge agents that support my work I want to go out to the web find the information that I need use it as context but kept keep on working so the data itself is only a step in the process for something bigger for the actions I
02:25need to take for the conclusions I need to draw and for every downstream application that follows uh the first one's to figure this one out oh sorry even before that but one thing that I want all of us to bear in mind because this is going to follow us through the rest of this conversation
02:41The web is messy.
02:42It's unstructured.
02:43And most importantly, it changes all the time.
02:46This is a chart that shows data decay, right?
02:49That it's an analysis done by our team that shows data decay.
02:51Basically, how long after a new page, a new piece of content goes live, it is no longer relevant, right?
02:58So, social media, it's easy for us to understand.
02:59It's far less than a day.
03:01But also news, finance, retail, 30 days later, data that was collected, it's mostly no longer relevant.
03:09And when we acknowledge that this simple notion, we understand that extracting context from the web or relying on the web is not a snapshot.
03:16It's not a one-time effort.
03:17It's not even a monthly effort.
03:19It's something that we need to keep on doing.
03:21It's something that we need to look at as an ongoing process and something that we need to be mindful of.
03:30The first ones to figure it out were of course search companies, right?
03:34Only what three years ago we were on the far left, right?
03:38This is this is in our lifetime, right?
03:41Three years ago, we were on the far left.
03:42Everything was Google.
03:43There was no there was complete and total dominance up until three years ago for the past 20 or so years, right?
03:50That's what we're talking about purely for humans.
03:53Search something, go collect the information you need, and carry on.
03:58Then fast forward maybe one and a half years ago, two years ago, search began to appear within the LLN within the chat bots, right?
04:07Which already started blurring the line between humans and and agents because now the same bots also have the same web search available.
04:13So the same LMS have the same access through API for the bot.
04:18So we started seeing that convergence happening, right?
04:20And so for the first time, we're seeing more and more traffic flowing down, search traffic, search intent flowing down these channels, not only to Google.
04:28And last but not least, right, we now see a whole breed of companies, the AI search companies.
04:33I'm sure you're familiar with them.
04:34I caught a talk yesterday by Will, the CEO of Exa and Parallel and you.com and Tavili and a bunch of others.
04:41They are purely built and indexing the web especially for agents.
04:45They're not even looking at the humans involved anymore.
04:48So Google's dominance when it comes to search, if Google was synonymous of web search, that is very much shaking.
04:55When there's blood in the water, the sharks come just last week.
05:00Amazon Amazon announced, I don't know how many of you saw it, that they developed their own index and started allowing the ability to retrieve data from the web to retrieve contacts for agents on agent core.
05:13Amazon developed their own search engine.
05:17Two weeks before that it was Microsoft.
05:20Microsoft always had skin in the game.
05:21I know one or two% of the world search traffic went to Microsoft but they have now repackaged it and launched it again as part of web byq as part of their suite
05:30for agentic development and orchestration.
05:34So we're seeing more and more this space becoming crowded.
05:40But when we're talking about context and we're looking at this through the lens of search, I believe it only tells us part of the story, right?
05:48I can search for, I don't know, what's the cost of a certain pair of sneakers this morning, right?
05:54In on a certain website.
05:57I cannot really search for how has that price changed over the last six months, what discounts it had, right?
06:04I can search for what open job positions we have at Bright Data.
06:08We do.
06:08I urge you to go have a look but I can't see how how that that was a chart and how that changed over time and how the headcount of the company changed over time
06:17and all of that information existed on the web simply back then.
06:21So when we actually start to think about it we understand that there's much more context in the web that than what web search allows us to extract and this is what we started seeing in the recent years a whole new breed of companies
06:34rising.
06:35We like to call them internally casts, context as a service because that's what they do.
06:39They allow agents to tap into them MCP, CLI, uh just pure good old API and actually start extracting data to to retrieve data so they can reason over for whatever knowledge work they are responsible for.
06:53We see this happening in e-commerce.
06:55We see this happening in travel.
06:56We see this happening in in finance, in market research, in uh in HR, in real estate, in a bunch of other domains.
07:03I'll show a few examples in a second.
07:05Right?
07:06What all of these have in common is that they don't only just discover the web, you know, in terms of think crawling, think searching, think all of that, accessing, extracting the data and indexing it.
07:15They take it a step further.
07:17They actually develop knowledge graphs to start structuring all of the entities and to ddup them and they start enriching them with a lot of different sources.
07:24So they actually start merging all that data.
07:27If you think about it, they kind of behave like vertical search engines, right?
07:31they are a very very very good search engine for something very specific and and it's it's already in full motion right so as I said we see this in finance and in market research and retail and e-commerce and GTM and sales intelligence
07:45what all of these companies by the way have in common they're all part of brighta's startup program if you are a builder and this is a hot space to go in because I think we're only tapping the surface
07:56I invite you to scan this and apply up to $20,000 in credits and all sorts of co-arketing but that's enough self-promotion.
08:06So CAS as as an industry is already in full bloom and as always with these situations right also the traditional players aren't left too much behind there at the bottom
08:20you see good old data as a service you see zoom info right by researching for this uh presentation today I also saw that they launched that thing at the top it's called gtm.ai AI.
08:30You can only imagine how much they paid for that domain.
08:33But they launched as a secondary brand for Zoom Info that is catering specifically for the need of agents.
08:39Look at look at the wording, right?
08:41They talk about GTM work, right?
08:42That knowledge work, that research that you do when you need to prospect, when you need to do headhunting, when whatever it is you need to do that involves people mostly
08:49straight from cloud code, straight from codeex or any other agent, they understand the gap, right?
08:54So yeah, it's funny to think of Cass as an evolution of DAS and it is in a way, but it's catering for a very specific need.
09:01As much as the AI search engines are different than Google, when agents need them, it's different than people.
09:10When we let this one sink that at the very least we have two different types of paths to complete knowledge work as an agent, we can start thinking about this in terms of web context engineering.
09:21We can start thinking about this in terms of how do I optimize for the specific task or more importantly when things come as it a as it is how do I optimize this for
09:32breeds of task how do I I do this for various parts of the organization that I'm building for if I'm an AI engineer I need to serve different teams they may have different needs it's very tempting to throw AI search at all of them but maybe that's not optimal
09:45maybe I need a combination of both maybe I can start seeing all sorts of cost efficiencies emerge emerge from that.
09:52So for the second half of this presentation, we actually went ahead and created a test.
09:56This is not a benchmark.
09:58You won't see any any something concrete that I can say with great confidence other than the actual research that we did because we wanted to start unraveling the different considerations and how do these two stack up against each other.
10:10So we designed a test.
10:11We went for something basic.
10:12We said okay let's take a company an entity and try and enrich it across 25 different fields.
10:18Some of them are very easy.
10:19you know the company domain, the name uh the headquarters, but some are more challenging, right?
10:25Things about hiring and people and something.
10:27And we build a simple agent, a loop in a loop that knows that uses Opus 4.8 as the harness and it starts to go over field by field, go out, search for it or retrieve it from the cast,
10:39do it again and again and again until it completes and brings back set some guardrails, you know, like budget and stuff just to keep it fair.
10:47and I want to share with you the results.
10:49So the first thing that we would care about right being knowledge work would be uh sorry we ran it 100 times on all of the sponsors of today's event.
11:00So the first thing that we saw in terms of coverage is that there's pretty good convergence.
11:04They all did fairly well, right?
11:07I'll get to the two at the bottom in a second.
11:09So search were consistent performance.
11:12One of the major cast providers were also very well.
11:15The third one by the way you can see unlocker and SER.
11:17SER is good old data uh good old Google.
11:21We basically did the same thing just with Google and it performed pretty well in extracting that information.
11:26Native is cloud's own uh search and you see that they converge really well.
11:32I was originally surprised about the two cast solutions at the bottom.
11:36It was counterintuitive.
11:37I expected cast to dominate this thing because that's you know you had one job right to map out all these companies.
11:44But but after diving into into it a bit more you you understand that well they are limited in the sense that they know what they have about an entity.
11:52If I ask it a question that is beyond that, they will never have that data, right?
11:57Unlike a searcher can go out and continue searching and exploring it.
12:00If they didn't collect data about the recent job hiring, it will never be there, right?
12:05So, it makes sense that they are a bit behind, but I'm sure at the same time that they have a lot of other advantages that we simply didn't ask for, a lot of other fields that they didn't have that aren't represented.
12:14So again it it creates some complexities when what how do we measure coverage when it relates to the specific job that we need to do rather than um in general
12:25when we the second thing we looked at was cost of course here it's we started seeing it spread out a bit so you can see that massive bulk in the center most of the search and the and the cast and even using uh Google right h
12:38converge to pretty much the same cost only different right the cast was just about the service itself what you pay the vendor, right?
12:46All of the other search solutions, you also needed a lot of token burn it burn to actually structure that data so you can actually act on it and and and use it as something retrievable,
12:55right?
12:56So, it's the same output native obscenely expensive.
12:59And the the cast on the right, I'm sure you're all familiar with the by far the most expensive in the industry.
13:04I will not name and shame them.
13:07Interesting.
13:08You see that small cast there at the the left that cast number two that were very cheap.
13:13They're also the ones that are here at the bottom, which is funny because what I believe is happening there is that we're seeing even within this industry niche players that have, you know, lower quality data but much cheaper.
13:26They're already carving that niche of the longtail, right, of small shops or small usage that don't want to pay as much and don't need as much data.
13:33And we're all seeing it them branch out there.
13:38Most of you here, I presume, are engineers.
13:41So there's a very evident question that we did not ask here which is what is the one thing that an engineer would care about?
13:52Thank you.
13:52Let's talk about scale.
13:55This is the cost not for the whole 100.
13:58This is the cost per one for one record.
14:05What happens if we need a million?
14:07Now yes a million records will not fit in the context with obviously we're not talking about a single run that needs a million you can think about million in terms of the frequency
14:16right if I am a market research I do due diligence for private equity I revisit these companies all the time I ask more questions about them as the time goes by was there any new news about them was there anything that changed somebody joined somebody leave do they have new hires I keep on asking the same thing
14:31so when I'm talking about this the multiply by a million it's not just about the number of companies it's the frequency in which I'm asking it frequency is the cost killer when we talk about these and we need to acknowledge that right we're we're thinking about this in terms of web context engineering we're starting to look at
14:46it differently every repeated query costs the same as the first even if it brought back the exact same answers nothing changed pay up right no is false positives for sure go in
15:01token cost right we saw them all that there's very high token we know that doesn't that doesn't shrink think well over time there's always some volume element in terms of the cost but it's not the same as you know flatlining
15:14right and if we bring this back to knowledge work this is where we see teams that are starting to cut corners so I won't research this company every day I'll look at it once a week or once a month I won't ask that question now I will I don't want all the results I'll only take 10 results
15:2920 results something we start so we already have the setup we have what we need to do the knowledge work but at the same time.
15:36We're not extracting all of the value because we're starting to be conscious about cost, right?
15:40Basically, we're renting context.
15:43We're not owning the context that we use.
15:45That is a very important distinction.
15:48Again, if we're good engineers and we ask ourselves, what about scale?
15:51The second and most obvious thing that will come to mind now, so how about we build it?
15:57What if we take all of that web data ourselves and stick it in some vector database and try and see what comes out of it?
16:06So I asked my AI engineer to do exactly that.
16:10Again, this is a test.
16:12This is not a benchmark or a full-blown operation.
16:14This is a day's work at best just to illustrate the concept and to show something about the cost efficiencies that you can generate potentially by doing it yourself potentially
16:24in specific scenarios.
16:26The test simple take the company name nothing but run it through Google find the relevant entries the relevant URLs of that company in various websites that have all of that information right you use search when you don't know the source but when we're talking about company enrichment we all know these sources we all know where that data
16:43comes from the cast also bring it from them zoom info bring it from them it's the same thing over and over again why not just go straight to the source why are we doing that middleman thing
16:53LinkedIn companies LinkedIn jobs crunchbas Right there we have scrapers for those.
16:57You just tap in and you start paying as a pay as you go.
16:59We build two dedicated scrapers.
17:01We have a new AI tool called scraper studio.
17:04It's basically lets you build a scraper for any website in less than 5 minutes.
17:09All powered by AI.
17:10And then it also have a self-healing function, right?
17:12So if the website changes, it fixes itself and keeps on going.
17:16Merge it all into one entity.
17:19Basic heristics.
17:20If there's conflict, choose that over that.
17:22And eventually we have a data set of these uh 100 companies.
17:28Zero AI cost involved.
17:31There's no tokens.
17:34Coverage fairly well.
17:36Not amazing.
17:38Not the best that we saw here, but stacking up pretty well.
17:41And again, this is just a day experiment.
17:44Probably not even as much.
17:46Okay, so again and again, very specific task, very limited context, very limited situation.
17:52you know tread uh tread lightly and then proceed with caution when it comes to conclusions.
17:58The real story is not this.
17:59The real story is this.
18:03That's what it cost to just go and fetch that data that is out there.
18:08We think about knowledge graphs, we think about entities, but if you think about for example LinkedIn, the data is already structured in form of entities.
18:15There's an entity for a company, there's entity for a person, there's entity for a job and they're connected between them.
18:21Sometimes the ontology is already there.
18:22Again, this is not the most complicated of scenarios, but this is pretty damn good.
18:29Now, yes, it took time to set up.
18:34So, it's not really fair to compare apples to apples when it comes to the cost because these are out of the box.
18:39You can just tap into the API.
18:41That one that I just showed you required some setup.
18:44Let's say it's a week.
18:45Let's price it at $5,000 just to give us some perspective.
18:48We can actually start thinking about this in terms of a tipping point.
18:52We can actually start thinking about what is that tipping point in which it makes more sense for me to build it myself, right?
19:00Then keep on renting it.
19:01Now again, everything to the left of that dot in this case it was just over 15,000 entities or queries, right?
19:08When we think about it, so it made sense to do it at this point.
19:11Maybe it's not 15, maybe it's 30, maybe it's 100,000, maybe it's 10,000.
19:15Really depends on the use case.
19:17But there is a tipping point in which it actually makes sense to do it yourself.
19:22H which leads us to the fact that both AI search and casts and all these solutions they're very get good in the sense that you can just plug and play.
19:31But if your knowledge work needs right are persistent and consistent and to a certain degree may even continue escalating and growing then this is perhaps a direction to start considering.
19:43Maybe I can just go ahead and build my own.
19:46Cuz the nice thing about it is that all of the things that we see on the left up until the third part is upfront investment.
19:55And the most important thing that whatever retrieval happens later on from the from the agents, right, is free.
20:02Not really free, but you get what I mean, right?
20:04There's no added cost.
20:05I can just ask that question over and over again.
20:07I did not like the first answer.
20:08I'll ask it again.
20:09I'll ask it a hundred times until I get what I need.
20:11I I have no more fear, no more cutting corners, which is maybe the most important thing.
20:16And I'm leaving aside the fact that this is also custom business logic.
20:18I can connect it with my own data.
20:20There's all sorts of other advantages of owning it.
20:22You know, we'll keep it to the imagination.
20:26[snorts]
20:28Remember, we asked about a million, not about 15,000.
20:32This compounds, this compounds greatly, right?
20:35We need that horizon.
20:36Remember, the web keeps changing.
20:38We saw the stainless data and how the data decays.
20:41So we need to be thinking about this in the long run and how this will evolve when we keep on asking the questions about the entities that we care about
20:50just to wrap it up.
20:52So AI search CAS they can get you very far when what you need is ad hoc when what you need is always changing when sometimes you look at different things
21:00even the mix and match of them for certain task use this for certain task use that you can I'm sure again I just showed a test there's a lot of ways to optimize it just like any other context engineering and make and use uh lighter models and use other stuff there's a lot of great stuff to be
21:13done but eventually the frequency will come and bite you in the ass when it comes to cost and that's something to be mindful of and there's a fair chance that that tipping point
21:23is much lower than you think.
21:25And that's something that when as we design these systems, when we think about web context engineering, we need to be mindful of that.
21:32And last but not least, the last slide we showed owned context compounds while rented decays, right?
21:39It's not a one-time task.
21:41Again, if it's a one-time question, use AI search.
21:43It will be amazing.
21:44when you need to do it over and over again, there's a fair chance that it will not that you're missing out on potential compounding effect and you are losing out.
21:57Thank you very much.
21:59[applause]