WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=iJVxxxHM_Oc

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/iJVxxxHM_Oc.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.960 --> 00:00:22.000
Okay, today I'm going to talk about where reinforcement learning will take search and some some background on me.

s3
00:00:22.000 --> 00:00:24.480
Uh I'm the founder and CEO of Citi.

s4
00:00:24.480 --> 00:00:27.359
We're a stealthish AI lab for search.

s5
00:00:27.359 --> 00:00:30.640
We're backed by some pretty amazing people.

s6
00:00:30.640 --> 00:00:33.439
Uh, and we're hiring.

s7
00:00:35.920 --> 00:00:36.800
Okay.

s8
00:00:36.800 --> 00:00:40.239
Agents are a new paradigm for search.

s9
00:00:40.239 --> 00:00:45.120
You can now get vastly higher quality results.

s10
00:00:45.120 --> 00:00:49.120
Um, twice as likely to find the right documents.

s11
00:00:49.120 --> 00:00:52.000
But it's incredibly expensive.

s12
00:00:52.000 --> 00:00:59.920
um about a hundred to a thousand times more expensive than what you'd get out of a classical search query and it is extremely slow.

s13
00:00:59.920 --> 00:01:05.359
You're looking at minutes and not milliseconds like turboroper.

s14
00:01:05.680 --> 00:01:11.600
And what this means is that agents spend 30 to 50% of their tokens on searching.

s15
00:01:11.600 --> 00:01:14.240
And this is usually at the beginning of some task.

s16
00:01:14.240 --> 00:01:19.600
It finds the right context to then do whatever you ask it to do.

s17
00:01:19.600 --> 00:01:22.000
And the idea here is quite simple.

s18
00:01:22.000 --> 00:01:32.720
First, instead of having the main agent do the searching, you pass the searching to a sub agent and you train a model to be a great sub agent.

s19
00:01:32.720 --> 00:01:40.560
And the question here that we'll answer today is how much cheaper and faster can we make this with reinforcement learning?

s20
00:01:40.560 --> 00:01:42.560
And this is really our target.

s21
00:01:42.560 --> 00:01:52.799
So this is a benchmark across legal, finance, knowledge basis, science, email, a bunch of different tasks, some academic benchmarks, some internal benchmarks.

s22
00:01:52.799 --> 00:01:58.079
And this is where you currently are.

s23
00:01:58.079 --> 00:02:09.599
You see reranker and vector only performance at the bottom and you can see frontier models essentially kind of like go through here at the cost of spending many many you know minutes

s24
00:02:09.599 --> 00:02:10.720
per question.

s25
00:02:10.720 --> 00:02:19.599
And can we get a model to kind of like be extremely accurate um have extremely high recall but also be incredibly fast and cheap.

s26
00:02:19.599 --> 00:02:23.360
And let's quickly look at classical search.

s27
00:02:23.360 --> 00:02:26.239
This is the pipeline that many of you guys will be familiar with.

s28
00:02:26.239 --> 00:02:30.720
A question comes in, you might have an LLM that rewrites the question.

s29
00:02:30.720 --> 00:02:33.360
You then execute that on a search backend.

s30
00:02:33.360 --> 00:02:35.120
You might have a reranker.

s31
00:02:35.120 --> 00:02:38.000
Um and you get your results at the end of the day.

s32
00:02:38.000 --> 00:02:51.280
Um, it is essentially a pipeline of chained locally optimized models and all of the decisions are baked in at design time and you expend a fixed amount of compute per question.

s33
00:02:51.280 --> 00:02:53.280
And this one is really important.

s34
00:02:53.280 --> 00:03:00.239
The re-ranker might know that the results are insufficient at answering the question, but the re-ranker can't take action.

s35
00:03:00.239 --> 00:03:03.920
It can only essentially return the results even when they're bad.

s36
00:03:03.920 --> 00:03:15.200
And what this means is that in practice a pipeline like this acrrues a long tail of failure where unexpected questions come that you know the designer didn't have something for.

s37
00:03:15.200 --> 00:03:21.040
And in practice this usually means people add lots of edge cases um to essentially fix these.

s38
00:03:21.040 --> 00:03:27.360
But of course you can't design infinite edge cases and you can't add infinite tweaks.

s39
00:03:27.360 --> 00:03:32.959
And so the strategy here is one that we've seen before.

s40
00:03:32.959 --> 00:03:35.840
machine design outperforms human design.

s41
00:03:35.840 --> 00:03:42.480
Um, and we saw this in computer vision where you had your, you know, primitive edge detection algorithms.

s42
00:03:42.480 --> 00:03:51.120
You then had the box around a dog generation of models um that were very good at this like very narrow task and locally optimized for it.

s43
00:03:51.120 --> 00:03:55.519
And then you had VLMs that were extremely good at all parts of the search pipeline.

s44
00:03:55.519 --> 00:04:01.280
You saw this again with chess with IBM Deep Blue being largely a collection of human written rules.

s45
00:04:01.280 --> 00:04:14.640
um stockfish bridging the two and then alpha zero and mu0ero essentially completely just putting it all inside of the model and we're going to see something similar happen to search where we have our existing algorithms like BM25

s46
00:04:14.640 --> 00:04:26.560
and page rank um then we had an evolution from that with small models that did some task very well like vectors and re-rankers and now essentially this new paradigm

s47
00:04:26.560 --> 00:04:32.080
of pure RL where we actually don't bake any design decisions into the model.

s48
00:04:32.080 --> 00:04:37.199
And what this looks like in practice is um something like this.

s49
00:04:37.199 --> 00:04:42.160
Um so you have one model uh it goes back and forth with the database.

s50
00:04:42.160 --> 00:04:45.040
It can search, it can read results, it can iterate.

s51
00:04:45.040 --> 00:04:47.680
Uh it can search again until it is happy.

s52
00:04:47.680 --> 00:04:50.639
It can set metadata filters on the fly.

s53
00:04:50.639 --> 00:04:52.880
It can constrain its search.

s54
00:04:52.880 --> 00:04:54.880
It can try as much as it wants.

s55
00:04:54.880 --> 00:04:59.680
Uh and in the end, it produces a ranked list of results.

s56
00:05:00.000 --> 00:05:00.479
Uhhuh.

s57
00:05:00.479 --> 00:05:11.680
And what you get is a model that makes all of the decisions and can adapt to any question on the fly and for example use much more compute if a user asks a very difficult question.

s58
00:05:11.680 --> 00:05:15.199
And what helps us here is that search is verifiable.

s59
00:05:15.199 --> 00:05:19.600
Um and reinforcement learning needs rewards that are verifiable and grindable.

s60
00:05:19.600 --> 00:05:23.680
Verifiable here means for a given question, did you find the correct document?

s61
00:05:23.680 --> 00:05:27.840
And we can design this and tell this quite easily.

s62
00:05:27.840 --> 00:05:39.759
Um, and is there an environment where the model can attempt this question loads and loads of times and in practice for us this means uh thousands of times per second during a training run.

s63
00:05:39.759 --> 00:05:49.280
Uh, and the second part is kind of like can we turn the models that are currently very general and very general purpose into something that is much more specialized

s64
00:05:49.280 --> 00:05:54.960
and it turns out we don't actually need most of the parts of a language model to be extremely performant at search.

s65
00:05:54.960 --> 00:06:04.720
Uh and similarly with like CPUs and GPUs and AS6, um a CPU in theory can do anything that a GPU can do, but you would never want to use a CPU

s66
00:06:04.720 --> 00:06:06.960
to do LLM inference, for example.

s67
00:06:06.960 --> 00:06:10.800
Um because the much more specialized version is much more effective.

s68
00:06:10.800 --> 00:06:14.479
And this really makes search an ideal target for RL.

s69
00:06:14.479 --> 00:06:19.039
Um and this is what happens when you train a model on this task.

s70
00:06:19.039 --> 00:06:22.639
And so again, here we added the vector and reranker only baselines.

s71
00:06:22.639 --> 00:06:24.160
This is of an earlier task.

s72
00:06:24.160 --> 00:06:28.160
And what we see is that search quality increases very predictably with compute.

s73
00:06:28.160 --> 00:06:35.840
And we can mix in other rewards like latency um and different kind of like retrieval strategies to make it even more performant.

s74
00:06:35.840 --> 00:06:38.639
Importantly, we don't really tell the model what to do.

s75
00:06:38.639 --> 00:06:44.880
Much like in Alpha Zero and chess, we want it to discover its own strategies and its own tricks to essentially search.

s76
00:06:44.880 --> 00:06:54.880
Well, uh, and we don't know whether this method has no ceiling, but we're definitely not yet seeing a ceiling to this approach.

s77
00:06:54.880 --> 00:06:56.400
And these are the results.

s78
00:06:56.400 --> 00:06:58.400
Uh, so this is the same chart as before.

s79
00:06:58.400 --> 00:07:04.240
And this is SID one and then SID one um with some parallel execution on the left hand side.

s80
00:07:04.240 --> 00:07:09.120
And so what this ends up meaning is you get you're about 20 times faster.

s81
00:07:09.120 --> 00:07:12.880
So instead of taking around 2 minutes, you take around 5 seconds on average.

s82
00:07:12.880 --> 00:07:17.039
And it's about a hundred times cheaper than using a frontier model for this task.

s83
00:07:17.039 --> 00:07:24.960
Uh there's some more detail here, but like yeah, the cost and kind of like speed are just completely incomparable.

s84
00:07:24.960 --> 00:07:36.319
Uh you can yeah um it's not quite at the latency of a vector and reranker pipeline, but in practice we think we can get there quite quickly.

s85
00:07:36.319 --> 00:07:38.319
And how does this look like in production?

s86
00:07:38.319 --> 00:07:41.199
So this is a usual kind of like agent execution trace.

s87
00:07:41.199 --> 00:07:42.720
The agent does some searching here.

s88
00:07:42.720 --> 00:07:43.680
It finds some good stuff.

s89
00:07:43.680 --> 00:07:45.280
It finds some bad stuff.

s90
00:07:45.280 --> 00:07:50.319
Um, but all of the bad stuff that it finds is essentially polluting its own context window.

s91
00:07:50.319 --> 00:07:58.080
And what we can instead do is use um a sub agent here that does all of the searching and thinking and iterating for the main agent.

s92
00:07:58.080 --> 00:08:00.639
And the main agent only ever sees great results.

s93
00:08:00.639 --> 00:08:06.160
And this means that the main agent sees more good stuff, which means it's more likely to be correct.

s94
00:08:06.160 --> 00:08:18.960
And it is also um extremely cost- effective uh where those 30 to 50% tokens that were earlier used by the main agent to do searching can now be passed off to this

s95
00:08:18.960 --> 00:08:23.120
just you know 100x cheaper search sub agent.

s96
00:08:23.440 --> 00:08:29.840
Uh and where will this take us?

s97
00:08:29.840 --> 00:08:42.959
Scaling RL will give us arbitrarily good search in any domain and RL models will become even faster which will allow them to be used in things like voice and e-commerce.

s98
00:08:42.959 --> 00:08:45.519
They'll become even cheaper than we are currently.

s99
00:08:45.519 --> 00:08:50.320
Um so the charts that you saw there but like I think we can move even further.

s100
00:08:50.320 --> 00:08:55.040
Uh and better search will unlock more knowledge work tasks.

s101
00:08:55.040 --> 00:09:02.399
Uh the web is actually quite small uh in comparison to the entirety of data that is there that is out there.

s102
00:09:02.399 --> 00:09:06.640
Uh and the most valuable information is not on the internet.

s103
00:09:06.640 --> 00:09:14.959
For example, how to run JP Morgan is nowhere on the web but it is deep inside of the databases at at JP Morgan.

s104
00:09:14.959 --> 00:09:16.240
That's it for me.

s105
00:09:16.240 --> 00:09:18.480
Thank you.

s106
00:09:18.721 --> 00:09:20.721
[applause]

s107
00:09:33.131 --> 00:09:35.131
[music]