Where RL Will Take Search — Maximilian-David Rumpf, SID.ai

AI Engineer · 9 min · 107 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:01[music]
  2. 00:12Okay, today I'm going to talk about where reinforcement learning will take search and some some background on me.
  3. 00:22Uh I'm the founder and CEO of Citi.
  4. 00:24We're a stealthish AI lab for search.
  5. 00:27We're backed by some pretty amazing people.
  6. 00:30Uh, and we're hiring.
  7. 00:35Okay.
  8. 00:36Agents are a new paradigm for search.
  9. 00:40You can now get vastly higher quality results.
  10. 00:45Um, twice as likely to find the right documents.
  11. 00:49But it's incredibly expensive.
  12. 00:52um about a hundred to a thousand times more expensive than what you'd get out of a classical search query and it is extremely slow.
  13. 00:59You're looking at minutes and not milliseconds like turboroper.
  14. 01:05And what this means is that agents spend 30 to 50% of their tokens on searching.
  15. 01:11And this is usually at the beginning of some task.
  16. 01:14It finds the right context to then do whatever you ask it to do.
  17. 01:19And the idea here is quite simple.
  18. 01:22First, instead of having the main agent do the searching, you pass the searching to a sub agent and you train a model to be a great sub agent.
  19. 01:32And the question here that we'll answer today is how much cheaper and faster can we make this with reinforcement learning?
  20. 01:40And this is really our target.
  21. 01:42So this is a benchmark across legal, finance, knowledge basis, science, email, a bunch of different tasks, some academic benchmarks, some internal benchmarks.
  22. 01:52And this is where you currently are.
  23. 01:58You see reranker and vector only performance at the bottom and you can see frontier models essentially kind of like go through here at the cost of spending many many you know minutes
  24. 02:09per question.
  25. 02:10And can we get a model to kind of like be extremely accurate um have extremely high recall but also be incredibly fast and cheap.
  26. 02:19And let's quickly look at classical search.
  27. 02:23This is the pipeline that many of you guys will be familiar with.
  28. 02:26A question comes in, you might have an LLM that rewrites the question.
  29. 02:30You then execute that on a search backend.
  30. 02:33You might have a reranker.
  31. 02:35Um and you get your results at the end of the day.
  32. 02:38Um, it is essentially a pipeline of chained locally optimized models and all of the decisions are baked in at design time and you expend a fixed amount of compute per question.
  33. 02:51And this one is really important.
  34. 02:53The re-ranker might know that the results are insufficient at answering the question, but the re-ranker can't take action.
  35. 03:00It can only essentially return the results even when they're bad.
  36. 03:03And what this means is that in practice a pipeline like this acrrues a long tail of failure where unexpected questions come that you know the designer didn't have something for.
  37. 03:15And in practice this usually means people add lots of edge cases um to essentially fix these.
  38. 03:21But of course you can't design infinite edge cases and you can't add infinite tweaks.
  39. 03:27And so the strategy here is one that we've seen before.
  40. 03:32machine design outperforms human design.
  41. 03:35Um, and we saw this in computer vision where you had your, you know, primitive edge detection algorithms.
  42. 03:42You then had the box around a dog generation of models um that were very good at this like very narrow task and locally optimized for it.
  43. 03:51And then you had VLMs that were extremely good at all parts of the search pipeline.
  44. 03:55You saw this again with chess with IBM Deep Blue being largely a collection of human written rules.
  45. 04:01um stockfish bridging the two and then alpha zero and mu0ero essentially completely just putting it all inside of the model and we're going to see something similar happen to search where we have our existing algorithms like BM25
  46. 04:14and page rank um then we had an evolution from that with small models that did some task very well like vectors and re-rankers and now essentially this new paradigm
  47. 04:26of pure RL where we actually don't bake any design decisions into the model.
  48. 04:32And what this looks like in practice is um something like this.
  49. 04:37Um so you have one model uh it goes back and forth with the database.
  50. 04:42It can search, it can read results, it can iterate.
  51. 04:45Uh it can search again until it is happy.
  52. 04:47It can set metadata filters on the fly.
  53. 04:50It can constrain its search.
  54. 04:52It can try as much as it wants.
  55. 04:54Uh and in the end, it produces a ranked list of results.
  56. 05:00Uhhuh.
  57. 05:00And what you get is a model that makes all of the decisions and can adapt to any question on the fly and for example use much more compute if a user asks a very difficult question.
  58. 05:11And what helps us here is that search is verifiable.
  59. 05:15Um and reinforcement learning needs rewards that are verifiable and grindable.
  60. 05:19Verifiable here means for a given question, did you find the correct document?
  61. 05:23And we can design this and tell this quite easily.
  62. 05:27Um, and is there an environment where the model can attempt this question loads and loads of times and in practice for us this means uh thousands of times per second during a training run.
  63. 05:39Uh, and the second part is kind of like can we turn the models that are currently very general and very general purpose into something that is much more specialized
  64. 05:49and it turns out we don't actually need most of the parts of a language model to be extremely performant at search.
  65. 05:54Uh and similarly with like CPUs and GPUs and AS6, um a CPU in theory can do anything that a GPU can do, but you would never want to use a CPU
  66. 06:04to do LLM inference, for example.
  67. 06:06Um because the much more specialized version is much more effective.
  68. 06:10And this really makes search an ideal target for RL.
  69. 06:14Um and this is what happens when you train a model on this task.
  70. 06:19And so again, here we added the vector and reranker only baselines.
  71. 06:22This is of an earlier task.
  72. 06:24And what we see is that search quality increases very predictably with compute.
  73. 06:28And we can mix in other rewards like latency um and different kind of like retrieval strategies to make it even more performant.
  74. 06:35Importantly, we don't really tell the model what to do.
  75. 06:38Much like in Alpha Zero and chess, we want it to discover its own strategies and its own tricks to essentially search.
  76. 06:44Well, uh, and we don't know whether this method has no ceiling, but we're definitely not yet seeing a ceiling to this approach.
  77. 06:54And these are the results.
  78. 06:56Uh, so this is the same chart as before.
  79. 06:58And this is SID one and then SID one um with some parallel execution on the left hand side.
  80. 07:04And so what this ends up meaning is you get you're about 20 times faster.
  81. 07:09So instead of taking around 2 minutes, you take around 5 seconds on average.
  82. 07:12And it's about a hundred times cheaper than using a frontier model for this task.
  83. 07:17Uh there's some more detail here, but like yeah, the cost and kind of like speed are just completely incomparable.
  84. 07:24Uh you can yeah um it's not quite at the latency of a vector and reranker pipeline, but in practice we think we can get there quite quickly.
  85. 07:36And how does this look like in production?
  86. 07:38So this is a usual kind of like agent execution trace.
  87. 07:41The agent does some searching here.
  88. 07:42It finds some good stuff.
  89. 07:43It finds some bad stuff.
  90. 07:45Um, but all of the bad stuff that it finds is essentially polluting its own context window.
  91. 07:50And what we can instead do is use um a sub agent here that does all of the searching and thinking and iterating for the main agent.
  92. 07:58And the main agent only ever sees great results.
  93. 08:00And this means that the main agent sees more good stuff, which means it's more likely to be correct.
  94. 08:06And it is also um extremely cost- effective uh where those 30 to 50% tokens that were earlier used by the main agent to do searching can now be passed off to this
  95. 08:18just you know 100x cheaper search sub agent.
  96. 08:23Uh and where will this take us?
  97. 08:29Scaling RL will give us arbitrarily good search in any domain and RL models will become even faster which will allow them to be used in things like voice and e-commerce.
  98. 08:42They'll become even cheaper than we are currently.
  99. 08:45Um so the charts that you saw there but like I think we can move even further.
  100. 08:50Uh and better search will unlock more knowledge work tasks.
  101. 08:55Uh the web is actually quite small uh in comparison to the entirety of data that is there that is out there.
  102. 09:02Uh and the most valuable information is not on the internet.
  103. 09:06For example, how to run JP Morgan is nowhere on the web but it is deep inside of the databases at at JP Morgan.
  104. 09:14That's it for me.
  105. 09:16Thank you.
  106. 09:18[applause]
  107. 09:33[music]