User Signal Dies at the Retrieval Boundary - Sonam Pankaj, StarlightSearch

AI Engineer · 15 min · 129 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01Hey everyone, I'm Surim.
2. 00:03I'm the CEO and co-founder of Starling Search and today my talk is a signal's die after retrieval boundary.
3. 00:10So, we'll look into uh what are agents essentially, why agents fail, what is uh the cause of fails in retrieval particularly, and how to make actually signals cross the retrieval boundary,
4. 00:24and how to make your agent basically outcome fair.
5. 00:29So, uh let's get started.
6. 00:31What is an agent?
7. 00:32An agent is an LLM that has agency to reason, invoke tools, uh interact with the real world, retrieve uh the memory to complete the task.
8. 00:43One major loop here is missing is uh learning.
9. 00:47It should also learn from what worked and what didn't work.
10. 00:50Suppose uh if I had to uh explain what is agent, I I can explain with react agent.
11. 00:58So, if I had to explain uh what agent does, I would explain with react agent.
12. 01:04So, basically use it from the agent and execute it in a loop, uh client to retrieval search, and then pause when the task is complete.
13. 01:13This is very basic react architecture.
14. 01:18One thing that is missing is how to make agent learn from the outcome.
15. 01:24So, agent keeps failing at the same task.
16. 01:26Gartner reported 85% of AI just failing traction.
17. 01:30So, it's in McKinsey's 2025 report.
18. 01:34The problem came out to be most of the time is that retrieval is static.
19. 01:4073% of our uh pipelines fails because of retrieval, not generation, and context stuffing.
20. 01:49So, a recent uh post from Ram Sriram, the ex-CTO of Pinecone, said, "Uh we have been optimizing for the wrong thing.
21. 01:59You are paying a lot for your agent's memory.
22. 02:02This is probably broken, and we have been optimizing for the wrong things.
23. 02:05We made wrong answers appear faster and cheaper, but we forgot to make retrieval learn."
24. 02:12So, why does this does does does it matter?
25. 02:16Again, uh, the third problem is agents are not outcome-informed.
26. 02:21So, there's a missing layer between evals and action.
27. 02:25Your observability has all the traces, uh, all the stack that capture observability is the stack that capture every tool call, every LM completion, every exceptions.
28. 02:37Your eval suite judges whether the final output was correct or wrong, basically pass or fail.
29. 02:44But, these evals are not reflected in agent, uh, context, skills, and defines my agent action in any ways.
30. 02:57Right?
31. 02:59So, the agent doesn't have any access to why yesterday's runs, uh, runs passed or failed.
32. 03:05The eval signal dies in the dashboard.
33. 03:08This is a missing layer, a system that consume traces, absorb eval, and convert both into retrieval guidance for future runs.
34. 03:17So, there's a manual improvement task, and engineer actually has to sit and see if the eval and observability perform well.
35. 03:25Rewrite the prompt, redeploy it, upgrade either and create your expensive model, restructure to our harness, or fine-tune the custom models.
36. 03:36Why are current memories failing?
37. 03:38Why memories was designed to actually address this, but, uh, uh, it's not.
38. 03:47So, let's see, uh, a what we have in our the current system uh, and current memory is that they basically store user preferences, profile, conversation history, or long-lived personalization.
39. 04:04So, chat experience is not self-improving learning systems for production.
40. 04:09If you see the already existing approach in the market, there's a LangChain, there's Mem0, which does extracted past preferences.
41. 04:21The user's retrieval signal is embedding similarity.
42. 04:26Does it learn from outcome?
43. 04:27No. Uh so, we have come up with something called utility score, which is a similarity weighted by how useful it is for the agent to execute the task.
44. 04:41It has actually the history of past pieces and past outcomes.
45. 04:47So, we came up with Agent RX, and that is agent's the runtime experience.
46. 04:51It's a runtime layer that let the large language agent improve from experiences without retraining, fine-tuning, or manual prompt engineering.
47. 05:01How it's a little different from compile time like DS5 because of you bake in all the lessons in the prompt, here is actually improving while it is executing the task.
48. 05:14So, which again introduces utility score.
49. 05:19So, you do not retrieve by keyword, you do retrieve by semantic similarity to the current task weighted by whether those memories have historically helped or hurt the execution or the outcome.
50. 05:32The event outcome becomes a first-class signal in the retrieval re-ranking and not just for retrieval.
51. 05:40One of the key things is it treat memory as reasoning, not as facts, statistics, fact with no context and no history, but reasoning.
52. 05:53Like suppose if if there's a if there is a customer support bot looking for a refund, you will not really say, "Hey, user prefers uh uh dark theme or user prefers
53. 06:09to be called by a shorter name."
54. 06:11It's actually reason about query like if some someone asked for refund, you should check the settlement before refunding it so that the the the customer doesn't get paid refund twice.
55. 06:26So, we act based on usefulness.
56. 06:28Context is updated based on task.
57. 06:31So, this is a very big thing because most of the agents deal with context stuffing and this has been brought up in the past and learn from history and reasoning,
58. 06:41right?
59. 06:42Talking about benchmarks.
60. 06:44So, we have benchmarks our memory system reflect on towel bench which essentially measure if agents have followed a policy well or not.
61. 06:56So, we have seen the performance improve from 66 to 76% without baking in uh skills and with skills reflect performance at 80.
62. 07:08So, once there are enough memory like 10 memories, uh what we do is we bake in the reasoning and the understanding into skills so that your agent always remains updated.
63. 07:23What happens most of the time we have seen suppose you have a product SQL agent and there's a system there's a column in system prompt.
64. 07:32Even though that column is no useful anymore, it remains in the system prompt.
65. 07:36So, there's no system right now that can update that, "Hey, there's no column uh right now called this.
66. 07:43So, maybe probably you shouldn't entertain it in the future.
67. 07:47And this is possible with skills that because it is always uses calls that skill updated skill all the time.
68. 07:58And the similar behavior has been seen in GPT-5.4.
69. 08:02Uh, we have also benchmarked it on agentic task.
70. 08:06Which essentially test a model's ability to reason, plan, and use tools or extended multi-step workflows rather than measuring a static Q&A.
71. 08:16So, you can see here, uh, when does it was the human last exam with drug, it get 47.5.
72. 08:25If it is starting from the baseline 35.7.
73. 08:29Uh, with the other memory system, it gets to 58.2.
74. 08:33But with the refined memory system, uh, it gets to 61.3%.
75. 08:38So, this this kind of trend is shown in another, uh, agentic benchmarks, uh, as well, like Big Code Bench, like Long TV, etc. So, of course, there are limitations
76. 08:52to this, uh, to this approach.
77. 08:56First of all, there's a cold start.
78. 08:58So, in the beginning, it's pure semantic search until, uh, enough reviews have been accumulated.
79. 09:06Uh, there's a utility drift.
80. 09:09Maybe sometimes, uh, similar memories could come up.
81. 09:12There are a lot of problems that could come at scale with this experience, but we have combat most of them.
82. 09:19Uh, there's a review quality.
83. 09:21So, noisy labels can make the utility noisy as well.
84. 09:25And there is, uh, hyperparameter called lambda that is associated with credit and re-ranking.
85. 09:32So, we have built reflect in such a way that most of these problems and most of these elements are now reduced except for cold start which we cannot do much about it.
86. 09:44Uh let's get into the demo.
87. 09:48So let's check this demo.
88. 09:57Um So it's a basically a product SQL demo.
89. 10:01I'll give ask it to search some product in SQL database and see if it is able to find it out.
90. 10:25So can you find me a gaming mouse?
91. 10:30Zero memories retrieved.
92. 10:31I couldn't find a gaming mouse.
93. 10:34Uh okay.
94. 10:35So maybe let's just go and see what's happening in the dashboard.
95. 10:43Okay.
96. 10:44It came out that I'm going on another benchmark that with the agent bench with actions essentially measures if agent has actually done the right reasoning, planning, and uh have followed.
97. 11:09Which essentially test a model's ability to reason, plan, and use tools over extended multi-step workflows rather than measuring a static Q&A.
98. 11:19So you can see here uh with the suppose the human last exam uh with drug you get 47.5.
99. 11:28If it is starting from the baseline 35.7.
100. 11:32Uh with the other memory system, it gets 258.2 but with the result memory system, uh it gets 261.3%.
101. 11:41So, this this kind of trend is shown in another uh adaptive benchmarks as well like big code bench, my long TV, etc. Wait, let's check.
102. 11:58I couldn't find I couldn't find any gaming mouse in the product catalog.
103. 12:03So, suppose I want to market fail and sell it give any relevant No. Because there is a there is a wireless mouse in the database.
104. 12:18And I have submitted the failure.
105. 12:21Uh this was the input.
106. 12:23This was the response.
107. 12:25I couldn't find any.
108. 12:26And this was a trajectory and tool quality.
109. 12:29So, you can see the product is empty with this product or with this query.
110. 12:35Let's uh let's check again what happens now.
111. 12:48Let's see now now what happens.
112. 12:56Okay, ID uh So, it searched a wireless mouse.
113. 13:08Uh Let's see what happens in the dashboard now.
114. 13:16Finally, a gaming mouse was input.
115. 13:19The response was I found a product related to mouse as the mouse failed to find any kind of relevant.
116. 13:25Let then, The most important thing is how the tool call evolved.
117. 13:30You can see that previously the tool calls or the trajectory used to look just with one search, that is search products it called.
118. 13:42And the product was empty.
119. 13:44Now, the trajectory has changed in the production and it found something in the product and it uh is still calling search product to call but it is getting some answer that we fed in as a feedback.
120. 14:02So, uh that's the demo and the most important thing that is happening over here is it's forming memories which is retrieved based on the utility score that is the score which basically keeps improving
121. 14:19keeps re-ranking itself on the basis of how useful that memory was.
122. 14:25So, uh You can see from this traces past traces which memory uh so this memory scores keep changing and after certain while after suppose uh five reviews maybe can bake in these
123. 14:48findings or these new updates in a skill.
124. 14:53Without so without changing the existing prompt you can actually update certain things that agent draws a lot from.
125. 15:05So, uh you can update the skill which is very cool.
126. 15:09So, uh I hope everything was I hope you get to try this new feature this new uh runtime experience that we have built.
127. 15:26You want more details you can visit our website and you can also contact me if I have still my grace@alishascollection.com.
128. 15:34I will reply.
129. 15:35Thank you.