WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Jx4ZFEAq6bY

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Jx4ZFEAq6bY.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.040 --> 00:00:03.000
Hey everyone, I'm Surim.

s2
00:00:03.000 --> 00:00:10.000
I'm the CEO and co-founder of Starling Search and today my talk is a signal's die after retrieval boundary.

s3
00:00:10.000 --> 00:00:24.200
So, we'll look into uh what are agents essentially, why agents fail, what is uh the cause of fails in retrieval particularly, and how to make actually signals cross the retrieval boundary,

s4
00:00:24.200 --> 00:00:29.080
and how to make your agent basically outcome fair.

s5
00:00:29.080 --> 00:00:31.880
So, uh let's get started.

s6
00:00:31.880 --> 00:00:32.880
What is an agent?

s7
00:00:32.880 --> 00:00:43.360
An agent is an LLM that has agency to reason, invoke tools, uh interact with the real world, retrieve uh the memory to complete the task.

s8
00:00:43.360 --> 00:00:47.000
One major loop here is missing is uh learning.

s9
00:00:47.000 --> 00:00:50.840
It should also learn from what worked and what didn't work.

s10
00:00:50.840 --> 00:00:58.800
Suppose uh if I had to uh explain what is agent, I I can explain with react agent.

s11
00:00:58.800 --> 00:01:04.080
So, if I had to explain uh what agent does, I would explain with react agent.

s12
00:01:04.080 --> 00:01:13.880
So, basically use it from the agent and execute it in a loop, uh client to retrieval search, and then pause when the task is complete.

s13
00:01:13.880 --> 00:01:18.040
This is very basic react architecture.

s14
00:01:18.040 --> 00:01:23.800
One thing that is missing is how to make agent learn from the outcome.

s15
00:01:24.120 --> 00:01:26.600
So, agent keeps failing at the same task.

s16
00:01:26.600 --> 00:01:30.880
Gartner reported 85% of AI just failing traction.

s17
00:01:30.880 --> 00:01:34.560
So, it's in McKinsey's 2025 report.

s18
00:01:34.560 --> 00:01:40.680
The problem came out to be most of the time is that retrieval is static.

s19
00:01:40.680 --> 00:01:49.040
73% of our uh pipelines fails because of retrieval, not generation, and context stuffing.

s20
00:01:49.040 --> 00:01:59.440
So, a recent uh post from Ram Sriram, the ex-CTO of Pinecone, said, "Uh we have been optimizing for the wrong thing.

s21
00:01:59.440 --> 00:02:02.120
You are paying a lot for your agent's memory.

s22
00:02:02.120 --> 00:02:05.920
This is probably broken, and we have been optimizing for the wrong things.

s23
00:02:05.920 --> 00:02:12.200
We made wrong answers appear faster and cheaper, but we forgot to make retrieval learn."

s24
00:02:12.200 --> 00:02:16.160
So, why does this does does does it matter?

s25
00:02:16.360 --> 00:02:21.400
Again, uh, the third problem is agents are not outcome-informed.

s26
00:02:21.880 --> 00:02:25.480
So, there's a missing layer between evals and action.

s27
00:02:25.480 --> 00:02:37.240
Your observability has all the traces, uh, all the stack that capture observability is the stack that capture every tool call, every LM completion, every exceptions.

s28
00:02:37.240 --> 00:02:44.280
Your eval suite judges whether the final output was correct or wrong, basically pass or fail.

s29
00:02:44.280 --> 00:02:57.840
But, these evals are not reflected in agent, uh, context, skills, and defines my agent action in any ways.

s30
00:02:57.840 --> 00:02:59.720
Right?

s31
00:02:59.720 --> 00:03:05.320
So, the agent doesn't have any access to why yesterday's runs, uh, runs passed or failed.

s32
00:03:05.320 --> 00:03:08.280
The eval signal dies in the dashboard.

s33
00:03:08.280 --> 00:03:16.959
This is a missing layer, a system that consume traces, absorb eval, and convert both into retrieval guidance for future runs.

s34
00:03:17.360 --> 00:03:25.080
So, there's a manual improvement task, and engineer actually has to sit and see if the eval and observability perform well.

s35
00:03:25.080 --> 00:03:35.680
Rewrite the prompt, redeploy it, upgrade either and create your expensive model, restructure to our harness, or fine-tune the custom models.

s36
00:03:36.800 --> 00:03:38.720
Why are current memories failing?

s37
00:03:38.720 --> 00:03:47.880
Why memories was designed to actually address this, but, uh, uh, it's not.

s38
00:03:47.880 --> 00:04:03.720
So, let's see, uh, a what we have in our the current system uh, and current memory is that they basically store user preferences, profile, conversation history, or long-lived personalization.

s39
00:04:04.880 --> 00:04:09.800
So, chat experience is not self-improving learning systems for production.

s40
00:04:09.800 --> 00:04:21.519
If you see the already existing approach in the market, there's a LangChain, there's Mem0, which does extracted past preferences.

s41
00:04:21.519 --> 00:04:26.320
The user's retrieval signal is embedding similarity.

s42
00:04:26.360 --> 00:04:27.600
Does it learn from outcome?

s43
00:04:27.600 --> 00:04:41.240
No. Uh so, we have come up with something called utility score, which is a similarity weighted by how useful it is for the agent to execute the task.

s44
00:04:41.240 --> 00:04:47.080
It has actually the history of past pieces and past outcomes.

s45
00:04:47.080 --> 00:04:51.840
So, we came up with Agent RX, and that is agent's the runtime experience.

s46
00:04:51.840 --> 00:05:01.040
It's a runtime layer that let the large language agent improve from experiences without retraining, fine-tuning, or manual prompt engineering.

s47
00:05:01.040 --> 00:05:14.680
How it's a little different from compile time like DS5 because of you bake in all the lessons in the prompt, here is actually improving while it is executing the task.

s48
00:05:14.680 --> 00:05:19.200
So, which again introduces utility score.

s49
00:05:19.200 --> 00:05:32.720
So, you do not retrieve by keyword, you do retrieve by semantic similarity to the current task weighted by whether those memories have historically helped or hurt the execution or the outcome.

s50
00:05:32.720 --> 00:05:40.200
The event outcome becomes a first-class signal in the retrieval re-ranking and not just for retrieval.

s51
00:05:40.919 --> 00:05:53.880
One of the key things is it treat memory as reasoning, not as facts, statistics, fact with no context and no history, but reasoning.

s52
00:05:53.880 --> 00:06:09.320
Like suppose if if there's a if there is a customer support bot looking for a refund, you will not really say, "Hey, user prefers uh uh dark theme or user prefers

s53
00:06:09.320 --> 00:06:11.880
to be called by a shorter name."

s54
00:06:11.880 --> 00:06:26.440
It's actually reason about query like if some someone asked for refund, you should check the settlement before refunding it so that the the the customer doesn't get paid refund twice.

s55
00:06:26.440 --> 00:06:28.760
So, we act based on usefulness.

s56
00:06:28.760 --> 00:06:31.160
Context is updated based on task.

s57
00:06:31.160 --> 00:06:41.520
So, this is a very big thing because most of the agents deal with context stuffing and this has been brought up in the past and learn from history and reasoning,

s58
00:06:41.520 --> 00:06:42.360
right?

s59
00:06:42.360 --> 00:06:44.760
Talking about benchmarks.

s60
00:06:44.760 --> 00:06:56.000
So, we have benchmarks our memory system reflect on towel bench which essentially measure if agents have followed a policy well or not.

s61
00:06:56.000 --> 00:07:08.640
So, we have seen the performance improve from 66 to 76% without baking in uh skills and with skills reflect performance at 80.

s62
00:07:08.640 --> 00:07:23.880
So, once there are enough memory like 10 memories, uh what we do is we bake in the reasoning and the understanding into skills so that your agent always remains updated.

s63
00:07:23.880 --> 00:07:32.000
What happens most of the time we have seen suppose you have a product SQL agent and there's a system there's a column in system prompt.

s64
00:07:32.000 --> 00:07:36.320
Even though that column is no useful anymore, it remains in the system prompt.

s65
00:07:36.320 --> 00:07:43.880
So, there's no system right now that can update that, "Hey, there's no column uh right now called this.

s66
00:07:43.880 --> 00:07:47.880
So, maybe probably you shouldn't entertain it in the future.

s67
00:07:47.880 --> 00:07:58.080
And this is possible with skills that because it is always uses calls that skill updated skill all the time.

s68
00:07:58.080 --> 00:08:02.360
And the similar behavior has been seen in GPT-5.4.

s69
00:08:02.360 --> 00:08:06.360
Uh, we have also benchmarked it on agentic task.

s70
00:08:06.360 --> 00:08:16.680
Which essentially test a model's ability to reason, plan, and use tools or extended multi-step workflows rather than measuring a static Q&A.

s71
00:08:16.680 --> 00:08:25.640
So, you can see here, uh, when does it was the human last exam with drug, it get 47.5.

s72
00:08:25.640 --> 00:08:29.080
If it is starting from the baseline 35.7.

s73
00:08:29.200 --> 00:08:33.080
Uh, with the other memory system, it gets to 58.2.

s74
00:08:33.080 --> 00:08:38.479
But with the refined memory system, uh, it gets to 61.3%.

s75
00:08:38.479 --> 00:08:52.680
So, this this kind of trend is shown in another, uh, agentic benchmarks, uh, as well, like Big Code Bench, like Long TV, etc. So, of course, there are limitations

s76
00:08:52.680 --> 00:08:56.240
to this, uh, to this approach.

s77
00:08:56.240 --> 00:08:58.000
First of all, there's a cold start.

s78
00:08:58.000 --> 00:09:06.440
So, in the beginning, it's pure semantic search until, uh, enough reviews have been accumulated.

s79
00:09:06.440 --> 00:09:09.320
Uh, there's a utility drift.

s80
00:09:09.320 --> 00:09:12.839
Maybe sometimes, uh, similar memories could come up.

s81
00:09:12.839 --> 00:09:19.280
There are a lot of problems that could come at scale with this experience, but we have combat most of them.

s82
00:09:19.280 --> 00:09:21.880
Uh, there's a review quality.

s83
00:09:21.880 --> 00:09:25.600
So, noisy labels can make the utility noisy as well.

s84
00:09:25.600 --> 00:09:32.080
And there is, uh, hyperparameter called lambda that is associated with credit and re-ranking.

s85
00:09:32.080 --> 00:09:44.480
So, we have built reflect in such a way that most of these problems and most of these elements are now reduced except for cold start which we cannot do much about it.

s86
00:09:44.480 --> 00:09:48.000
Uh let's get into the demo.

s87
00:09:48.680 --> 00:09:51.920
So let's check this demo.

s88
00:09:57.240 --> 00:10:01.680
Um So it's a basically a product SQL demo.

s89
00:10:01.680 --> 00:10:10.520
I'll give ask it to search some product in SQL database and see if it is able to find it out.

s90
00:10:25.800 --> 00:10:29.440
So can you find me a gaming mouse?

s91
00:10:30.160 --> 00:10:31.800
Zero memories retrieved.

s92
00:10:31.800 --> 00:10:34.120
I couldn't find a gaming mouse.

s93
00:10:34.120 --> 00:10:35.240
Uh okay.

s94
00:10:35.240 --> 00:10:41.080
So maybe let's just go and see what's happening in the dashboard.

s95
00:10:43.640 --> 00:10:44.360
Okay.

s96
00:10:44.360 --> 00:11:07.080
It came out that I'm going on another benchmark that with the agent bench with actions essentially measures if agent has actually done the right reasoning, planning, and uh have followed.

s97
00:11:09.320 --> 00:11:19.560
Which essentially test a model's ability to reason, plan, and use tools over extended multi-step workflows rather than measuring a static Q&A.

s98
00:11:19.560 --> 00:11:28.560
So you can see here uh with the suppose the human last exam uh with drug you get 47.5.

s99
00:11:28.560 --> 00:11:31.960
If it is starting from the baseline 35.7.

s100
00:11:32.080 --> 00:11:41.360
Uh with the other memory system, it gets 258.2 but with the result memory system, uh it gets 261.3%.

s101
00:11:41.360 --> 00:11:58.360
So, this this kind of trend is shown in another uh adaptive benchmarks as well like big code bench, my long TV, etc. Wait, let's check.

s102
00:11:58.360 --> 00:12:03.080
I couldn't find I couldn't find any gaming mouse in the product catalog.

s103
00:12:03.080 --> 00:12:18.080
So, suppose I want to market fail and sell it give any relevant No. Because there is a there is a wireless mouse in the database.

s104
00:12:18.080 --> 00:12:21.360
And I have submitted the failure.

s105
00:12:21.360 --> 00:12:23.960
Uh this was the input.

s106
00:12:23.960 --> 00:12:25.120
This was the response.

s107
00:12:25.120 --> 00:12:26.920
I couldn't find any.

s108
00:12:26.920 --> 00:12:29.800
And this was a trajectory and tool quality.

s109
00:12:29.800 --> 00:12:35.720
So, you can see the product is empty with this product or with this query.

s110
00:12:35.720 --> 00:12:40.000
Let's uh let's check again what happens now.

s111
00:12:48.720 --> 00:12:52.280
Let's see now now what happens.

s112
00:12:56.400 --> 00:13:08.280
Okay, ID uh So, it searched a wireless mouse.

s113
00:13:08.280 --> 00:13:14.920
Uh Let's see what happens in the dashboard now.

s114
00:13:16.680 --> 00:13:19.560
Finally, a gaming mouse was input.

s115
00:13:19.560 --> 00:13:25.760
The response was I found a product related to mouse as the mouse failed to find any kind of relevant.

s116
00:13:25.760 --> 00:13:30.160
Let then, The most important thing is how the tool call evolved.

s117
00:13:30.160 --> 00:13:42.720
You can see that previously the tool calls or the trajectory used to look just with one search, that is search products it called.

s118
00:13:42.720 --> 00:13:44.960
And the product was empty.

s119
00:13:44.960 --> 00:14:02.200
Now, the trajectory has changed in the production and it found something in the product and it uh is still calling search product to call but it is getting some answer that we fed in as a feedback.

s120
00:14:02.200 --> 00:14:19.720
So, uh that's the demo and the most important thing that is happening over here is it's forming memories which is retrieved based on the utility score that is the score which basically keeps improving

s121
00:14:19.720 --> 00:14:25.880
keeps re-ranking itself on the basis of how useful that memory was.

s122
00:14:25.880 --> 00:14:48.440
So, uh You can see from this traces past traces which memory uh so this memory scores keep changing and after certain while after suppose uh five reviews maybe can bake in these

s123
00:14:48.440 --> 00:14:53.560
findings or these new updates in a skill.

s124
00:14:53.560 --> 00:15:05.360
Without so without changing the existing prompt you can actually update certain things that agent draws a lot from.

s125
00:15:05.360 --> 00:15:09.720
So, uh you can update the skill which is very cool.

s126
00:15:09.720 --> 00:15:25.480
So, uh I hope everything was I hope you get to try this new feature this new uh runtime experience that we have built.

s127
00:15:26.000 --> 00:15:34.360
You want more details you can visit our website and you can also contact me if I have still my grace@alishascollection.com.

s128
00:15:34.360 --> 00:15:35.920
I will reply.

s129
00:15:35.920 --> 00:15:38.280
Thank you.