WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=2IxD9OB3XuQ NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/2IxD9OB3XuQ.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.560 --> 00:00:02.120 Hi everyone. s2 00:00:02.120 --> 00:00:03.520 My name is Soheil Feizi. s3 00:00:03.520 --> 00:00:05.960 I'm founder and CEO at Rely. s4 00:00:05.960 --> 00:00:10.800 I'm also an associate professor in the computer science department at University of Maryland. s5 00:00:10.800 --> 00:00:15.200 Today I'm going to talk about continual learning for AI agents. s6 00:00:15.200 --> 00:00:19.160 How we can go from failures to durable improvements. s7 00:00:19.160 --> 00:00:28.120 And if you're interested in any of the tools that I'll be talking in this presentation, you can visit at our web at our website rely.ai. s8 00:00:28.360 --> 00:00:30.240 Let's get this started. s9 00:00:30.240 --> 00:00:36.960 Humans learn mainly from experience by interacting with the world and getting feedback. s10 00:00:37.240 --> 00:00:42.440 The goal of continual learning is to imitate the same for agents. s11 00:00:42.440 --> 00:00:50.320 So they can also learn from experience by acting, getting feedback, and improving without forgetting. s12 00:00:50.440 --> 00:00:50.640 All right. s13 00:00:50.640 --> 00:00:58.880 So here's basically a bigger picture of how continual learning for agents look like. s14 00:00:58.880 --> 00:01:07.160 So here's an agent that interact with the world, with diverse users, with complex tools, uh with various data policies. s15 00:01:07.160 --> 00:01:12.840 And as I mentioned, the goal is to continuously improve the agent from its experience without forgetting. s16 00:01:12.840 --> 00:01:16.680 And this learning can happen in different layers of the agent. s17 00:01:16.680 --> 00:01:27.480 It can happen in the model layer where potentially we can change weights of LLMs or other models used in the agent, or use different types of models in the agent. s18 00:01:27.480 --> 00:01:39.680 It can happen in the harness layer where it brings the context, proper context to the LLM uh with components like prompts, skills, tools, code, workflow. s19 00:01:39.680 --> 00:01:46.200 And it can also happen in the memory, either in session memory or persistent memory of the agent. s20 00:01:48.240 --> 00:01:53.360 But there are two, I would say, fundamental challenges in continual learning for agents. s21 00:01:53.360 --> 00:01:55.840 The first challenge is how to get feedback. s22 00:01:55.840 --> 00:01:58.400 How do we know if the agent did well? s23 00:01:58.400 --> 00:02:01.840 And if not, what should it have done instead? s24 00:02:01.840 --> 00:02:04.560 That's basically the first part, getting the feedback. s25 00:02:04.560 --> 00:02:07.520 And the second part is how we can act upon that feedback. s26 00:02:07.520 --> 00:02:12.920 How we can optimize and improve agent and learn from that feedback. s27 00:02:12.920 --> 00:02:17.680 Which layer, which component do we need to change and also how. s28 00:02:17.680 --> 00:02:28.200 I'll be talking about these two challenges, current approaches in order to deal with them and also provide some perspective of how we think about these two problems. s29 00:02:28.200 --> 00:02:31.400 So, let's get it started with the first problem. s30 00:02:31.400 --> 00:02:34.520 Where does the feedback come from? s31 00:02:34.520 --> 00:02:40.480 So, the easy case is when we have a benchmark and some evaluators on that benchmark. s32 00:02:40.480 --> 00:02:43.160 So, the agent can run tasks from the benchmark. s33 00:02:43.160 --> 00:02:55.240 Now, we have the evaluators in order to score and we can get grades like pass, fail, or reward, as well as potentially some feedback on the agent behavior and agent performance. s34 00:02:55.240 --> 00:03:06.959 This is usually what is happening during the development time that different teams they curate benchmarks in order to understand the performance of the agent in certain applications. s35 00:03:08.200 --> 00:03:11.160 But in production, we don't have such benchmark. s36 00:03:11.160 --> 00:03:12.760 We have logs. s37 00:03:12.760 --> 00:03:17.120 Here's an example of a session log where a user is interacting with the agent. s38 00:03:17.120 --> 00:03:23.760 Maybe the user is not very happy with the way way the agent is behaving, but we don't have any explicit feedback. s39 00:03:23.760 --> 00:03:29.000 So, there are two ways of getting such feedback on such session logs. s40 00:03:29.000 --> 00:03:37.840 One is automatic using some other models or LLMs or code in order to analyze the log and provide feedback. s41 00:03:37.840 --> 00:03:43.840 In some cases, even the agent itself can look at it is log and provide some critics of it. s42 00:03:43.840 --> 00:03:47.000 It is automatic and it is scalable. s43 00:03:47.000 --> 00:03:59.800 And the second approach is where we have human experts to look at some handful of these logs and provide some domain expert expert feedback on those agent outputs. s44 00:03:59.800 --> 00:04:12.600 Uh this is lower in the volume, but it is critical because it provides expert knowledge on the behavior of the agent and its is alignment with the way that we want agent to behave in those applications. s45 00:04:12.600 --> 00:04:19.079 Either way, now we have session log plus some feedback on those logs. s46 00:04:19.079 --> 00:04:21.000 Is it enough? s47 00:04:21.000 --> 00:04:25.760 The answer is no because it is still not testable. s48 00:04:25.760 --> 00:04:38.960 Here we have log and feedback, but what we really need is a replayable learning environment, a simulation that we can rerun with defined grading on what success looks like, not one instance of what happened s49 00:04:38.960 --> 00:04:42.200 and the feedback on top of it. s50 00:04:43.080 --> 00:04:45.000 So what is a learning environment? s51 00:04:45.000 --> 00:04:53.240 Here we are inferring a distribution from one observation that replaces what happened and what success means. s52 00:04:53.240 --> 00:04:58.600 Uh the input is what we have some session logs and feedback. s53 00:04:58.600 --> 00:05:13.680 This is basically one observation of what happened and now we want to create a simulation and evaluation environment from that information that involves, for example, understanding how tools in the agent s54 00:05:13.680 --> 00:05:25.120 behavior in the agent log should behave, should we use real tools, should we use mock tools, and if so, what kind of data brought to the mocking process of those those tools. s55 00:05:25.120 --> 00:05:32.640 If the agent is interacting with some users, how we can infer synthetic users from that data. s56 00:05:32.640 --> 00:05:37.320 And also how success looks like in that learning environment. s57 00:05:37.320 --> 00:05:39.960 What are the evaluators that needs to be inferred. s58 00:05:39.960 --> 00:05:50.520 So there are lots of technical challenges in any of these components, but if we could do this successfully, they could use this the output as executable. s59 00:05:50.520 --> 00:06:04.520 We can now run different candidates of the agents against such learning environment, understand the behavior and the performance of the agent in those scenarios and in those patterns, and we can fix the issues s60 00:06:04.520 --> 00:06:12.240 um based on the feedback, based on the information that we observe because now everything becomes testable and verifiable. s61 00:06:15.200 --> 00:06:15.440 All right. s62 00:06:15.440 --> 00:06:24.760 So, the second problem is now we have this feedback, how we can act upon it, how we can optimize the agent. s63 00:06:25.200 --> 00:06:31.120 And from a high-level point of view, there are three layers that we can improve the agent. s64 00:06:31.120 --> 00:06:42.720 Uh there's a model layer where we can change the weights of the model, and there are methods like SFT, so supervised fine-tuning, uh RL-based post-training in order to make those changes. s65 00:06:42.720 --> 00:06:51.040 And these are usually expensive because that requires uh more intensive compute in terms of changing the model weights. s66 00:06:51.040 --> 00:07:03.800 The second layer is the harness layer, harness engineering, context engineering, where we can potentially pre-write prompts, maybe learn some skills, uh change tools or add tools, maybe change code around s67 00:07:03.800 --> 00:07:12.880 around the LLM, and there are different methods like get past race to harness, uh which provides a lot of flexibility in terms of learning from that feedback. s68 00:07:12.880 --> 00:07:23.720 And the last layer is the memory layer, where we store facts and learn skills uh in order to not repeat those issues uh and failures in in the future. s69 00:07:23.720 --> 00:07:30.080 But a good learning is not going to be focusing on any of these components exclusively. s70 00:07:30.080 --> 00:07:39.560 A good learning engine should ask for the smallest durable change at the right layer of the agent. s71 00:07:41.960 --> 00:07:42.200 All right. s72 00:07:42.200 --> 00:07:47.280 So, let me uh dig a little bit deeper into each of these layers. s73 00:07:47.280 --> 00:07:58.720 So, first in terms of updating the model weights, there are various approaches uh in order to do that, uh including SFT, supervised fine-tuning, where we imitate correct trajectories. s74 00:07:58.720 --> 00:08:06.240 Uh we often need labeled samples in order to um fit those samples, uh fit the model to those samples. s75 00:08:06.240 --> 00:08:19.000 Other approaches are based on RL, um reinforcement learning post-training, like DPO, GRPO, RLV R, where we sample score against the reward or preference signals, and reinforce what wins. s76 00:08:19.000 --> 00:08:25.800 And there are some categories based on LoRA, low-rank adaptation, that limits the set of parameters that can potentially change. s77 00:08:25.800 --> 00:08:32.960 It makes this uh the learning in this layer cheaper, and also safer um in terms of the updates. s78 00:08:32.960 --> 00:08:37.360 But these methods, they usually need benchmarks and explicit evaluators. s79 00:08:37.360 --> 00:08:48.040 They cannot be directly applied on, let's say, if you have a log and feedback, unless we turn those into uh replayable learning environments. s80 00:08:48.680 --> 00:08:57.520 Right, in terms of uh updating the harness, uh I would highlight the two uh categories in in this domain, in this layer. s81 00:08:57.520 --> 00:08:59.600 One is trace-to-harness approaches. s82 00:08:59.600 --> 00:09:09.320 Let's say you observe a log, you have some feedback on top of it, you can effectively ask a coding agent in order to analyze the log and improve the agent. s83 00:09:09.320 --> 00:09:15.280 So, this works on uh the case where we have log and feedback, but it is wipe-based. s84 00:09:15.280 --> 00:09:24.800 We don't know if even for that particular uh sample, if the change is effective, because it is not testable. s85 00:09:24.800 --> 00:09:30.400 And we don't know what is the impact of it on other samples and other scenarios. s86 00:09:30.400 --> 00:09:38.840 What uh might have been working previously, but with these changes might not work properly, and create some hidden regressions. s87 00:09:38.840 --> 00:09:52.880 The other category that I want to highlight is methods like GEPPO and prompt search where they mutate prompts, they score different candidates, and they keep the winners using some search algorithms like evolutionary algorithms. s88 00:09:52.880 --> 00:10:03.440 And these are These methods are testable, but they need a benchmark and explicit evaluators in order to have those have those scorings. s89 00:10:04.040 --> 00:10:13.840 And in the memory layer, we effectively write down facts and we distill skills so the agent doesn't rediscover them. s90 00:10:13.840 --> 00:10:25.640 It can happen in the information memory layer, methods like LETA and MemZero where they can effectively store a fact or correction. s91 00:10:25.640 --> 00:10:36.680 And we have also methods to skill distillation that compresses successful trajectory into reusable how-to do skill for the for the agent. s92 00:10:36.680 --> 00:10:41.560 So, this layer in terms of the update is cheapest and fastest. s93 00:10:41.560 --> 00:10:53.160 It works directly on the cases where you only have log and feedback, but usually it is unverified because you don't have a way in order to test whether or not s94 00:10:53.160 --> 00:11:02.240 writing in the memory will resolve the issues that you have dealt with and whether or not it can potentially create some regressions on some other cases. s95 00:11:02.240 --> 00:11:09.800 With that, let me introduce a new subcategory of continual learning called verifiable continual learning. s96 00:11:10.320 --> 00:11:22.000 In a verifiable continual learning, the goal is to improve an agent from its his own experience where every fix is proven to help and proven to break nothing that already worked. s97 00:11:22.000 --> 00:11:25.480 And usually it involve it involves three steps. s98 00:11:25.480 --> 00:11:31.440 Where we need to have an executable test where the failure becomes a task you can replay and create. s99 00:11:31.440 --> 00:11:36.680 Then we need to have a measured delta where the update is scored on the test before and after. s100 00:11:36.680 --> 00:11:39.040 And then we have a regression test. s101 00:11:39.040 --> 00:11:46.400 So prior test still pass even after we make such changes to the agent. s102 00:11:47.360 --> 00:11:54.920 So let's think about what are the principles of a practical verifiable continual learning first. s103 00:11:54.920 --> 00:12:00.320 And I will argue there are four important principles that we need to keep in mind. s104 00:12:00.320 --> 00:12:03.240 So the first principle is replayability. s105 00:12:03.240 --> 00:12:07.480 We need to turn a one of failure into a test that we can rerun. s106 00:12:07.480 --> 00:12:14.240 Here, as I had mentioned previously, many cases we have log and feedback, but that is not testable. s107 00:12:14.240 --> 00:12:24.160 We need to lift it in a learning environment to simulate and evaluate the agent on a similar pattern on a similar scenario. s108 00:12:24.160 --> 00:12:30.960 So everything becomes testable based on that simulation and evaluation environment. s109 00:12:30.960 --> 00:12:34.960 So that's basically the first principle that we need to have. s110 00:12:34.960 --> 00:12:37.720 The second principle is holisticness. s111 00:12:37.720 --> 00:12:41.560 One failure may have several causes and several possible repairs. s112 00:12:41.560 --> 00:12:42.920 You know, let me give you an example. s113 00:12:42.920 --> 00:12:48.120 Let's say you have an agent that cites a stale policy and skips the required escalation. s114 00:12:48.120 --> 00:12:51.360 The issue might come from the memory where you have some stale fact. s115 00:12:51.360 --> 00:12:54.880 It can come from uh not optimized prompt. s116 00:12:54.880 --> 00:12:59.320 It might come from a tool that doesn't normalize the policy. s117 00:12:59.320 --> 00:13:03.520 Might come from the workflow that we need to add escalation gate before we run. s118 00:13:03.520 --> 00:13:05.280 It might come from the model. s119 00:13:05.280 --> 00:13:09.880 Maybe the model is not a good model in order to have a strong reasoning. s120 00:13:09.880 --> 00:13:20.480 So here we need to route the fix to the right layers that explains the failure with the smallest durable change to the agent. s121 00:13:20.480 --> 00:13:26.800 And that is basically the principle of holisticness in verifiable continual learning. s122 00:13:26.800 --> 00:13:29.160 The third principle is lifelongness. s123 00:13:29.160 --> 00:13:33.480 A new fix must improve the new case without breaking the past. s124 00:13:33.480 --> 00:13:44.640 Let's consider the setup where we already optimize the agent on K past learning environments and a new failure comes and we turn that into a learning environment EK plus one. s125 00:13:44.640 --> 00:13:47.160 What do want to change? s126 00:13:47.160 --> 00:13:59.600 So, the first approach is okay, just focus on this new learning environment, but that can create regression on the past behavior on the past learning environments that the agent was successful. s127 00:13:59.600 --> 00:14:11.160 A better approach is a regression aware learning where the regression is not be treated as a post-hoc approach, but as a mechanism within the optimization itself. s128 00:14:11.160 --> 00:14:19.800 So, here we are uh fixing the recent failures subject to having no regression on the past uh learning environments. s129 00:14:19.800 --> 00:14:33.080 And obviously, this needs to be done in a efficient manner, so it doesn't scale even linearly with K because K can grow and the complexity of this approach can uh can s130 00:14:33.080 --> 00:14:37.240 uh can be very can be very high. s131 00:14:38.640 --> 00:14:44.360 And the last but not least principle is the efficiency. s132 00:14:44.360 --> 00:14:54.760 This continual learning loop needs to run frequently and we need to have efficiency in uh different layers in updates to the agent. s133 00:14:54.760 --> 00:15:10.320 So, sometimes the change can be cheap, like for example, writing something in the memory can be medium in terms of the complexity by changing the prompt or hardness, and sometimes it can be very expensive s134 00:15:10.320 --> 00:15:13.960 by changing the weights of the model. s135 00:15:13.960 --> 00:15:29.480 Also, efficiency should be in the optimization loop itself, especially when we have regression aware optimization and regression is treated within the loop not as a post hoc approach. s136 00:15:31.480 --> 00:15:37.280 To sum up, these are the four principles of a practical verifiable continual learning. s137 00:15:37.280 --> 00:15:43.480 Replayability, holisticness, lifelongness, and efficiency. s138 00:15:44.360 --> 00:15:56.400 And this is what we have been working on at Rely to create a verifiable continual learning engine for AI agents based on these four principles. s139 00:15:56.800 --> 00:16:01.240 In particular, here's how Rely's learning loop runs. s140 00:16:01.240 --> 00:16:06.120 You can start with some signals to this loop. s141 00:16:06.120 --> 00:16:10.960 It can be logs, feedback, or even instructions and prompts. s142 00:16:10.960 --> 00:16:16.400 We lift those in those signals to replayable learning environments. s143 00:16:16.400 --> 00:16:20.320 So, that's based on the replayability principle. s144 00:16:20.320 --> 00:16:25.280 This makes everything that follows testable and verifiable. s145 00:16:25.280 --> 00:16:31.760 Then we do root cause analysis and route the fixes to the right layer of the agent. s146 00:16:31.760 --> 00:16:35.520 It can be memory, it can be model, or it can be harness. s147 00:16:35.520 --> 00:16:40.120 That touches the holisticness principle that I described. s148 00:16:40.120 --> 00:16:43.080 We have regression aware optimization. s149 00:16:43.080 --> 00:16:51.160 Regression is not being treated as a post hoc approach, so that touches the lifelongness principle that I mentioned. s150 00:16:51.160 --> 00:16:55.040 And obviously, this loop should run efficiently. s151 00:16:55.040 --> 00:16:57.800 That touches the efficiency principle. s152 00:16:57.800 --> 00:17:16.000 So, the output of this is a reviewable version update to the agent explaining what changes in the agent during this loop and why those changes are are improving the agent without creating regression. s153 00:17:17.400 --> 00:17:26.520 The beauty of it is you can add a VCL verifiable continual learning to your current agent in just two comments. s154 00:17:26.520 --> 00:17:30.120 So, the first one is a one-time setup. s155 00:17:30.120 --> 00:17:33.680 You create a learning harness in your agent. s156 00:17:33.680 --> 00:17:44.480 You can use your own LLM and your agent can be built on top of any of available major agent frameworks. s157 00:17:44.480 --> 00:17:49.720 And then after that, you need two commands in order to activate this learning loop. s158 00:17:49.720 --> 00:18:03.240 You can create learning environments using various type of signals that you can have either log log feedback or some instructions, and then you can call to like optimize in order to use holistic lifelong s159 00:18:03.240 --> 00:18:05.360 optimizer to improve the agent. s160 00:18:05.360 --> 00:18:15.840 And the output is a optimized version pull request that you can review and you can use it in order to improve your agent. s161 00:18:16.720 --> 00:18:20.560 So, let's look at the how it actually works in practice. s162 00:18:20.560 --> 00:18:36.240 We build a continual learning benchmark on a fictional support agent case where we have reproducible test beds for continual learning in a tool using support agent. s163 00:18:36.240 --> 00:18:45.960 So, we have a single source of truth and the policies are interacting for this agent to be handling. s164 00:18:45.960 --> 00:18:54.160 So, we have deterministic evaluators and we also build this benchmark in a way that it has some regression trap. s165 00:18:54.160 --> 00:19:07.440 So, if the optimizer focuses on overfitting on the latest fix, it can potentially break what are what the agent was previously successful on other tasks. s166 00:19:08.320 --> 00:19:11.200 So, let's say we have an agent. s167 00:19:11.200 --> 00:19:16.080 We don't even have logs or anything, and we want to just see how the agent is behaving. s168 00:19:16.080 --> 00:19:21.040 Uh, let's say when um, caller is rude and adversarial. s169 00:19:21.040 --> 00:19:25.920 So, simply we can create a learning environment using such instruction. s170 00:19:25.920 --> 00:19:31.720 And what it will do, it will create a learning environment to simulate and evaluate the agent. s171 00:19:31.720 --> 00:19:45.840 Uh, the simulator will include personas, intent, mock, uh, or real tools and also the learning environment contain evaluators in order to define success metrics. s172 00:19:45.840 --> 00:19:50.600 All of these are produced from just one interactive comment. s173 00:19:51.480 --> 00:19:58.120 And after that, we can just simulate the agent using this learning environment, see how it behaves. s174 00:19:58.120 --> 00:20:01.280 Okay, the score is uh, not too uh, high. s175 00:20:01.280 --> 00:20:13.360 It is 78% and in particular, there are two um, evaluators that uh, basically show very low scores uh, of agent in this environment. s176 00:20:13.360 --> 00:20:15.240 So, these are some of the failures that we observe. s177 00:20:15.240 --> 00:20:17.120 How to improve such failures? s178 00:20:17.120 --> 00:20:21.840 We can do that by calling Rely Optimize with certain number of rollouts. s179 00:20:21.840 --> 00:20:28.120 And as you can see, the average improvement uh, can be um, quite high. s180 00:20:28.120 --> 00:20:37.400 Uh, it is 10% improvement on average just with one loop and score increases to 97% from 87%. s181 00:20:37.760 --> 00:20:41.920 Okay, now let's consider the case that now the agent is in production. s182 00:20:41.920 --> 00:20:48.840 Now, you have a log, you have an agent session that isn't um, not desired and you have a feedback. s183 00:20:48.840 --> 00:20:55.760 For example, you can say keep fast eligible refunds, but do not generalize generosity beyond refund thresholds. s184 00:20:55.760 --> 00:20:58.680 So, that's a feedback on the agent behavior. s185 00:20:58.680 --> 00:21:07.000 Again, the flow is the same, so we lift it into a replayable learning environment and we call Rely Optimize in order to mitigate this uh, issue. s186 00:21:07.000 --> 00:21:14.400 Use this feedback without creating regression of the agent behavior in past environments. s187 00:21:14.760 --> 00:21:23.480 And this is lifelong, so you can keep doing that uh to improve the agent without breaking what already works, and it is compounding. s188 00:21:23.480 --> 00:21:35.000 This is verifiable continual learning in practice, where each update is tested, every gain is measured, and nothing that already works breaks during this optimization. s189 00:21:35.400 --> 00:21:39.160 So, that's it for uh today and for this talk. s190 00:21:39.160 --> 00:21:42.960 So, there are three key takeaways that I want to highlight. s191 00:21:42.960 --> 00:21:47.960 The first one is agent continual learning is not necessarily model fine-tuning. s192 00:21:47.960 --> 00:21:52.520 The updates and many useful updates can happen in the harness and memory layer. s193 00:21:52.520 --> 00:21:56.560 So, the second takeaway is production logs are not learning environments. s194 00:21:56.560 --> 00:22:04.720 We need to transform them into replayable learning environments to simulate and evaluate the agent on the same patterns and scenarios. s195 00:22:04.720 --> 00:22:16.360 And the third takeaway is that the frontier is regression-aware continual improvement, where when fixing the new failure, we verify that we don't forget the old ones. s196 00:22:16.360 --> 00:22:18.040 We don't create regression. s197 00:22:18.040 --> 00:22:26.000 So, that's verifiable continual learning built on four principles: replayability, holisticness, lifelongness, and efficiency. s198 00:22:26.000 --> 00:22:33.840 And if you want to try um our VCL and apply it to your agent, you can use it today at rely.ai. s199 00:22:33.840 --> 00:22:36.080 Thank you.