Beyond Static Intelligence: Evaluating Continual Learning — Parth Asawa, UC Berkeley

AI Engineer · 20 min · 223 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hi everyone.
00:13Uh my name is Partasawa.
00:15I'm a PhD student at UC Berkeley.
00:18We're all at these AI and machine learning conferences, but everyone is just talking about how smart these agents are.
00:26We're not actually talking about learning ability or how much these agents learn.
00:30And so today I want to talk about that.
00:35You've seen the way we evaluate language models today.
00:38Every time there's a new model release, we see charts and graphs that look something like this, right?
00:43The way we evaluate these language models is we ask them to do one task and then completely independently we ask them to do another task and then another.
00:53And then we repeat this across a variety of different benchmarks, right?
00:56And so you've probably seen a variety of all of these.
00:59In the end, those benchmarks in aggregate when we take all of those independent evaluation scores give us leaderboards that look like this.
01:08Now, the challenge here is that if you look at the way we actually come to these single independent scores, what we've done is we've kind of told the models, imagine that every time you do something, you completely forget your memory.
01:21Okay?
01:22Like imagine if your life was like that.
01:23You're restarting from scratch every single time you do anything.
01:26That's the premise under which we're evaluating language models today.
01:31I'm going to argue that it's not necessarily what it should look like.
01:34If you were to look at learning ability, you might see charts where across different instances, performance doesn't look like a scattered line, but it looks like as a function of prior experience,
01:43models are actually able to improve their performance in contrast to what they've done in the past.
01:49Let's take a step back.
01:51What is continual learning?
01:53Continual learning to me is sample efficient online learning that is stable over long horizons.
01:59It's a challenging problem because you have to deal with both your ability to retain prior information without forgetting while simultaneously being able to update from new information and adjust your priors to new new data you're receiving over time.
02:12In language models specifically, you know, the way we train language models is we take a bunch of data we want to train them on and we kind of put them in a box in some offline training process and then maybe for a few weeks or something we train the model and then we extract a frozen checkpoint,
02:27right?
02:27And this frozen checkpoint is what's deployed to the world.
02:29The weights don't really change after that in today's paradigm.
02:32And continual learning is all about trying to change that, trying to enable mechanisms for models to learn over time.
02:38So in language models, people have looked to approaches like in context learning.
02:43Put more information in the model's context and the model can simulate some amount of learning in context.
02:48People have looked towards external memory stores, right?
02:51You could externalize some of your memory to a notepad that you then get to refer to every single time or a key value store or your choice of external memory.
02:59Or people look towards the parametric approaches.
03:03What would it actually look like if we had ways to update the model's weights online such that it could improve and learn from the new information?
03:11Now, today I'm not going to make a case for which one of these can solve continual learning, though I do have an opinion and I'll share it at the end.
03:18But what I am going to make a case for is that we're actually not evaluating it at all today.
03:23And the question that we need to ask ourselves is that if continual learning doesn't look like point capabilities, we need to be measuring it the right way to optimize for the right objective as a field.
03:33So let's get into it, right?
03:35What does continual learning kind of look like today when you read the literature?
03:39To be clear, continual learning by itself is not a new idea.
03:41Continual learning for neural networks has been around for decades, right?
03:45There have been famous problems in catastrophic forgetting that people have spent years trying to solve.
03:50But with language and you know an example of that is some of these sorts of papers where you'll see graphs like this where you train the models on a particular task distribution task A then you train it on the task B distribution and then you train it on task C. And the objective you're looking for is the
04:06does the model's performance not degrade on prior tasks while it's still able to learn the new tasks.
04:12And people have taken this similar methodology and applied it to language models too.
04:17Right?
04:17In some of the more recent papers we've seen, you're you're starting to see similar graphs of you train on one task and you train on the next.
04:23There's also a different class of evaluations for language models that look more like a factual recall sort of test where over these extremely long horizons, can models remember particular parts of the conversation and improve from that.
04:36The case I'm going to make though is that these sorts of evaluations aren't actually sufficient for continual learning.
04:42Okay, they don't consider sample efficiency a first order requirement.
04:46They don't necessarily always measure learning.
04:48And perhaps most importantly, they won't work for frontier language models.
04:52And the reason being that frontier language models are pre-trained on vast distributions of the entire internet or economically valuable tasks we care about.
05:00And so if you're expecting to see improvement on new tasks, it's it's actually really hard to come up with what those sorts of tasks should look like.
05:08So given all of that, what should a continual learning benchmark look like?
05:13The biggest question that I get when I when I talk about this with people is why can't I just chain existing benchmarks together?
05:20Like we've we've seen all of these Amy data sets.
05:23Why can't I just take my AM problems and solve them in a sequence?
05:26And the fundamental problem there is that benchmark instances are in traditional language model evaluation are designed to be independent.
05:34That means they don't have shared structure across tasks and as a result you can't meaningfully expect them to improve from earlier experience in future instances.
05:44And this is the fundamental problem that exists with any approach that tries to chain prior benchmarks together.
05:51In contrast, when we're designing continual learning benchmarks, we looked at three main design criteria as things that are kind of a fundamental requirement for evaluating continual learning.
06:01The first one here is headroom.
06:03And this kind of goes to my last point.
06:05Frontier language models are trained on almost everything in the internet.
06:08If you want to evaluate continual learning, you need some sort of task that actually requires online adaptation or learning from the language models.
06:16And that doesn't exist for a wide variety of the benchmarks where the models are just trained offline on that data.
06:22If the model can improve on your tasks by just training offline and not actually require any online learning, then it's not a good task for measuring continual learning.
06:32The second point that I'm going to talk about is shared structure.
06:36What this means is when going back to my point of pre prior language modeling benchmarks all being independent instances.
06:43If you want there to be an expectation of improvement in how you're evaluating continual learning, then there needs to exist some shared latent structure between the tasks.
06:52You can kind of think of this as some sort of shared latent in the environment that your models or agents are learning over time and they're seeking to exploit
07:00these these latent structures that exist in the environments to improve their performance on future tasks as a result of their prior information.
07:08And the third criteria that I'll talk about is a learning mechanism.
07:11And what this means is just there has to be some realistic expectation that the models are able to learn as a result of their prior experience.
07:19This could look like scalar reward.
07:21It could look like error messages.
07:22It could look like textual feedback.
07:24The point being, there needs to exist something in the environment that's giving agents signal to learn and improve on future tasks.
07:31Otherwise, it's not really a fair measure of continual learning.
07:37Let's talk a little bit more about the metrics before we get into examples.
07:42Obviously, there's a simple metric for a lot of tasks that we've seen recently, which is reward.
07:46And in our benchmark, it's no different.
07:48Every task defines a per instance reward metric where reward serves as a proxy for learning.
07:53Higher reward is better.
07:54You could think of this as efficiency.
07:56You could think of this as profit.
07:57You could think of this as how close you were to the actual predictions.
08:00There's some single unit of reward that operates on a per instance basis.
08:05And you might ask like why can't I just evaluate continual learning by looking at cumulative reward across these sequences of tasks we're constructing?
08:13And the answer is that total reward alone might confound [clears throat] continual learning ability with base model strength.
08:22Let's look at a diagram to kind of explain why this might work.
08:25Here we have two different continual learning systems.
08:27Okay, the the red one and the black one, the ones connected by the dotted lines.
08:31The gray system at the bottom is the black system except it wasn't allowed to maintain state across time.
08:37Okay, so that's what the black system looks like if its memory was wiped every single time.
08:41But if its memory wasn't wiped, if it had some mechanism of learning, then you can kind of see that it tends to improve over time.
08:47The challenge here is that if we looked at the cumulative reward metric alone, the red system would look like it was the best the best performing system on this task, right?
08:57It let's assume for a second that there is per room to do better and it hasn't saturated it.
09:00It has a higher cumulative reward, but you can kind of tell that in comparison to uh to the black system, it doesn't necessarily improve over the stateless baseline.
09:10It's just a better system to begin with.
09:12And this is one of the confounds we run into when we're trying to measure cumulative reward.
09:17To deal with this, we look at the gain metric in addition to reward.
09:22Gain refers to the difference between stateful reward and stateless reward.
09:26What that means practically in our benchmark is that for any system, we run it through the benchmark twice.
09:32Once in the normal way where it's allowed to maintain state across all of the instances in the benchmark.
09:37That might mean it's learning and updating its policy.
09:40It might mean that it's updating its notes.
09:42It might mean that it's just growing its context length, but it's allowed to maintain state.
09:46The second thing we run is a stateless baseline.
09:48What that means is we reset the system between every instance of a task so that the model isn't actually allowed to continually learn in any meaningful way.
09:57The difference between these two numbers then for any instance intuitively is kind of a question.
10:03It answers the question of how much did if we're looking at gain on task five, how much did my prior experience on the first four tasks actually lead to an improvement in my performance on the fifth task.
10:13It isolates out what your benefit of from actually learning was versus your base model's initial capability.
10:19To to be clear, I think reward, gain, and cost are all measured on Parto Frontiers.
10:24There isn't one single metric that I think defines continual learning because we still care about the base model strength.
10:30We still care about the ability to learn and we still care about the cost we're expending for these systems and so we try to measure everything on PTO Frontiers in this benchmark.
10:39Let's jump into an example of a concrete task to ground some of what we've been talking about here.
10:43I'm going to walk through the database exploration task which is hopefully a little approachable for uh which is hopefully approachable to understand.
10:51If you guys have ever worked in data engineering or know data engineers, you know that one of the things they do is operate over these vast databases with large set many different sets of tables, schemas they're unfamiliar with and idiosyncrasies
11:02in the data.
11:04And if you're a human data engineer, you learn those schemas over time.
11:07You learn how tables link together.
11:08You learn the idiosyncrasies in the data.
11:11But if I was an agent where my memory was being reset every single time, I wouldn't learn any of that, right?
11:16And so here what we can use as a measure for our ability to continually learn is the efficiency in the the task here is you're given a natural language question over the data and you have to answer it by performing some set of SQL queries right that gives you the answer from the from the databases
11:31but if you don't know anything about the databases it's going to take you a lot of SQL queries at the start and at the first questions you're going to be doing a lot of different exploration into figuring out what do the schemas look like what is the data format how do these link together
11:44but as you progress through the benchmark.
11:46If you're a continually learning system, maybe let's say by the 10th instance of a task, it'll look a little different, right?
11:52You'll you'll be able to know a lot more about the schemas, the idiosyncrasies of the data.
11:56You'll know how things link together, and it'll take you much less to actually answer the question.
12:01But there's a bit of a there's a bit of a conundrum here.
12:04We're going to add something a little more challenging to this task, too.
12:08In the real world, there's often concept drift in the things you're doing.
12:11In the database example, let's say there's a migration of your database, right?
12:14Columns get dropped.
12:15There's new columns with different names added.
12:17The data format changes.
12:18As a human, you reason through all this uncertainty and you're able to update your priors.
12:23You know that there's still some information from the past that may be relevant, but from your exploration, you learn what to forget and you learn what's actually relevant to maintain over time.
12:30You kind of have an innate ability to maintain the stability and plasticity trade-off in your mind.
12:35But this isn't native to a lot of language models.
12:38And so if we give it a task after the database migration, it might struggle to require it might struggle in its ability to detect and discard stale experience and simultaneously
12:47update from new experience.
12:49And so this n this notion of concept drifts is something we try to add a lot to our tasks to further test the limits of what memory and continual learning might look like in language model systems.
13:01In continual learning bench 1.0, know which is the benchmark I'm talking about today.
13:05We have tasks from six different domains of tasks.
13:08Okay, we have blind spectrum monitoring which is a signal processing task, codebase adaptation which is a software engineering efficiency, cohort studies and epidemiology task, exploitable poker, a strategic game playing task and because my friends like to play poker, and a database exploration task, the one I just showed you, and sales prediction, which is more of a data
13:27sciency prediction error style task.
13:29Across each of these task domains, we construct sequences of individual task instances with defined reward metrics that agents or models are tasked with solving gradually over time.
13:41And all of these task instances across domains we validate with domain experts to see is this learnable?
13:46Is this realistic?
13:47Is these the sorts of drifts you would expect and things you would expect to remember in an environment.
13:52Let's jump quickly into the results.
13:55In the first in initial release of the continual learning bench, we evaluated a lot of these context management systems to begin with and there's some parametric systems down the line too.
14:04If we look at just aggregate reward for a second and ignore gain and cost, you'll kind of see that the incontext learning systems, this is vanilla in context learning where you just put the experience in the context and you don't do any of the fancy context management that some of these other systems do.
14:17It tops the leaderboard and and it's not just on reward.
14:21It's actually also when we look at the Pareto frontiers, this kind of holds across reward uh versus cost and gain versus cost.
14:28I don't necessarily think this is what the end state of continual learning might look like.
14:32You could argue that these were medium horizon tasks and they didn't push the frontiers of the incontext learning systems enough and I I would say that's fair and that's one of the things on our road map to push those even further.
14:42But it was still surprising that these more expensive context management systems perform a lot poorly compared to just vanilla in context learning on these sets of tasks where you have to do real learning.
14:52These are real tasks that you might expect a normal person to do.
14:56I think it's also interesting to chat briefly about what some of the failure modes look like when we observed in this initial version of the benchmark.
15:04I'm going to argue that most failure modes in continual learning fall on one side of the stability plasticity trade-off.
15:11Okay.
15:11Okay.
15:11And stability is your ability to retain new information in a stable way and use that for future tasks.
15:16While plasticity is your ability to actually learn from new information.
15:20We see usually that with most continual learning methods, any sort of failure mode comes from the inability to do one of these things.
15:27And I'll dive into examples from both.
15:29You don't have to read too much of the text here, but this is our sales prediction task where the model's tasked with making some predictions or forecasting for sales over the course of I think five years.
15:37The model starts by getting feedback that it overpredicted for the last uh the last submission it set.
15:43And so what it does is that it then leads to it revises its prediction downwards to a lower prediction for the set of five years.
15:49And then it gets feedback that actually its underprediction was too much of an underprediction.
15:54And so as a human might naturally do you would go for the middle.
15:58But that's not what the model does.
15:59The model kind of forgot that it had the over prediction to begin with at the start and it just re jumps back right to the over prediction.
16:05This is kind of example of what stability failures look like where it's not retaining past information in improving its future experience.
16:12Plasticity is a bit of a different story.
16:14This is from our epidemiology task.
16:16And the only words I really want you guys to read this is a this is a system with with a notepad.
16:20Okay?
16:20It's a context management system that's given a notepad to externalize its memory to the highlighted text is is the important thing to read here.
16:27Quote, "These seem to be cohort definitions from a different study schema that doesn't apply here.
16:33the study schema did in fact apply here that the the model was unable to update its priors and data from new information and meaningfully improve over time.
16:41It just didn't even recognize that this is something that's relevant to the task and use that to update its priors.
16:48Before I finish, I want to offer a little bit of my broader thoughts on continual learning.
16:52I alluded to this before that I have some opinions on what methods for continual learning might actually look like.
16:57I'm personally quite excited about the parametric methods for continual learning that look at alternative architectures, data, and algorithms jointly to optimize what continual learning should look like.
17:07If you look at language models today, the training stack used to look something simple like this, or maybe it never did, but I'm simplifying for for the sake of this pre-training, then we did supervised fine-tuning, and then we do RHF.
17:19But the stack kind of adapted over the course of the last few years.
17:23Now it might look something more like this where you have mid-training, you have RL for different teacher models and you finish everything off with multi multi-teer on policy distillation.
17:32A lot of the work in continual learning today is how do we take these models that are already trained and then kind of that frozen checkpoint.
17:39How do we figure out methods for continual learning that work after the fact?
17:43But these models were never designed to be continual learners to begin with.
17:47And one of my hypotheses here at least is that we're operating in a bit of a sunk cost fallacy that because we've trained models the way we are today, we need continual learning methods that work on top of that.
17:56If you were to think of designing what continual learning looks like as a first order principle and a first order requirement, it might look nothing like this.
18:04In the purest sense, continual learning might just be one set, one phase of training for continual learning and everything after that is deployment.
18:11You're just the model is interacting in the environment.
18:13It's updating its weights.
18:15there's only one phase of learning and we don't necessarily have to assume that everything is built on top of the way we do language modeling today just because we've spent so much doing that
18:24there there are there are different ways to explore where if you think about continual learning as a first order requirement for your systems more broadly here for a second I think that continual learning is just one example of my group's broader effort to challenge the way we do things in training today
18:40I think that there are larger questions going on though and opportunities to challenge how we do AI research as a whole.
18:46There's a lot of questions around open science, consolidation of power, safety, what the future of things like academia or these third party institutions might look like.
18:55And I think that there's a huge opportunity for us to rethink a lot of that.
18:59And I've spent some time doing some more writing about that that I'd encourage people to check out.
19:03But the crux being there there are opportunities to reimagine what third party institutions for AI research look like and what the future of open science should look like that I think we all need to spend more time getting involved in.
19:14To wrap up, uh, I want to thank a lot of my contributors on continual learning bench.
19:18This is a collaboration across Berkeley, Snorkel, Udub, Madison.
19:22Uh, thanking the snorkel AI for their open their grant through the open benchmarks grant program and the LOD institute via their LOD slingshots program.
19:30Uh, in the road map, we have more domain specific and longer horizon tasks on the benchmark.
19:35We want to add more of the OSS models and the parametric approaches.
19:38We want to improve what simulation for task environments outside of easily verifiable deterministic domains look like.
19:44What would continual learning where you're personalizing to a user model do?
19:48I don't think we have models that are capable enough of simulating that right now to figure it out, but I think we should.
19:53If there's one thing to take away, continual learning doesn't look like point capabilities.
19:57We need to measure it the right way to optimize for the right objective as a field because that's a history of how machine learning has progressed.
20:03Thank you guys very much for listening.
20:05Please feel free to email me or follow for updates.
20:07I would love to stick around, but I'm going to catch a flight, but I'm happy to take anything afterwards via email.
20:11Thank you guys very much.