Video Has No Memory. Here's How We Built One. — James Le, TwelveLabs

AI Engineer · 20 min · 232 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Thanks so much for having me and inviting me to to be a speaker at the the welfare.
00:16You know, I I attended last year and was so impressed by the quality of presenters.
00:20So so glad to to have a chance to be here and to present.
00:23So the title of my talk is, you know, video has no memory, right?
00:26And you know, this might sound strange because video is already in like a preservation of the past, right?
00:32If you think about like you have footage, you preserve recording training data, incident, creative work, history, etc. But actually most of the video AI systems these days do not have memory in the system sense.
00:44So actually for this talk, I will try to answer the question like what could it take to build a memory layer for video intelligence?
00:51To start, I want to be clear about what makes video different from other data type, right?
00:55So this is the first mental model that I want to highlight, which is that video is not a stack of frames.
01:01Um so in in a many of my conversations with like developers, you know, who are using our product, a lot of them still treat video as like a stack of images,
01:09maybe a transcript being attached or you know, like you know, but essentially like like a frame level, right?
01:16And that is a useful approximation for some tasks, but it throw away the thing that makes video very unique, which is continuity, right?
01:23So meaning in video derives from space, time, modalities, and sequence.
01:29So a better mental model for video is a special temporal volume.
01:33So what I mean that inside that volume, you have visual information, speech, sound, motion, OCR, camera changes, scene transition, metadata, and time, right?
01:43So the hard part here is really well, how can you preserve any relationship across this volume so that later an application can traverse it?
01:51And then you know, especially at the enterprise scale, you know, across like industry like entertainment, sport, you know, um, short-form content, then you're sitting on petabytes of of footage, right?
02:01So, finding moment is already hard.
02:03So, how can you preset meaning across millions of moments in the deeper platform?
02:07So, I work at Twelve Labs, which is a uh series B uh startup.
02:10Um, we build foundation models that understand, you know, video the way that human do.
02:15Um, and the way we talk about our positioning is like the existing uh stack of of dealing with video is not equipped to do that, right?
02:24Obviously, language model are very powerful.
02:27They are good reasoning interfaces.
02:29They are increasingly multimodal, as well.
02:32But, the supporting stack, right, around that is very um I'd say limited, and that create three problem.
02:38Number one, is wrong context, right?
02:40So, video is not naturally a sequence of text token.
02:44If we force it into that sequence by sampling frames, by extracting a transcript, uh by dumping everything into a prompt, you lose the spatiotemporal relationships, right, that actually define the event.
02:55Second, uh wrong memory.
02:57So, if you think about text system memory here is often mean retrieval and generation, vector search, or probably like larger context window.
03:04Uh those are very useful, but video memory has a different requirement.
03:07It needs to link today's scene for something that happened in another file, another episode, another camera angle, another season, another year.
03:14So, it actually need durable continuity, And the last part here is wrong reasoning.
03:18As I said, you know, text-first system cannot reason over, you know, um, natively over motion, causality, all that.
03:25So, uh you know, they do not automatically build like a persistent structure uh on, you know, who appear, what happened, what changes, etc. And so, my argument is that video intelligence need a memory layer that decide what to preserve, how to connect it, and how to retrieve later.
03:41So, um, I want to kind of ground it into the properties of video, right, to to make it even clearer.
03:48There's five challenges in dealing with video.
03:50Number one is temporal, right?
03:52So, depends on before and after.
03:55So, frame by itself can can be misleading, right?
03:58The same expression, product shot, physical action can mean different things depending on the sequence around it, right?
04:03Second is that video is obviously multimodal as I I explained already.
04:07Um you know, a transcript alone may miss, you know, the logo, a frame alone may miss the spoken claim.
04:13Video Video is also very dense, right?
04:15So, a few minutes can contain dozens of shots, people, objects, action, location claims.
04:21The useful signal is uneven across the distribution on on the frame.
04:25Some seconds are decisive, others are noisy.
04:28Fourth is that uh video is also ambiguous, right?
04:31Um people reappear under different lighting and angles, brands are partially visible, location are implied, concepts emerge over time rather than being named in a single moment.
04:41And lastly, it is uh expensive uh because in a lot of uh big enterprise and in complex workflow, you need to, you know, point back to the source moment, like where it come from, right?
04:52So, these are the five properties explaining why video memory um is is very complex.
04:56You need to preserve temporal span, multimodal evidence, continuity, all of that.
05:00Uh this is a very simple uh stack of how we build things at Twelve Labs.
05:04Um at the bottom, we have these semantic chunks that capture, you know, meaningful temporal units.
05:10Above that is our uh multimodal embedding coder called Marengo, which essentially turn those spans into spatial-temporal relations.
05:19Um basically, vector embeddings that represent video content.
05:22And then we have a special special-temporal context store, which is where it preserve pre-usable structure, like moment, entities, metadata, all of that.
05:32Uh we also build our own uh VLM, video context-aware language model called Pegasus, that essentially serve as the the reasoning layer, right?
05:39That can preserve uh prepare over video content, so think about summaries, like um metadata, synthesis, comparison.
05:45And we expose our our models as API because, you know, we want to get developers to use them as infrastructure.
05:51Now, moving beyond like kind of the stack right here, I want to talk about the difference between search and memory, right?
05:56Very quickly speaking, uh search is obviously super important.
05:59It's how you recover relevant moments from large video library.
06:03But then it gives candidate.
06:04It actually not give you like any continuity.
06:06So, memory on the other hand is is all the um you know, uh the things that enable the system to answer different class of question as you see here on the right side of of my screen.
06:16So, these are not the single retrieval call, right?
06:19They require the system to reason entities, timeline, evidence across an entire corpus.
06:24Um and so like you can actually build product moving beyond from like show me something like this to, you know, tell me what this collection knows, right?
06:33And so that that might sound, you know, simple and subtle, but uh the the the the output is completely different.
06:38Like with search, you you get like an output like a time-bounded moment, but with memory, you actually return like structured knowledge, timeline, uh explanation, composable output.
06:47And that like, you know, is very important because we can now move um the the unit output from clip retrieval to corpus memory, right?
06:56Um there are two scaling dimensions shown here in the slide.
06:59The first is time scaling.
07:01So, a real video system should be able to reason over years of footage without reprocessing the whole archive every time, right?
07:07That means memory first retrieval, uh be reusable representation once and then support multi-hop timeline, episodic recall, follow-up question at lower latency and cost.
07:16And then the second uh dimension is in space, right?
07:19So, many real workflow actually um involve multiple perspective like different camera angles, uh you know, live stream, creator broadcasting content, body cams, stock cameras, uh event feed, right?
07:29So, how can you build a system that can fuse evidence across all the sources and then maintain current understanding, right?
07:35And so, that is the challenges here.
07:37How can you build a representation that let application traverse video across time and across sources?
07:43Um since this is um you know a track on on graph, right?
07:46So the the best mental model that I can come up with is to represent you know, video collection as a context graph.
07:52So a context graph is a durable, queryable representation that connects video moment, entities, appearances, relationship, time span, metadata, and corpus level context, right?
08:02So if you take a look here on on the screen all the way in the bottom, you got time bounded moment.
08:05These are like the the scene, the shot, right?
08:08Uh these are evidence unit.
08:10One one level up are the appearances, where and when each entity show up.
08:14And then you got the actual entity itself.
08:16So think about the people on the video, the brand, the places, the concept.
08:20Next you have relationship, uh co-occurrences to the same brand, sequences between different places, the causality, and timeline.
08:28And finally at the top you have corpus level context.
08:30What are the main themes, the patterns, the gap, the coverage that this video collection cover, right?
08:35Uh this matter because different question travels different part of the graph.
08:39If you ask a simple search question, then that might go directly into the moment.
08:43But like an entity workflow might start with a person and then it expand into appearances, right?
08:47And if you ask question like a storyline, like narrative storytelling of certain you know you know person, then it may follow relationship across time, right?
08:55Uh so the key idea here is that memory in the context of video understanding is a navigable structure over the entire video volume.
09:03From that concept, I come up with these five principles as a building you know a memory layer for video intelligence.
09:08Number one is to ingest once and reason many times.
09:11So um you don't want to like do every single query from scratch, right?
09:16You want to pay the cost up front, do one interpretation from the video content up up front, pay the cost, and then you move expensive understanding into injection.
09:25So this is same mental model uh database, right?
09:28You you're not repeatedly have to parse your entire source of data um for you know every application request.
09:34Um second principle is to store primitive, not just answer.
09:38So, uh, you know, moments, entities, appearances, as we talked about that, those are the the primitives, right?
09:42That allows you to, uh, do downstream workflow, like search, editing, um, you know, analytics, all of that.
09:49Third is to ground every claim.
09:51They like basically, if you ask a question, you need to cite back into where that scene happening in the video.
09:56So, uh, evidence, like, you know, should be grounded to a specific timestamp within the video, right?
10:02Uh, fourth is to let intent shape memory.
10:05Um, this is important because the same footage mean different thing in different workflow.
10:09We work across sports, uh, application, brand safety, compliance review, clear analytics.
10:13All of them require different primitives from the same video.
10:15So, the memory layer should be configurable, right?
10:18Developers should, uh, should be able to tell the system what matters.
10:21And lastly, uh, keep the layer composable.
10:24Um, so, basically, being API first, you know, um, it should provide, uh, the layers that allows those application, uh, on top of that to to serve it, uh, structured grounded metadata
10:33that can be plugged into any sort of application.
10:36Um, so, moving beyond these five principles, I want to talk about like, kind of the the harnesses around building a memory layer, right?
10:45There's a lot of talk these days about, um, you know, building the right harnesses for the context of language model.
10:49So, what does it look like for for video understanding model, right?
10:53Um, a model could produce a single answer.
10:56It is stateless.
10:56It start fresh each time, start fresh each time, and doesn't have any constraint.
11:01So, the output is largely based on what the model is set to produce.
11:04A video worker, on the other hand, operate inside a a very deterministic system.
11:09Understand what is available, uh, it can plan the task, could ship evidence, inspect, uh, the relevant moments, synthesize, validate, return output, and then the entire workflow can be evaluated, right?
11:21Um, so, so for video understanding, this is very important because the worker need to know what memory is is available, what evidence matters, and also like how deep to inspect, because that will depend,
11:30uh, determine how much cost to spend, what output constraints to satisfy, right?
11:35Um talking about harness engineering for for video understanding, um I come up with these like different capabilities for for like a video worker, right?
11:43Um number one is memory, I talked about that already.
11:47Number two is task planning.
11:48So, given given a query from from an user, uh you have to decide like what task to execute.
11:53Is it like search or is it like summarization or like, you know, multi multi-step reasoning?
11:59Uh third is retrieval, like every single system should be able to like select the right evidence from from your video corpus to read for a specific task.
12:07Uh expert tools, right?
12:09So, we work with customer where they require like, you know, zoom in, zoom out, uh comparing different uh uh you know, frames, enriching uh content with like additional metadata.
12:18So, building expert tools inside uh like a like a video worker uh is very important.
12:23Operating envelope, so these are like explicit limit on time, cost, depth, scope, autonomy.
12:29Uh and output contract.
12:30So, sometimes natural language is not, sometimes the patient needs structured data with references and time stamp.
12:35And of course, finally you have evaluation, right?
12:38Uh like, you know, did the retrieval find the right evidence?
12:41Did the synthesis preserve it, right?
12:43Did the worker stay within the budget?
12:45All right, so so that's a lot of like, you know, uh slide and and and talk.
12:48I want to quickly jump into some demos uh that I actually built using Two Labs uh you know, video agent product.
12:55So, the uh there'll be three demos.
12:57Um the the video agent product that we've been building is called Jockey.
13:01So, this first example here is for sport understanding.
13:05Uh you know, obviously everyone is super excited about the World Cup that happening right now.
13:09So, what I did is I ingest um 67 videos from the 2022 uh World Cup in Qatar.
13:16And you can see here I asked it to find the near misses, uh the shot that almost become goal but did not.
13:21For each, explain why it was not a goal, uh but do not include actual goals, right?
13:27So, these are the the the top output that it returned.
13:35So, that that's hitting the woodwork.
13:37This is saved from the goalkeeper.
13:40I don't know if the sound is up, but like I'm playing the the video, by the way.
13:45Um Right, this is another save from the goalkeeper.
13:52It even catch like, you know, the upside uh from one of the goals.
14:01And then I ask question, okay, when is a goal?
14:03Uh Find the most dramatic actual goals.
14:05Show the build-up play and the finish.
14:08For each goal, describe the sequence, right?
14:11So, if you know this one, this is the um the first goal of the World Cup final like 4 years ago.
14:19And it it actually like returned like, you know, you you don't understand who who are the passer like it was Alvarez passing to Mac Allister and passing to Di Maria to score the goal.
14:30Um Take a look at this one from Richarlison.
14:32This is goal of the tournament uh from Brazil again uh South Korea, I believe, right?
14:40And it returned like, you know, um an outrageous skill in a build-up, right?
14:44The name the player who did the return pass.
14:48You can even do player tracking.
14:50So, I asked it to track Lionel Messi across this entire corpus including the shot where he's won the man of the match figure on the screen.
14:56Describe the camera framing, right?
14:58Uh so, this is uh hilarious of all the important moment in the game and this is a scene where Messi dribble past a sliding defender.
15:06You can see here.
15:09It pick up the scene where he scored the first goal against Australia in the round of 16, I believe.
15:15Right?
15:16This is another scene where he scored the third goal in the final.
15:25Yeah, so that one example on spot spot understanding.
15:29But then you can obviously build more interesting and more like real practical application of which security is one that we encounter a lot.
15:37So on the on this example I ingest it you know publicly available camera footage.
15:42For context this are the clip you have traffic jams suburban you know urban area and given this footage where I asked Jockey to count and classify every vehicle in the intersection
15:55break it down by type plus pedestrian.
15:57And it returned the number of vehicles and the big foot traffic as well.
16:02You can detect safety events right?
16:07So you see that a red SUV turn and almost gets struck.
16:13Another scene here.
16:16Turn left into an upcoming car.
16:22Yeah, so that a clear straight line entry.
16:25It works well in you know different scenario.
16:29This scene is very crowded area in Bangkok.
16:36It asked it I also asked it to work on the you know the rain right?
16:45So this is another scene where it understanding the um rainy condition.
16:55It identify the basis intersection vehicle window.
17:03So yeah, those are some example for for camera security surveillance footage.
17:07Finally advertising so um you probably seen this Adidas clip in all commercial leading up to the World Cup.
17:16Recently is a 5 minutes Adidas footage and I asked it to classify all the point where you can put an ad on.
17:22So, it find the review, the hard cut, the impact, energy big rate.
17:27It find the scene where a certain player appear on the screen.
17:32It identify like, you know, high impact action like this.
17:37Condition, um, the hard cut to Nike football underlies.
17:41And of course, it it point into the logo.
17:44Um, of Adidas.
17:48So, you know, uh, from from perspective of an advertiser, these are very important moments because they can, you know, find the scene with the slow motion hero.
18:01Hard cut on a beat.
18:06Or pick action.
18:10In which they can advertise their brand content against this footage, right?
18:15Um, yeah, so those are three sample demo application, um, that I want to highlight, uh, I've using two labs.
18:22Um, and again, um, you know, what can you build with with this sort of video memory layer based on the case example?
18:29These are the categories of of application that I believe developers can build.
18:33You can discover things.
18:34You can be reasoning experience.
18:36You can organize your content across different video library and it can be action workflow, assemble uh, different scene together, do compliance review, data operation, etc. The same framework apply for different verticals
18:50in media and entertainment spot, segmentation, highlight generation, in commercial security, evidence review, contextual analysis.
18:58In advertising, uh, brand safety, uh, creative intelligence, right?
19:03And, uh, yeah, so this is our product, uh, that I coming up.
19:07Um, one quick highlight is that we we try to as a video cognition infrastructure.
19:12So we have a knowledge style that basically become the video memory layer.
19:16Web configurable injection that let video shape what the system can can extract.
19:20Corpus digest so that you can understand in what is in the library and the resolution as you search responses API.
19:26So the thing we want to highlight here is it's not an application layer.
19:29It's not an editing platform, not a compliance product.
19:32It's the cognition infrastructure with the layer and the harnesses that enable like those product being to become available.
19:39Um and if you found the content of this talk interesting, definitely recommend you to scan this QR code.
19:46The product is currently in private beta right now.
19:48If you bring any sort of workflow that touch video content especially around content assembly, content organization, you know, think about media archive, content creator, YouTube, TikTok, sport analysis, media workflow,
20:01definitely add the scan this QR code and register for the interest or come talk to me after the talk.
20:06So that should be my time.
20:07Thanks a lot.