WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=jNE8No-wvok

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/jNE8No-wvok.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.719 --> 00:00:13.679
Hello everyone.

s3
00:00:13.679 --> 00:00:20.240
Uh I am Adita and we will be talking about uh two main problems that happen on the short form surfaces.

s4
00:00:20.240 --> 00:00:23.680
We are going to talk about how do we deal with these things at a scale.

s5
00:00:23.680 --> 00:00:35.120
Uh so to get started first thing is we will understand what are the characteristics of the data that we are trying to deal with what are the two main problems that we are actually working on

s6
00:00:35.120 --> 00:00:47.520
uh and what are the multi- aent systems to solve those problems at a scale what are the different specialized small scale WM that we can uh build uh to solve those individual agentic problem

s7
00:00:47.520 --> 00:01:02.800
and how do we build those agents what are the different ways to actually optimize it like to make it like scalable at a very big scale and then we will be looking into the evaluation in a very holistic 360 manner not just like a precision recall what is out there how do we understand

s8
00:01:02.800 --> 00:01:19.600
the entire multi-agenting pipeline both from LLM level tools MCP and whatn not everything out there and then we will be getting into the optimization techniques which this which are very vision specific and metadata specific which will allow us and give us some intelligence to tell that

s9
00:01:19.600 --> 00:01:32.640
we really don't need to do intell like this intelligent like workflow So for all the videos we can figure out what are the small set of videos which should be the candidate for doing these things and we will conclude with the takeaways.

s10
00:01:32.640 --> 00:01:35.119
So the real life data is very messy.

s11
00:01:35.119 --> 00:01:40.079
It is we are talking about at a scale of 100 million plus and a lot more viral content.

s12
00:01:40.079 --> 00:01:43.520
There's a lot of adversal content people are trying to gain the system.

s13
00:01:43.520 --> 00:01:48.720
There are a lot of uh multilingual text on the screens and in their videos and images.

s14
00:01:48.720 --> 00:01:50.880
And the uh data is very dynamic.

s15
00:01:50.880 --> 00:02:01.040
it keeps on changing from like month to month different AI tools are coming everything out there so it's it's really really dynamic that means there is a lot of drift issues and other thing that comes with a video data

s16
00:02:01.040 --> 00:02:10.879
and then we know that there is no clear ground truth for the problem that we're trying to solve so these are like existing problem which happens in a real world video data set

s17
00:02:10.879 --> 00:02:24.720
we will be the first problem that we will be talking about is the modality misalignment where the first is intramodality which is a very solved problem you can have a clip model you can have a modality on uh uh images, videos,

s18
00:02:24.720 --> 00:02:29.360
uh audio, text and figure out what exactly is the cosign similarity on those embedding and figure it out.

s19
00:02:29.360 --> 00:02:31.599
So this is a much simpler problem.

s20
00:02:31.599 --> 00:02:37.280
What we will be talking about how we are doing intraodality u issues.

s21
00:02:37.280 --> 00:02:49.440
So this is consider that like a video we have a big video and then uh suddenly you are seeing something some ad agenda some political things or something out there which shouldn't be there in the video as when you clicked on it.

s22
00:02:49.440 --> 00:03:01.040
So how do we actually look into the small segments of the uh video understand this problem and figure out where there is an anomaly or kind of like adversarial behavior out there which shouldn't be there in the first place.

s23
00:03:01.040 --> 00:03:10.480
So this is the first problem which requires a lot of granular understanding vision understanding and a video understanding to see what is happening at a very clip and a very frame level.

s24
00:03:10.480 --> 00:03:14.239
The second problem is uh to understand the unoriginal content.

s25
00:03:14.239 --> 00:03:19.840
In today's economy with the AI tools available, it's really easy to actually duplicate the content.

s26
00:03:19.840 --> 00:03:29.840
When someone uh upload a video or something, we see that okay, it is being copied, it is being transformed and with tools, it is getting much easier to transform these videos.

s27
00:03:29.840 --> 00:03:33.200
U so how do we actually detect this kind of unoriginal content?

s28
00:03:33.200 --> 00:03:35.920
How do we figure out the source of the video?

s29
00:03:35.920 --> 00:03:47.840
And um this has actually caused a problem in attribution credit u ecosystem imbalance and lot of like user fatigue where we are seeing a lot of repetitive videos which shouldn't be there in the first place.

s30
00:03:47.840 --> 00:03:57.680
So uh coming to the multi- aent system uh the first thing is that uh why we are going to multi-agent not a single agent or single lenm because the problem is really complex.

s31
00:03:57.680 --> 00:04:14.480
It requires a really specialized uh nodes and uh kind of like an understanding at each part of retrieval content understanding and then reasoning and if you can solve a problem with one agent one LLM you we don't need to actually do it on multi- aent systems.

s32
00:04:14.480 --> 00:04:22.079
So this is where the centralized brain how we are actually thinking kind of like how the uh decomposition happens of this problem.

s33
00:04:22.079 --> 00:04:28.560
The first is basically you have a video and kind of like a clip video ID and everything given to a reviewer agent.

s34
00:04:28.560 --> 00:04:29.919
This is a centralized agent.

s35
00:04:29.919 --> 00:04:31.520
This is basically the orchestrator.

s36
00:04:31.520 --> 00:04:38.800
Consider that as an API gateway of uh doing a sing signal decomposition and finding what is happening around in the image.

s37
00:04:38.800 --> 00:04:52.160
So what this guy do is basically this guy takes ask the perceiver agent and perceiver agent is the one which is basically considered that a very sophisticated VLM expert with lot of image and the video tools at

s38
00:04:52.160 --> 00:04:53.600
uh disposal.

s39
00:04:53.600 --> 00:05:01.840
Uh perceiver agent will take the ID from the retriever reviewer agent and it fetches the video from the database.

s40
00:05:01.840 --> 00:05:19.840
It decomposes into smaller parts using different tools, different like semantic embeddings and different kind of like temporal change that happens which is a little bit beyond our scope but of this talk but it's uh a technique which is making sure that we are not doing at a fixed frame rate but we are finding where the temporal change

s41
00:05:19.840 --> 00:05:27.759
happen and compressing those similar frames into one or two frames and then the perceiver agent get all the data from a clip level.

s42
00:05:27.759 --> 00:05:43.120
The video embedding level get all the information about a clip embedding tags OCR whatever is present out there what is the natural language description and what are the time stamp from which frame to which frame what are those this metadata looks like it gives that data to the reviewer

s43
00:05:43.120 --> 00:05:58.240
reviewer will look into all the temporal this JSON object provided by the uh perceiver agent in a raw form along with embeddings along with semantic ids tags OCR and everything and it does a temporal analysis it looks into okay you know for from

s44
00:05:58.240 --> 00:06:13.360
first frame to frame or frame number 360 or like till 6 second it was a video talking about a sports and suddenly we are seeing from 6 second to 6.5 second or frame number this to this we are seeing that like this is changing into some

s45
00:06:13.360 --> 00:06:14.560
political thing.

s46
00:06:14.560 --> 00:06:30.479
So just by looking into the metadata the reviewer agent is able to understand comprehend and figure out what exactly is the anomaly coming in the this temporal space and once it do that it figures out whether this is actually a modality misalignment or there is some bigger issue out there on there.

s47
00:06:30.479 --> 00:06:36.639
So review agent will look and talk to the retriever agent and it tells that hey this is the video I'm looking into.

s48
00:06:36.639 --> 00:06:42.240
These are the some of the metadata that I have already uh kind of like ddup and kind of like post-processed.

s49
00:06:42.240 --> 00:06:47.039
Now give me some understanding of this similar clips which are available in the corpus.

s50
00:06:47.039 --> 00:06:51.680
So retriever agent will look into all the signals provided by perceiver agent.

s51
00:06:51.680 --> 00:06:58.000
All the metadata of a clip level and a whole video level and index it into different databases adaptively.

s52
00:06:58.000 --> 00:07:03.280
For example the topics has to be maybe like a inverted index where we have topics and a lot of videos.

s53
00:07:03.280 --> 00:07:07.440
For the embedding it will be a vector databases which is available out there.

s54
00:07:07.440 --> 00:07:17.840
And for different kind of entities which I've extracted we have a graph databases kind of like where you review your agent will figure out okay these are the multiple databases these are the entities and these are the metadata

s55
00:07:17.840 --> 00:07:29.039
let me index it so that in the online inference time I can figure out for a particular clip what are the similar clips what are the similar entities and topics which are available to fetch and improve the recall.

s56
00:07:29.039 --> 00:07:39.840
So going into individual agent we talked about perceiver it looked into the entire video do a temporal decomposition into small clips based on the semantic embedding and couple of algorithms

s57
00:07:39.840 --> 00:07:51.120
and it fine-tuned the VLM to actually emit all the real uh concrete and a very um granular information about like each clips and the entire video.

s58
00:07:51.120 --> 00:07:56.639
Uh because we are dealing at a very big scale that means we cannot just go with the standard VLM which are out there.

s59
00:07:56.639 --> 00:07:58.000
There has to be a compression.

s60
00:07:58.000 --> 00:08:05.360
There has to be a cost effective way to actually serve this these models to do it at like billions of frames.

s61
00:08:05.360 --> 00:08:19.120
That's where we are getting into the pre-training fine-tuning and knowledge distillation quantization to actually deploy and solve the very very specialized VLM based on this particular problem and we will be talking a little bit brief about that.

s62
00:08:19.120 --> 00:08:34.880
U the retriever agent is the one which we talked at a very high level and then what it does in an offline processing is that all the signal which is decomposed by the perceiver agent in an offline fashion consider that rather an agent you are actually taking that library and doing an offline analysis on a big scale

s63
00:08:34.880 --> 00:08:51.200
let's say ray cluster or something like that and then once you have all those metadata it is basically indexing it into different databases doing an uh periodic offline clustering and finding what are the similar content what should be the embedding ID what should be cluster ID for each clips and entire video.

s64
00:08:51.200 --> 00:09:05.519
So that is a data which is basically used by online inference to find similar videos and in online manner it given a query a clip ID and every all the metadata details it's figured out what is out there in the corpus which this guy has

s65
00:09:05.519 --> 00:09:09.839
is similar to what like what are the different things which has a high similarity.

s66
00:09:09.839 --> 00:09:27.760
So what it does it looks into databases is it find similar clips and like similar authors and different other metadata information rerank those candidates find uh and use some of the tools about like for example the traditional model span score like a classifier score to see what are the different candidates which might be spam which might not

s67
00:09:27.760 --> 00:09:38.399
be high quality and those things out there and once it has those top end candidates is give back it to the reviewer and reviewer is where it actually process this is my main content signals

s68
00:09:38.399 --> 00:09:42.320
and clip embeddings These are a similar video which is given by retriever.

s69
00:09:42.320 --> 00:09:48.720
Let me think over it and whether I need to retraate and get more data from retriever.

s70
00:09:48.720 --> 00:09:52.959
So uh that's where reviewer have all the temporal signals from a single clip.

s71
00:09:52.959 --> 00:09:58.320
It has all the information of similar clips understanding and what is similar authors and other thing.

s72
00:09:58.320 --> 00:10:03.680
It also has a realtime information about how the users are actually interacting with this video.

s73
00:10:03.680 --> 00:10:08.080
What are the different kind of like a reports or like likes, dislike, the comments?

s74
00:10:08.080 --> 00:10:25.519
what is the u sentiment of those comments out there right so a lot of these signals which are actually missed by the offline uh signals and the VLMs and other kind of agents are actually also incorporated to see if there is a change happen in the sentiment what is the response I'm getting in the real time so

s75
00:10:25.519 --> 00:10:38.560
there are different tools available for this to actually understand the video not just from the content and from the semantic perspective but understanding from the user interaction perspective those signals are really really important

s76
00:10:38.560 --> 00:10:50.560
So once we have this judge and everything we build this uh agentic framework and each of these agentic frameworks uh all all these three agents are actually powered by specialized

s77
00:10:50.560 --> 00:10:54.560
uh VLMs and small scale VLMs in a way.

s78
00:10:54.560 --> 00:10:58.399
So we will be talking about uh pre-training first.

s79
00:10:58.399 --> 00:11:07.600
Now uh in most of the cases you see that okay you have a pre-training you fine-tune your vision transform a little bit here and there and basically tune it for your specific purpose.

s80
00:11:07.600 --> 00:11:20.640
Uh the thing is that these VLMs what we have outside and available the foundational model the front end models they are trained on a very clean very nice data set very web data which is very well tuned cleaned and everything

s81
00:11:20.640 --> 00:11:26.560
but uh data inhouse for a specific purpose is not having the same data characteristics.

s82
00:11:26.560 --> 00:11:27.600
It's it is messy.

s83
00:11:27.600 --> 00:11:28.880
It is generated by users.

s84
00:11:28.880 --> 00:11:30.800
It is for your specific workflow.

s85
00:11:30.800 --> 00:11:42.079
That means you need to pre-train on those image tokens and language to fine-tune your vision transformer from scratch and see whether there is a delta in fine-tuning it and tuning it from the scratch.

s86
00:11:42.079 --> 00:11:44.079
That's where the pre-training is helpful.

s87
00:11:44.079 --> 00:11:50.640
It's little expensive but um it if if it can get a delta that that actually works really well.

s88
00:11:50.640 --> 00:11:54.959
The second is instruction fine-tuning where we have the two problems.

s89
00:11:54.959 --> 00:11:57.600
We have a certain policies and certain guidelines.

s90
00:11:57.600 --> 00:12:09.760
We know the content out there, signal out there and it understand and tune it on that particular data set with it with a particular output which is schema like a JSON schema to understand what is the modality

s91
00:12:09.760 --> 00:12:16.399
uh misalignment or duplication of scores and other chain of thought reasoning provided by reviewer agent.

s92
00:12:16.399 --> 00:12:23.760
So here we have a video clip we have a vision encoder which we are already pre-trained a little bit like now we are doing a little bit more training on that.

s93
00:12:23.760 --> 00:12:36.800
There's a projector which is sitting between the vision transformer and the L language models and it is actually basically a bridge between them and then we have our output depending on what is the instruction finetuning data we have on this side.

s94
00:12:36.800 --> 00:12:45.519
So this is the critical part of actually making your uh model performance like go up for your domain specific.

s95
00:12:45.519 --> 00:12:59.920
So uh the context is basically you have a role, you have a policy, what are the tools available to ground that into some of the metadata or and what are the other things available out there give it entire things into uh in in a very brief manner into the context

s96
00:12:59.920 --> 00:13:05.279
and let it figure out like what the structured label you have for these labels are actually in-house label.

s97
00:13:05.279 --> 00:13:19.040
These are the one where we have created like what exactly modality things are, what are the different uh issues we are seeing, what are the different chain of thought reasoning there should be there in the model and how does the output looks like for a human reviewer.

s98
00:13:19.040 --> 00:13:25.200
So this is like a very high quality data data set that we are fine-tuning it on for different agents.

s99
00:13:25.200 --> 00:13:25.920
Right?

s100
00:13:25.920 --> 00:13:42.560
So once we have done with the pre-training just to understand the vision aspect of or the other modality aspects of the videos then we go into the finetuning to make it understand and provide the context and the output in the manner that we would want to actually process the data set on.

s101
00:13:42.560 --> 00:13:52.639
So u the next phase is the DPO phase where uh which is basically this technique is used a lot in the uh post- training to actually fine-tune our models into a specific

s102
00:13:52.639 --> 00:13:57.440
uh like realm domain policy understanding or something.

s103
00:13:57.440 --> 00:14:07.040
But this can also be used a lot in the prediction for actually understanding what are the samples which are actually getting uh not so good by the by your multi-ent systems

s104
00:14:07.040 --> 00:14:09.760
and by your u llms.

s105
00:14:09.760 --> 00:14:23.440
So what uh can be done is that like uh you have this uh production data set which is coming out you have a lot of inference happening you take a subsample of these uh data which is from the production

s106
00:14:23.440 --> 00:14:34.399
you pass it through LLM as a judge which is trained inhouse on the human label data set and then you have the human review Q to see what is the performance coming up on on the actual

s107
00:14:34.399 --> 00:14:35.360
system.

s108
00:14:35.360 --> 00:14:45.040
Once you have this thing if your performance is coming amazing and it is above like your whatever is the prediction threshold you have 95% for each of the problem it's great

s109
00:14:45.040 --> 00:15:00.240
but if it is not that means there is a way there has to be a way to loan on these samples where the model did not do well that's where you have a human in the queue it understand those all the traces which is there from all the agents the LLM call MCP and it figured out where

s110
00:15:00.240 --> 00:15:07.360
the problem happened is it like a chain of thought reasoning or is it like some of the wrong tool tools are called the retrieval did not work.

s111
00:15:07.360 --> 00:15:24.800
So that entire validations and basically understanding of each and every hook and node both at a model intelligence level plus at a harness level is what you figure out and say that okay these are the improvement that we I need to do on these samples which was incorrectly

s112
00:15:24.800 --> 00:15:28.000
um sampled by our um system.

s113
00:15:28.000 --> 00:15:40.480
So once you have this thing you have a positive sample negative sample what needs to be updated and that's where you retrain your model to see what uh how can we improve it better and this is the continuous improvement

s114
00:15:40.480 --> 00:15:54.399
where we are looking into these samples reiterating re um improving the models and having a new data set out there from the production u like samples and trying to understand whether the drift happen or what exactly the model is doing.

s115
00:15:54.399 --> 00:16:02.480
So this human in the loop is like always a continuous thing where you have a daily sampling from the production and trying to understand like how the model and the

s116
00:16:02.480 --> 00:16:05.279
L&M as a judge are doing.

s117
00:16:05.279 --> 00:16:18.240
So uh because of the cost and ability to solve at this scale we cannot go with the standard like a like a VLM models from frontier sizes because it's not scalable.

s118
00:16:18.240 --> 00:16:19.440
It requires a lot of inference.

s119
00:16:19.440 --> 00:16:26.320
It's a lot of complexity and we are solving a very specific problem like I really don't care if the model can solve a coding problem.

s120
00:16:26.320 --> 00:16:29.279
I only care about my domain specific problem.

s121
00:16:29.279 --> 00:16:30.240
That is all I care.

s122
00:16:30.240 --> 00:16:31.600
This is not exposed to the customer.

s123
00:16:31.600 --> 00:16:33.120
This is an internal thing.

s124
00:16:33.120 --> 00:16:46.000
So I would actually do a off policy and on policy knowledge distillation and do a little bit of quantizations depending on like some experimentation quantization to understand whether a forbit work braining float what actually work really well.

s125
00:16:46.000 --> 00:16:59.440
Then I would have a table of different sizes of distillations and quantization and understand and see where exactly is my performance like up to the mark and where I'm gaining a lot of compute and the cost resources

s126
00:16:59.440 --> 00:17:04.000
um savings from on inference production by doing these optimization.

s127
00:17:04.000 --> 00:17:12.720
So this is really critical because as a problem what we have solved it's all good but in production it has to be scalable.

s128
00:17:12.720 --> 00:17:14.079
It has to be cost effective.

s129
00:17:14.079 --> 00:17:24.959
It cannot just be like something which is um out of the um which is just out in the market because we are solving a very specific problem out here.

s130
00:17:24.959 --> 00:17:28.880
Coming back to the evaluation basically first is the task of success.

s131
00:17:28.880 --> 00:17:34.480
Of course these are the binary things the modality happen it does not happen like there is alignment or disalignment.

s132
00:17:34.480 --> 00:17:40.240
So precision, recall, F1 are obvious metrics to understand what the task is success is.

s133
00:17:40.240 --> 00:17:42.880
But we want to look at a system in a very holistic manner.

s134
00:17:42.880 --> 00:17:46.240
Not just like the end goal but what is happening at each nodes.

s135
00:17:46.240 --> 00:17:48.559
How well like the retrieval system is working.

s136
00:17:48.559 --> 00:17:53.360
What is the latency and recall of the system and how well we are able to reason it?

s137
00:17:53.360 --> 00:18:03.360
What is the chain of thought reasoning coming from these models at each agentic level or wherever the its reasonings are applicable which is basically a reviewer agent in our case.

s138
00:18:03.360 --> 00:18:04.799
And what is the quality of that?

s139
00:18:04.799 --> 00:18:06.000
Is it like overthinking?

s140
00:18:06.000 --> 00:18:16.400
Can we reduce a budget somewhere like to make sure that like the models is actually doing a good job with less uh budget or can we increase it the budget of the planning or the reasoning

s141
00:18:16.400 --> 00:18:24.640
things to have it a higher budget and make sure that the complex problem that we are solving is maybe having a much higher performance and accuracy.

s142
00:18:24.640 --> 00:18:30.480
So having an adaptive adaptive reasoning budgeting is also uh quite important.

s143
00:18:30.480 --> 00:18:31.840
And then we have the robustness.

s144
00:18:31.840 --> 00:18:33.520
what are the edge cases we are seeing?

s145
00:18:33.520 --> 00:18:35.360
Uh what are the error rates we are having?

s146
00:18:35.360 --> 00:18:38.320
What are the different nodes and hooks where they are happening?

s147
00:18:38.320 --> 00:18:40.559
Is it like a tool call is not working really well?

s148
00:18:40.559 --> 00:18:45.679
It's a retrieval part or LLM is not doing a good job where or those kind of thing.

s149
00:18:45.679 --> 00:18:54.240
And then we have a system efficiency where we are not looking uh just at the performance of the output and everything but we are looking at each and every aspects

s150
00:18:54.240 --> 00:18:59.919
of the token cost which LLM are calling can you optimize LLM little bit more to actually save the cost.

s151
00:18:59.919 --> 00:19:06.480
What is the efficiency which is we are saying and the latency of each agents and the entire system combined together.

s152
00:19:06.480 --> 00:19:16.240
Then we have a LLM as a judge where we see that like in any of the prediction system there is always a data drift happening especially when you have the user generated content.

s153
00:19:16.240 --> 00:19:22.080
So how our LLM as a judge which is used for evaluation and everything is doing with respect to the human queue.

s154
00:19:22.080 --> 00:19:23.440
Is there a drift happening?

s155
00:19:23.440 --> 00:19:31.600
Do we need to retrain our LLM as a judge on new data set which is coming from labeling team or how do we actually do on those parts in optimization?

s156
00:19:31.600 --> 00:19:38.720
We have the spatial temporal reduction optimization which actually look into the frame which are similar and just compress them into the one aspect.

s157
00:19:38.720 --> 00:19:45.039
This reduces the total processing of the videos by a huge extent.

s158
00:19:45.039 --> 00:19:59.520
The second one is basically you have a caching you have a viral content which is basically coming up and you do not want to have the same content out there which is doing going through the entire pipeline and that's where you have a high similarity score and just skip the multi- aent system and just make a call

s159
00:19:59.520 --> 00:20:00.240
on that.

s160
00:20:00.240 --> 00:20:03.280
The third one which is really important is the metadata pruning.

s161
00:20:03.280 --> 00:20:08.160
This is where we actually shrink the space from lot of candidates based on some of the metadata.

s162
00:20:08.160 --> 00:20:12.320
For example, some topics and some creator which already have a really good record.

s163
00:20:12.320 --> 00:20:26.160
That means we really don't need to process these all these images all all the videos from the creator which has a really good high authenticity score which are really doing really well on this where the video quality is really high engagement is good those kind of things.

s164
00:20:26.160 --> 00:20:32.960
A metadata is something which can be used to filter out the video which shouldn't even go to the systems.

s165
00:20:32.960 --> 00:20:36.480
So some of these flags u would be helpful.

s166
00:20:36.480 --> 00:20:45.440
Um so the last takeaway that u u we basically we can take out of this room is that like decomposition t is a key.

s167
00:20:45.440 --> 00:20:49.039
Uh decompose uh a problem as and when necessary.

s168
00:20:49.039 --> 00:20:54.000
Adaptive optimization is really important for uh ROI and cost feasibility.

s169
00:20:54.000 --> 00:20:55.679
Good evaluation is paramount.

s170
00:20:55.679 --> 00:20:57.600
Everything in and out depends on this.

s171
00:20:57.600 --> 00:21:08.480
This is the foundation of your entire system and entire VLMs and production monitoring and qualitative improvement are really essential to make sure this is sustainable in long term.

s172
00:21:08.480 --> 00:21:11.760
With that, I'll end it and thank you very much for listening.

s173
00:21:11.760 --> 00:21:12.679
Thank

s174
00:21:12.679 --> 00:21:14.679
[applause]