WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=dKcTBQzR7jI NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/dKcTBQzR7jI.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:13.519 --> 00:00:17.600 Hello and welcome to this talk on skill issue. s2 00:00:17.600 --> 00:00:20.000 It's actually no longer a skill issue. s3 00:00:20.000 --> 00:00:27.760 So uh by the end of this talk you will be able to build a lot with the vision models if you are already not doing it. s4 00:00:27.760 --> 00:00:30.960 So shortly about me, I'm Marv. s5 00:00:30.960 --> 00:00:36.399 Uh I've been working on computer vision since Lava in the fix times. s6 00:00:36.399 --> 00:00:46.079 Uh and lately I work on more agents and ondevice stuff because I'm a bit fascinated and I love vision language models so much that I even wrote a book on it. s7 00:00:46.079 --> 00:00:51.039 But I don't want developers to directly use vision language models anymore. s8 00:00:51.039 --> 00:00:59.520 And uh I want every single developer to start uh building vision langu vision applications end to end. s9 00:00:59.520 --> 00:01:04.960 And this talk is going to give you like a good baseline to do so. s10 00:01:04.960 --> 00:01:15.200 And common behaviors I observe with the developers is the fact that they try to use vision language models for everything but they you will never get real time. s11 00:01:15.200 --> 00:01:25.680 And when I mentioned real time, it's like you have a toaster and you get like 30 40 fps uh on it like whatever whether you are building like image classification, s12 00:01:25.680 --> 00:01:29.040 instance segmentation or whatever. s13 00:01:29.040 --> 00:01:40.079 And um they are not super robust in terms of like if you were to train like an RFDTR which Joseph spoke in the I spoke about in the first talk s14 00:01:40.079 --> 00:01:46.960 um it will always outperform your vision language model and I'm going to prove it today. s15 00:01:47.360 --> 00:01:53.360 And uh on the right hand side you can actually see me uh doing stuff with RFDTR. s16 00:01:53.360 --> 00:01:57.280 Um and another thing is that they don't read the licenses. s17 00:01:57.280 --> 00:02:02.640 Everybody's like whenever I post something about object detection, they always ask me about YOLO. s18 00:02:02.640 --> 00:02:04.399 Like YOLO is a good model. s19 00:02:04.399 --> 00:02:08.399 Uh but it's a it has AGPL 3.0 license I think. s20 00:02:08.399 --> 00:02:15.599 And I could swear on my life that there is like some developers that actually deploy it without knowing that they have to pay for it. s21 00:02:15.599 --> 00:02:24.000 Um so yeah like uh I want you to migrate to Apache 2.0 models today. s22 00:02:24.000 --> 00:02:30.959 And for this I have built something called web vision and it's kind of inspired by this post by Mazar. s23 00:02:30.959 --> 00:02:40.480 Basically what he does is that he gives um SAM 3.1 model as a tool to Gemma 4 to call and I find it super impressive. s24 00:02:40.480 --> 00:02:49.040 Um and today I built like a tool kit uh where you can do this with like even more things. s25 00:02:49.599 --> 00:02:52.160 So I'm kind of distilling myself. s26 00:02:52.160 --> 00:03:03.920 Um first off uh this toolkit has my favorite models as tools uh so that you can give it to your agent because your coding agent is a bit of a clueless computer vision engineer s27 00:03:03.920 --> 00:03:05.040 basically. s28 00:03:05.040 --> 00:03:10.239 Um and when I distill myself basically like whenever I pick a model I always check the following. s29 00:03:10.239 --> 00:03:14.159 So first up the license is the biggest priority for me. s30 00:03:14.159 --> 00:03:18.319 It has to be Apache 2.2 MIT or something non-commercial. s31 00:03:18.319 --> 00:03:25.680 Uh secondly uh the performance has to be on par uh depending on its size or architectural choices. s32 00:03:25.680 --> 00:03:43.200 So I check from the benchmarks uh whenever a model from a computer vision conference comes out and third vibes obviously and so this toolkit has a second part to it and it's a bit like a vibe training part which is s33 00:03:43.200 --> 00:03:44.799 the most exciting part. s34 00:03:44.799 --> 00:03:46.879 So I will go through it first. s35 00:03:46.879 --> 00:03:51.680 So I put myself in developers shoes to build like a vision application. s36 00:03:51.680 --> 00:04:05.200 Um if I have labeled images easy I can just train a model or like if I I can give my computer computer vision agent some tutorials uh to do that because it's all out there like we built transformers s37 00:04:05.200 --> 00:04:06.400 for it. s38 00:04:06.400 --> 00:04:14.400 Uh but if I have images only I have to annotate uh and then uh evaluate the annotations and then just train a model. s39 00:04:14.400 --> 00:04:16.079 But how can you do it on scale? s40 00:04:16.079 --> 00:04:24.800 You can actually use a vision language model as a labeler and the vision language models as judge and then train what you want. s41 00:04:24.800 --> 00:04:28.160 But how does this pipeline look like? s42 00:04:28.160 --> 00:04:37.440 So basically I built this and it has like it has VLM for labeling VLM as a judge and then training. s43 00:04:37.440 --> 00:04:43.840 Uh it's a bit of a long horizon tasks for coding agents and it has infra many infrastructure support. s44 00:04:43.840 --> 00:04:45.280 You can do this locally. s45 00:04:45.280 --> 00:04:47.199 You can do this uh remotely. s46 00:04:47.199 --> 00:04:49.759 So basically it runs on hugging face infrastructure. s47 00:04:49.759 --> 00:04:55.280 We have jobs that allow you to do oneoff batch processing or training. s48 00:04:55.280 --> 00:05:08.479 We also have like a serverless routing system called inference providers where you can use multiple providers and you we also have like buckets to dump intermediate data on top of the data set repositories, s49 00:05:08.479 --> 00:05:11.840 model repositories and so on. s50 00:05:11.840 --> 00:05:13.680 Um but what enables this work? s51 00:05:13.680 --> 00:05:18.080 First off, my favorite model, RFDTR, RFDTR segmentation. s52 00:05:18.080 --> 00:05:21.360 I'm currently working on segmentation at the moment. s53 00:05:21.360 --> 00:05:25.440 Um, we have better agents for long horizon tasks. s54 00:05:25.440 --> 00:05:39.039 Uh, that, you know, you have to babysit the labeling process, the training process, etc. Uh, and smaller yet more capable vision models enable you to label stuff very cheaply. s55 00:05:39.039 --> 00:05:47.919 And also with transformers we went for V5 refactoring etc. So like it performs better for vision models at the moment. s56 00:05:47.919 --> 00:05:50.320 And this is how the pipeline actually looks like. s57 00:05:50.320 --> 00:05:56.560 So first up I label the data set like I take an image data set like any image data set. s58 00:05:56.560 --> 00:06:04.880 I label that image data set with Q1 3.59B and then I pass the labeled data set to two judges. s59 00:06:04.880 --> 00:06:20.319 First one is Gemma 4 E4B which is like an 8B judge and then the second one is LFM 2.5VL which is nearly 2B uh it's relatively smaller basically I checked the research and it's better to have like an ensemble s60 00:06:20.319 --> 00:06:39.199 smaller judges and on top of it I merged the judgments so I also checked the research around it and most of the people ask the VLM or LLM to assign some score to it but those scores absolutely absolutely don't work especially if your models are of different size with judges. s61 00:06:39.199 --> 00:06:43.759 Um and then I pass it to train RFDTR medium or large. s62 00:06:43.759 --> 00:06:51.440 So I I chatted a bit with Rob people and they encouraged to use that and I'm going it it actually works. s63 00:06:51.440 --> 00:06:54.240 I'm going to show you very shortly. s64 00:06:54.240 --> 00:06:55.680 U but how does it work? s65 00:06:55.680 --> 00:07:01.199 So you take the repository and then you just ask it okay can you train it? s66 00:07:01.199 --> 00:07:05.680 Can you do the training on this data set on hub? s67 00:07:05.680 --> 00:07:17.599 And then it will start like if if the data set has labels like you can actually just get to training but if it doesn't have it you can just uh start annotating s68 00:07:17.919 --> 00:07:24.880 and I basically like the trick is I I pass the overlaid bounding boxes on images to the judge. s69 00:07:24.880 --> 00:07:29.440 So basically Qan technically outputs bounding boxes as tokens. s70 00:07:29.440 --> 00:07:30.319 I don't pass them. s71 00:07:30.319 --> 00:07:42.960 I just overlay the bounding boxes and I pass that image on top of some labels and label descriptions and I say okay if this label description has bounding box on it then just s72 00:07:42.960 --> 00:07:56.400 um tell me if you approve or not and then the judge I I merge the judge verdicts over minimum agreement and not consensus which I will come to why I did that that way s73 00:07:56.400 --> 00:08:00.160 and these label descriptions are also generated by coding agents. s74 00:08:00.160 --> 00:08:03.840 and you just approve it as a human being. s75 00:08:03.919 --> 00:08:18.479 And the the models that I used in this pipeline all have Apache 2.0 of licenses except for LFM model which has type of license where you are kind of um if you have like certain amount of s76 00:08:18.479 --> 00:08:30.800 revenue after that you pay for it but you can comfortably use it it's large and for the coding agents that are babysitting this pipeline uh I initially built with oppus 4.6 s77 00:08:30.800 --> 00:08:44.959 uh 8 and then ran the workflow with the G gl GLM 5.2 2 which is doing a good job on long horizon tasks to be honest and for infra I actually work at hugging face I have a lot of compute credits s78 00:08:44.959 --> 00:09:03.200 and I'm super impatient in life so like uh I use a good amount of hardware for experimentations but I benchmarked it and overall it takes like three four dollars if you want to run this entire pipeline to train models which to me is crazy s79 00:09:03.200 --> 00:09:17.040 initially for Q1 3.5 5 I used serverless because I was like okay this is convenient and it's super cheap so I use deep infra which is super super cheap if you were to use together it's better if you s80 00:09:17.040 --> 00:09:30.080 do batch processing over jobs and then for the judging I used hugging face jobs which cost less and then for training again I used like an L4 but the model is super small like RFDTR s81 00:09:30.080 --> 00:09:35.200 is super small and you can just use something else you can do it locally if you wanted to. s82 00:09:35.200 --> 00:09:36.160 I'm just impatient. s83 00:09:36.160 --> 00:09:37.680 I want a big batch size. s84 00:09:37.680 --> 00:09:42.320 So yeah, and I tested in two problems. s85 00:09:42.320 --> 00:09:45.200 First of the road sign detection. s86 00:09:45.200 --> 00:09:47.519 Uh secondly, document parsing. s87 00:09:47.519 --> 00:09:50.720 For the road sign detection one, I already have the labels. s88 00:09:50.720 --> 00:09:56.160 So like I actually compared against the ground truth annotations if my pipeline works or not. s89 00:09:56.160 --> 00:10:06.240 And for document parsing, I actually uh couldn't do it because basically I use like a docqa data set and I the problem is like I want to extract the images, s90 00:10:06.240 --> 00:10:08.959 tables, I don't know signatures and stuff. s91 00:10:08.959 --> 00:10:15.200 So it's like a novel task and I wanted to see if RFDTR can actually learn it. s92 00:10:15.200 --> 00:10:17.200 So first result it works. s93 00:10:17.200 --> 00:10:18.800 So yay. s94 00:10:18.800 --> 00:10:32.480 um we have like a good uh mean average precision over um 50 um and I compare it against the basically I have a test set and I take that test set and pass through QN s95 00:10:32.480 --> 00:10:44.480 and then I compare against the pseudo notations and the ground root annotations of that test set there is a bit of a gap but it's kind of expected because it's learned from QN s96 00:10:44.480 --> 00:10:52.079 and also uh rock oak is also like a good value to be frank for like such use case. s97 00:10:52.079 --> 00:10:57.519 Um and for the document parsing it actually generalizes which to me is crazy. s98 00:10:57.519 --> 00:11:04.320 Basically uh here the trained model output you can see that it detected the signature. s99 00:11:04.320 --> 00:11:08.880 Meanwhile the QN annotation of that test set missed it. s100 00:11:08.880 --> 00:11:12.880 So I would like to say it actually generalizes very well as well. s101 00:11:12.880 --> 00:11:25.680 uh but we owe this to like the how good RFDTR is as a backbone in a way and here you can also see how it uh captures the images technically s102 00:11:25.680 --> 00:11:37.440 um and it's like oneonone and while I was building this I actually noticed that I was clueless about building with vision agents um so I have bunch of findings around it s103 00:11:37.440 --> 00:11:42.320 um so first up there is a huge judge imbalance so for depending on the problem. s104 00:11:42.320 --> 00:11:45.440 LFM tends to reject a lot. s105 00:11:45.440 --> 00:11:58.399 That's why I couldn't take the consensus because if I were to if I were to eliminate everything that both LFM and Gemma agreed to remove, um I would left with very very little number of examples s106 00:11:58.399 --> 00:12:01.360 which would leave me with very poor generalization. s107 00:12:01.360 --> 00:12:07.279 So what I did was that okay, if one of them says yes, I'm going to take that example and nevertheless it worked well. s108 00:12:07.279 --> 00:12:20.560 If you have a large data set and if you care about the recall, I suggest that you take the consensus or like just observe for document parsing the gap isn't as big. s109 00:12:21.440 --> 00:12:26.240 And secondly, uh the prompt generation is a bit hard. s110 00:12:26.240 --> 00:12:43.440 So like this is the only part where as a human you have to approve uh okay the model generates the prompts for you for the judge uh and then you will say okay this I I approve this because you need to to take it still take a look at it uh take a look at your data set s111 00:12:43.440 --> 00:12:56.959 a little there's no escaping that um and thirdly and this is super interesting because your co your coding agent despite no matter how uh good it is. s112 00:12:56.959 --> 00:13:01.600 Like you take OPUS 4.8 which is like a very good coding agent. s113 00:13:01.600 --> 00:13:06.880 Um it's clueless as a computer vision engineer as well as it misses common sense. s114 00:13:06.880 --> 00:13:19.839 For instance like it was doing like horizontal flip over the traffic signs or it was doing like jitter over the traffic lights which will definitely corrupt your data set and break it. s115 00:13:19.839 --> 00:13:21.680 So I patched this later. s116 00:13:21.680 --> 00:13:25.920 So you can just say okay I don't want to augment or I can augment. s117 00:13:25.920 --> 00:13:30.320 So Angular coding agent will help you with that. s118 00:13:30.320 --> 00:13:35.519 And lastly the second part of this toolkit is my preferred models as tools. s119 00:13:35.519 --> 00:13:42.639 Um so this repository covers my favorite models from depth estimation to zero shot segmentation. s120 00:13:42.639 --> 00:13:49.839 Uh and this is partially powered by first of hugging face benchmarks uh which we recently rolled out maybe a few months ago. s121 00:13:49.839 --> 00:14:04.399 Basically every single basically we have like a benchmark leaderboard and over there you have the open models as well as their evaluation results and you can compare different models of different sizes. s122 00:14:04.399 --> 00:14:16.560 So um I keep it up to date as well but like also it's partially powered by me who likes to read the um computer vision conference papers. s123 00:14:16.560 --> 00:14:21.920 Um, so I would like to give a shout out to this model because not a lot of people know about this. s124 00:14:21.920 --> 00:14:32.000 So basically Sam cannot do open-ended reference se uh segmentation like you can do like okay segment this red car and it will do it. s125 00:14:32.000 --> 00:14:37.920 But if you say okay the red car next to the orange car that is next to the blue car it will not do that. s126 00:14:37.920 --> 00:14:48.000 and Falcon perception which is a model by TIA um can actually do it and it's only like 600 million parameters with Apache 2.0 license. s127 00:14:48.000 --> 00:14:55.199 So this one does the zero shot segmentation for me and this is like a non-exhaustive list. s128 00:14:55.199 --> 00:15:09.600 So for posing we have the sapiance family um for the human centric tasks where you need to do human koid detection human um depth estimation and so on and for zero shot detection s129 00:15:09.600 --> 00:15:16.880 uh I have moon dream 3 and mm grounding dynino which is like a model with apache 2.0 license is also very good. s130 00:15:16.880 --> 00:15:19.440 It's very small compared to Moon Dream. s131 00:15:19.440 --> 00:15:29.760 I give you the multiple models in multiple sizes depending on your hardware that you can pick like if you want to go fast just pick the tiny alternative. s132 00:15:29.760 --> 00:15:41.600 Um for OCR I actually took them from the OCR benchmark in different sizes and for depth estimation I actually uh discovered that the large model doesn't have a non-commercial s133 00:15:41.600 --> 00:15:43.279 license and the rest of them have it. s134 00:15:43.279 --> 00:15:44.720 So you can actually use it. s135 00:15:44.720 --> 00:15:47.440 That one has like an Apache 2.0 license. s136 00:15:47.440 --> 00:15:50.480 And it also comes with um supervision and tracker support. s137 00:15:50.480 --> 00:16:02.000 They are both uh libraries from Roboflow that um allow you to do tracking of instances, bounding boxes and so on and future plans. s138 00:16:02.000 --> 00:16:11.440 So first off, I I could hear you say, okay, this will definitely not work for the industry use cases because industry use cases have different parts. s139 00:16:11.440 --> 00:16:18.079 it has like nondescribable parts like natural language is not a good gateway to it. s140 00:16:18.079 --> 00:16:30.079 Um so I think in that sense image guided detection could help like if you don't know about image guided detection you basically have like instance of an image like a haggi here as an example s141 00:16:30.079 --> 00:16:38.160 and then you ask the model okay detect this object in this image um across all of the images. s142 00:16:38.160 --> 00:16:46.800 I feel like it could actually somewhat help in the industry use cases where you cannot describe it by like um natural language. s143 00:16:46.800 --> 00:17:07.919 Um as well as I want to try um intersection over union merger sort of basically you have like labeled boxes and then judges boxes like you ask the judge to actually generate a box and then you take the intersection over union instead of asking judge to reject or accept s144 00:17:07.919 --> 00:17:18.720 and I'm working currently working on the segmentation support and thanks for listening if you want to actually learn more so basically I We have a small vision repository. s145 00:17:18.720 --> 00:17:27.039 It has everything about like fine-tuning models, quantizing models, multimodel models, everything around vision. s146 00:17:27.039 --> 00:17:29.360 Uh as well as transformers task guides. s147 00:17:29.360 --> 00:17:33.760 We keep them up to date has many um tutorials. s148 00:17:33.760 --> 00:17:45.360 Um, we also have hugging face skills that actually have computer vision specific skills as well as the infras skills that you can just do like a one prompt training. s149 00:17:45.360 --> 00:17:54.960 Again, um, and this is my Twitter profile and this repo is actually at GitHub Marvin uh, vision intern. s150 00:17:54.960 --> 00:17:59.440 I think I have qu um, time for one question. s151 00:17:59.440 --> 00:18:02.000 Thank you so much. s152 00:18:05.120 --> 00:18:05.840 Yes, s153 00:18:05.840 --> 00:18:08.720 you have plans for training. s154 00:18:08.720 --> 00:18:17.039 Um, he's asking if I have uh plans to train train VLMs themselves like a self-improvement type of thing. s155 00:18:17.039 --> 00:18:18.320 That would be super exciting. s156 00:18:18.320 --> 00:18:28.960 But first, I want to solve this thing of like developers actually training task specific models and then deploying on edge and then that could come perhaps. s157 00:18:28.960 --> 00:18:30.799 Maybe one more. s158 00:18:30.799 --> 00:18:32.799 Yes. s159 00:18:38.320 --> 00:18:39.760 Um, not really. s160 00:18:39.760 --> 00:18:40.559 I don't think so. s161 00:18:40.559 --> 00:18:45.520 I just use the because I wanted because a coding agent actually has the context. s162 00:18:45.520 --> 00:18:49.120 I wanted it to generate the prompt. s163 00:18:49.120 --> 00:18:51.760 Maybe one more. s164 00:18:52.640 --> 00:18:53.120 Okay. s165 00:18:53.120 --> 00:18:55.679 Thank you so much.