WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=A_I8mw8yfns

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/A_I8mw8yfns.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.840 --> 00:00:15.920
Hi, thanks for coming everyone.

s3
00:00:15.920 --> 00:00:27.760
Um, I'm the co-founder and CEO of Elorium and I'm here to talk about some of the issues with current models, current frontier models.

s4
00:00:27.760 --> 00:00:44.480
This includes um Claude Chat GBD and Gemini um and how they handle visual problems and um uh this might be new to some of you who don't work in the visual space but actually there's quite a big gap between

s5
00:00:44.480 --> 00:00:49.760
how these models handle visual reasoning and how humans deal with it.

s6
00:00:49.760 --> 00:00:58.559
And you will see that we're actually quite far away from any definition of AGI for visual reasoning.

s7
00:00:58.879 --> 00:01:05.760
So here are some examples of where uh how easy it is to find where models break down.

s8
00:01:05.760 --> 00:01:08.479
And um you can find these examples yourself.

s9
00:01:08.479 --> 00:01:10.080
Just takes a few minutes.

s10
00:01:10.080 --> 00:01:26.080
Uh in this first example uh we have a chessboard hallucination and we give the models um this picture and ask how many white squares are in the image and any ordinary person

s11
00:01:26.080 --> 00:01:30.880
uh who doesn't hallucinate would probably not say it's 32.

s12
00:01:30.880 --> 00:01:41.280
So 32 of course the models say this because they see part of the chessboard and they hallucinate the complete board and as a result they give the wrong number

s13
00:01:41.280 --> 00:01:46.960
and you see this quite a lot that models rely a lot current models rely a lot on pattern matching.

s14
00:01:46.960 --> 00:01:53.680
That's what makes them so good at identifying plants and animals and flowers uh in the real world.

s15
00:01:53.680 --> 00:01:57.600
But when it comes to like complex questions that part hurts them.

s16
00:01:57.600 --> 00:02:01.280
So the pattern matching is actively hurting them in this case.

s17
00:02:01.280 --> 00:02:07.040
Um so in their in what's going on in their reasoning is that oh this is a chess board.

s18
00:02:07.040 --> 00:02:09.200
Chess boards all have 32 squares.

s19
00:02:09.200 --> 00:02:12.959
Therefore this one must have 32 Y squares too.

s20
00:02:12.959 --> 00:02:17.360
On the right example um I'm a big board game player.

s21
00:02:17.360 --> 00:02:18.879
Have quite a collection.

s22
00:02:18.879 --> 00:02:22.239
Um so you can see there's a a board game theme going on here.

s23
00:02:22.239 --> 00:02:27.520
And actually you can uh reproduce this outside if you just go to you know outside the talks.

s24
00:02:27.520 --> 00:02:29.280
It's a ball game area there.

s25
00:02:29.280 --> 00:02:30.800
There are chess boards.

s26
00:02:30.800 --> 00:02:42.720
Um I will bet if any of you uh place the pieces in some kind of random position, no frontier model will be able to tell you where those pieces are located, where all those pieces are located.

s27
00:02:42.720 --> 00:02:46.000
Um and then another example here is katan.

s28
00:02:46.000 --> 00:02:48.959
Um here another very simple question.

s29
00:02:48.959 --> 00:02:51.200
How many rows does the blue player have?

s30
00:02:51.200 --> 00:02:54.800
Um, these frontier models think extensively about this problem.

s31
00:02:54.800 --> 00:03:02.800
Uh, one response I've seen is that, oh, the the guy has uh 10 blue rows off to the side of the board.

s32
00:03:02.800 --> 00:03:07.440
Therefore, there must be five uh blue rows on the can board.

s33
00:03:07.440 --> 00:03:09.280
Uh, but obviously that's not true.

s34
00:03:09.280 --> 00:03:11.680
There's seven if you actually count.

s35
00:03:11.680 --> 00:03:24.720
So these models um again are great at guessing uh great at pattern matching but they are not very spatially grounded and they just can't handle any kind of detailed questions.

s36
00:03:24.720 --> 00:03:40.319
Um and then finally um we have this example where it actually um affects robots where here you have a robot arm uh manipulating this uh cup and um uh cooker basically

s37
00:03:40.319 --> 00:03:53.280
and the state-of-the-art models today they miss the fact that the uh robot arm lifted the lid um and at the end they also missed the fact that the robot is turning on the stove like right now.

s38
00:03:53.280 --> 00:03:57.519
So there's uh essentially context amnesia happening.

s39
00:03:57.519 --> 00:04:09.120
Um and this is because these models can't maintain consistency um across long videos and they very easily lose track of what's happening.

s40
00:04:09.120 --> 00:04:17.120
And a very common question I get is how do you define a visual reasoning problem versus a visual understanding problem?

s41
00:04:17.120 --> 00:04:21.840
Um or you could say like visual thinking um compared to visual understanding.

s42
00:04:21.840 --> 00:04:28.320
I think a very simple way to do it is just um ask yourself the same question.

s43
00:04:28.320 --> 00:04:34.560
If you looked at an image or a video, how long would it take you to answer the question?

s44
00:04:34.560 --> 00:04:43.919
So uh for example, in both of these cases, I doubt anyone in this room would be able to give an answer uh if they were only allowed one second to look at the image.

s45
00:04:43.919 --> 00:04:53.680
So 1 second isn't enough to do these kind of like complex questions also called like system two um kind of thinking in Daniel Canon's book.

s46
00:04:53.680 --> 00:05:05.280
But if I asked you what game is this uh or similarly what flower is this or if there are only three pieces on the chessboard if I asked you how many pieces are there those questions

s47
00:05:05.280 --> 00:05:11.680
uh I'm sure all of you would be able to answer in less than a second and similarly all the frontier models would get that kind of question right.

s48
00:05:11.680 --> 00:05:23.440
So that is the distinction um that we make between what is understanding uh what is like pattern recognition versus what is reasoning where you actually have to look in detail

s49
00:05:23.440 --> 00:05:30.880
at the picture and um look at various things and this is exactly where frontier models uh fall apart today.

s50
00:05:30.880 --> 00:05:42.720
So as you are designing your own systems uh that's something to keep in mind keep these visual tasks very simple otherwise you will have hallucinations a lot and a lot of hallucinations

s51
00:05:44.000 --> 00:05:58.639
um so uh this leads into evals of course um frontier models there are already a bunch of multimodal reasoning evals or visual reasoning evals some that you might have heard of is arc agi

s52
00:05:58.639 --> 00:06:12.560
this is uh quite often brought up to um people saying oh we the frontier models are 85% or 90% on RKGI therefore we are 90% of the way to a uh to AGI itself

s53
00:06:12.560 --> 00:06:22.479
uh but I think these people they haven't really looked at any of the benchmark data because if you actually look at the data you will notice that the images are only 32x

s54
00:06:22.479 --> 00:06:35.120
32 or 64x 64 pixels and I would challenge anyone uh to give me like a real world complex task that can be reduced to a 32x 32 pixel problem.

s55
00:06:35.120 --> 00:06:42.880
Um I think you'll very quickly realize almost no tasks almost no interesting tasks can be reduced to that kind of resolution.

s56
00:06:42.880 --> 00:06:47.039
Another eval that people commonly uh bring up is MMU.

s57
00:06:47.039 --> 00:06:50.800
Uh this is the massive multiddiscipline multimodal understanding.

s58
00:06:50.800 --> 00:06:57.520
This is a step up from MMLU because it has um images rather than just pure text science questions.

s59
00:06:57.520 --> 00:06:59.840
This is science questions based on images.

s60
00:06:59.840 --> 00:07:03.680
But still images are a minor part of a lot of these questions.

s61
00:07:03.680 --> 00:07:11.199
A lot of the questions you can just answer without looking at the image or just doing some pattern recognition just knowing roughly what the image is about.

s62
00:07:11.199 --> 00:07:25.759
So what we really need is new visual reasoning benchmarks in the industry that really target the things that people care about like geometric align alignment, spatial intelligence, um object terminus

s63
00:07:25.759 --> 00:07:31.919
and these are really critical for AI to be deployed in these visual use cases.

s64
00:07:31.919 --> 00:07:47.440
And you might have noticed that still in a lot of industries that uh are primarily visual um which I will go into there isn't much uptake of AI right a lot of the AI uptake has been in the software engineering world and in the mathematician world

s65
00:07:47.440 --> 00:08:03.395
um and in like documents um document handling etc. But this uh there is actually a huge gap huge opportunity that is just being looked over right now um based on the interest in coding.

s66
00:08:03.395 --> 00:08:08.879
[clears throat] And so the missing paradigm in visual AI is thinking.

s67
00:08:08.879 --> 00:08:15.919
So we have generation models very high quality generation models like bite dances seance model.

s68
00:08:15.919 --> 00:08:24.560
Um, so we have these very high fidelity models and they look great, but they lack actual physical grounding um and causal logic.

s69
00:08:24.560 --> 00:08:41.120
So you will you probably notice that if you ask these models to produce a a picture um a video of a some like uh something blowing up like um or a building falling down or these things they look very cartoonish they look Hollywood style kind of things and that's because

s70
00:08:41.120 --> 00:08:52.720
they are just outputting what was in the training data and a lot of disaster videos um a lot of like action kind of videos on the internet are just going to be from Hollywood or game engines.

s71
00:08:52.720 --> 00:09:00.160
So they're working to reproduce that and that's fundamentally a problem because it means they can be no better than those kind of uh videos.

s72
00:09:00.160 --> 00:09:11.360
Um on understanding the what we are where we currently are is we have lot of tools that can map pixels to semantic labels like Google lens is obviously great to identify

s73
00:09:11.360 --> 00:09:14.560
plants and flowers and I use that all the time.

s74
00:09:14.560 --> 00:09:20.959
The SAM 3 for segmentation, YOLO for uh object recognition detection, mascaras CNN.

s75
00:09:20.959 --> 00:09:25.839
These are of course highly robust and they're used everywhere in the industry, but they're fundamentally passive.

s76
00:09:25.839 --> 00:09:27.839
So there's no reasoning capability to them.

s77
00:09:27.839 --> 00:09:30.160
So they can't answer more complex questions.

s78
00:09:30.160 --> 00:09:35.040
Um and really where the frontier is is uh with thinking visual thinking models.

s79
00:09:35.040 --> 00:09:39.040
These models will have active spatial and temporal intelligence.

s80
00:09:39.040 --> 00:09:47.680
They can extract actional logic for planning uh agentic workflows and physical execution.

s81
00:09:48.240 --> 00:09:52.000
And so our approach uh is uh four stage.

s82
00:09:52.000 --> 00:09:59.440
So we are collecting and generating our own uh multimodal data uh visual reasoning specific data.

s83
00:09:59.440 --> 00:10:03.600
This this kind of data we found you just can't uh get online.

s84
00:10:03.600 --> 00:10:09.680
Uh we have a synthetic data flywheel using evals agents SFT and RL to improve the model.

s85
00:10:09.680 --> 00:10:20.560
We're making some uh we made some advances to the architecture um in terms of uh various different time um advance various different improvements on top of the transformer-based architecture

s86
00:10:20.560 --> 00:10:31.120
and we're also enabling visual chain of thought reasoning and this is one of the key things that humans have that no frontier model has today since the frontier models are only textual

s87
00:10:31.120 --> 00:10:36.000
uh chain of thought based um and this is one example of a visual chain of thought.

s88
00:10:36.000 --> 00:10:41.120
So the question is like how many red hotels are built in this photo?

s89
00:10:41.120 --> 00:10:45.680
Then the model realizes oh uh first we need to identify all the hotels.

s90
00:10:45.680 --> 00:10:54.640
So it draws boxes around hotels um and other objects and then uh it reduces that to the red hotel.

s91
00:10:54.640 --> 00:11:00.800
So it's this multi-step uh process happening in the visual space natively.

s92
00:11:01.519 --> 00:11:06.720
So um about our company um I'm the co-founder and CEO.

s93
00:11:06.720 --> 00:11:10.240
I spent the last 12 years at Google Brain and Deep Mind.

s94
00:11:10.240 --> 00:11:14.640
Um I developed a lot of the foundational techniques for the model for modern LLMs.

s95
00:11:14.640 --> 00:11:20.160
11 years ago I was the first author of the work that introduced pre-training and fine-tuning.

s96
00:11:20.160 --> 00:11:25.760
That's the work when combined with the transformer paper in 2017 led to the GBT series of models.

s97
00:11:25.760 --> 00:11:28.800
So all the GBT uh papers site our paper.

s98
00:11:28.800 --> 00:11:44.560
Um I co-led the earlye models uh the first model that was state-of-the-art called glam and then um more recently I co-led the palm to 2 pre-training architecture and I was co-lead for the gemini data area and my co-founder info

s99
00:11:45.120 --> 00:12:01.120
and Google research he led research for Apple's first public multimodal model MM1 and he's has a a lot of experience in visual reasoning um across uh language as And this is our team.

s100
00:12:01.120 --> 00:12:04.000
So we're roughly 20 people now.

s101
00:12:04.000 --> 00:12:07.920
Um we've also have a chief reasoning architect Dustin Tran.

s102
00:12:07.920 --> 00:12:11.279
Previously he was lead of post training at XAI.

s103
00:12:11.279 --> 00:12:23.760
Um and we've hired a world-class team um across uh many other uh companies like Apple uh XAI um deep mind Amazon and so on.

s104
00:12:24.160 --> 00:12:30.880
Um and in terms of the uh use cases that I mentioned, robotics is one primary use case.

s105
00:12:30.880 --> 00:12:40.240
So robots have uh really critical bottlenecks performing complex real-time physical actions uh in these kind of like dynamic environments.

s106
00:12:40.240 --> 00:12:47.839
Uh but existing vision models uh you probably realize are trained from static images and very directed videos.

s107
00:12:47.839 --> 00:12:51.600
They are not like act they don't have active physical interaction.

s108
00:12:51.600 --> 00:12:55.600
So existing methods are overengineered and brittle.

s109
00:12:55.600 --> 00:13:02.639
Um and uh we are planning to release a model API available uh by the end of this year.

s110
00:13:02.639 --> 00:13:16.880
Um and at that point the API can be used to deliver action relevant uh scene understanding into existing um planning and control systems for these um robots.

s111
00:13:16.880 --> 00:13:23.040
Another important use case for uh visual reasoning um is construction.

s112
00:13:23.040 --> 00:13:29.360
So construction sites they have these very complex zone specific safety rules.

s113
00:13:29.360 --> 00:13:35.120
Um and uh computer vision can't adapt fast enough to changing safety rules.

s114
00:13:35.120 --> 00:13:48.079
They also can't interpret things like OSHA policy language and match the that language to what's actually going on at the site or understand the spatial relationships that are important there.

s115
00:13:48.079 --> 00:14:00.480
For example, like how many of these workers are wearing helmets or like is the construction um happening according to the plans that uh were defined earlier.

s116
00:14:00.480 --> 00:14:10.320
And um currently enforcing these rules require training separate models for different use cases uh because they're very these models as I said before are very brittle.

s117
00:14:10.320 --> 00:14:13.040
So you constantly have to do uh retraining.

s118
00:14:13.040 --> 00:14:26.560
Um and our approach uh with the video um understanding capabilities that we are building into our models is um allows you to um ground these video streams in the safety regulations.

s119
00:14:26.560 --> 00:14:29.680
And of course the safety regulations are in text, they're in language.

s120
00:14:29.680 --> 00:14:35.760
So you have to be um the model has to manipulate both language and vision very well.

s121
00:14:35.760 --> 00:14:46.480
And um yeah uh this will allow these models to interpret uh site policies using the current camera infrastructure that they have.

s122
00:14:46.800 --> 00:14:54.399
Um and then finally architecture and design we think is also a very uh promising use case here.

s123
00:14:54.399 --> 00:14:58.959
Uh this is exactly the use case where you need to be very detail oriented.

s124
00:14:58.959 --> 00:15:03.839
So back to the board game example around counting spatial relationships.

s125
00:15:03.839 --> 00:15:06.959
This shows up a lot in architecture and design.

s126
00:15:06.959 --> 00:15:15.120
Uh like if you design a if you design a house with with four bedrooms instead of three, that homeowner is going to be very angry, right?

s127
00:15:15.120 --> 00:15:18.959
Um so obviously counting is actually important.

s128
00:15:18.959 --> 00:15:28.880
Um and also just understanding these spatial constraints, real world constraints is uh is a very manual process uh these days.

s129
00:15:28.880 --> 00:15:41.760
We spoke to a mechanical engineering company just a few weeks ago and they said to design one small part of a robot testing platform takes 100 to 200 hours

s130
00:15:41.760 --> 00:15:46.000
uh of the time to design the entire testing platform.

s131
00:15:46.000 --> 00:15:53.199
Um, I believe it takes 2,000 to 3,000 hours of human uh time there.

s132
00:15:53.199 --> 00:15:58.880
And they've uh a lot of these places they've tried frontier models, but they just don't work for these use cases.

s133
00:15:58.880 --> 00:16:07.440
They really struggle to understand uh visual context across these like architecture blueprints, 3D CAD, CAM files.

s134
00:16:07.440 --> 00:16:10.240
Um, and so there are lots of errors there.

s135
00:16:10.240 --> 00:16:24.880
Um and similarly we believe that this can be useful useful for other kinds of design as well not just um architecture and engineering but maybe like designing um yeah for the web or fashion or other things

s136
00:16:24.880 --> 00:16:34.320
and our approach is uh we're using multimodal reasoning to uh to extract um this uh geometric logic that's important.

s137
00:16:34.320 --> 00:16:39.279
We're allowing programmatic validation or simulation validation.

s138
00:16:39.279 --> 00:16:43.040
Just like in code, you can run code against unit tests.

s139
00:16:43.040 --> 00:16:54.079
You can also run u mechanical devices through simulators that have been developed through seammens um and uh v various other companies to see if something will work in the real world.

s140
00:16:54.079 --> 00:16:59.920
So there's a lot of parallels actually between uh this kind of like mechanical design and coding itself.

s141
00:16:59.920 --> 00:17:04.640
But mechanical design is still relatively untouched by AI.

s142
00:17:04.640 --> 00:17:16.400
Um and yeah, CAD CAM quality control is another potential use case and ultimately we believe that this is going to be a critical step to the future of mechanical design where

s143
00:17:16.400 --> 00:17:25.679
the where AI can make faster cars, more efficient rockets, better batteries and all these things cannot be done just with code.

s144
00:17:25.679 --> 00:17:31.039
Uh people are not coding up the next iPhone or coding up the next uh SpaceX rocket.

s145
00:17:31.039 --> 00:17:34.640
It's all fundamentally very visual.

s146
00:17:36.240 --> 00:17:49.520
So, um you can find out more about us through our website um lauren.ai, our Twitter page xx.comai or our LinkedIn uh page.

s147
00:17:49.520 --> 00:17:52.000
And yeah, happy to take any questions.

s148
00:17:52.000 --> 00:17:55.760
I'll be standing around here for for a little bit.

s149
00:17:55.760 --> 00:17:57.760
Thanks.

s150
00:17:58.142 --> 00:18:00.142
[applause]