WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=VBCDhRrvlYo

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/VBCDhRrvlYo.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.720 --> 00:00:14.040
Hello.

s3
00:00:14.040 --> 00:00:14.640
Hi everyone.

s4
00:00:14.640 --> 00:00:16.240
Thank you for being here.

s5
00:00:16.240 --> 00:00:19.160
Um let's start the presentation.

s6
00:00:19.160 --> 00:00:23.720
Uh I really appreciate you having interest in in learning more about uh game development.

s7
00:00:23.720 --> 00:00:33.440
So, today I'm going to talk about how I think the next game engine won't have a manual and uh at the end you'll see what I mean by by this.

s8
00:00:33.440 --> 00:00:34.640
My name is Arturo.

s9
00:00:34.640 --> 00:00:43.600
I'm uh working on this tool called Nereo and uh first I want to show you kind of like how it works today.

s10
00:00:43.600 --> 00:00:47.520
Uh so you get a glimpse of what I mean by all this.

s11
00:00:47.520 --> 00:00:58.160
Um So, we we we we can start asking for what we want and describing something like, "Okay, I want to add a robot."

s12
00:00:58.160 --> 00:01:06.200
Um okay, so I ask my assistant to help me add a robot.

s13
00:01:06.200 --> 00:01:13.720
It found some assets that are robots or having tagged as robot or the description are uh a robot.

s14
00:01:13.720 --> 00:01:16.280
In this case, I want to have a character.

s15
00:01:16.280 --> 00:01:19.840
It's going to be the character that I will be controlling.

s16
00:01:19.840 --> 00:01:22.880
Um if I try to play right now, it won't do anything.

s17
00:01:22.880 --> 00:01:24.880
I still need to tell what I want.

s18
00:01:24.880 --> 00:01:26.280
I need to describe, "Oh, okay.

s19
00:01:26.280 --> 00:01:34.600
I I want this to be uh to move with WASD and I want it to be animated uh have some sort of animation."

s20
00:01:34.600 --> 00:01:40.880
So, I will describe that to my um assistant uh which is called Bibi, by the way.

s21
00:01:40.880 --> 00:01:47.080
And I'm saying, "Make this robot move with WASD and animate it as I as I mentioned."

s22
00:01:47.080 --> 00:01:53.480
These descriptions are pretty common if you know or if you've played games.

s23
00:01:53.480 --> 00:01:57.160
You just describe what you want the the thing to to do.

s24
00:01:57.160 --> 00:02:01.160
And it's language that uh people who play games kind of understand.

s25
00:02:01.160 --> 00:02:07.400
So, you don't need to to code or you don't need to learn about importing models or anything like that.

s26
00:02:07.400 --> 00:02:08.920
The focus is on making the game.

s27
00:02:08.920 --> 00:02:11.080
So, now we have our character moving.

s28
00:02:11.080 --> 00:02:14.959
I want to continue describing my game, building my game.

s29
00:02:14.959 --> 00:02:17.600
So, I want to add some buildings.

s30
00:02:17.600 --> 00:02:24.600
In this case, we have like a library of assets that might match the description of a building.

s31
00:02:24.600 --> 00:02:31.560
We could have like futuristic buildings, something like this warehouse or or store.

s32
00:02:31.560 --> 00:02:34.280
Maybe we can use a a castle.

s33
00:02:34.280 --> 00:02:39.520
The idea here is that we can let our imagination go wild.

s34
00:02:39.520 --> 00:02:45.280
It doesn't need to be you know, like the the the the the perfect game or anything.

s35
00:02:45.280 --> 00:02:51.240
It's kind of like playing a game, playing with toys, playing with with Legos or or something like that.

s36
00:02:51.240 --> 00:02:55.120
I'm saying I want it to to to to have some rain.

s37
00:02:55.120 --> 00:03:03.160
So, I just described the rain and it will spawn a particle system that's set up or configured as as as rain.

s38
00:03:03.160 --> 00:03:08.800
The last thing is I want a camera to follow my character.

s39
00:03:08.800 --> 00:03:14.440
Because if I move around, it will go out of the of the of the view and I won't be able to to see it.

s40
00:03:14.440 --> 00:03:27.640
So, it added a camera, it configured it, it's looking at my main character and I have a game without knowing how to code or import assets or anything besides just knowing

s41
00:03:27.640 --> 00:03:31.920
what I want because I I know the the language of of building building games.

s42
00:03:31.920 --> 00:03:33.640
That's the idea.

s43
00:03:33.640 --> 00:03:41.880
So, making games or getting to this point with a a regular tool is very difficult.

s44
00:03:41.880 --> 00:03:47.800
I I recorded this demo, but you could do this in just a couple of minutes without knowing anything.

s45
00:03:47.800 --> 00:03:49.120
Just describing.

s46
00:03:49.120 --> 00:03:51.959
All the assets are there for you.

s47
00:03:51.959 --> 00:03:57.000
But if you were to follow the traditional workflow, you you will need to know other things.

s48
00:03:57.000 --> 00:03:58.800
Sorry.

s49
00:03:58.800 --> 00:04:09.840
Before jumping continuing, I I want to mention why am I speaking here and you can decide if you you want to to pay more attention or less attention to to myself.

s50
00:04:09.840 --> 00:04:14.040
I worked at Unity, a game engine company for almost 10 years.

s51
00:04:14.040 --> 00:04:19.239
So, I saw a lot of people building games and struggling with the same things over and over.

s52
00:04:19.239 --> 00:04:20.440
At first, it was fun.

s53
00:04:20.440 --> 00:04:26.320
The challenges were fun, but after seeing them thousands of times, it was like, "Okay, this is not fun.

s54
00:04:26.320 --> 00:04:33.120
I don't think people should spend their energy and and and time uh reinventing the same wheel."

s55
00:04:33.120 --> 00:04:35.720
So, then I I I was part of MongoDB.

s56
00:04:35.720 --> 00:04:43.160
Then I was learning about more AI topics and how to you know, manage data and all that stuff.

s57
00:04:43.160 --> 00:04:49.480
And I was helping a startup to manage and handle a version control of assets of of games.

s58
00:04:49.480 --> 00:04:53.000
So, as I said, I've been I've seen people make games.

s59
00:04:53.000 --> 00:05:07.000
I've made games and I think I want to to to to to let more people have fun and and experience the joy of making games without getting frustrated and without

s60
00:05:07.000 --> 00:05:14.760
you know, spending two three years releasing something, realizing that it's difficult to to to to sell a game.

s61
00:05:14.760 --> 00:05:16.800
There's a lot of competition.

s62
00:05:16.800 --> 00:05:28.280
And of course, if you want to do that, I'm not preventing you from doing that, but a lot of people I think it's it's just an a creative outlet making games and the game itself should be the the the thing, right?

s63
00:05:28.280 --> 00:05:33.240
Like enjoying the process rather than the end product.

s64
00:05:33.240 --> 00:05:33.800
Okay.

s65
00:05:33.800 --> 00:05:40.080
So, as I was I was saying before, making a game requires a lot of skills, a lot of knowledge.

s66
00:05:40.080 --> 00:05:41.680
It's very it's very difficult.

s67
00:05:41.680 --> 00:05:56.760
You need to know programming, 3D modeling, rendering, music, animation, and so many other things, you either have a huge team that complement each other, or if it's just you, or you have a small team,

s68
00:05:56.760 --> 00:05:59.120
people need to wear multiple hats.

s69
00:05:59.120 --> 00:06:08.760
And honestly, it's very difficult to find someone that's great at game design, and also great at rendering, and great at composition of cameras in real time.

s70
00:06:08.760 --> 00:06:14.440
So, on top of that, the game has to be fun, right?

s71
00:06:14.440 --> 00:06:21.640
If If it works and technically works, if it's not fun, honestly, people won't play it.

s72
00:06:21.640 --> 00:06:25.880
And I think fun should be for the player, but also for the developers.

s73
00:06:25.880 --> 00:06:36.720
In recent years, I think it's become more about producing more and trying to sell more, rather than than than the the craft of making games.

s74
00:06:36.720 --> 00:06:42.120
And I don't think you need to be a a professional game developer to be able to to make games.

s75
00:06:42.120 --> 00:06:50.480
The same as with with AI assistants to write code, now you don't need to be a programmer to to build whatever you want.

s76
00:06:50.480 --> 00:06:51.560
The same for for games.

s77
00:06:51.560 --> 00:06:55.919
There are other challenges, but that's my my thought.

s78
00:06:55.919 --> 00:07:04.440
Now, there are powerful engines, Unreal, Unity, and there are powerful LLMs and agents, but I think still it's hard.

s79
00:07:04.440 --> 00:07:15.880
It's hard because I think we're just building a bridge between two worlds, and it's not optimal, and you still need to know kind of like what to ask in vocabulary

s80
00:07:15.880 --> 00:07:19.240
of an engine, in the vocabulary of of code.

s81
00:07:19.240 --> 00:07:28.200
Otherwise, the the the the the LLM goes rogue and does stuff, and reinvents the wheel over and and over.

s82
00:07:28.200 --> 00:07:29.200
This is the the point, right?

s83
00:07:29.200 --> 00:07:38.200
Like, if I say, "I want a camera that follows this character," I've seen the the demos, and the LLM reinvents the wheel every single time.

s84
00:07:38.200 --> 00:07:41.960
Well, when the result is going to be essentially the the same.

s85
00:07:41.960 --> 00:07:51.280
Uh, by default, the context is on the game engine uh rather than on the game design part, and I think we should flip that idea.

s86
00:07:51.280 --> 00:08:01.920
So, in a current engine, if you want to control a character, you need to have a mesh uh uh import a mesh and think about uh so many things, a renderer,

s87
00:08:01.920 --> 00:08:09.520
an animator, a rigid body, and collider for physics, an audio source, and then you add your movement logic and your game rules.

s88
00:08:09.520 --> 00:08:20.080
Uh most of that is just boilerplate that every single game out there has or every character in every single game out there has, but somehow developers need to understand

s89
00:08:20.080 --> 00:08:27.919
and read the descriptions of those components and what the hundreds of of of of sliders do uh in in in a game.

s90
00:08:27.919 --> 00:08:35.520
So, I think the the the the the goal should be just to think, "Okay, everything is just an asset.

s91
00:08:35.520 --> 00:08:38.520
Everything has to be rendered on screen.

s92
00:08:38.520 --> 00:08:47.320
Everything has physics most of the time uh and you just add tags to describe the intent of those those assets in your game.

s93
00:08:47.320 --> 00:08:51.839
So, I want this to be a character, to be animated, to double jump."

s94
00:08:51.839 --> 00:08:55.720
And this is the language of that we use when we play games, right?

s95
00:08:55.720 --> 00:09:03.080
Like uh the tutorial tells you, "Oh, press A to jump and press A again while you're in the air to do a double jump."

s96
00:09:03.080 --> 00:09:06.160
That's that's the language that we should be using.

s97
00:09:06.160 --> 00:09:10.480
And uh also defining what happens when when a an event occurs.

s98
00:09:10.480 --> 00:09:14.800
Like in this case, "Oh, when you collect a coin, I want you to increase the score."

s99
00:09:14.800 --> 00:09:28.200
And uh of course, the physics and the rendering still happen, but uh we're we're we're building uh on a layer on top to make it more accessible for more people to to build uh games.

s100
00:09:28.200 --> 00:09:35.400
Um and the engine knows what to do with these tags because these these systems.

s101
00:09:35.400 --> 00:09:41.600
And this system comes from something that we're calling the ATS or asset tax system.

s102
00:09:41.600 --> 00:09:55.800
Comes from the idea from game development called the entity component system data oriented design, which means we just describe objects with with components or in this case with tags.

s103
00:09:55.800 --> 00:10:08.120
And then there are systems that query for all the assets in the in the world and say, "Okay, this I'm going to move all the objects that have the vehicle,

s104
00:10:08.120 --> 00:10:10.560
the player, and the drivable tag."

s105
00:10:10.560 --> 00:10:14.720
So, that's how how it works and all the games can recycle this.

s106
00:10:14.720 --> 00:10:17.560
In the on the on the right, you see that there's a building.

s107
00:10:17.560 --> 00:10:20.000
It doesn't have tags, so it's not going to move.

s108
00:10:20.000 --> 00:10:30.800
But nothing prevents you from adding the the vehicle and drivable tag to your building and then you have a a building that you can put in a Mario Kart style of game.

s109
00:10:30.800 --> 00:10:36.440
Nothing prevents you from from doing that and as I said like letting your imagination go wild.

s110
00:10:36.440 --> 00:10:40.520
Uh the assistant the AI assistant essentially helps you get unstuck.

s111
00:10:40.520 --> 00:10:48.800
Like if I don't know how to make the car moves, I just ask it and it knows understands like what are the tools available, what are the tags available,

s112
00:10:48.800 --> 00:10:54.280
and then it applies them to to the to the to the asset that I'm talking about.

s113
00:10:54.280 --> 00:11:01.440
Uh how it is driven and this is this is pretty common concept from from um agents.

s114
00:11:01.440 --> 00:11:03.720
You just type your query.

s115
00:11:03.720 --> 00:11:16.200
You well, we build the the the prompt using context from the scene, but also some extra context based on for example, what type of assets are on your game.

s116
00:11:16.200 --> 00:11:24.800
If you're using robots, if you're using something like medieval, etc. We we append that to the to the to the context.

s117
00:11:24.800 --> 00:11:30.000
We also allow the the user to describe the type of game that they want to to make.

s118
00:11:30.000 --> 00:11:35.600
So, that's also part of what is is appended into the into the the context.

s119
00:11:35.600 --> 00:11:41.600
And then the agent just performs calls and appends or removes the the tags.

s120
00:11:41.600 --> 00:11:48.160
All these tags and all these systems are built into the the the the engine.

s121
00:11:48.160 --> 00:11:50.960
We don't have a scripting system in there.

s122
00:11:50.960 --> 00:11:52.240
That's on purpose.

s123
00:11:52.240 --> 00:11:55.280
But it's just JavaScript and runs on the on the browser.

s124
00:11:55.280 --> 00:11:58.480
So, whoever wants to extend, they can.

s125
00:11:58.480 --> 00:12:00.480
Nothing prevents them from from doing that.

s126
00:12:00.480 --> 00:12:05.240
But for most users that shouldn't be the the case.

s127
00:12:05.560 --> 00:12:09.880
Yeah, all the updates to the scene happen and that's how how it works.

s128
00:12:09.880 --> 00:12:18.640
Honestly, it's the the the challenging part is the composing games into these tags and into these systems.

s129
00:12:18.640 --> 00:12:26.280
Because there are a lot of genres, there are a lot of ways to describe the same game, there are things like the mood of the game.

s130
00:12:26.280 --> 00:12:31.240
Like I want to make a platformer but that feels like scary.

s131
00:12:31.240 --> 00:12:43.720
Well, that also touches on things like like the post-processing effects and things like the lighting, etc. So, those those things are the things that we're decomposing to to to

s132
00:12:43.720 --> 00:12:45.640
drive the the engine.

s133
00:12:45.640 --> 00:12:47.240
How I'm building this?

s134
00:12:47.240 --> 00:12:59.000
Well, it's it's the first time that I built something of this scale at this speed and using you know, mostly working with with an AI daily.

s135
00:12:59.000 --> 00:13:02.440
And I'm sure everyone who uses Cloud Code or something similar.

s136
00:13:02.440 --> 00:13:16.880
The way it works is I bring all this stuff that is on my mind and I talk to the to to Cloud and we discuss that part and I go to let's say if I'm focusing on something related to to to

s137
00:13:16.880 --> 00:13:29.160
lighting, I talk to some friend on the industry who focuses on that and I ask questions and then I bring that in and then we build those tools and I kind of like say I don't think we need this setting.

s138
00:13:29.160 --> 00:13:30.680
I don't think we need this feature.

s139
00:13:30.680 --> 00:13:33.320
I haven't seen a lot of users using those things.

s140
00:13:33.320 --> 00:13:37.880
So, let's get rid of them in order to simplify the the engine.

s141
00:13:37.880 --> 00:13:39.480
Uh we build it.

s142
00:13:39.480 --> 00:13:44.640
Then, as we build, we give uh the definition of the tools to the to the assistant.

s143
00:13:44.640 --> 00:13:50.400
So, every time uh we add new tools, the assistant knows the the tools that we're we're adding.

s144
00:13:50.400 --> 00:13:56.400
And um yeah, right now it's it's kind of like still in closed alpha.

s145
00:13:56.400 --> 00:13:58.440
I don't even know how to call it at this point.

s146
00:13:58.440 --> 00:14:04.440
But, some people are using it, giving feedback, and uh then we go back to square one.

s147
00:14:04.440 --> 00:14:16.520
Um one thing that I I think it's worth sharing also in in in in this presentation is the the way of assembling these contexts because yeah, we're using an LLM mostly.

s148
00:14:16.520 --> 00:14:25.120
Uh I've used uh also vision models mostly to tag the the assets that we have because it's like six or 7,000 assets.

s149
00:14:25.120 --> 00:14:34.600
I could not manually tag them all and explain like, "Oh, this is a an astronaut and this is a a knight and this is a castle and this is whatever."

s150
00:14:34.600 --> 00:14:38.920
I just have the names and the and the and the 3D file.

s151
00:14:38.920 --> 00:14:43.680
So, I took a screenshot and ran a a vision model to describe the those.

s152
00:14:43.680 --> 00:14:48.000
But, it's mostly an an LLM what's uh what we're using.

s153
00:14:48.000 --> 00:14:54.400
And if we feed the entire scene to the LLM, the context grows a lot.

s154
00:14:54.400 --> 00:15:00.360
Let's say in this in this scene I think I had like 100 assets at 100 things uh being there.

s155
00:15:00.360 --> 00:15:04.440
But, most of them are like grass that we could ignore.

s156
00:15:04.440 --> 00:15:06.640
It doesn't really make sense.

s157
00:15:06.640 --> 00:15:09.160
But, uh this is simple like like scene.

s158
00:15:09.160 --> 00:15:18.280
But, what we're using here to to assemble the context is something that's very well known in in the in the game dev world, which is called level of details,

s159
00:15:18.280 --> 00:15:30.360
uh which means if I'm close to an object uh or or it's close in inside to to myself, I'm going to render it with a higher quality texture, with a higher quality material,

s160
00:15:30.360 --> 00:15:32.920
with a higher quality model in many cases.

s161
00:15:32.920 --> 00:15:42.280
Something that's too far away from the camera, I'm just going to maybe just put a cube and the user won't be able to to tell because it's so far away.

s162
00:15:42.280 --> 00:15:45.360
So, we're using something similar to assemble the context.

s163
00:15:45.360 --> 00:15:54.280
In this case, if the user is editing the game, you can see that maybe well, it's it's it's clicking on the on the knight, right?

s164
00:15:54.280 --> 00:16:06.000
So, the things that are around it might have a higher priority, so we feed them information about the tags that they they have and the the values of the of the settings on on those tags.

s165
00:16:06.000 --> 00:16:18.960
Things that are nearby, we just say, "Okay, there's a something that's a player here at this position, but I'm not going to send you the entire entire context in there."

s166
00:16:18.960 --> 00:16:19.520
Okay?

s167
00:16:19.520 --> 00:16:33.640
And as as a user keeps moving around and modifies things, then we update that and feed the the assistant with more relevant information about about the game.

s168
00:16:33.920 --> 00:16:49.680
Okay, so just to start wrapping up, I I I think this assistant called Weeby should be kind of like the the something that gets people unstuck from making the game.

s169
00:16:49.680 --> 00:16:56.680
I don't want us to one-shot games that nobody is going to play and I don't see the point in in that.

s170
00:16:56.680 --> 00:17:07.640
There's in the industry, of course, the need for for games that are one-shotted, but here the idea is that we allow people to make games and experience that and have fun

s171
00:17:07.640 --> 00:17:11.800
and share that those games with their families and friends and and and that.

s172
00:17:11.800 --> 00:17:20.920
And of course, that they learn along the way the language of making games, the language of game design, not necessarily the coding or programming.

s173
00:17:20.920 --> 00:17:28.840
Although people have used it this and say like, "Oh yeah, I can understand now concepts that I didn't understand before of programming."

s174
00:17:28.840 --> 00:17:32.000
But that was not the the the initial goal.

s175
00:17:32.000 --> 00:17:50.400
Um there's another kind of like branch of of engines or tools that are are being used called world models, which are being generated on the on the fly and and some people I think there's another session later talking about these and how games can be achieved with world models.

s176
00:17:50.400 --> 00:17:54.920
I think that's going to be a different medium even if we call them video games.

s177
00:17:54.920 --> 00:17:56.520
Uh there are many challenges.

s178
00:17:56.520 --> 00:18:01.960
Games for instance in these days they have to render 60 frames per second.

s179
00:18:01.960 --> 00:18:10.280
And doing that at 4K resolutions in real time uh with a with a world model, I think it's still far away.

s180
00:18:10.280 --> 00:18:20.240
And on top of that rendering physics and on on well sorry, simulating physics and stuff, it's very difficult, but I think it's exciting what's going on.

s181
00:18:20.240 --> 00:18:23.360
Um so that's it.

s182
00:18:23.360 --> 00:18:33.280
If you want to play around, if you want to share this with your kids or friends or or whoever, uh I I can add you to the to the wait list.

s183
00:18:33.280 --> 00:18:44.200
Um it's honestly it's been very fun to make, but also whenever I want to test something and I just play around and and and bring in uh an astronaut

s184
00:18:44.200 --> 00:18:48.880
and bring in a dinosaur and make them just walk around, it's it's so much fun.

s185
00:18:48.880 --> 00:18:55.080
It's uh it's it's kind of like what got me into game development I think like 20 years ago.

s186
00:18:55.080 --> 00:18:57.920
And I progressed throughout my career.

s187
00:18:57.920 --> 00:19:02.520
I started getting like kind of disappointed like, "Oh I don't This is not fun anymore."

s188
00:19:02.520 --> 00:19:06.280
So I'm having fun again and I want to share that with uh more people.

s189
00:19:06.280 --> 00:19:12.720
So thank you so much and I'll be outside if you have questions, want to chat more about games or whatever.

s190
00:19:12.720 --> 00:19:14.778
Thank you so much for your time.

s191
00:19:14.778 --> 00:19:16.778
[applause]

s192
00:19:30.498 --> 00:19:32.498
[music]