WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=418t26CVz-w

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/418t26CVz-w.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.560 --> 00:00:12.960
Awesome.

s3
00:00:12.960 --> 00:00:21.240
So, yes, as great introduction took us to we are going to be talking about local agentic theory for accessible mobile games.

s4
00:00:21.240 --> 00:00:25.480
And we realize this is the graph track, but there are graphs here in this presentation.

s5
00:00:25.480 --> 00:00:28.240
So, fret not if you're missing them.

s6
00:00:28.240 --> 00:00:30.200
So, yeah.

s7
00:00:30.200 --> 00:00:33.120
So, a few disclaimers before we continue.

s8
00:00:33.120 --> 00:00:40.400
Since we we do work at The New York Times and there's a few things we have to say about our puzzles AI etc. Our puzzles are made by people.

s9
00:00:40.400 --> 00:00:41.560
They're not made by AI.

s10
00:00:41.560 --> 00:00:47.480
So, that's just a thing we this true and it'll always be true.

s11
00:00:47.480 --> 00:00:49.480
There's no AI in the games themselves.

s12
00:00:49.480 --> 00:00:51.760
So, there's no AI features.

s13
00:00:51.760 --> 00:00:54.120
You might have seen Wordle Bot, that's not an AI feature.

s14
00:00:54.120 --> 00:00:58.080
There's so all our games are are pretty much AI free.

s15
00:00:58.080 --> 00:01:03.360
And the work that we're going to be talking about now is experimental work.

s16
00:01:03.360 --> 00:01:20.560
So, it's stuff that we look at in terms of solvability and other capabilities of agents in the space that we can potentially take advantage of on a local device to to do other things than actually generate games more so things about playability and other kinds of features we can

s17
00:01:20.560 --> 00:01:23.680
utilize agents for in an intelligent way.

s18
00:01:23.680 --> 00:01:26.360
So, introduce a little bit of an introduction.

s19
00:01:26.360 --> 00:01:36.080
So, I also if anyone has seen the last year's World Fair talk that I did on connections, there's a similar vibe to this.

s20
00:01:36.080 --> 00:01:40.520
I did a deeper dive into how a solver can be built for that game.

s21
00:01:40.520 --> 00:01:42.560
And if you're interested, you can take a look.

s22
00:01:42.560 --> 00:01:50.200
It's on YouTube and it has a lot of interesting deeper dive specifics for RL solvers, which I'll go into in a bit as well.

s23
00:01:50.200 --> 00:01:52.800
So, little bit of a history.

s24
00:01:52.800 --> 00:02:03.479
Everybody here and just in case you aren't familiar with AI in gaming, which for you know, it's not the most common topic here at at the World's Fair right now, but

s25
00:02:03.479 --> 00:02:08.360
if you are familiar with the history of it, AI has been in games for a long time.

s26
00:02:08.360 --> 00:02:16.040
And since this is since the '80s, it's been established as a core pattern in games like Pac-Man.

s27
00:02:16.040 --> 00:02:19.440
This is a simple version of an AI called a finite state machine.

s28
00:02:19.440 --> 00:02:23.920
And if anybody everyone's familiar with a finite state machine here, probably.

s29
00:02:23.920 --> 00:02:25.400
I would imagine enough people are.

s30
00:02:25.400 --> 00:02:33.680
It's a basic AI system which tells the ghosts in the game what to do when Pac-Man gets a dot or not.

s31
00:02:33.680 --> 00:02:38.440
But basically it is a conditional kind of symbolic AI.

s32
00:02:38.440 --> 00:02:43.440
And then moving forward we'll talk more about RL and the advancements of where that went to.

s33
00:02:43.440 --> 00:02:54.360
But firstly, the core thesis of what we're going to really focus on is the model of running the devices running AI on our local devices.

s34
00:02:54.360 --> 00:03:05.959
So most AI infrastructure today runs really on the cloud and if you're running mobile application, most practical scenarios involve cloud architecture.

s35
00:03:05.959 --> 00:03:13.840
And that that essentially is expensive because you have AI calls and inference calls that you have to make upstream and then they have to come to your device.

s36
00:03:13.840 --> 00:03:24.519
So there's things like latency, etc. So in an ideal world when we evolve to that place, we can offload a lot of the intelligence onto a device.

s37
00:03:24.519 --> 00:03:33.480
What that does is make a lot of the make a lot of the calls faster when we get to that world because you're not doing a round trip to the cloud to get that information.

s38
00:03:33.480 --> 00:03:39.160
So you can actually compress that time to the time within the device frame computation.

s39
00:03:39.160 --> 00:03:41.840
That it's private, too.

s40
00:03:41.840 --> 00:03:46.000
So your AI computation is with on it's within the device itself.

s41
00:03:46.000 --> 00:03:53.520
And if it's set up properly, it won't leave the device because there's some things that for the gameplay experience that you don't actually need it to leave the device.

s42
00:03:53.520 --> 00:03:58.880
Local computation is the way to go and it stays within your confined security zone uh locally.

s43
00:03:58.880 --> 00:04:02.480
And uh it's not needed upstream for telemetry or anything like that.

s44
00:04:02.480 --> 00:04:04.200
Um and then it should work anywhere.

s45
00:04:04.200 --> 00:04:17.519
So, one thing is that games uh you know, some a lot of games rely on the internet and and many they they will for a live con- connection point, but at least in in the in many cases where you need advanced AI compute,

s46
00:04:17.519 --> 00:04:23.200
you can develop games that can work locally on your uh machine, on your mobile device, rather.

s47
00:04:23.200 --> 00:04:31.760
And uh that's great because if you're in a subway tunnel or something like that, you can still have a a game with a very smart AI that is um building uh

s48
00:04:31.760 --> 00:04:38.200
doing something uh without being disrupted due to um uh HTTP calls that are not getting through.

s49
00:04:38.200 --> 00:04:46.120
Um And then it's uh again, the personalization is a big thing in general with on-device models because personalization is important.

s50
00:04:46.120 --> 00:04:49.720
Uh and a lot of people want their game experience to be tailored to them.

s51
00:04:49.720 --> 00:04:57.760
And in the world where we can do things um on device, we have that capability more so because the the features are local.

s52
00:04:57.760 --> 00:05:01.280
So, now we're going to talk about agentic AI for games.

s53
00:05:01.280 --> 00:05:07.880
And um as I'm sure everybody's familiar with AlphaGo, AlphaZero, etc. here.

s54
00:05:07.880 --> 00:05:09.320
Um the Alpha series models.

s55
00:05:09.320 --> 00:05:20.000
So, basically um a reinforcement learning was the way that uh games were in the past um trained in order to uh do intelligence, right?

s56
00:05:20.000 --> 00:05:31.320
So, basically what you would do is that you would take a you take a model, uh have many iterations, and then that uh that model would then um be really tuned to a particular game.

s57
00:05:31.320 --> 00:05:38.800
Um and that's still happening, but uh what I'm doing is setting up the kind of history in the school of thought of that uh all the way up to 2024.

s58
00:05:38.800 --> 00:05:42.400
Um has anybody heard of EfficientZero before here?

s59
00:05:42.400 --> 00:05:44.080
Uh if not, it's fine.

s60
00:05:44.080 --> 00:06:01.160
EfficientZero and EfficientZero V2 are the reinforcement models uh reinforcement learning models that are state-of-the-art um and they're called sample efficient reinforcement learning models because they learn uh very fast with a lot of a lot less data than traditional reinforcement learning models do on games.

s61
00:06:01.160 --> 00:06:14.040
Um and there's just uh so that was up to 2024 and that that uh mechanic is continuing, but it lives alongside uh our reinforcement learning and agentic game uh development.

s62
00:06:14.040 --> 00:06:25.400
So, uh reinforcement learning, again, is on the left is a bunch of cycles that are used to train a model in order to change its weight so it becomes more intelligent and able to play a particular game.

s63
00:06:25.400 --> 00:06:38.760
An agentic system is one where you actually have a game that is a negotiated by agents, which are essentially language models, which are using in-context learning to figure out the state of the world in the game.

s64
00:06:38.760 --> 00:06:42.480
So, RL changes the model itself and changes the weights in the model.

s65
00:06:42.480 --> 00:06:49.919
Um the agentic system reasons over a space in the game, um and there's no reward system to grind out.

s66
00:06:49.919 --> 00:07:01.160
Uh the loop There's an agentic loop that's at play, which then learns the system through tool calls and other things that are local to the device and using local device functionality and adapts to a new situation.

s67
00:07:01.160 --> 00:07:03.160
So, it's much more dynamic.

s68
00:07:03.160 --> 00:07:10.600
So, um So, this is uh basically one example of an agentic-based game using Have everybody seen Space Invaders before?

s69
00:07:10.600 --> 00:07:12.040
I I presume.

s70
00:07:12.040 --> 00:07:12.480
Okay.

s71
00:07:12.480 --> 00:07:14.000
Well, if not, you will There it is.

s72
00:07:14.000 --> 00:07:30.000
So, this is a agentic version of Space Invaders, which um is designed to use a local agentic framework which we worked on to um do basically the basic things like kill the spaceships as they get close and avoid the bullets as the spaceships attack.

s73
00:07:30.000 --> 00:07:39.440
And you've probably seen simulations where our models have been trained to do the same thing, but this is specifically an agentic system that we built to actually play this game.

s74
00:07:39.440 --> 00:07:42.320
Um and then you You see the agentic states on the right.

s75
00:07:42.320 --> 00:07:46.560
And so, this is the inside the agent itself and how the loop works.

s76
00:07:46.560 --> 00:07:58.360
So, basically, you have a very few basic states where you're trying to perceive what's happening in this where the agent is trying to perceive what's happening in the scene, where it's trying to predict the next move of the spaceships attacking it, where it's deciding

s77
00:07:58.360 --> 00:08:06.720
what to do next because if the bullet is too close or if the spaceship is too close, it will take a it will make some sort of action and then it will act in that

s78
00:08:06.720 --> 00:08:09.880
way in the appropriate way and then loop again and then see the state.

s79
00:08:09.880 --> 00:08:18.919
So, that's a very simple system, but it's it's basically a simple agent that's using an on-device model to actually execute its steps.

s80
00:08:18.919 --> 00:08:21.735
So, some challenges in this space are space

s81
00:08:21.735 --> 00:08:22.560
[snorts]

s82
00:08:22.560 --> 00:08:26.560
is health is a challenge because on the device you have a limited amount of resources.

s83
00:08:26.560 --> 00:08:39.440
You have a device budget where you have to fit the weights, you have to compress the state history of the agent, you have to compress the working set or the planning artifacts of the agent if you even think of the way that an agent will create the way that it navigates a system,

s84
00:08:39.440 --> 00:08:44.839
and you have to have enough render headroom in order to actually render the game and play the game itself.

s85
00:08:44.839 --> 00:08:46.720
So, this is basically that.

s86
00:08:46.720 --> 00:08:56.200
And then, you also have a time constraint because time itself is budgeted in a particular way so that you are essentially using like a refresh rate of 60 Hz.

s87
00:08:56.200 --> 00:09:03.360
So, in each frame that passes, you have to which you get a new frame every 16 ms a pound if your refresh rate is 60 Hz.

s88
00:09:03.360 --> 00:09:14.240
You need to actually fit the agent planning within that, and if you go outside of that budget, what will happen in the agent execution, you're going to see jank and other things on mobile devices that that are very hard to

s89
00:09:14.240 --> 00:09:15.160
pull out.

s90
00:09:15.160 --> 00:09:19.560
And then, you have energy constraints, and energy constraints are basically battery life.

s91
00:09:19.560 --> 00:09:27.920
So, battery life you know, the basic devi- devices these days are not optimized to um really handle agentic workloads.

s92
00:09:27.920 --> 00:09:37.640
Uh yet, they're getting to that point where you have NPUs, neural processing units, and AI chips that are specifically designed to handle that inference, but at this point in time, um

s93
00:09:37.640 --> 00:09:49.760
the agentic design of the loop on the device has to be very curated to do as minimum energy processing as possible, otherwise your phones die fast as it is and they'll die even faster.

s94
00:09:49.760 --> 00:10:02.080
So essentially there's a constraint function in of this for example constraint graph that takes into account all three of these areas and so there's different ways of managing the constraints in this constraint graph so you can actually

s95
00:10:02.080 --> 00:10:04.040
make sure that your agent is performing well.

s96
00:10:04.040 --> 00:10:20.800
So you can handle space constraints, time constraints, energy constraints and there's some techniques that we use to optimize those constraints so that we're not we're we're penalizing over focus on one area and then we're using like soft constraining on another area so that we're not penalizing

s97
00:10:20.800 --> 00:10:29.080
too much if we need a little extra space but if we use too much time, we want to penalize that because we don't want the agent to disrupt the user experience.

s98
00:10:29.080 --> 00:10:36.480
So we we look at things like gaze estimation models as well because gaze is another new factor.

s99
00:10:36.480 --> 00:10:46.680
It's been around for a while but it's evolving to the point where it can actually um be useful and an agent has to take into account gaze model data which is basically

s100
00:10:46.680 --> 00:10:53.160
convolutional neural net which takes gaze data and then can translate it to agentic behavior.

s101
00:10:53.160 --> 00:11:04.880
And so it just begs to mention that we need to consider things like the Arc AGI score because we also have the base intelligence of the model and the base intelligence of a lot of these models aren't

s102
00:11:04.880 --> 00:11:15.440
capable yet of being fully cognizant of all the complex decisions that an agent can make in the most intelligent and possibly interesting games in the future but we have to

s103
00:11:15.440 --> 00:11:18.160
essentially surmount this to get to that point.

s104
00:11:18.160 --> 00:11:25.080
And the last thing I'll show you before I hand it over to Joanne is this is another agent that we built to solve the mini crosswords.

s105
00:11:25.080 --> 00:11:38.880
So if you play the mini crosswords, this is called a constraint satisfaction agent which uses a satisfaction graph to figure out how to put the words in the crossword puzzle at the right place so that um that it fills the puzzle properly.

s106
00:11:38.880 --> 00:11:49.080
And it backtracks in the puzzle and finds the right word, sometimes finds the wrong word, but the constraint graph here is optimized so that if the constraints are getting too um

s107
00:11:49.080 --> 00:11:59.080
uh too crowded or too incorrect, then it will backtrack efficiently and the agent can use natural language from the graph to backtrack and find the correct crossword uh crossword layout.

s108
00:11:59.080 --> 00:12:05.320
Um so basically where we now, we have a lot of different models that we are considering.

s109
00:12:05.320 --> 00:12:13.960
So we have generalist models um like for example Google Seema, which will be the agentic framework uh agentic framework that can do a lot of different things in this space.

s110
00:12:13.960 --> 00:12:17.720
Um we have agents that do visual estimation on the phone.

s111
00:12:17.720 --> 00:12:22.200
Uh and then we have ins- entire generative worlds that are being built using AI.

s112
00:12:22.200 --> 00:12:25.960
Um and that will be a whole different interesting agentic space to consider.

s113
00:12:25.960 --> 00:12:29.760
Now I'll hand it over to Joanne to talk about accessibility.

s114
00:12:30.440 --> 00:12:31.640
Oh, is this thing on?

s115
00:12:31.640 --> 00:12:32.600
Yes.

s116
00:12:32.600 --> 00:12:34.320
Um let's talk about accessibility.

s117
00:12:34.320 --> 00:12:37.880
How can we build a system that truly serves everyone?

s118
00:12:37.880 --> 00:12:44.560
Um like Shafiq mentioned, traditionally games live in a world of fixed state models, rigid, hand-authored.

s119
00:12:44.560 --> 00:12:47.800
A player's choice is often limited to a toggle or a static menu.

s120
00:12:47.800 --> 00:12:55.960
The flaw here, as you could have guessed, is that it's entirely deterministic, predictable, and completely blind to the real-time needs.

s121
00:12:56.839 --> 00:13:01.800
But we don't need to reinvent the wheel here to battle to build the battle-tested accessible game.

s122
00:13:01.800 --> 00:13:10.839
We ground our design in an existing international standard, WCAG 2.2. This framework evaluates interactions across four human-centric pillars.

s123
00:13:10.839 --> 00:13:15.680
Is this experience perceivable, operable, understandable, and robust?

s124
00:13:15.680 --> 00:13:20.640
Giving us a proven global foundation to build our dynamic player states.

s125
00:13:21.640 --> 00:13:23.800
But you can't have rules without breaking them.

s126
00:13:23.800 --> 00:13:26.560
Some common pitfalls where our break these standards.

s127
00:13:26.560 --> 00:13:29.480
A mobile game like crosswords, as an example.

s128
00:13:29.480 --> 00:13:41.240
There are motor challenges we come across where tiny grid cells can lead to mistaps, active cell highlights are invisible to low vision players, uh cognitive barriers can be forcing manual logins or timers that are unforgiving

s129
00:13:41.240 --> 00:13:45.200
and alienating to process speed uh to processing speed needs.

s130
00:13:45.200 --> 00:13:51.160
A static easy mode can't fix a crossword grid that remains fundamentally blind to our players.

s131
00:13:52.920 --> 00:14:02.000
Although it's still in draft, WCAG 3.0, spoiler alert, is following the trends of dropping the binary pass/fail to a graded bronze, silver, and gold scoring.

s132
00:14:02.000 --> 00:14:07.520
Accessibility is developing into a more diverse scale and moving away from our standard checkbox.

s133
00:14:07.520 --> 00:14:11.040
The same shift should apply for gameplay.

s134
00:14:11.960 --> 00:14:17.160
This scale visualizes the values that can change depending on the needs of the player.

s135
00:14:17.160 --> 00:14:22.560
Here we have the core dials that scale depending on the player's real-time needs.

s136
00:14:24.720 --> 00:14:33.120
If we tune for a player needing more assistance, we have input tolerance that controls the flexibility around physical constraints, making interactions more forgiving.

s137
00:14:33.120 --> 00:14:41.400
The step granularity adjusts for the number of operations in a sequence, helping users break complex tasks and tuning to players who needs more nuance.

s138
00:14:41.400 --> 00:14:51.000
The agent's time The agent's entire job is to dynamically calculate these dials as they rise and fall together, balancing the scale against the player's live context.

s139
00:14:58.040 --> 00:15:02.360
With on-device models, the agent can get a real-time feel for how the player is doing.

s140
00:15:02.360 --> 00:15:11.640
For example, we can use vision models to track eye gaze and figure out when the user might be facing search friction, analyze shaky taps for someone having a hard time with controls,

s141
00:15:11.640 --> 00:15:16.480
meet players where they're at by allow by allowing handwriting recognition.

s142
00:15:17.080 --> 00:15:18.440
Um focus traps.

s143
00:15:18.440 --> 00:15:21.480
But we don't want the agent to just watch, we want it to act.

s144
00:15:21.480 --> 00:15:28.760
We simulate on the right um well, the simulation on the right is a broken dialogue forcing a user into a keyboard trap.

s145
00:15:28.760 --> 00:15:34.000
For a user using a switch or a keyboard instead of touch, focus order is very important.

s146
00:15:34.000 --> 00:15:39.440
If we have a broken dialogue, tab and focus can be an endless loop with no way to navigate back.

s147
00:15:39.440 --> 00:15:46.280
Here the agent can monitor that focus path, flag the loop, and inject an exit route live on the device.

s148
00:15:48.320 --> 00:15:54.200
When you're navigating with limited dexterity or just fat fingers, tiny targets can mean a miss.

s149
00:15:54.200 --> 00:16:01.440
Instead of using players instead of making players play operation, our agent is a live layout auditor.

s150
00:16:01.440 --> 00:16:07.160
It measures the interface on the fly, catches violations, and dynamically resizes the controls.

s151
00:16:07.160 --> 00:16:12.720
The agent rewrites the layout live adapting to the human and not the other way around.

s152
00:16:13.840 --> 00:16:19.360
Think of it like a real-time tug-of-war where the system is no longer fixed but tunes the game to the player.

s153
00:16:19.360 --> 00:16:27.320
Accessibility and challenge stop being treated separately and become two ends of one dial constantly tuning for the moment.

s154
00:16:28.160 --> 00:16:32.960
Ultimately, this is uh there this is where on-device AI is heading.

s155
00:16:32.960 --> 00:16:42.600
By building systems that can actively sense human frustrations, we can move past fixed menus and turn our devices into responsive empathetic partners.

s156
00:16:42.600 --> 00:16:44.440
But there's still groundwork to be laid.

s157
00:16:44.440 --> 00:16:49.480
Looking ahead for local agents to truly understand games, there's still a lot we need to build.

s158
00:16:49.480 --> 00:16:51.680
First, they have to be faster.

s159
00:16:51.680 --> 00:16:57.720
We need a plan we need plans and decisions within a 16-ms frame to prevent stuttering for games.

s160
00:16:57.720 --> 00:17:03.760
We need models to predict the game so that we can see exactly what a layout change will do before they actually make it.

s161
00:17:03.760 --> 00:17:10.520
And the real magic though will be the long-term memory learning one specific person's unique habits and needs over time.

s162
00:17:10.520 --> 00:17:18.880
We need a shared game state language so one agent can work across multiple games instead of being rebuilt from scratch for different releases releases.

s163
00:17:18.880 --> 00:17:22.000
And then finally, we need better chips and honest testing.

s164
00:17:22.000 --> 00:17:27.199
Faster chips paired with real benchmarks to prove that the agent makes things better.

s165
00:17:28.160 --> 00:17:32.400
And the future of AI doesn't have to be one giant centralized brain.

s166
00:17:32.400 --> 00:17:40.200
It can be billions of small local brains, each running on a personal device, each shaped entirely by the individual it serves.

s167
00:17:40.920 --> 00:17:43.120
Thank you.

s168
00:17:43.732 --> 00:17:44.387
[applause]

s169
00:17:44.387 --> 00:17:45.000
[cheering]

s170
00:17:45.000 --> 00:17:47.000
Woo!

s171
00:18:01.417 --> 00:18:03.417
[music]