WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=0I6aoPSRzVc

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/0I6aoPSRzVc.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.040 --> 00:00:14.400
Hello everyone.

s3
00:00:14.400 --> 00:00:19.160
Um So, I want to start with a simple question.

s4
00:00:19.160 --> 00:00:24.360
What if your team or your org or company moves like a single body?

s5
00:00:24.360 --> 00:00:28.120
I'm Abdullah Muhammad, the VP of AIML at A Data Chef.

s6
00:00:28.120 --> 00:00:34.800
And today was supposed to be Khalid with me to present this, but he's he's down with our development partner at the moment.

s7
00:00:34.800 --> 00:00:38.160
So, I will be presenting the whole presentation for today.

s8
00:00:38.160 --> 00:00:41.280
So, let's go for the next slide.

s9
00:00:41.280 --> 00:00:47.480
So, how many of you have been attending the World Cup soccer or watching some games on ice?

s10
00:00:47.480 --> 00:00:49.240
We have a couple of fans.

s11
00:00:49.240 --> 00:00:51.000
Yeah, it's over all over the place.

s12
00:00:51.000 --> 00:00:58.000
And imagine for a moment, just a single moment, you are a soccer player, all right?

s13
00:00:58.000 --> 00:01:07.840
And if you are a soccer player, you have this intent the moment you go into the field, you just going to run and score a goal.

s14
00:01:07.840 --> 00:01:10.160
This is what you want to do.

s15
00:01:10.160 --> 00:01:21.920
And for the second thing, you have this knowledge that you've been accumulated through your training the whole day, your exercises with your coach, and the best practices and the videos you have watched.

s16
00:01:21.920 --> 00:01:30.480
And you at the moment in the field like the moment of truth that you are there, you combine both of the intent and knowledge and compound both of them.

s17
00:01:30.480 --> 00:01:35.000
And through your nervous system, you execute to achieve your goal.

s18
00:01:35.000 --> 00:01:43.480
And we can call this, in a sense, you are being self-aligned as a single entity by yourself.

s19
00:01:44.400 --> 00:01:51.800
And accept the fact that a soccer team or a football team, depending where you're coming from, is not a single player.

s20
00:01:51.800 --> 00:01:54.720
It's actually 11 players.

s21
00:01:54.720 --> 00:02:01.240
And on the field, you are up against another team with 11 players they playing against you.

s22
00:02:01.240 --> 00:02:11.000
And at this moment it's not about your individual skills, it's about how your team working together well.

s23
00:02:11.000 --> 00:02:24.440
So, in general like the team keep changing and everything is getting harder and harder and the team that wins actually the team that the most aligned in both of the both of teams.

s24
00:02:24.440 --> 00:02:31.280
So, in short we can say alignment beats individual skills.

s25
00:02:31.306 --> 00:02:32.040
[snorts]

s26
00:02:32.040 --> 00:02:33.480
Okay.

s27
00:02:33.480 --> 00:02:38.800
Now, what if your team is over 50 engineers or 50 players?

s28
00:02:38.800 --> 00:02:42.400
This is completely changes the whole scene right now.

s29
00:02:42.400 --> 00:02:52.400
So, everyone at these days we empower the engineers with AI tools, AI agents and we want to increase the productivity.

s30
00:02:52.400 --> 00:03:01.560
But we know from literature that the more people you have, the quadratic term of communication between them and alignment them keep growing and keep growing.

s31
00:03:01.560 --> 00:03:07.840
And at a specific point actually it actually starts to going declining.

s32
00:03:07.840 --> 00:03:10.480
Your throughput actually is not what you getting.

s33
00:03:10.480 --> 00:03:12.200
It's diminishing cost.

s34
00:03:12.200 --> 00:03:20.040
So, everyone trying to solve this linear problem of more tools and more stuff but nobody actually tackling the quadratic term over there.

s35
00:03:20.040 --> 00:03:21.880
And this is why the alignment is important.

s36
00:03:21.880 --> 00:03:31.200
If you are able to change this quadratic term into a linear term or build a multi-layer AI system that will solve this problem.

s37
00:03:32.880 --> 00:03:35.000
Okay, moving into ship design.

s38
00:03:35.000 --> 00:03:39.160
Ship design is a different story.

s39
00:03:39.160 --> 00:03:45.360
If you are in software company, you have a bug in your software, you can ship a batch to fix it.

s40
00:03:45.360 --> 00:03:47.080
You can roll out a new version.

s41
00:03:47.080 --> 00:03:49.239
It's most of the time is doable.

s42
00:03:49.239 --> 00:03:51.239
But in ships, you can't do this in ships.

s43
00:03:51.239 --> 00:03:54.560
It's hardware, fixed it on silicon has been printed.

s44
00:03:54.560 --> 00:04:00.080
And if you're going to do this, there is a cost actually, we call it the risk band cost.

s45
00:04:00.080 --> 00:04:05.720
On average between chip design companies about $50 million.

s46
00:04:05.720 --> 00:04:12.200
And for some companies like being 1 month late in the market, it's a make or break for them.

s47
00:04:12.200 --> 00:04:14.920
And we spoke to many practitioners in the field.

s48
00:04:14.920 --> 00:04:20.480
On average like 15 practitioner and we found that most of them pointed towards the same problem.

s49
00:04:20.480 --> 00:04:26.520
That we spend 70% of our time doing alignment.

s50
00:04:26.520 --> 00:04:30.280
Alignment to make sure that once we print the chip, nothing is there.

s51
00:04:30.280 --> 00:04:41.560
And one of the key words that we heard and still resonating that the most successful chip organization are not the one with the best engineers, but they are the most aligned

s52
00:04:41.560 --> 00:04:43.560
organized.

s53
00:04:44.440 --> 00:04:48.400
So, how chip design today works?

s54
00:04:49.560 --> 00:04:53.520
We start with bottom figure like the fragmented intent and decision.

s55
00:04:53.520 --> 00:04:57.919
You attend couple of meetings, you talk about decisions, what you're going to do next.

s56
00:04:57.919 --> 00:05:04.560
You have the specs written everywhere, you have the Slack messages, you have emails, everything is fragmented over there.

s57
00:05:04.560 --> 00:05:08.680
And then we go into a second part, which is the knowledge.

s58
00:05:08.680 --> 00:05:10.560
Nobody updates wikis, right?

s59
00:05:10.560 --> 00:05:12.280
Many of us has wikis.

s60
00:05:12.280 --> 00:05:18.560
They've been collecting dust for years and the code keep evolving outside the wikis, it's not over there.

s61
00:05:18.560 --> 00:05:24.200
And now we have the tools that you execute with, which comes with many many fractions.

s62
00:05:24.200 --> 00:05:31.560
And these tools like the data is lost over there, what input, what output, what results, most of the time are not being captured.

s63
00:05:31.560 --> 00:05:38.080
And what you see here is not something we came with like draw from our imagine, this is actually how is it today.

s64
00:05:38.080 --> 00:05:44.400
We wrote from inside the companies and from the backgrounds of the people we have in our team.

s65
00:05:44.560 --> 00:05:50.600
And what we're trying to solve here is building a multi-layer AI with a shared nervous system.

s66
00:05:50.600 --> 00:05:58.760
Instead of having scattered knowledge or scattered intent all over the place, we build a living graph.

s67
00:05:58.760 --> 00:06:00.760
We call it the system of intent.

s68
00:06:00.760 --> 00:06:08.480
And this living graph actually has all the constraints of the system, has all the decisions over there.

s69
00:06:08.480 --> 00:06:09.920
It keep evolving.

s70
00:06:09.920 --> 00:06:19.680
And as an AI person actually, we don't allow the agents to touch it except with human in the loop approval for specific changes.

s71
00:06:19.680 --> 00:06:22.000
And this thing is like the Bible of the whole system.

s72
00:06:22.000 --> 00:06:26.680
This is where the whole org is going or whole company is going.

s73
00:06:26.680 --> 00:06:28.919
And the next one is the tribal knowledge layer.

s74
00:06:28.919 --> 00:06:39.160
The tribal knowledge layer we can think about it as a memory that keeps evolving with day-to-day usage and the knowledge base that capture all the information and documents.

s75
00:06:39.160 --> 00:06:45.280
And it's keep evolving from a project to project and keeping the best practice over there.

s76
00:06:45.280 --> 00:07:00.600
And lastly, instead of having this general coding agent that everyone uses today, we have a special designed agent that being developed by subject matter experts to help the engineers doing their work.

s77
00:07:00.600 --> 00:07:08.040
So for example, like we have digital design agent, analog design agent, and so on.

s78
00:07:08.040 --> 00:07:15.680
And by combining all of this, you will have this shared nervous system that allows you to move fast and move forward.

s79
00:07:18.040 --> 00:07:18.919
Okay.

s80
00:07:18.919 --> 00:07:21.200
So it's easy to say an idea on a slide.

s81
00:07:21.200 --> 00:07:21.840
It's nice.

s82
00:07:21.840 --> 00:07:23.440
Everyone makes slides.

s83
00:07:23.440 --> 00:07:30.040
But I want to show you like a demo from what we have today and showing the intent, knowledge, and execution.

s84
00:07:30.040 --> 00:07:32.080
It will be short demos.

s85
00:07:32.080 --> 00:07:35.240
And we'll start with the first one.

s86
00:07:35.520 --> 00:07:37.400
Yeah, the Yeah.

s87
00:07:37.400 --> 00:07:38.400
Okay, cool.

s88
00:07:38.400 --> 00:07:45.120
So we can see that each engineer gets a role-based AI teammate specific to their role.

s89
00:07:45.120 --> 00:07:54.400
They can check the knowledge base of the whole project that being contained and being growing and compounding over time and now they have their own intent.

s90
00:07:54.800 --> 00:08:05.120
And you have single place for design where it captures all the tooling you have.

s91
00:08:05.120 --> 00:08:06.560
It captures the results.

s92
00:08:06.560 --> 00:08:11.000
It captures what you did and what you're going to do next and analysis of everything.

s93
00:08:11.000 --> 00:08:14.560
So, everything being contained in one place.

s94
00:08:17.080 --> 00:08:23.360
And here we see a human finishing their work.

s95
00:08:23.360 --> 00:08:32.520
This human signing off the the results of some space simulation and the system of intent realizes, okay, this person is done with this.

s96
00:08:32.520 --> 00:08:40.960
I'm going to notify the next stakeholders of what they should do and signal to them that they are done with this.

s97
00:08:42.680 --> 00:08:50.320
And now the system of intent, which actually the nervous system or the Bible of the system, it's a graph living graph that keep compounding with time.

s98
00:08:50.320 --> 00:09:01.640
We see in this example, like it realizes like there is something off, like some value out of constraints that shouldn't be there that might cost you $50 million actually to suspend the whole ship

s99
00:09:01.640 --> 00:09:07.080
and it notified the system and the notification goes and some engineers start working on it.

s100
00:09:07.080 --> 00:09:14.040
And once it got it fixed, it submit a game into the system and it keep evolving over time.

s101
00:09:15.280 --> 00:09:16.640
Okay.

s102
00:09:16.640 --> 00:09:18.640
Good.

s103
00:09:18.680 --> 00:09:29.040
So, let's say for example, like um you were working in the system, you look at the Bible, you find, oh, there there's something wrong about it.

s104
00:09:29.040 --> 00:09:31.200
Uh I don't like this value.

s105
00:09:31.200 --> 00:09:32.960
And then you propose a change.

s106
00:09:32.960 --> 00:09:45.840
So, the system of intent and this big graph captures all the values over there, all the stakeholders, and you start doing this modification, and it gather all the shared knowledge,

s107
00:09:46.120 --> 00:09:53.920
and then it fire a request, as you can see here, and this request goes to an architect or an owner of the system.

s108
00:09:53.920 --> 00:10:01.400
The owner can approve or decline it, and the moment they approve that this is a valid change, it actually goes and echo in the whole system.

s109
00:10:01.400 --> 00:10:04.480
Like everyone will know that this decision has been made.

s110
00:10:04.480 --> 00:10:08.880
There is that change that advises everything over there.

s111
00:10:08.880 --> 00:10:10.880
What?

s112
00:10:14.440 --> 00:10:15.040
Good.

s113
00:10:15.040 --> 00:10:27.680
So, moving to a very difficult topic we have like how we going to evaluate uh our claims and measure the success of the system.

s114
00:10:28.520 --> 00:10:34.520
The philosophy we are using this or the philosophy toward this, we don't grade the agents.

s115
00:10:34.520 --> 00:10:36.560
We try to grade alignment itself.

s116
00:10:36.560 --> 00:10:40.160
So, we have four axes, two horizontal, two vertical.

s117
00:10:40.160 --> 00:10:49.080
The horizontal axes like qualitative, the vertical axes like qualitative and quantitative values, which is typical in this domain at the moment.

s118
00:10:49.080 --> 00:10:53.520
And then horizontal ones, which is bare component and the system into it.

s119
00:10:53.520 --> 00:10:53.697
And

s120
00:10:53.697 --> 00:10:54.120
[snorts]

s121
00:10:54.120 --> 00:11:05.200
if we're going to zoom into the bare component, you can measure like if that agent giving you the correct output for this voltage, like known values versus golden answers.

s122
00:11:05.200 --> 00:11:11.600
Or you can use LNM judge and measure the golden answer versus the expert we have for this one.

s123
00:11:11.600 --> 00:11:12.320
Which is okay.

s124
00:11:12.320 --> 00:11:18.520
You can measure how good my memory, like if the recall state of art, which is the case in our thing.

s125
00:11:18.520 --> 00:11:21.000
Are we doing inference really good?

s126
00:11:21.000 --> 00:11:26.880
But then it comes into the harder question, which is basically are we doing a task completion?

s127
00:11:26.880 --> 00:11:33.040
Like if someone uses this whole thing, is he really completing the task he want to do.

s128
00:11:33.040 --> 00:11:35.160
Is he frustrated while using this?

s129
00:11:35.160 --> 00:11:39.120
Are our agent overstepping human in the loop approval or not?

s130
00:11:39.120 --> 00:11:42.400
Sometimes the agent go goes out on that end.

s131
00:11:42.400 --> 00:11:50.000
And we measure also does does our system allow you to work concurrently on multiple task in parallel?

s132
00:11:50.000 --> 00:11:53.240
This is a success metric or success goal we have.

s133
00:11:53.240 --> 00:11:54.960
And the last one is token tax.

s134
00:11:54.960 --> 00:12:01.560
We don't want to overload you once you use this with all the lovely tokens and increase your budget.

s135
00:12:01.560 --> 00:12:15.480
And there is hard frontier here like in the future now the topic of memory or graph memory or graph rag, whatever the title is, is there is around like 150 papers in this area at the moment

s136
00:12:15.480 --> 00:12:17.400
and all of them are addressing in a nice way.

s137
00:12:17.400 --> 00:12:20.080
You can measure the recall there is data sets.

s138
00:12:20.080 --> 00:12:27.400
But there is no work and research at the moment that targets tribal memory or institutional memory.

s139
00:12:27.400 --> 00:12:28.880
Like what does it mean exactly?

s140
00:12:28.880 --> 00:12:32.280
How do you measure tribal memory success?

s141
00:12:32.280 --> 00:12:41.160
And also for the chip design domain, it's actually even harder because there is not enough data sets like computer vision domain, there is many data sets over there.

s142
00:12:41.160 --> 00:12:42.360
So there is nothing collected.

s143
00:12:42.360 --> 00:12:49.520
So we have our own wheel and going with SMEs collecting this kind of data sets.

s144
00:12:49.680 --> 00:12:50.760
Cool.

s145
00:12:50.760 --> 00:12:50.943
So

s146
00:12:50.943 --> 00:12:51.200
[snorts]

s147
00:12:51.200 --> 00:12:52.120
what broke?

s148
00:12:52.120 --> 00:12:57.440
Which actually when I attend any talk I like to hear what broke, how do you fix it?

s149
00:12:57.440 --> 00:13:00.000
First, agent overstepped.

s150
00:13:00.000 --> 00:13:11.000
In early design phases of the system, we found that an analog agent that's specifically for analog design actually overstepping and doing RTL agent work.

s151
00:13:11.000 --> 00:13:13.120
Which wasn't really great.

s152
00:13:13.120 --> 00:13:16.240
Even we tried to enforce it, but it was a difficult problem.

s153
00:13:16.240 --> 00:13:20.839
And then another thing is we noticed that truth has drifted.

s154
00:13:20.839 --> 00:13:28.920
An agent modifying something in the system not necessarily means it modifies it everywhere it should be modified.

s155
00:13:28.920 --> 00:13:29.920
And that make it harder.

s156
00:13:29.920 --> 00:13:38.800
Like we have the cases specifically where one agent were modifying a parameter, it updated it in one place, five other places were forgotten.

s157
00:13:38.800 --> 00:13:44.720
And the third one is one of my favorite is we asked the agent do not write into specs.

s158
00:13:44.720 --> 00:13:46.840
Just don't don't change the specs.

s159
00:13:46.840 --> 00:13:48.800
They said, "Okay, I obey you.

s160
00:13:48.800 --> 00:13:51.120
I'm not going to write into specs."

s161
00:13:51.120 --> 00:13:55.440
But then they moved into bash and they used set to write into specs.

s162
00:13:55.440 --> 00:13:57.480
We blocked, bash we blocked set.

s163
00:13:57.480 --> 00:13:58.640
They said, "Okay, cool.

s164
00:13:58.640 --> 00:14:01.440
I will use cat actually to write over the specs."

s165
00:14:01.440 --> 00:14:07.800
So we're being like a cat chasing a mouse around to just to prevent it from writing over specs.

s166
00:14:07.800 --> 00:14:14.960
And based on these three failures we have, we came up with principles that we are working today.

s167
00:14:14.960 --> 00:14:24.080
First, we have a spec hierarchy with agent scope and file isolation to allow them only to work on this specific task or specific domain.

s168
00:14:24.080 --> 00:14:28.200
That's all this our problem of agents stepping on each other.

s169
00:14:28.200 --> 00:14:39.120
Second one is we have a single source of truth with automatic conflict detection that is not element based but actually rule based that can detect that this agent did this issue.

s170
00:14:39.120 --> 00:14:44.920
And we can or want to change this value and actually resonate in the whole system immediately.

s171
00:14:44.920 --> 00:14:50.320
And thirdly, which I think of it as an IT administration for agent, we block at the source.

s172
00:14:50.320 --> 00:14:57.080
Like we block from system level, not about level like tool by tool, but just we try to block it over there.

s173
00:14:57.080 --> 00:15:09.320
And the key lesson we learned here that agents care about like if you have your agents which are intelligent, it what matters is substrate layer that they are living in.

s174
00:15:09.320 --> 00:15:13.160
Like the world they living in is more important than the agents itself.

s175
00:15:13.160 --> 00:15:18.760
Like what they can do, what they cannot do, what you allow and what you don't allow.

s176
00:15:19.040 --> 00:15:19.640
Cool.

s177
00:15:19.640 --> 00:15:22.320
So, I'm going to use the word bottleneck.

s178
00:15:22.320 --> 00:15:26.040
It's been used many times, but actually it's bottleneck in our case.

s179
00:15:26.040 --> 00:15:27.520
It wasn't missing intelligence.

s180
00:15:27.520 --> 00:15:29.480
It was missing alignment.

s181
00:15:29.480 --> 00:15:33.640
And a shared nervous system lets your team move like a one body.

s182
00:15:33.640 --> 00:15:42.760
As we see at the moment, one of the things I like hearing from our subject matter experts that they're saying that at the beginning of system is not working fine.

s183
00:15:42.760 --> 00:15:44.600
Now it is good.

s184
00:15:44.600 --> 00:15:46.360
Now I feel it's racing me.

s185
00:15:46.360 --> 00:15:48.520
This is success for our case.

s186
00:15:48.520 --> 00:15:52.760
And we think that this gives you four x leverage from our measurement at the moment.

s187
00:15:52.760 --> 00:15:55.440
And alignment is universal.

s188
00:15:55.440 --> 00:16:00.160
We're building it for the hardest case, which is ship design.

s189
00:16:00.680 --> 00:16:06.120
So currently we're in alpha stage with our development partners.

s190
00:16:06.120 --> 00:16:09.240
And the sign ups for beta are open.

s191
00:16:09.240 --> 00:16:15.400
And you can actually join now and we expect it to release it in October 26.

s192
00:16:15.560 --> 00:16:24.080
If you want to reach out us, sign up for the beta, just use this QR code or the link over there.

s193
00:16:24.080 --> 00:16:26.360
Thank you everyone.

s194
00:16:27.751 --> 00:16:29.751
[applause]

s195
00:16:43.471 --> 00:16:45.471
[music]