WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=vUJDyDnqT4c

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/vUJDyDnqT4c.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.880 --> 00:00:13.440
Hi everybody.

s3
00:00:13.440 --> 00:00:14.720
Good morning.

s4
00:00:14.720 --> 00:00:16.000
Um, my name is Adam Nahari.

s5
00:00:16.000 --> 00:00:29.720
I'm going to share a few thoughts today on driving a successful AI transformation and I want to start somewhere specific, a scheduling desk for a large cargo airline.

s6
00:00:29.720 --> 00:00:38.000
An aircraft experiences a failure, the scheduler has 15 minutes to re-crew the flight.

s7
00:00:38.000 --> 00:00:47.240
The process is entirely manual and the scheduler has to consider rest rules, qualifications, where each pilot is.

s8
00:00:47.240 --> 00:00:53.480
They identify one legal solution and it's not necessarily the cheapest one.

s9
00:00:53.480 --> 00:01:03.520
Often, they identify no legal solutions, in which case the release valve is premium pay, paying more to make the problem go away.

s10
00:01:03.520 --> 00:01:12.400
This happens every day across the entire network and crew is the second largest cost after fuel.

s11
00:01:12.400 --> 00:01:13.520
Remember the scheduler?

s12
00:01:13.520 --> 00:01:16.360
I'm going to come back to him.

s13
00:01:16.600 --> 00:01:17.480
Let's zoom out.

s14
00:01:17.480 --> 00:01:18.920
AI tools work.

s15
00:01:18.920 --> 00:01:22.000
Everybody in this room knows that they work.

s16
00:01:22.000 --> 00:01:28.760
Over 55% of US adults have used generative AI within the last 3 years.

s17
00:01:28.760 --> 00:01:39.440
At the same point in the curve, the PC was at 20% and the internet was at 30%.

s18
00:01:39.600 --> 00:01:47.720
Almost 90% of businesses report using AI in at least one business function.

s19
00:01:47.760 --> 00:01:55.720
And yet, only 6% report significant EBIT impact.

s20
00:01:55.880 --> 00:01:58.400
Time savings is real.

s21
00:01:58.400 --> 00:02:00.080
People report it.

s22
00:02:00.080 --> 00:02:11.812
If you add it up, it amounts to about 2% of total US working hours, and yet almost none of that shows up in the P&L.

s23
00:02:11.812 --> 00:02:19.492
Robert Solow famously said, "You can see the computer age everywhere but in the productivity statistics."

s24
00:02:19.492 --> 00:02:22.252
He said that in 1987.

s25
00:02:22.932 --> 00:02:31.772
We are seeing the same thing happen again today on a larger scale and faster speeds.

s26
00:02:31.772 --> 00:02:33.292
Why is that?

s27
00:02:33.292 --> 00:02:39.412
And what do the 6% that see material EBIT impact do differently?

s28
00:02:39.772 --> 00:02:42.212
Here's my answer.

s29
00:02:42.212 --> 00:02:51.132
For every AI transformation that I've led or watched up close, engineering was the smaller part.

s30
00:02:51.132 --> 00:02:58.492
For every five units of change management, there's only one unit of engineering.

s31
00:02:58.492 --> 00:03:01.572
That's my mental model.

s32
00:03:01.572 --> 00:03:04.412
And this ratio is moving.

s33
00:03:04.412 --> 00:03:10.492
As models get better and better, the engineering share is only shrinking.

s34
00:03:10.492 --> 00:03:25.252
Picking the right problem, creating buy-in, driving adoption, measuring impact, that's where the real time and effort goes, and that's where failures tend to happen.

s35
00:03:25.252 --> 00:03:31.852
So, for the rest of this talk, I'm going to focus on those five units of change management.

s36
00:03:32.572 --> 00:03:38.092
So, why doesn't the P&L move even though the tools work?

s37
00:03:38.092 --> 00:03:42.452
I like to think of AI in businesses in these three layers.

s38
00:03:42.452 --> 00:03:49.972
The first layer is general productivity, the copilots and chatbots that we all use.

s39
00:03:49.972 --> 00:04:00.652
Because everyone has access to them, gains tend to show up in price but not in margin.

s40
00:04:00.652 --> 00:04:03.734
They get competed away.

s41
00:04:03.734 --> 00:04:07.174
The second layer is domain-specific AI.

s42
00:04:07.174 --> 00:04:19.174
You can think of these as um business function-specific co-pilots or retrieval over your business documents.

s43
00:04:19.414 --> 00:04:28.414
These tools allow for faster retrieval and better and deeper use of your enterprise knowledge.

s44
00:04:28.414 --> 00:04:39.614
And finally, the third layer, core operational AI, these are the vertical systems embedded in the business processes that make your business what it is.

s45
00:04:39.614 --> 00:04:47.414
So, crew scheduling for airlines is a great example of this.

s46
00:04:47.574 --> 00:04:59.094
And because you can't buy these tools, this change management off the shelf, these gains tend to remain in margin.

s47
00:04:59.094 --> 00:05:17.654
Most businesses are investing in layer one, but the businesses that see material EBIT impact are three times as [clears throat] likely to redesign an entire workflow end-to-end rather than just adopting a new tool.

s48
00:05:17.654 --> 00:05:24.934
They're work They're investing in layer three in addition to investing in layer one.

s49
00:05:27.014 --> 00:05:39.534
I'd like to go through the six moves that in my mind are key to driving a successful AI transformation and moving the P&L.

s50
00:05:39.534 --> 00:05:41.374
Move number one.

s51
00:05:41.374 --> 00:05:45.654
Pick one or two problems, not 40.

s52
00:05:45.654 --> 00:05:50.574
Let's start with a P&L and work our way backwards.

s53
00:05:50.574 --> 00:05:55.614
So, for an airline, that North Star metric might be EBITDA.

s54
00:05:55.614 --> 00:06:09.015
The four drivers of that metric are routes, which is demand, tails, which is supply, maintenance, which dictate constraints, and crew, which is the cost driver.

s55
00:06:09.015 --> 00:06:17.335
It's such a significant cost driver that we decided to start and focus there.

s56
00:06:18.295 --> 00:06:25.975
As I mentioned, it's the second highest cost driver after fuel.

s57
00:06:25.975 --> 00:06:31.255
We did not work on 40 pilots at this company.

s58
00:06:31.255 --> 00:06:38.775
We chose this one initiative that, if we got right, would be truly needle moving.

s59
00:06:38.775 --> 00:06:47.375
So, on the right, you can see the five things that I look for in a good use case or problem.

s60
00:06:47.375 --> 00:06:51.775
The first is it moves a line the CFO recognizes.

s61
00:06:51.775 --> 00:06:58.135
It's measurable with a baseline agreed upfront, not in retrospect.

s62
00:06:58.135 --> 00:07:05.135
The tech and data infrastructure is ready for the transformation, and the company owns the data.

s63
00:07:05.135 --> 00:07:11.895
A named user wants it and can champion the initiative, a person, not a department.

s64
00:07:11.895 --> 00:07:16.495
And this is one of the CEO's top five priorities.

s65
00:07:16.695 --> 00:07:18.415
Okay, move number two.

s66
00:07:18.415 --> 00:07:21.855
Reimagine the process end to end.

s67
00:07:21.855 --> 00:07:37.855
These are illustrative numbers, but the idea is that when you take an existing process and just try to automate it, the gains are incremental.

s68
00:07:37.855 --> 00:07:47.095
Whereas, when you redesign an entire process around the new technology, the gains can be truly transformational.

s69
00:07:47.095 --> 00:08:04.175
So, in airlines, the 10% play would be taking that existing process, scheduling process, that I described to you, and giving the scheduler a tool to make them somewhat faster.

s70
00:08:04.335 --> 00:08:25.177
On the other hand, the 10x play was building an optimization agent engine that took all of these constraints into account, produced the entire list of possible solutions, and ranked them by cost.

s71
00:08:25.177 --> 00:08:30.217
And this system was able to do all of that in under a minute.

s72
00:08:30.217 --> 00:08:41.017
In addition to building that system, we also redesigned the process, the workflow that the scheduler went through.

s73
00:08:41.017 --> 00:08:56.697
This took consolidating all of the company data, including flights, crew, maintenance, building this optimization age engine that I described to you, and building a decision support app for the schedulers.

s74
00:08:57.137 --> 00:09:02.217
All of this engineering work was still the smaller part of this project.

s75
00:09:02.217 --> 00:09:06.257
The bigger job was getting it used.

s76
00:09:06.977 --> 00:09:09.937
Move number three, start with the CEO.

s77
00:09:09.937 --> 00:09:16.657
Before I put any money at risk, the one question I ask is where is the CEO?

s78
00:09:16.657 --> 00:09:24.297
Have they committed publicly that this initiative is a priority to the company?

s79
00:09:24.297 --> 00:09:27.737
If they haven't, I don't fund the initiative.

s80
00:09:27.737 --> 00:09:32.977
The correlation with success is that strong.

s81
00:09:32.977 --> 00:09:43.537
Without it, the company doesn't necessarily have to say no, but you experience scheduling conflict after scheduling conflict.

s82
00:09:43.537 --> 00:09:46.697
Data requests take weeks and weeks.

s83
00:09:46.697 --> 00:09:51.177
Pilots get rescheduled and delayed.

s84
00:09:51.217 --> 00:09:54.457
Everybody takes their cue from the CEO.

s85
00:09:54.457 --> 00:10:06.177
The P&L owner staffs it, the CFO who needs to fund and track it, the CTO who needs to prioritize engineering and technical resources for it.

s86
00:10:06.177 --> 00:10:12.498
And this alignment wears off over time, so we need to keep renewing it.

s87
00:10:12.858 --> 00:10:17.258
Move number four, building with the business.

s88
00:10:17.858 --> 00:10:24.898
Significant transformation requires the P&L owner to drive change.

s89
00:10:24.898 --> 00:10:32.138
In the airline, the VP of crew scheduling was the co-sponsor of this transformation.

s90
00:10:32.138 --> 00:10:40.778
We also identified AI champions, crew schedulers, that were embedded in the solution design from day one.

s91
00:10:40.778 --> 00:10:44.778
They met with the engineering team every other day.

s92
00:10:44.778 --> 00:10:48.978
They designed the interface together with the engineering team.

s93
00:10:48.978 --> 00:10:55.978
They were part of AB testing, and they drove this change to their peers.

s94
00:10:55.978 --> 00:11:00.698
So, they adopted it because they had a hand in building it.

s95
00:11:00.698 --> 00:11:09.538
And likewise, talent, we recruited new leadership to build the platform and the AI at the point of decision-making.

s96
00:11:09.538 --> 00:11:15.738
We hired a chief information officer, a VP of data and AI, and a senior director of data science.

s97
00:11:15.738 --> 00:11:33.098
These folks had both the technical expertise and the domain expertise in airlines, so they understood how schedulers worked and could work with them to design a tool that fit their needs.

s98
00:11:33.098 --> 00:11:42.378
So, the takeaway here is that the organizational investment exceeded the techno- the technology investment.

s99
00:11:42.778 --> 00:11:46.218
Move five, put the tools in everyone's hands.

s100
00:11:46.218 --> 00:11:59.658
I was in a AI workshop with 30 or so other executives, and we were asked to think how we might redesign our business with AI.

s101
00:11:59.658 --> 00:12:09.138
But three out of four people in that room had never used any AI tool until that week.

s102
00:12:09.138 --> 00:12:16.219
It's impossible to redesign a workflow with AI if you've never used AI.

s103
00:12:16.219 --> 00:12:27.939
The intuition on where value lies and which use cases can be enabled with AI only comes if you're using the tools.

s104
00:12:28.339 --> 00:12:37.459
Experimentation with the tools is the absolute minimum requirement to drive this type of transformation.

s105
00:12:38.379 --> 00:12:45.899
And move number six, set the kill metrics first and let the CFO keep keep score.

s106
00:12:45.899 --> 00:12:58.579
Before we start a pilot, we write down what are the kill metrics that will determine if a pilot continues or stops.

s107
00:12:58.579 --> 00:13:03.899
Three types of gates that we used in this initiative.

s108
00:13:03.899 --> 00:13:06.099
The first is a time gate.

s109
00:13:06.099 --> 00:13:14.339
We gave each stage 90 days and if we didn't see value in those 90 days, we would stop the initiative.

s110
00:13:14.339 --> 00:13:16.819
The second is a metric gate.

s111
00:13:16.819 --> 00:13:23.459
AB testing at each stage relative to a baseline that we decided on in advance.

s112
00:13:23.459 --> 00:13:25.779
And the third is an adoption gate.

s113
00:13:25.779 --> 00:13:31.499
Were the crew schedulers using this tool at over 50%?

s114
00:13:31.499 --> 00:13:34.579
Over 50% of crew schedulers.

s115
00:13:34.579 --> 00:13:50.579
The other important test that we put in place, working hand-in-hand with the CFO, was calculating whether the cost of this initiative at scale, including compute and token and the individuals

s116
00:13:50.579 --> 00:13:59.459
involved with this work, was significantly lower than the impact that the initiative would generate.

s117
00:13:59.459 --> 00:14:05.699
Many pilots die at this stage because they don't pass this test.

s118
00:14:06.779 --> 00:14:12.259
So, let's recap the six moves that I just described.

s119
00:14:12.259 --> 00:14:16.019
First, you want to be thoughtful about choosing your problem.

s120
00:14:16.019 --> 00:14:29.941
You want to pick one or two things to tackle, not 40, and you want to reimagine the entire process end-to-end, rather than building on processes that may already be obsolete.

s121
00:14:29.941 --> 00:14:38.301
Second, you want to choose the owners, starting with the CEO and building with the business.

s122
00:14:38.301 --> 00:14:41.221
Third, you want to make it stick.

s123
00:14:41.221 --> 00:14:47.421
Make sure that relevant stakeholders are experimenting with the tools.

s124
00:14:47.421 --> 00:14:58.661
Set kill metrics in advance before starting a pilot and work with the CFO on the unit economic scalability of your projects.

s125
00:14:59.981 --> 00:15:21.541
While the tools are available to everybody, models are available to all of us, the ability to absorb these models is a competitive differentiator, and the businesses that are able to do that and truly take advantage

s126
00:15:21.541 --> 00:15:28.301
of the models as they continue getting better and better, are going to thrive.

s127
00:15:29.741 --> 00:15:42.221
So, before it took 15 minutes to find one answer, now it takes less than a minute to find the answer.

s128
00:15:42.221 --> 00:15:44.520
Thank you.

s129
00:15:44.520 --> 00:15:46.520
[applause]