WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=WJRdLNhrsLQ

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/WJRdLNhrsLQ.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.440 --> 00:00:13.280
It's time.

s3
00:00:13.280 --> 00:00:14.640
We can get it started.

s4
00:00:14.640 --> 00:00:15.520
I'm Dan.

s5
00:00:15.520 --> 00:00:17.240
I'm from Maven Clinic.

s6
00:00:17.240 --> 00:00:24.800
Today we'll share the experience how we transition from a traditional technology company to AI native company.

s7
00:00:24.800 --> 00:00:28.040
Before I started, I would like to do a little exercise.

s8
00:00:28.040 --> 00:00:32.680
Raise your hand if you think you are already an AI native company.

s9
00:00:32.680 --> 00:00:34.200
Okay, we saw a few.

s10
00:00:34.200 --> 00:00:39.640
Raise your hand if you thought about it, but haven't started the journey yet.

s11
00:00:39.920 --> 00:00:40.840
Okay, we saw a few.

s12
00:00:40.840 --> 00:00:43.720
That means most of us is in between.

s13
00:00:43.720 --> 00:00:47.760
Hopefully this talk can help you with that one.

s14
00:00:47.760 --> 00:00:52.520
So, Maven Clinic is the largest digital health platform.

s15
00:00:52.520 --> 00:00:55.920
We are focused on women and their families.

s16
00:00:55.920 --> 00:01:01.840
So, we specialize in like maternity, fertility, parenting, menopause.

s17
00:01:01.840 --> 00:01:05.280
We started our AI journey just 2 years back.

s18
00:01:05.280 --> 00:01:08.720
And this moment we built something called a Maven Intelligence.

s19
00:01:08.720 --> 00:01:16.920
It's a orchestration layer across all our product to enable AI for everybody in this company and for our clients.

s20
00:01:17.120 --> 00:01:21.000
So, AI is here and improving every day.

s21
00:01:21.000 --> 00:01:23.440
I think adopting it is not an optional.

s22
00:01:23.440 --> 00:01:26.640
Even you choose not to, your competitors will do.

s23
00:01:26.640 --> 00:01:29.560
This is a quote I heard like a couple years back.

s24
00:01:29.560 --> 00:01:31.880
I would like to share here again.

s25
00:01:31.880 --> 00:01:39.680
Like a tractors aren't to replace farmers, but the farmers who can operate the tractor will replace the ones who cannot.

s26
00:01:39.680 --> 00:01:44.280
Hopefully everybody here will become farmers who can operate your tractors.

s27
00:01:44.280 --> 00:01:46.480
That's the goal.

s28
00:01:46.480 --> 00:01:52.480
So, first of all, I don't think there's a one single definition what it means by AI native.

s29
00:01:52.480 --> 00:01:59.840
And more importantly, there's no predefined playbook you can just follow and bingo, you become AI native.

s30
00:01:59.840 --> 00:02:03.120
For us, it is really come down to three parts.

s31
00:02:03.120 --> 00:02:07.720
One is internally with our AI tools, whenever it's possible.

s32
00:02:07.720 --> 00:02:19.840
It can be as simple as like generating your daily summary, managing your meeting, create a Jira task, anything you need to do today manually, you should think about to say can use AI to do it.

s33
00:02:19.840 --> 00:02:26.480
Whenever you want to ask other people to do something for you, you should be saying with yourself I can use AI to do it.

s34
00:02:26.480 --> 00:02:36.520
A lot of leaders in fact today at the Maven include our sales sale, they use AI tool to solve those task by themselves now instead delegate to other people.

s35
00:02:36.520 --> 00:02:37.959
That's for internally.

s36
00:02:37.959 --> 00:02:43.880
Externally, we want to build AIs into our product which achieve two goals there.

s37
00:02:43.880 --> 00:02:47.320
One is really focused on improve our user experience.

s38
00:02:47.320 --> 00:02:51.600
Second maybe help us reduce our operational cost.

s39
00:02:51.600 --> 00:02:55.959
Like a like AI based like chatbot is really good example.

s40
00:02:55.959 --> 00:03:03.360
It's 24/7, always available, can help our address issues, help our customer instantly.

s41
00:03:03.360 --> 00:03:07.200
It's way better and cheaper compared to human agents.

s42
00:03:07.200 --> 00:03:18.760
Thirdly, I think is more important, we need to think about our culture, process, the way we work, how we can change it so we can maximize what AI offers for us.

s43
00:03:18.760 --> 00:03:22.320
I will touch it more on the following slides.

s44
00:03:22.840 --> 00:03:29.519
So, when we come to adopting new technologies, there's always like screw three groups of users.

s45
00:03:29.519 --> 00:03:32.720
One there's some early adopters, right?

s46
00:03:32.720 --> 00:03:34.760
For them, we don't need to do too much.

s47
00:03:34.760 --> 00:03:43.160
Only thing we need to do is enable the tools for them and encourage them to share what they learn with the company.

s48
00:03:43.160 --> 00:03:46.519
And what we need to really focus on is the one in the middle.

s49
00:03:46.519 --> 00:03:48.000
That's a majority.

s50
00:03:48.000 --> 00:03:53.720
We should build a shared AI infrastructure for them, build easy to use tools for them.

s51
00:03:53.720 --> 00:03:57.720
Just make the adoption as seamlessly as possible.

s52
00:03:57.720 --> 00:04:03.240
More importantly, we should really listen to them, get feedback, consistently inputting.

s53
00:04:03.240 --> 00:04:08.040
For example, last year, most of folks and Maven they are using cursor.

s54
00:04:08.040 --> 00:04:11.160
This year, a lot of them switch to cloud codes.

s55
00:04:11.160 --> 00:04:13.280
For us, we need to support the both.

s56
00:04:13.280 --> 00:04:15.000
We need to meet where they are.

s57
00:04:15.000 --> 00:04:17.760
Just make feel they comfortable to use it.

s58
00:04:17.760 --> 00:04:22.200
And for an older places, you always have a few slow adopters.

s59
00:04:22.200 --> 00:04:26.000
They always have concerns, worries for the new technologies.

s60
00:04:26.000 --> 00:04:28.919
For them, we just should have meet where they are.

s61
00:04:28.919 --> 00:04:31.000
Understand what's their concern is.

s62
00:04:31.000 --> 00:04:37.720
But more important, we should be crystal clear with them where the company is heading to.

s63
00:04:38.919 --> 00:04:45.400
So, AI is really good and execution if we know what we want to do.

s64
00:04:45.400 --> 00:04:50.800
So, this is change how we should hire new people and how should we reward them.

s65
00:04:50.800 --> 00:05:01.480
We used to the way we used to work is we have a senior engineer who sense the problem, come up with solution, and delegate to other engineers for implementation.

s66
00:05:01.480 --> 00:05:05.720
So, we can work on it in parallel and be faster.

s67
00:05:05.720 --> 00:05:16.080
But these days, we found the NASA and NASA engineers wouldn't like to delegate the implementation work to other people because they they already figure out how to solve the problem.

s68
00:05:16.080 --> 00:05:19.480
They just use AI to solve it instantly.

s69
00:05:19.480 --> 00:05:24.320
Delegating to other people means more overheads and less efficient.

s70
00:05:24.320 --> 00:05:31.040
And also means like when you have a new people, you want to make sure they can solve the problem independently.

s71
00:05:31.040 --> 00:05:35.280
They pretty much has to work on the traditional technical lead memo.

s72
00:05:35.280 --> 00:05:40.720
We can We cannot afford other people delegate implementation task for them.

s73
00:05:40.720 --> 00:05:44.680
Also, when we hire a new people, we should think about what we are looking for.

s74
00:05:44.680 --> 00:05:48.440
We definitely want to look for somebody genuinely interested in AI.

s75
00:05:48.440 --> 00:05:50.480
The domain is moving so fast.

s76
00:05:50.480 --> 00:05:52.160
We want them to keep learning.

s77
00:05:52.160 --> 00:05:55.400
Also, help the team to stay on track.

s78
00:05:55.400 --> 00:06:00.920
Secondly, is um with AI, engineers can do way more than they used to do.

s79
00:06:00.920 --> 00:06:06.000
The boundaries between PM and engineers is getting blur blurry.

s80
00:06:06.000 --> 00:06:10.240
We found like a engineers who really understand the product.

s81
00:06:10.240 --> 00:06:17.880
In fact, they can have more way more contribution than a traditional engineer who only focus on software side.

s82
00:06:17.880 --> 00:06:19.680
And this is what we are looking for.

s83
00:06:19.680 --> 00:06:28.360
And those deep understanding of the system, the ability to can handle complicated ambiguous problem is also very valuable.

s84
00:06:28.360 --> 00:06:30.360
This is where AI land off.

s85
00:06:30.360 --> 00:06:35.280
When we hire new people, this is also the people we are interested in bring on board.

s86
00:06:35.280 --> 00:06:40.240
For people we bring in, we want to reward them in the proper way.

s87
00:06:40.240 --> 00:06:45.680
Even in our performance with review, we start ask, "Okay, what do you have done for AI side?"

s88
00:06:45.680 --> 00:06:51.160
We definitely want to reward the people who leverage AI to multiple their impact.

s89
00:06:51.160 --> 00:06:55.480
Although this is impactful everybody in the company.

s90
00:06:56.600 --> 00:06:58.480
So, now we have the right tools.

s91
00:06:58.480 --> 00:07:00.760
We get the right talent in the place.

s92
00:07:00.760 --> 00:07:06.640
And we need to change how we work to maximize the benefit of AI.

s93
00:07:06.640 --> 00:07:18.640
The we used the way we used to work is say, "Okay, we spend the weeks, sometimes it's the months to flash out the business requirements, finalize the design, and then

s94
00:07:18.640 --> 00:07:20.000
do the implementation."

s95
00:07:20.000 --> 00:07:23.480
Because implementation can be really expensive.

s96
00:07:23.480 --> 00:07:30.360
If we didn't get the other part right in the beginning, it's it can be very costly to change it later.

s97
00:07:30.360 --> 00:07:37.280
But in fact, we never get the sense and right in the beginning anyway for any of big projects.

s98
00:07:37.280 --> 00:07:39.680
With AI, building is super fast.

s99
00:07:39.680 --> 00:07:43.080
It's probably couple minutes you can get it done.

s100
00:07:43.080 --> 00:07:45.920
Argument is really expensive one.

s101
00:07:45.920 --> 00:07:51.400
So, we should really think about what's the best we can work, how can we deliver fast.

s102
00:07:51.400 --> 00:07:55.960
It's still okay, you can think about what you want to deliver in one year.

s103
00:07:55.960 --> 00:08:00.560
You can assume AI models can do anything you want in one year.

s104
00:08:00.560 --> 00:08:11.160
Based on that one, really dream big to think what you can do in one year, but it should only serve as inspiration, inspiring, and directional.

s105
00:08:11.160 --> 00:08:17.400
What we really need to focus on is what we want to deliver in the next two to four weeks, right?

s106
00:08:17.400 --> 00:08:23.040
What we want to get to the PMs and designers is say, "Okay, tell me what I need to deliver in this sprint."

s107
00:08:23.040 --> 00:08:29.840
And the engineer will focus on it and get it released if at the end of the sprint, if not sooner.

s108
00:08:29.840 --> 00:08:35.760
Meanwhile, and the PMs, they have time to flesh out the next bunch of the requirements.

s109
00:08:35.760 --> 00:08:40.840
If at the end of this sprint, they say, "No, what we decided two weeks ago is wrong."

s110
00:08:40.840 --> 00:08:41.800
It's totally okay.

s111
00:08:41.800 --> 00:08:45.680
We can switch the gear, get it fixed quickly.

s112
00:08:45.680 --> 00:08:52.440
That also means like we prefer people not write pages or pages of PRD or TDD anymore.

s113
00:08:52.440 --> 00:08:56.560
We prefer them to write just a short one or two pages.

s114
00:08:56.560 --> 00:09:01.440
That one is really serve as communication, so we can iterate on it.

s115
00:09:01.440 --> 00:09:05.240
The really awkward part is mid-term goals.

s116
00:09:05.240 --> 00:09:07.640
Those like a three months, six months.

s117
00:09:07.640 --> 00:09:09.960
It's very hard to plan these days.

s118
00:09:09.960 --> 00:09:14.240
The reason is I don't know what AI models will be capable in three months.

s119
00:09:14.240 --> 00:09:16.640
There may be multiple releases already.

s120
00:09:16.640 --> 00:09:19.320
So, we prefer not focus on this one.

s121
00:09:19.320 --> 00:09:30.440
But this one can maybe make it not very easy for most of folks who has been in this domain for a long time because traditionally we get used to have a quarterly planning

s122
00:09:30.440 --> 00:09:32.839
or we plan it for six months.

s123
00:09:32.839 --> 00:09:41.160
But it's our job to get used to the new AI era and learn how to work it efficiently.

s124
00:09:42.040 --> 00:09:46.720
So, I want to talk about the coding and software development a little bit more here.

s125
00:09:46.720 --> 00:09:52.120
AI coding tools is probably the most successful AI application.

s126
00:09:52.120 --> 00:09:55.880
And it's really good and implementation.

s127
00:09:55.880 --> 00:10:00.760
So, you probably heard a lot of people say, "Okay, I have this AI tools.

s128
00:10:00.760 --> 00:10:06.240
Now I can even use my phone to implement software and automate every stage."

s129
00:10:06.240 --> 00:10:09.080
If they feel comfortable do that, it's totally okay.

s130
00:10:09.080 --> 00:10:10.800
But you don't have to.

s131
00:10:10.800 --> 00:10:16.840
What I'm trying to say here is And maybe this is our journey, how we adopt those AI tools.

s132
00:10:16.840 --> 00:10:23.080
We started with the lowest risk task, like starting with writing unit tests, documentation.

s133
00:10:23.080 --> 00:10:27.600
Those things are very easy to verify and the risk is super low.

s134
00:10:27.600 --> 00:10:35.720
By doing that, one we build the confidence and we start to construct our own rules, skills, and build our barriers.

s135
00:10:35.720 --> 00:10:42.520
And then we push to the whole engineering team say, "Now you should use this AI coding tools for all the tasks."

s136
00:10:42.520 --> 00:10:47.200
When they choose not to do, it's the time we really want to learn say, "Why you don't do it?"

s137
00:10:47.200 --> 00:10:53.040
And at this moment, we pretty much use the AI coding tools to do all our implementation.

s138
00:10:53.040 --> 00:10:59.200
Engineers really focus on reviewing, architecturing, and evaluation.

s139
00:10:59.592 --> 00:10:59.880
[snorts]

s140
00:10:59.880 --> 00:11:05.320
So, and with AI coding tools, we are writing so much code these days.

s141
00:11:05.320 --> 00:11:08.440
Code review becomes really challenging.

s142
00:11:08.440 --> 00:11:13.600
So, for good engineer, used to they probably write hundreds of lines code every day.

s143
00:11:13.600 --> 00:11:16.800
These days, they can easily write like thousands.

s144
00:11:16.800 --> 00:11:22.400
If we keep do the code review as we used to do, we won't be able to keep up.

s145
00:11:22.400 --> 00:11:27.280
We also tried the multiple like AI coding review review tools.

s146
00:11:27.280 --> 00:11:32.120
It helps a little bit, but we don't feel comfortable 100% rely on them yet.

s147
00:11:32.120 --> 00:11:36.520
We still found those feedbacks from our engineers are very, very valuable.

s148
00:11:36.520 --> 00:11:43.040
And that means we need to really change the way we are doing code review to meet where we are now.

s149
00:11:43.040 --> 00:11:45.120
And couple things we have done.

s150
00:11:45.120 --> 00:11:51.120
One is we allow engineers to self-identify whether they still need code review.

s151
00:11:51.120 --> 00:11:56.680
If they think this PR is simple enough, I feel very confident, I don't need anybody to take a look.

s152
00:11:56.680 --> 00:11:58.320
And we are fine with that one.

s153
00:11:58.320 --> 00:12:01.720
We let them merge, but we still hold them accountable.

s154
00:12:01.720 --> 00:12:06.840
And if they do want code review, we want them stay with the best practices.

s155
00:12:06.840 --> 00:12:17.800
For example, each PR shouldn't have more than 500 lines of code because nobody can do a meaningful code review with the ones has like thousands of lines code.

s156
00:12:17.800 --> 00:12:21.200
And we also enable the like stack the PR.

s157
00:12:21.200 --> 00:12:31.680
What it means is that for big feature, and the engineers can bring break it into multiple PRs where people review the PRs and they can keep working on it.

s158
00:12:31.680 --> 00:12:36.440
One thing we really want to avoid is a rubber stamp, we call it.

s159
00:12:36.440 --> 00:12:40.320
Means like people submit code review, you cannot really do anything to it.

s160
00:12:40.320 --> 00:12:42.240
You just say blindly approve it.

s161
00:12:42.240 --> 00:12:47.400
This is the worst case, we should really avoid because that's just give us false confidence.

s162
00:12:47.400 --> 00:12:51.560
We think we reviewed it, it's good, and we release it.

s163
00:12:51.560 --> 00:12:58.240
Meanwhile, we should keep working on our AI code review tools because we are thinking that's the future.

s164
00:12:58.240 --> 00:13:05.680
So, at this moment, we use AI tools pretty much assist each step of our software development.

s165
00:13:05.680 --> 00:13:15.800
Our goal is it will be automate the whole life cycle from end to end, from designing, implementation, and here is a fully release it.

s166
00:13:15.800 --> 00:13:24.880
More important, we want the AI tools be able to monitor the live traffic and be able to catch the issue early and automatically fix it.

s167
00:13:24.880 --> 00:13:29.520
That's what we are still working on, and we're not there yet.

s168
00:13:29.640 --> 00:13:35.640
The last thing I want to touch a little bit for this presentation is about reliability.

s169
00:13:35.640 --> 00:13:43.440
So, what it means is like for the traditional software, it does what we implement there, no more, no less.

s170
00:13:43.440 --> 00:13:48.080
But for the Genex solutions, hallucination is there.

s171
00:13:48.080 --> 00:13:49.800
We cannot ignore it.

s172
00:13:49.800 --> 00:13:54.600
And there completely eliminating them is can be very costly.

s173
00:13:54.600 --> 00:13:57.800
Sometimes is not necessary, either.

s174
00:13:57.800 --> 00:14:03.320
So, the way we should do is really have a holistic solution even from the get beginning.

s175
00:14:03.320 --> 00:14:11.760
For example, we can start with identify which failures is acceptable, which ones are not acceptable.

s176
00:14:11.760 --> 00:14:18.480
For our AI system, for example, we have the functionality to help our customers to schedule appointment.

s177
00:14:18.480 --> 00:14:21.800
If we fail well to 1,000, probably it's okay.

s178
00:14:21.800 --> 00:14:27.960
I'm not saying it's a good experience, but the users really can just click the button again, we will reschedule for them.

s179
00:14:27.960 --> 00:14:29.120
Probably it's okay.

s180
00:14:29.120 --> 00:14:42.960
But if we help user to like submit their reimbursement claim, we cannot tolerate a failure because if people ask of $200, we issue them 50 or they ask 50, we give them 200.

s181
00:14:42.960 --> 00:14:46.760
Each case will cause a escalation right away.

s182
00:14:46.760 --> 00:14:49.839
For those cases, we have to put in extra stamps.

s183
00:14:49.839 --> 00:14:56.600
For example, when we receive their receipt, we will use different models to review the same receipt.

s184
00:14:56.600 --> 00:15:01.880
We only move forward if the results from different models agree with each other.

s185
00:15:01.880 --> 00:15:10.520
If we really have trouble to figure it out which one is right, it's it's easy it's okay to tell the customer, say, "Hey, we have trouble to process your stuff.

s186
00:15:10.520 --> 00:15:13.839
Do you want us to get you connect to a human agent?"

s187
00:15:13.839 --> 00:15:15.080
We will move from there.

s188
00:15:15.080 --> 00:15:17.240
That's acceptable solutions.

s189
00:15:17.240 --> 00:15:21.920
And also we have should have a rigorous process to release our software.

s190
00:15:21.920 --> 00:15:32.560
For us, we have like hundreds of integration tests, for which pretty much covered all the use cases we know, and we are keep adding to the integration test the suite.

s191
00:15:32.560 --> 00:15:40.920
And the when we run the integration test, not only pass once is not good enough anymore, because the LLM can do different things.

s192
00:15:40.920 --> 00:15:44.160
So, for each test case, we run it to many times.

s193
00:15:44.160 --> 00:15:50.880
We consistently requires the high pass rate, like for example, 90% for all the time.

s194
00:15:50.880 --> 00:15:52.380
And the more important,

s195
00:15:52.380 --> 00:15:52.720
[snorts]

s196
00:15:52.720 --> 00:16:01.320
and the after we launch the software, we have our auto evolve system evaluate carefully evaluate each conversation.

s197
00:16:01.320 --> 00:16:06.440
We have predefined a lot of rubrics, what we think is good, what is bad.

s198
00:16:06.440 --> 00:16:10.680
And then we will generate results, we will review the score.

s199
00:16:10.680 --> 00:16:18.240
Besides this one, we also have dedicated a group, their job is mainly review those conversations.

s200
00:16:18.240 --> 00:16:30.320
We will spot a check our conversations, that helps us to say whether we need to come back to improve our systems, or our rubrics is too strict or too loose, and we need consistently

s201
00:16:30.320 --> 00:16:31.480
improve it.

s202
00:16:31.480 --> 00:16:41.120
When we launch new features, then the time we say not only spot a check probably not enough, we really want to review like say 20%, and we can do it.

s203
00:16:41.120 --> 00:16:50.280
This whole process make [clears throat] sure we we are really confident whenever we release something, although we know hallucination is there.

s204
00:16:50.280 --> 00:16:58.839
And that's pretty much what I have for today, and I can stay here to take up questions, and if you have other things, you can reach out to me.

s205
00:16:59.191 --> 00:17:01.191
[applause]