How to build an AI-Native Health Company — Dan Feng, Maven Clinic

AI Engineer · 17 min · 205 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12It's time.
00:13We can get it started.
00:14I'm Dan.
00:15I'm from Maven Clinic.
00:17Today we'll share the experience how we transition from a traditional technology company to AI native company.
00:24Before I started, I would like to do a little exercise.
00:28Raise your hand if you think you are already an AI native company.
00:32Okay, we saw a few.
00:34Raise your hand if you thought about it, but haven't started the journey yet.
00:39Okay, we saw a few.
00:40That means most of us is in between.
00:43Hopefully this talk can help you with that one.
00:47So, Maven Clinic is the largest digital health platform.
00:52We are focused on women and their families.
00:55So, we specialize in like maternity, fertility, parenting, menopause.
01:01We started our AI journey just 2 years back.
01:05And this moment we built something called a Maven Intelligence.
01:08It's a orchestration layer across all our product to enable AI for everybody in this company and for our clients.
01:17So, AI is here and improving every day.
01:21I think adopting it is not an optional.
01:23Even you choose not to, your competitors will do.
01:26This is a quote I heard like a couple years back.
01:29I would like to share here again.
01:31Like a tractors aren't to replace farmers, but the farmers who can operate the tractor will replace the ones who cannot.
01:39Hopefully everybody here will become farmers who can operate your tractors.
01:44That's the goal.
01:46So, first of all, I don't think there's a one single definition what it means by AI native.
01:52And more importantly, there's no predefined playbook you can just follow and bingo, you become AI native.
01:59For us, it is really come down to three parts.
02:03One is internally with our AI tools, whenever it's possible.
02:07It can be as simple as like generating your daily summary, managing your meeting, create a Jira task, anything you need to do today manually, you should think about to say can use AI to do it.
02:19Whenever you want to ask other people to do something for you, you should be saying with yourself I can use AI to do it.
02:26A lot of leaders in fact today at the Maven include our sales sale, they use AI tool to solve those task by themselves now instead delegate to other people.
02:36That's for internally.
02:37Externally, we want to build AIs into our product which achieve two goals there.
02:43One is really focused on improve our user experience.
02:47Second maybe help us reduce our operational cost.
02:51Like a like AI based like chatbot is really good example.
02:55It's 24/7, always available, can help our address issues, help our customer instantly.
03:03It's way better and cheaper compared to human agents.
03:07Thirdly, I think is more important, we need to think about our culture, process, the way we work, how we can change it so we can maximize what AI offers for us.
03:18I will touch it more on the following slides.
03:22So, when we come to adopting new technologies, there's always like screw three groups of users.
03:29One there's some early adopters, right?
03:32For them, we don't need to do too much.
03:34Only thing we need to do is enable the tools for them and encourage them to share what they learn with the company.
03:43And what we need to really focus on is the one in the middle.
03:46That's a majority.
03:48We should build a shared AI infrastructure for them, build easy to use tools for them.
03:53Just make the adoption as seamlessly as possible.
03:57More importantly, we should really listen to them, get feedback, consistently inputting.
04:03For example, last year, most of folks and Maven they are using cursor.
04:08This year, a lot of them switch to cloud codes.
04:11For us, we need to support the both.
04:13We need to meet where they are.
04:15Just make feel they comfortable to use it.
04:17And for an older places, you always have a few slow adopters.
04:22They always have concerns, worries for the new technologies.
04:26For them, we just should have meet where they are.
04:28Understand what's their concern is.
04:31But more important, we should be crystal clear with them where the company is heading to.
04:38So, AI is really good and execution if we know what we want to do.
04:45So, this is change how we should hire new people and how should we reward them.
04:50We used to the way we used to work is we have a senior engineer who sense the problem, come up with solution, and delegate to other engineers for implementation.
05:01So, we can work on it in parallel and be faster.
05:05But these days, we found the NASA and NASA engineers wouldn't like to delegate the implementation work to other people because they they already figure out how to solve the problem.
05:16They just use AI to solve it instantly.
05:19Delegating to other people means more overheads and less efficient.
05:24And also means like when you have a new people, you want to make sure they can solve the problem independently.
05:31They pretty much has to work on the traditional technical lead memo.
05:35We can We cannot afford other people delegate implementation task for them.
05:40Also, when we hire a new people, we should think about what we are looking for.
05:44We definitely want to look for somebody genuinely interested in AI.
05:48The domain is moving so fast.
05:50We want them to keep learning.
05:52Also, help the team to stay on track.
05:55Secondly, is um with AI, engineers can do way more than they used to do.
06:00The boundaries between PM and engineers is getting blur blurry.
06:06We found like a engineers who really understand the product.
06:10In fact, they can have more way more contribution than a traditional engineer who only focus on software side.
06:17And this is what we are looking for.
06:19And those deep understanding of the system, the ability to can handle complicated ambiguous problem is also very valuable.
06:28This is where AI land off.
06:30When we hire new people, this is also the people we are interested in bring on board.
06:35For people we bring in, we want to reward them in the proper way.
06:40Even in our performance with review, we start ask, "Okay, what do you have done for AI side?"
06:45We definitely want to reward the people who leverage AI to multiple their impact.
06:51Although this is impactful everybody in the company.
06:56So, now we have the right tools.
06:58We get the right talent in the place.
07:00And we need to change how we work to maximize the benefit of AI.
07:06The we used the way we used to work is say, "Okay, we spend the weeks, sometimes it's the months to flash out the business requirements, finalize the design, and then
07:18do the implementation."
07:20Because implementation can be really expensive.
07:23If we didn't get the other part right in the beginning, it's it can be very costly to change it later.
07:30But in fact, we never get the sense and right in the beginning anyway for any of big projects.
07:37With AI, building is super fast.
07:39It's probably couple minutes you can get it done.
07:43Argument is really expensive one.
07:45So, we should really think about what's the best we can work, how can we deliver fast.
07:51It's still okay, you can think about what you want to deliver in one year.
07:55You can assume AI models can do anything you want in one year.
08:00Based on that one, really dream big to think what you can do in one year, but it should only serve as inspiration, inspiring, and directional.
08:11What we really need to focus on is what we want to deliver in the next two to four weeks, right?
08:17What we want to get to the PMs and designers is say, "Okay, tell me what I need to deliver in this sprint."
08:23And the engineer will focus on it and get it released if at the end of the sprint, if not sooner.
08:29Meanwhile, and the PMs, they have time to flesh out the next bunch of the requirements.
08:35If at the end of this sprint, they say, "No, what we decided two weeks ago is wrong."
08:40It's totally okay.
08:41We can switch the gear, get it fixed quickly.
08:45That also means like we prefer people not write pages or pages of PRD or TDD anymore.
08:52We prefer them to write just a short one or two pages.
08:56That one is really serve as communication, so we can iterate on it.
09:01The really awkward part is mid-term goals.
09:05Those like a three months, six months.
09:07It's very hard to plan these days.
09:09The reason is I don't know what AI models will be capable in three months.
09:14There may be multiple releases already.
09:16So, we prefer not focus on this one.
09:19But this one can maybe make it not very easy for most of folks who has been in this domain for a long time because traditionally we get used to have a quarterly planning
09:30or we plan it for six months.
09:32But it's our job to get used to the new AI era and learn how to work it efficiently.
09:42So, I want to talk about the coding and software development a little bit more here.
09:46AI coding tools is probably the most successful AI application.
09:52And it's really good and implementation.
09:55So, you probably heard a lot of people say, "Okay, I have this AI tools.
10:00Now I can even use my phone to implement software and automate every stage."
10:06If they feel comfortable do that, it's totally okay.
10:09But you don't have to.
10:10What I'm trying to say here is And maybe this is our journey, how we adopt those AI tools.
10:16We started with the lowest risk task, like starting with writing unit tests, documentation.
10:23Those things are very easy to verify and the risk is super low.
10:27By doing that, one we build the confidence and we start to construct our own rules, skills, and build our barriers.
10:35And then we push to the whole engineering team say, "Now you should use this AI coding tools for all the tasks."
10:42When they choose not to do, it's the time we really want to learn say, "Why you don't do it?"
10:47And at this moment, we pretty much use the AI coding tools to do all our implementation.
10:53Engineers really focus on reviewing, architecturing, and evaluation.
10:59[snorts]
10:59So, and with AI coding tools, we are writing so much code these days.
11:05Code review becomes really challenging.
11:08So, for good engineer, used to they probably write hundreds of lines code every day.
11:13These days, they can easily write like thousands.
11:16If we keep do the code review as we used to do, we won't be able to keep up.
11:22We also tried the multiple like AI coding review review tools.
11:27It helps a little bit, but we don't feel comfortable 100% rely on them yet.
11:32We still found those feedbacks from our engineers are very, very valuable.
11:36And that means we need to really change the way we are doing code review to meet where we are now.
11:43And couple things we have done.
11:45One is we allow engineers to self-identify whether they still need code review.
11:51If they think this PR is simple enough, I feel very confident, I don't need anybody to take a look.
11:56And we are fine with that one.
11:58We let them merge, but we still hold them accountable.
12:01And if they do want code review, we want them stay with the best practices.
12:06For example, each PR shouldn't have more than 500 lines of code because nobody can do a meaningful code review with the ones has like thousands of lines code.
12:17And we also enable the like stack the PR.
12:21What it means is that for big feature, and the engineers can bring break it into multiple PRs where people review the PRs and they can keep working on it.
12:31One thing we really want to avoid is a rubber stamp, we call it.
12:36Means like people submit code review, you cannot really do anything to it.
12:40You just say blindly approve it.
12:42This is the worst case, we should really avoid because that's just give us false confidence.
12:47We think we reviewed it, it's good, and we release it.
12:51Meanwhile, we should keep working on our AI code review tools because we are thinking that's the future.
12:58So, at this moment, we use AI tools pretty much assist each step of our software development.
13:05Our goal is it will be automate the whole life cycle from end to end, from designing, implementation, and here is a fully release it.
13:15More important, we want the AI tools be able to monitor the live traffic and be able to catch the issue early and automatically fix it.
13:24That's what we are still working on, and we're not there yet.
13:29The last thing I want to touch a little bit for this presentation is about reliability.
13:35So, what it means is like for the traditional software, it does what we implement there, no more, no less.
13:43But for the Genex solutions, hallucination is there.
13:48We cannot ignore it.
13:49And there completely eliminating them is can be very costly.
13:54Sometimes is not necessary, either.
13:57So, the way we should do is really have a holistic solution even from the get beginning.
14:03For example, we can start with identify which failures is acceptable, which ones are not acceptable.
14:11For our AI system, for example, we have the functionality to help our customers to schedule appointment.
14:18If we fail well to 1,000, probably it's okay.
14:21I'm not saying it's a good experience, but the users really can just click the button again, we will reschedule for them.
14:27Probably it's okay.
14:29But if we help user to like submit their reimbursement claim, we cannot tolerate a failure because if people ask of $200, we issue them 50 or they ask 50, we give them 200.
14:42Each case will cause a escalation right away.
14:46For those cases, we have to put in extra stamps.
14:49For example, when we receive their receipt, we will use different models to review the same receipt.
14:56We only move forward if the results from different models agree with each other.
15:01If we really have trouble to figure it out which one is right, it's it's easy it's okay to tell the customer, say, "Hey, we have trouble to process your stuff.
15:10Do you want us to get you connect to a human agent?"
15:13We will move from there.
15:15That's acceptable solutions.
15:17And also we have should have a rigorous process to release our software.
15:21For us, we have like hundreds of integration tests, for which pretty much covered all the use cases we know, and we are keep adding to the integration test the suite.
15:32And the when we run the integration test, not only pass once is not good enough anymore, because the LLM can do different things.
15:40So, for each test case, we run it to many times.
15:44We consistently requires the high pass rate, like for example, 90% for all the time.
15:50And the more important,
15:52[snorts]
15:52and the after we launch the software, we have our auto evolve system evaluate carefully evaluate each conversation.
16:01We have predefined a lot of rubrics, what we think is good, what is bad.
16:06And then we will generate results, we will review the score.
16:10Besides this one, we also have dedicated a group, their job is mainly review those conversations.
16:18We will spot a check our conversations, that helps us to say whether we need to come back to improve our systems, or our rubrics is too strict or too loose, and we need consistently
16:30improve it.
16:31When we launch new features, then the time we say not only spot a check probably not enough, we really want to review like say 20%, and we can do it.
16:41This whole process make [clears throat] sure we we are really confident whenever we release something, although we know hallucination is there.
16:50And that's pretty much what I have for today, and I can stay here to take up questions, and if you have other things, you can reach out to me.
16:59[applause]