Which AI startups actually land enterprise contracts? — Brian Lewis, Millennium

AI Engineer · 18 min · 249 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Welcome everybody.
00:14Sorry for everybody who was already here and missed the Coinbase guy.
00:17I have no idea where he went or why he didn't come.
00:20I was actually pretty excited to hear about his his comments.
00:23But today I'm going to talk about which AI startups actually win enterprise contracts.
00:28So to begin I thought this was going to be a different audience.
00:32I didn't realize this was going to be mostly people on the leadership track.
00:35I thought I was going to be speaking to more AI engineers.
00:38So maybe just by show of hands, how many of you represent like the engineering or startup or like seller side?
00:45And then how many of you represent maybe like the buyer side?
00:47Like you're in the enterprise, you're trying to get these tools in.
00:50Okay, so we got a good mix.
00:51I'm going to try to balance that out today.
00:53I work on product stuff at Millennium, which is a hedge fund.
00:57We build a lot of stuff.
00:58I can't talk about any of it.
01:00So I'm going to talk about stuff that we we look at and evaluate.
01:04It's going to be pretty generic.
01:05I tried to make it as interesting as possible while still getting my compliance department to be okay with me doing this.
01:11But also need to say legally that I'm speaking as an individual.
01:15I am not representing my company and all opinions are my own.
01:18So with that we can dive in.
01:21So I ran through this with my parents last week.
01:25I don't I grew up not that far from here.
01:27And my mom basically said, "Why are you spending your time teaching vendors how to sell to you?
01:31Aren't you busy enough already?"
01:33And the real answer to that question is I really like how stuff works.
01:37I like seeing stuff come together.
01:39My bachelor's degree was in economics.
01:42And I really love seeing things just work well.
01:44So AI's been really interesting because it's kind of a whole new paradigm of how businesses are doing work.
01:51That's the whole point of this track, this AI native enterprise track.
01:54And so even though I don't need more people DMing me on LinkedIn, um I'm actually really excited to talk about this.
02:02So, my hypothesis in short is basically at current model intelligence, most of the value available is already being left on the table.
02:10Um this is not a hot take for most people, I think who work in enterprise.
02:14You've probably seen this problem.
02:16This little stat at the bottom is uh pretty heavily uh repeated for a lot of people who work inside of business circles, and they all kind of like laugh, and they're like, "Yeah, yeah, you know, all these AI tools,
02:26how much are they actually doing?"
02:27Um and I want to talk about why.
02:29So, there's kind of two sides to this becoming gen AI native.
02:33Um you have models and products, which are one side, that's the seller side, and then you also have systems and all of what's inside of the enterprise, that's the buyer side.
02:41So, that's the side that I deal with a lot.
02:44Um so, we're going to talk first about the seller side, and then we're going to talk about the buyer side.
02:47So, per pain point, uh a lot of my job is kind of go around the company and figure out like what are the pain points?
02:53Uh what are we trying to solve for?
02:55Uh can we buy it?
02:55Can we build it?
02:57So, let's say for a given pain point, maybe I identify 10 to 15 startups that look really interesting.
03:03Like, "Huh, maybe these guys can solve our problem for us, we don't have to build it."
03:07Um of those, after doing a little bit of due diligence on my own, I might schedule two to three demo calls.
03:14Of those, we probably will land zero or one pilots.
03:19And of those, probably one in four of those longer term will actually end up with a contract.
03:25So, what does this mean?
03:26This means about 5% of all of our demo calls actually end up in a signed contract.
03:31Um and this tracks with the industry.
03:32I had no idea that this was actually a benchmark, um but it turns out that there's quite a bit out there that indicates that this is really similar across the board.
03:43So, I want to talk about what enterprise ready actually means from the inside, uh because we have a lot of startups that tell me what enterprise ready means, and And we go through all of our requirements, and then we have a very different idea of what enterprise ready actually means.
03:55Um so we're going to talk about what breaks down and why.
03:59So 40% of this is efficacy, so just value, uh commercial issues.
04:04Then there's a lot that dies in security.
04:06There's other things that die in reliability, and then there's some stuff that dies in legal.
04:12So we're going to start with the requirements that we put forward and then some of the things that we've seen go wrong across various AI companies that we work with.
04:19So our requirements for efficacy, maybe unsurprisingly, the product actually needs to solve the problem.
04:24Um that seems pretty clear, but that's not always super clear.
04:29Uh the next one is pricing models that need to reflect real value.
04:33Clear demonstration of integrations on day one, not a hypothetical.
04:38And we define the success criteria, not the vendor.
04:42So things we've seen go wrong, uh vaporware in short.
04:45Um we've had a lot of startups who come in, they pitch us an idea, uh and it's something that our platform team can rebuild in about 6 weeks.
04:52So this is not a knock, uh this is actually just what's going on in the industry everywhere, um on all sides of the equation.
04:59Um sometimes it's actually better for us to build, and sometimes it is still better for us to buy even if we could rebuild.
05:06Upside down pricing, so this one's crazy.
05:08Um We had a startup just recently tell us, "Hey, um we know that all of the LLM traffic that we're using for our wrapper is passing through your LLM gateway,
05:18but we want you to report your gateway telemetry to us so that we can then price a huge margin on top of that, even though none of it's running through our infrastructure."
05:29Um that did not work.
05:31Another one is promises in demo calls, but no ETAs after 2 months.
05:35Um this is pretty common.
05:36Um not a lot to say here.
05:40Um and then repitching features we've already declined.
05:42So if you're a salesperson, um my best advice to you is listen to your customers.
05:47It's not novel, but uh it still seems to be a struggle for some.
05:51Uh it's really just better to address the things that we've asked for.
05:54So, the other thing I want to point out at the very bottom of this slide is the pilot window collapsing.
05:58So, uh I've been at Millennium for a little over 2 years, and when I started, a lot of these pilot timelines that people were used to were like, "Oh, maybe we'll run a pilot for 6 months."
06:07And then, not that long after that, it was like, "Oh, maybe we only need it for 3 months."
06:12And anymore, it's like, "Maybe we can do this pilot for 2 weeks."
06:15Uh because it's just accelerated so rapidly.
06:20Um so, then moving on to security.
06:21Uh this is a huge one.
06:22I'm not a security expert, but I do run kind of frontline defense on talking to a lot of startups about security.
06:28And so, these are a lot of the things that that come up over and over.
06:31Uh ZDR.
06:32So, this is a really hot topic.
06:33Obviously, a lot going on with Fable, uh mandatory data retention requirements, uh and then a whole other battleground around customer-managed encryption keys.
06:42So, ZDR is always best, of course.
06:45If that's not possible, customer-managed encryption keys and, with a big parentheses, that don't break the product.
06:52Um there are a lot of things that people are like, "Oh, yeah, it's fine.
06:54It'll work with customer-managed encryption keys."
06:56And then, it breaks the product.
06:58Uh so, that's a big product uh issue that we have to work through with people.
07:03Other requirements, bring your own gateway.
07:05We prefer to route all of our own traffic through our own gateway and BYO infrastructure.
07:09Uh we would prefer to host it in our own cloud infrastructure and have something that's deployable in our systems.
07:15This is another really big one.
07:16Um SCIM-tied RBAC.
07:18So, for all of you who who get that jargon, um it's really important that we can tie our AD groups or other permission and entitlement groups to role-based access control.
07:29We want to make sure that we don't just turn on features for everybody across the board.
07:33A lot of people don't think about this when they're designing their systems.
07:35They're like, "Oh, this is a great feature.
07:37We should just turn it on for everybody."
07:39Um when you work at a a enterprise, that's not something that people want to do.
07:43Um there are usually different groups who should have different access at different times, and most of all we want it to be configurable via API.
07:51Um for smaller companies, we want to see at least one real security hire.
07:56So, this is something that's really important.
07:58We know that security is not the first thing that people hire for.
08:01Um but in the age of AI, this is a very real problem, and we need to make sure that the startups we're working with actually have somebody who can understand what's going on from the security standpoint.
08:11Uh so, some of the things we've seen go wrong, um outright people just sending data to their vendors, uh cloud servers, and not following any of what we've asked for.
08:22Um this has been a problem in pilots.
08:24Uh thankfully, all of our pilots run non-production data.
08:28Another one, like we kind of talked about, um read write all default scopes.
08:32So, there's a lot of really cool tools out there, integrations, features.
08:36They're really flashy.
08:38You can click a button, and it'll integrate with everything.
08:41And then you get a little bit deeper and find out the only way that it'll work is if you literally give it read write all to everything, uh which is a huge problem.
08:49Another one, uh kind of along the same lines, all or new beta features on by default with each release.
08:55So, if you're an enterprise, you don't want everything just turned on with each release.
09:00Um so, being able to control that, and then the line that we hear a lot, which is we'll get you the security architecture diagram next week.
09:08Uh we do weekly check-in calls during a pilot, and then we hear this over and over.
09:12Uh it's not usually a great sign.
09:17Uh the question that we often have our CISO end up asking, which is um what are you going to do if there's a breach?
09:24And we get this response, well, we haven't had a breach yet.
09:27Uh with the subtext of we don't know what we would do if we did.
09:31Um okay, reliability, this is another one.
09:33So, a control plane that actually works.
09:36We want to see every admin setting available via API.
09:39We want to see audit logs on config changes.
09:41So, if there are five different people who are given admin access and somebody accidentally changes something or does it because uh maybe it was really late at night and maybe they had too many drinks.
09:51Uh we actually want to see what happened.
09:54Uh we want to be able to control the rollout on these changes.
09:57Uh we want to see real SLAs and a reachable support engineer.
10:00That goes a very, very long way.
10:02So, uh things we've seen go wrong, a lot of apps that are rapidly prototyping, they're shipping so quickly that they are maybe shipping updates multiple times a day and there's a really attractive little button that says relaunch to update
10:14and it happens across 3,000 people.
10:16We have no way of tracking what's going wrong.
10:18Maybe then like SSL certificates break in one of the new releases and then we have no way of tracking because everybody's on a different version um and we have no way of being able to deploy at scale.
10:28Um that's really challenging.
10:30No documentation versioning.
10:32So, support articles with new terms or risks that are not actually in the legal contract but show up in the website somewhere in a random support page and then we have no way of tracking what they were before versus after
10:42and it just says updated yesterday.
10:44All of these are real examples, by the way.
10:46I am not naming and shaming.
10:47Um I'm just shaming.
10:49So, uh maybe if any of you are familiar, you can put it together.
10:53Um core API's down for multiple hours during a busy trading day.
10:57Uh that is a really big problem for us because we run production systems.
11:02We are trading billions of dollars.
11:04Um this is a really big issue for us.
11:07And then lastly, no SLA roadmap or status page.
11:10Um the status page is a big one.
11:12Okay, last, legal issues.
11:14So, we don't want anybody training on our data regardless of what type of feature or product it is.
11:20Uh we also want to see a lot of transparency in the sub processors.
11:24Um any fourth-party risk becomes our risk.
11:28We want to see IP indemnification with reasonable liability caps.
11:31Uh we do not control the models, so if there's output that is IP infringing, we don't want to be held liable for it.
11:37So, we have seen in pilots that people claim they have ZDR, they have it legally, but then they find out or we find out later that they actually retain some of our data because they say, "Hey, we were looking at something and we noticed this thing."
11:48And we're like, "How did you notice that?
11:50You weren't supposed to have this data."
11:52And they're like, "Oh, yeah, you're right."
11:54Um so, that's not great.
11:55If you say ZDR, do ZDR.
11:58Um next, every feature that is conveniently beta with permissive data retention clauses.
12:04So, we've seen some vendors who they will stop releasing new features in general availability.
12:10They will only make them beta, and then the beta comes with a secret little clause that says that they're allowed to retain our data, which is a very sneaky way of trying to get our data.
12:19We don't like that.
12:20Um not great.
12:22Another one kind of similar is fourth-party risk that's tucked away on a random website page that's not listed in the contract.
12:29Uh this is a really big problem for us managing risk.
12:34So, um it was the best of times, it was the worst of times.
12:37As a recap, the best startups have security architecture that actually works, support engineers who respond, an admin API from the beginning, a 90-day plan that deploys into our infrastructure and cloud,
12:49and success criteria that we write.
12:51The worst AI startups don't have any security architecture diagrams, no path to a support engineer, no deployment control or audit logs, no ETAs, and salesmanship over solid product building.
13:02Um this is really just kind of a recap of like what I have been through over the last 2 years.
13:08Um I actually don't think that any of this is novel, um but it is codifying a lot of what I feel like is good and best practice.
13:15Um okay, so a new frontier model comes out on average every 11 days, but your architecture might be a decade or more old.
13:22So, you've got a bunch of cool new models, there's some amazing capabilities is there, and then you have profitability on the other side of it.
13:28And what's in the middle?
13:29Maybe it's your legacy architecture, probably a lot of security and privacy issues, and a lot of change management.
13:35Um ChatGPT has only been out for 43 months.
13:37There are a lot of companies who are still doing an ERP migration that might have been from 5 years ago.
13:42Um so, the timelines are very asymmetric.
13:45Uh and I think that sometimes we forget about that.
13:49Um okay.
13:49So, my thesis again, half or more of getting to AI native is unsexy and has absolutely nothing to do with AI.
13:57Um AI models and products today can't fix your legacy architecture.
14:00Although, if any of you are startup people, that's a great one to go for.
14:04Um and it also can't run your change management.
14:07These are unscientific numbers that I'm putting up here, but I hypothesize that 40% of getting to AI native is AI models and products.
14:15The other 60% is all the other stuff that no one really likes talking about anymore, uh which is like data hygiene, clean architecture, having good integration, strong enablement, and change management.
14:26Um I really look at AI as a flashlight, not a band-aid.
14:30Um I really think that AI shines a light on a lot of what's already working or not working.
14:34It can accelerate what's working really well, and it breaks down very quickly when things don't work well.
14:39Um I don't think that it's a band-aid, and I think that for everybody who's in tech leadership, it's really important to remember that if you have issues in your technology estate,
14:48those need to be addressed before trying to plug in AI and just having everything rip.
14:53Um it's it's not going to work.
14:55Um so, hehehe again, maybe an unpopular message, but I really believe that we all need to start with the boring 60%.
15:03I think that's where we all need to start to get to the other side of the road.
15:07So, what did we learn as we shine the flashlight internally?
15:12Again, not revealing anything super proprietary, but I do think these are big picture lessons.
15:16Number one, entitlements.
15:18Entitlements need a new paradigm.
15:20Uh there are a lot of people in a lot of large enterprises who are over entitled, under entitled.
15:25The entitlements model and how it works and how it's managed, all of that breaks down when you think about agents and how quickly you want agents to work and what you want them to work on and their ability to exercise judgment.
15:35Um the entire paradigm just shifts.
15:38Another one is cross-platform integration moved up the stack.
15:41So, AI is only as good as what it reaches and we want it everywhere.
15:46Uh so, having things that can integrate across platforms is really important uh even more than it already was.
15:53Another one is centralized knowledge.
15:54So, this is something that um Emil brought up this morning in his keynote, which is that basically we need thinner agents and a smarter substrate.
16:03Um centralized knowledge is really key to that.
16:05So, all of your documentation, all your support articles, everything that's going on inside of your company that's making it work, um all of that needs to be centralized and easily consumable.
16:13Even better if AI can help write that in real time in a feedback loop.
16:17Uh that's something we've been talking about with some of our vendors.
16:21Another one is a separate ecosystem for experimentation.
16:24Um some companies may need to get here.
16:27That gap between your legacy architecture and where you want to go might be so vast that you actually just decide, "Hey, maybe we need a separate ecosystem to do a lot of this work, figure out what does work and what doesn't and then kind of go from there."
16:39Um and that's something that we thought about as well.
16:42So, to just put a finer point on the agents and the entitlement thing, uh agents inherit your foundations.
16:48So, I strongly recommend that everybody fix their entitlements if they are not working really well now um because this is something that if you think about the problems that you run into when things go rogue, processes go rogue, people go rogue,
17:01agents are going to like 100X that problem.
17:03Um so, this is really something that's worth figuring out now.
17:08So, to kind of recap uh as I wrap up here, the recipe, if you are one of the people in the first half who are raising your hand on like, "What do I need to do if I'm a startup and I want to work with a really difficult large customer?
17:20Millennium's got like 8,000 people.
17:23We have very, very tight security, compliance, regulatory requirements.
17:27Um this is the stuff that we care about.
17:30And we want to see more startups doing work that allows us to work with them.
17:35Um I really view this as like one of the highest bars.
17:38We're probably not the highest, um although we're probably pretty close.
17:43Um and I think if you can architect your startup to work with companies like this with this kind of architecture, um you're probably going to be able to satisfy
17:50basically everybody else.
17:52Um on the other side for anybody who's buying, uh these are the things that I think again that kind of that boring 60% that really deserves a lot of work.
18:01Um off entitlements, governance, audit logging, etc. Um these are the things that I think we need to have in terms of systems to get it working on the other side of the equation.
18:12So, that's it.
18:14Um my only motivation here is to getting stuff working better and having better enterprise grade AI.
18:21Uh that's a QR code to my LinkedIn and I appreciate all of your time.
18:25Thank you.
18:26[applause]
18:40[music]