How do you diffuse AI into the real world? — Varun Shenoy, Long Lake

AI Engineer · 17 min · 248 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hi everyone.
00:13I'm Varun.
00:14I'm one of the co-founders at Long Lake and I'm excited to share a little bit about what we've been up to for the last 2 years.
00:21It all comes back to a question all of us have asked time and time again.
00:27The models are getting better, but the real question is how do you actually deploy the AI into the real world?
00:35How do you get the models to complete economically relevant tasks?
00:43Let me start by saying everyone has seen the demo.
00:46Think of the agent automatically booking a flight, the agent automatically completing a ticket in some kind of customer service portal.
00:55Think of an agent completing a block of code ready to commit and go.
01:00The reality is we've all seen this and it feels like magic.
01:052 years ago any of this would have been complete science fiction.
01:09The capabilities are real.
01:12Now, walk with me into a 200-person property management firm.
01:16Real people, real properties, real dollars, real customers all across the US.
01:24You would expect AI to show up by now, but the reality is nothing has changed at all.
01:34Here's the thing.
01:36This is totally normal and maybe in fact I'd argue this is what we should expect.
01:42This is true for every general-purpose technology.
01:45You know, take electricity for example.
01:48Electricity was invented in the 1880s and it was first demoed at Edison's Pearl Street Station Dynamo Room over in Manhattan.
01:57This was the magic demo of its time.
02:01The reality is it took a long time for electricity to be fully adopted.
02:07Consider a Ford factory.
02:09It's not enough to just have electricity.
02:12You have to rip out the existing motors and equipment.
02:15You have to bring in the new equipment.
02:16You have to go and train everybody to use that very same equipment.
02:21Here's a picture of a Ford electrified moving assembly in 1924.
02:26These things take time.
02:28Diffusion of any technology takes a generation.
02:33And since everyone here in this room today is talking about AI, I would argue AI diffusion is perhaps the single most important problem for the next 20 years.
02:44The models are going to keep getting better.
02:46The big question is how do we actually get these models to be in the real world, complete real tasks, uh and make people more efficient, happier, and provide better service.
02:57So taking a quick step step back, who are we?
03:00Uh we are Long Lake.
03:01Over the last 2 years, we've raised over $3 billion from Elad Gil, General Catalyst, and AlphaWave since our founding.
03:10Here's the strange part.
03:11We we don't sell software.
03:13We actually go out and acquire and partner with real services businesses in the world.
03:18Uh we've acquired 35 businesses across HOA and property management, architecture, HR services, and a lot more.
03:26To give you a little bit more flavor, we have roughly a 40% team right now split between technology, finance, and operations.
03:33More than half our team is part of the technology team focused on uh building products, data, and deploying the core products into the field.
03:42Uh we're in a collected group of folks, a bunch of ex-founders who've worked in the services before, ex-military, folks from Palantir, Ramp, Glean, uh and from the finance side, Blackstone, H.I.G.,
03:54et cetera.
03:55We are we are not selling them to these companies above from the outside.
04:00We're actually deploying into these companies and figuring out how to get the technology to work.
04:05And just to show you the scale we're playing at, we announced recently our $6.3 billion take private of American Express Global Business Travel, the world's largest corporate travel platform.
04:16We own these businesses.
04:19So, when the AI doesn't work, it's not their problem.
04:23We're not the vendor.
04:24It's our problem.
04:27Concretely, again, we are not the vendor.
04:29We are the operator owners.
04:31And we work very closely with our teams within the businesses to drive real outcomes.
04:37Now, I want to step back and get to the concrete about the how.
04:41What are the lessons we've learned over the last 2 and 1/2 years?
04:44And what we've learned from deploying AI into companies we've owned.
04:51Three quick lessons.
04:53One, how we move agents from co-pilots to co-workers.
04:57Two, how we leverage real-world data within these businesses.
05:02Remember, we're seeing all of the work that's being done in these real services businesses.
05:07There's a lot of interesting problems and solutions embedded within that.
05:11And then finally, perhaps the most interesting and exciting is how do you actually get all of this technology to compound over time by learning loops in the enterprise.
05:19We'll get to that at the end over here.
05:23So, starting off from co-pilots to co-workers.
05:27There's a spectrum of how much autonomy you can give an agent.
05:31On the left here, you see a co-pilot.
05:32This is, you know, your simple rag chatbot from 2 years ago.
05:36It's very quick.
05:36You can ask a question.
05:37Maybe it's integrated with some systems.
05:39It can give you information back very, very quickly.
05:44The second step is a synchronous agent.
05:45Consider something like Claude code, Codex, Claude co-work.
05:49It's real-time.
05:50There's this two-way interaction.
05:52It's a bit more sophisticated than a co-pilot.
05:54You can go let it run off for 1 to 5 minutes.
05:56Uh it'll call tools, maybe use its skills.
05:59Uh it's still synchronous.
06:00You still need to step in and ask a query.
06:02So, the next obvious rung of the ladder is the asynchronous agent.
06:06You can come in here, still ask a query.
06:08The agent will go off into the background, do some work, and then come back.
06:12Uh and what's really interesting about asynchronous agents is that the user does not have to be the one that triggers them.
06:19You can have external triggers as well.
06:21Maybe someone completes a certain task and there is an async job queue uh that allows the async agent to pull off from and proactively offer advice to the end user.
06:31Then, I'd argue the next step is a long-running agent.
06:35How do you get these agents to work for hours, days, weeks, months, etc.?
06:40I think this is currently a very core problem that a lot of the labs are focused on, as are we.
06:48And then finally, at the end, the holy grail, an AI co-worker.
06:53This is where most people start off.
06:55You want a proactive partner that gets work done just alongside you.
07:00This is what everyone wants to sell you, but what we've learned from owning the outcomes in this business is you have to earn the right to do more.
07:09It's it's not enough to jump to the co-worker immediately, right?
07:13For for a bunch of reasons.
07:14One, for certain tasks, the models might not quite be there yet.
07:18And two, you actually have to work with these companies in the field, interact and iterate very, very closely, so that they understand that this is the beginning of AI, and you can work up the rungs over time.
07:33I think a really unique lens to look at this problem through is the that of the jagged frontier.
07:38We all know that agents are incredibly good at writing code.
07:42So, what does the, for example, synchronous agent for code generation look like?
07:47This is super simple.
07:48This is just your coding agent, maybe it's Codex, Cloud Code, just running on your desktop.
07:52It has access to a file system.
07:54You collaborate within real time.
07:55You get instant feedback and you iterate.
07:59The next step is, you know, if you look at code code generation, what is the async agent?
08:03This is also fairly straightforward and largely solved.
08:06You take the exact same coding agent, you wrap it in a sandbox, and you just let it go run.
08:11It can build, it can test, and once it's done with its work, it can provide the code in the form of a PR.
08:16One thing that's really unique about engineers is folks are incredibly good at already paralyzing their work.
08:23It's very commonplace to launch 10 jobs and be comfortable with the fact that job seven might finish before job three.
08:30So, engineers are incredibly good at using these async agents.
08:35Now, when we come to services, the equivalent of a synchronous agent, what we talked about a little bit earlier, it's a co-working agent.
08:41It's an agent that has deep context about your enterprise.
08:44It interacts potentially with MCPs, custom tools, custom integrations, uh and you can chat with it synchronously just like any of these other products.
08:53I think this is a frontier here in the bottom right.
08:56What does it mean to build an asynchronous agent for the services?
09:00What does it mean to paralyze work in industries where work is traditionally done in a very, very serial manner?
09:08This is where we spend a lot of time and this is what I wake up every morning really excited thinking about, you know, we've we've figured out what the async and forking mechanism for code is.
09:17You just spin up a bunch of sandboxes and do work.
09:20What does that look like for the rest of the world?
09:24So, here's a couple questions we think about pretty seriously.
09:27One, you know, the models are trained on code, they want to write code, they're incredibly good at writing code.
09:32How do we leverage these coding agents for actual knowledge work?
09:35You You rather than wait for the models to catch up on doing services knowledge work, what if we just use that code knowledge and represent knowledge work as code?
09:44Two, as I mentioned, engineers are used to paralyzing work.
09:47How do you paralyze work that's traditionally serial?
09:49You know, people clean out their inbox one email by one email, not 10 emails at once.
09:54And finally, how do you move up the ladder here both in terms of product and user enablement?
09:59What are the right form factors?
10:01And I'd argue this varies dramatically from industry to industry.
10:05Just because you have one way of launching an async agent for code, doesn't mean that same way is going to work for architecture or property management.
10:13The second point I want to cover today is leveraging real-world data.
10:17We all know this.
10:18Frontier models have learned from everything humanity has written down, but the most valuable tasks are not on the internet.
10:26How do you actually close the books when you're missing receipts?
10:29[snorts]
10:29How do you scope a building for construction in a blueprint, potentially collaboratively?
10:35How do you coordinate vendors for fixing a broken roof?
10:39All of this knowledge lives in people's heads, in 20-year-old software, uh in the way that one senior person on one of these teams just knows how to do it.
10:47How do you make this information explicit and create tasks that you can actually learn from?
10:52So, we've constructed a little bit of a flywheel.
10:54We get our agents to collaborate with our employees to do real work.
10:59And this allows us to generate rich traces of data and information.
11:02Tool calls, the hiccups, the papercuts, everything that goes wrong with doing real work.
11:08This in turn allows us to build real-world evals.
11:12There is a ground truth here.
11:14In the case of the roofing example, the question is, did the roof get repaired?
11:19Did the books get closed?
11:21And this allows us to hill climb and build better agents, which leads to more and more impact.
11:26And what's really exciting is it ratchets up.
11:29Every week our hill climbing benchmarks become a regression test.
11:34So, our agents get better and better over time.
11:37Just to drive a little bit deeper here on the traces, there's three upshots of being able to collect these rich traces.
11:44One, we get to generate amazing evals that are built and scored automatically.
11:49Uh and we're able to gather both implicit and explicit feedback.
11:52Explicit feedback in the sense of thumbs ups and thumbs down, maybe people provide a note telling us whether this response was good or not.
11:59Uh and also implicit feedback.
12:01Right?
12:01Again, we have the ground truth.
12:02Maybe there's some data that the AI generated and there's a real diff between the data that the AI generated and what was ultimately submitted.
12:10That's rich information that almost no one else has.
12:14Two, we've started post training models internally on all of the data that these businesses operate on and produce, generally speaking.
12:23This is all data that is completely out of distribution for most frontier labs.
12:27Think of the task I showed at the beginning.
12:29A lot of the models A lot of the frontier models today just can't do these tasks yet and we're trying to post train our own models internally to be able to do that on the rich source of data that we own.
12:40And then finally, the actual agents themselves.
12:43The real world is incredibly hairy and messy and you want customization per company.
12:48Every company does things very differently.
12:51Customization per user.
12:52The way each user does their work is very unique.
12:55And customization per client.
12:57The way you work with every client is different.
12:59It's a services business and you want to uphold those standards.
13:04I love this picture because it's the whole thing in a single image.
13:08Um the the way we usually talk about LLM tasks is the top panel.
13:13Right?
13:13You just It's It's a slope.
13:15You got a bike.
13:16And but there's clear sight to success.
13:19The reality is most work is not like that.
13:22And And you and I both know that.
13:24Uh there are hills and ravines.
13:27Uh there's death by a thousand paper cuts.
13:29But But that's what real work looks like.
13:32That's the entire job.
13:33The exceptions are the job.
13:37That's That's the demo.
13:39That's the actual job.
13:43Now, on to the final thing I want to chat with you guys today is learning loops within the enterprise.
13:49I'd argue there's two hot trends everyone's talking about in 2026.
13:54One, it's continual learning.
13:56How do you make an agent better over time with feedback?
13:58I think there are plenty of sessions uh this week on how you can use continual learning, whether it's in the prompt or in the weights.
14:06And two, enablement.
14:07How do you get in these enterprises and actually get them to adopt and use AI?
14:12Traditionally speaking, these two initiatives are owned by two separate teams.
14:17Right?
14:18The continual learning is owned by your research team, your platform engineering team.
14:21Enablement's owned by growth or deployment or customer experience.
14:25Uh usually pretty siloed, not much interaction between the two.
14:29We think these are part of the exact same loop.
14:32The agent only improves if people actually use it.
14:37And people only use the agent if it's worth adopting.
14:41So, here's a little graphic of a snowball.
14:43More usage drives continual learning, which drives a better agent, which drives more usage again.
14:49All this to say, there's still a really big elephant in the room.
14:53How do you get the initial usage?
14:55I think a lot of people, you know, will use Claude Code or or give it to their whole enterprise, expect folks to just start using it.
15:04Everyone assumes the usage just shows up.
15:08But as we all know, that's simply not the case.
15:11It never does.
15:12Right?
15:12Getting a 100-year-old firm to change its processes is hard.
15:17You could have the best AI coworker on the internet or on Earth.
15:22And if the people if the person who's closed the books for the last 20 years continues to do things the same way, nothing changes.
15:29Nothing happens.
15:32So, what can you actually do about it?
15:33What you know, this this seems like incredibly hard.
15:37What what's the upshot?
15:38How do you actually get this stuff to work?
15:39Well, I think a lot about Jensen and how he dominated the market in his words with extreme hardware software co-design.
15:47Designing the chips and the software together as one system.
15:52We look at this through the lens of extreme software service co-design.
15:57How do you co-design our products with the people and the processes at our businesses?
16:02And I'd argue this is only possible from being within under the same roof.
16:08We need to meet the people within these companies both metaphorically, for example, bringing products to their systems so that the energy required for enablement is kept low, and also physically.
16:20Get on a plane, show up, say hi, learn what people actually do.
16:25You know, maybe you build a product that's natively embedded into Excel or into their ERP system, maybe their 3D design software, or or maybe even their Microsoft products like Outlook, Gmail, etc.
16:38Or you show up in person.
16:39You do a lunch and learn with a bunch of folks at one of the companies.
16:41You go to their conferences and you create cotton candy and run a stand for them.
16:45You go mountain biking and ask them about all the difficulties that they have with their actual day-to-day jobs.
16:50Or you show up in person one-on-one or sometimes even two-on-one in this case and just show them how to use the tools and learn from the feedback because this is what the rest of the world really looks like.
17:02It's not like the folks in this room or in San Francisco.
17:04It's a lot more like this.
17:06You cannot co-design software with the services business over Zoom or over a support ticket.
17:11You you have to be there.
17:12You have to be in person.
17:14And I'd argue this is the part that actually makes it work.
17:17In order to get AI diffusion to work, you have to touch some grass.
17:22Thank you so much.
17:23I'll be around for the rest of day if there's anything I can help with.
17:25My email is up there.
17:27And yeah, thank you.
17:43[music]