"The biggest challenge in your stack? Evals, Evals, Evals" - 2026 State of AI Engineering results

AI Engineer · 19 min · 245 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:13Now joining us on stage is the partner at Amplify Barren.
00:37Fantastic.
00:38You did a great job practicing.
00:40I feel very very loved.
00:42Um, let's get started.
00:44So, like you just heard, my name is Bar.
00:48I run a survey every year on the state of AI engineering.
00:53And the funny thing about running a survey on the state of AI engineering is that the field changes as you make the slides.
01:01Just in the past week, we've had Frontier releases treated like national security events.
01:06Meta reportedly exploring selling AI compute.
01:10By the time I get off stage, maybe something else will happen.
01:12So, if I miss a major announcement while I'm up here, please come find me after.
01:18But that's exactly why we run the survey every year to cut through the noise, take a moment, step back and understand what AI engineers are actually doing.
01:29Uh for the first time this year, we were thrilled to partner with Notion and Verscell to run this survey.
01:36Very quickly on me, uh this is the least interesting slide.
01:40I'm an investment partner at Amplify.
01:42Very lucky to invest in companies built by and for AI engineers.
01:46And I'll make the same promise that I make every single year, which is short time on bar, long time on bar charts.
01:53So, let's get right into it with lots of bar charts.
01:59First, let's talk about well, maybe raise your hand.
02:02Did you fill out the survey?
02:03This is a very large group.
02:05Okay.
02:06Yes, I see you in the front.
02:07Um, if the answer is you, thank you so much.
02:12If the answer is not you, I will find you in 2027.
02:15But genuinely, this only exists because a thousand of you gave your time.
02:19So, thank you.
02:20We had 1,048 respondents this year, which is a lot of AI engineers.
02:26And to be precise, this is not just AI engineers, as I'm sure you see at the conference.
02:31Every year, we see that AI engineering is more of a discipline than a job title.
02:35It touches founders, CTO's, engineers, product people, folks across company sizes and experience levels.
02:43And that range shows up in experience too.
02:45Um for the third year running we see the same pattern which is skew towards senior engineers but newer to AI.
02:53Of those with over 10 years of software experience over half have three years or less of AI experience which tracks uh these are very experienced engineers learning a new paradigm in real time.
03:06And the newest cohort, the ones who just started uh engineering, the median new engineer has nearly as much AI experience as the median 10-year software veteran.
03:16Uh so the newest engineers have never known software without this.
03:21But doing AI doesn't mean one thing.
03:23We talked about all these different titles, all these different roles.
03:26Before we get into models and agents, I have a more basic question, which is when people say they're doing AI at work, what are they actually doing?
03:36So, first up, like to start with the modalities.
03:38We asked, which modalities are you actively building with at work?
03:43Can anyone take a guess?
03:44Text dominates.
03:46I know.
03:46Hold your applause.
03:47Um, but one piece of this chart that I always find very interesting and I always look at is the ratio of nope, I'm not using this modality to I'm not using it, but I do plan to.
04:00I call this the intent to adopt ratio.
04:03Of the people who are not building with a modality today, how many say they plan to use it?
04:10And audio has the strongest intent to adopt this year.
04:14Among AI engineers who are not building with audio today, a whopping 56% say they plan to adopt it in the AI applications they build.
04:23And this is not a brand new signal.
04:26Last year audio also had the highest intent to adopt across modalities but 37%.
04:31So audio continues to take the lead and have high interest but that interest is accelerating.
04:38Now there has been an audio swing but if we look at what changed most from the last year in the survey the biggest jump is actually in people using image generation.
04:49The share of respondents using generative AI for images and feeling really good about it doubled from 18% last year to 36% this year.
04:59Makes sense if you look at what we launched in the same window.
05:02Over the past year plus survey time, uh we've had models Nano Banana, Nano Banana 2, Chat GPT images 2.0. The products have gotten much better.
05:13What used to feel like an efficient way to generate cursed hands is just increasingly becoming a part of real work.
05:20Audio may have the strongest intent to adopt, but image generation shows us what happens when a modality crosses that threshold.
05:27So, I'm excited to continue watching these adoption curves every single year.
05:31I think we're going to see a lot this year.
05:34Uh, now models.
05:35If you've Who here spends time on Twitter?
05:39All right.
05:39Yes.
05:40I imagine this is a very Twitter pilled uh crowd.
05:43If you spend any time on Twitter in this uh in this circle, you've seen a lot written about openweight models these past few months and I think we'll see it even more in the next year.
05:53Um so we asked what models are you actually using in production.
05:5894% use closed models.
06:0145% are using openweight models.
06:04But here's the thing, you know, openweight models are not replacing closed models for the most part.
06:09at least not yet.
06:11The respondents using openweight models, over 90% of them are also using closed models.
06:16So they're looking like an augmentation.
06:19Teams are mixing and matching.
06:23We also asked just to double click on this for the top three considerations when choosing a model.
06:29If you're choosing a model, what is important to you?
06:32Um and despite the airtime of the open versus closed, it's not what drives model choice.
06:38It was a top three consideration for only 5% of the respondents.
06:43What matters is actually more straightforward.
06:45It's quality.
06:46Quality dominates.
06:47Followed by agentic capabilities like tool calling and cost tied right with it.
06:52We'll money money.
06:53We'll get back to that.
06:55Um, one thing that I found very interesting is that reliability is not near the top.
07:01Only one in five named reliability.
07:03That doesn't mean teams stopped caring about reliability.
07:06Uh there are different ways to interpret this data.
07:08My guess is that it's more likely to become a threshold requirement and the models they're choosing are reliable enough so the decision moves up the stack outside of certain circumstances to quality, capability, cost,
07:21but we could talk after.
07:24All right, so here's where the model story all comes together.
07:28Like I said, teams are not choosing one model and calling it a day.
07:33Earlier I showed that 87% of teams are using more than one model.
07:38Uh the model that's the opposite of standardization.
07:41Uh and the way that they choose models for given tasks varies.
07:45Most popular is routing by task type.
07:48Some run multiple models compare outputs.
07:50Some route based on cost.
07:52Uh but models are good at different things.
07:55What was interesting was that more than half of respondents said that their organizations starting to standardize on fewer AI tools.
08:02They're trading standardiz flexibility for standardization.
08:06A share of those are mixed.
08:07They say they're standardizing on some layers while staying flexible on others.
08:11But the headline here is that there's we're in the early great standardization of the platform and tools, not the models.
08:20All right, this is the slide where anyone who's opened an AI bill in the last year starts nodding.
08:26So it turns out that infinite intelligence still comes with a usagebased bill.
08:31Once teams are managing many models and AI workflows, the next question becomes cost.
08:38Cost is now a first class engineering constraint.
08:42We see this in the data.
08:4340% of respondents say that cost regularly shapes how ambitiously they use AI and another 36% say that it sometimes does.
08:54Well, this is pretty straightforward.
08:55So all in about uh three out of four respondents are adjusting their AI usage based on cost and maybe the fourth has a company card.
09:06That might be surprising or maybe it's obvious but 12 months ago it was not.
09:11Token maxing is cool.
09:14Being able to find real use cases is amazing but cost is becoming a real big part of the product decision today.
09:21And it shows up in monitoring too.
09:24What folks are monitoring in production includes cost and token usage as the number two thing they watch for.
09:30It's being monitored like an SLA right under quality itself.
09:36Which brings us to the biggest line item of them all.
09:39Agents.
09:40We've been talking about agents for a while.
09:42Uh this year, as you've seen, as you'll see today, as you've seen in previous days, you're going to talk a lot about harness engineering.
09:49They're escaping demo world.
09:53So, we asked respondents what level of tool permissions their agents typically have.
10:00And this is where agents start to look more real.
10:03There are two things happening at once.
10:05First, and I don't think this is surprising, relative to last year, there are far more teams using agents.
10:13This year, 95%, this seems high to me, 95% say they're using agents, roughly double last year.
10:21Second, amongst the teams that are using agents, those agents are much more likely to have write access.
10:28Last year, 52% of folks building with agents said their agents could actually write data.
10:35This year, that number is 89%.
10:38So when you combine these two shifts, more teams using agents and more of those agents having write permissions, the share of all the respondents and again it's a survey
10:48using write enabled agents is up more than three times relative to last year.
10:53So this is really the big shift.
10:55Agents are no longer reading, summarizing, drafting.
10:58They're taking actions inside of systems.
11:00And that raises the obvious question, how are we controlling all of this?
11:06um with pretty blunt instruments.
11:08Uh very there are many ways that folks are controlling agents today.
11:12The top two are human in the loop approvals and gating permissions which are the right instincts but kind of the same toolkit you'd use to manage an intern.
11:23Below that the results scatter.
11:25Task decomposition, retrieval, memory, sandboxing.
11:28People are trying everything.
11:30Nobody has settled the control layer for agents.
11:33uh memory and persistent context is one that I'm watching very carefully right now.
11:37I think it's going to evolve a lot in the next year.
11:40And when agents fail or when people complain about agents failing to be more precise, it's usually the thinking, not the plumbing.
11:49So, uh you know, like twothirds say that hallucination or losing context mid task is what frustrates them the most.
11:57All right, so agents are out in the wild, which makes it a good time to look at what everyone's actually running underneath.
12:04So let's take a peek at the stack.
12:07Um, we asked, what is the biggest challenge in your stack?
12:09Every single year that I ask this, the answer, the number one answer is eval.
12:14Um, so Eval's lead here is same as always, but by a very thin margin, like that margin is getting smaller.
12:20And I'll say the quiet part here, which is that 96% of the people in the survey in this room have a problem with the stack.
12:28You just can't agree on which one.
12:30Um, so if you're deciding what to build next, if you're interested in infrastructure, that scatter is the map.
12:37And the leading challenge, how to evaluate your AI outputs requires many different methods, but as always, the vibe review is number one.
12:46So there are some consistent things that we'll see if they change over the time, but they they have not changed.
12:53Okay, this is interesting.
12:54So across eight layers of the stack, we asked what do people build versus buy.
13:00Um again, maybe the corporate card is is going to play a part in this, but there is a wide range and mix for every layer of the stack and a few clear takeaways.
13:11So the first is that inference and model serving is the layer that people buy the most.
13:16Many people don't want to build inference infrastructure and fair enough.
13:22Uh prompt management is the opposite.
13:2461% build it themselves.
13:26Um apparently everyone's prompts are special.
13:30And this is true of a lot of the product logic prompts rag eval.
13:33They tend to stay inhouse on a relative basis.
13:37fine-tuning is the clearest not yet.
13:40Like most people don't have it at all and uh folks are pretty locked in.
13:46So those who bought aren't looking as much to build.
13:50Those who built aren't looking as much to buy.
13:53Uh but those are those are the core takeaways from the usage in our stack.
13:59So many of you work on teams and like we said at the start these range from solo founders to large enterprises.
14:07What is this doing to teams?
14:10And remember this is a builderheavy sample.
14:13But among builders the vibes are good which you know I'm sure if you look to your left and your right you're feeling that the vibes are pretty good.
14:2097% report a net positive effect on their organization.
14:26The top effect isn't really just speed.
14:29It's cheaper failure, more experimentation, more prototypes, more bets.
14:35It didn't just make engineers faster, but it made trying things nearly free.
14:39And so there's some happy campers as a result of that.
14:44But it's not free free.
14:45You know, there's no free lunch as nothing is.
14:48So the same tool that increases experimentation also increases review burden.
14:53Both can be true.
14:54And um you know o over nine and 10 respondents are feeling negative downstream effects in some way.
15:03The most common ones being wid you know widely discussed at this conference uh online and anywhere that you see AI engineers erosion of deep technical skills and understanding of the codebase.
15:16And these are consequences of cheap code generation.
15:22And the org chart is really feeling it.
15:25So many folks, 81% are saying that AI is blurring the line between their role as engineers and product design and marketing.
15:36These stats shocked me.
15:38Um, where you feel it the most is shipping software once exclusively the engineers domain.
15:44I know folks talk about vibe coding and how that's accessible to more folks than ever before in different roles, but today over a third of teams have non-developers shipping features,
15:56which was pretty wild to me.
15:59Mostly smaller, mostly internal, but 17% say that non-developers are regularly shipping customerf facing features across the stack.
16:09And even when non-developers aren't shipping, a third of teams see them building really useful things, prototypes, front-end mocks, and more.
16:17So, shipping software is not gated on being an engineer.
16:20We knew this, but uh the extent to which it's being pushed is is higher than I expected.
16:27All right, so where does all of this go?
16:29We always ask people to place bets rapid fire.
16:33So, let's talk about those results.
16:36Um, so present tense first.
16:4176% say AI boosted their job satisfaction.
16:45So that's good for most of this crowd.
16:47I hope you're uh as uh Alphaba and Glenda say, I hope you're happy now.
16:52Um, that's great.
16:53But 59% fear today's AI code creates long-term liabilities.
17:00Only a third call software engineering a solved problem.
17:04Although uh when I have conversations with folks sometimes the way in which they define software engineering is different.
17:11So you can read into that stat as you will.
17:13Um happier faster but embracing the maintenance build is the TLDDR and people are unsure what's going to happen with hiring.
17:22And for the five-year bets we have 67% expect a leading lab will declare AGI in the next five years.
17:31Note the wording.
17:33We said will, we asked about the press release, not the achievement.
17:36So, will they declare it?
17:37Yes.
17:37What does that mean?
17:38Not sure.
17:40Uh, only 9% bet on Transformers being state-of-the-art in 5 years.
17:46Most are unsure.
17:48Uh, but that was interesting.
17:49And then my favorite, will there be more AI compute in space or on land?
17:5436 yes.
17:5538 no. The most divisive question in the survey is about outer space.
18:02I promised you a lot of bar charts and that was a lot of information.
18:06So a review or our 2026 wrapped impact is overwhelmingly positive.
18:13Image genen doubled or happy image gen doubled while audio has the highest adoption intent the same as last year.
18:23cost really became a first class constraint and we see that everywhere in monitoring in how ambitious folks that are going out and building AI products are behaving.
18:35Open weights augment but they don't replace.
18:38So we're seeing a multimodel future with a consolidation of the stack.
18:43Agents got right access more than ever before, tripling relative to last year.
18:48While the guardrails stayed pretty primitive, and inference is the buy market, everything closer to product logic tends to relatively stay more in-house.
18:58It is a very exciting time to be an AI engineer.
19:01I cannot wait to see how the next year unfolds.
19:04So, you can find the full report in the link up here.
19:07Every chart plus some cuts that we didn't have time for today.
19:11Um, I won't ask you to fill out a survey about the survey, but if there's something that you want on the books for 2027, something you're curious about, you can come find me here on the internet.
19:21I'm easy to spot.
19:22Thank you so much.
19:24Uh, we will see you next year or per 36% of you, maybe in orbit.
19:29Thank you.