{ "video": { "id": "RGe6EjucbzI", "title": "\"The biggest challenge in your stack? Evals, Evals, Evals\" - 2026 State of AI Engineering results", "duration": 1187, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "RGe6EjucbzI", "title": "\"The biggest challenge in your stack? Evals, Evals, Evals\" - 2026 State of AI Engineering results", "one_liner": "Amplify's 1,048-respondent 2026 State of AI Engineering survey: evals are still the #1 stack problem (but only barely), agents with write access more than tripled while guardrails stayed at 'human in the loop', and cost has become a first-class engineering constraint.", "summary": "Barr, an investment partner at Amplify, presents the third annual State of AI Engineering survey — 1,048 respondents, run this year with Notion and Vercel — as a way to cut through weekly frontier-model noise and see what AI engineers actually do. The headline shifts: agents escaped demo world (95% use them, 89% of those can write data, so write-enabled agents are up more than 3x year over year) while the control layer is still human-in-the-loop approvals and permission gating; cost now regularly shapes how ambitiously 40% of teams build and is the #2 thing monitored in production after quality. Open-weight models augment rather than replace closed ones, teams are standardizing on platforms and tools but not on models, and evals remain the top stack challenge — though by a shrinking margin, with 96% of respondents naming some stack problem and no agreement on which.", "key_points": [ "1,048 respondents; third year running the same skew shows up — of engineers with 10+ years of software experience, over half have 3 years or less of AI experience, while the median brand-new engineer has nearly as much AI experience as the median 10-year veteran.", "Text dominates modalities, but audio has the strongest 'intent to adopt': 56% of those not building with audio plan to, up from 37% last year. The biggest actual jump is image generation, where confident use doubled from 18% to 36% — credited to Nano Banana, Nano Banana 2 and ChatGPT Images 2.0.", "94% use closed models in production, 45% use open-weight — but over 90% of open-weight users also use closed models, so open weights augment rather than replace. Open-vs-closed was a top-three model selection criterion for only 5%; quality dominates, then agentic capabilities like tool calling tied with cost, while only one in five named reliability (the speaker reads this as reliability becoming a threshold requirement).", "87% use more than one model, most commonly routing by task type; yet more than half say their org is starting to standardize on fewer AI tools — 'the early great standardization of the platform and tools, not the models.'", "Cost is now a first-class engineering constraint: 40% say it regularly shapes how ambitiously they use AI and 36% say sometimes — about three in four adjusting usage on cost. Cost and token usage is the #2 thing monitored in production, right under quality.", "Agents went real: 95% report using agents (roughly double last year), and among those, agents with write access jumped from 52% to 89% — write-enabled agents up more than 3x across all respondents. Control is still blunt: human-in-the-loop approvals and permission gating on top, then a scatter of task decomposition, retrieval, memory and sandboxing. Two-thirds say hallucination or losing context mid-task is the most frustrating failure — 'the thinking, not the plumbing.'", "Evals is the #1 stack challenge for the third year, but by a thinning margin, and 96% of respondents report some stack problem with no consensus on which. The top eval method is still vibe review.", "Build vs buy across eight stack layers: inference and model serving is the layer people buy most; prompt management is most built in-house at 61%, along with product logic, RAG and eval. Fine-tuning is the clearest 'not yet' — most don't have it at all — and people are locked in both directions (buyers aren't moving to build, builders aren't moving to buy).", "Team effects cut both ways: 97% report net positive organizational impact, with the top effect being cheaper failure and more experimentation rather than raw speed — but over 9 in 10 report negative downstream effects, most commonly erosion of deep technical skills and understanding of the codebase. 81% say AI blurs the line between engineering and product/design/marketing; over a third of teams have non-developers shipping features and 17% say non-developers regularly ship customer-facing features.", "Bets: 76% say AI boosted job satisfaction, 59% fear today's AI code creates long-term liabilities, only a third call software engineering a solved problem; 67% expect a leading lab to declare AGI within five years (the question asked about the press release, not the achievement), only 9% bet Transformers are still state-of-the-art in five years, and the most divisive question was whether there will be more AI compute in space than on land (36% yes / 38% no)." ], "takeaways": [ "Treat cost as an engineering constraint, not a finance afterthought: monitor cost and token usage in production alongside quality, the way the majority of surveyed teams now do.", "If you're deploying agents, assume write access is the norm now and that approvals-plus-permission-gating is not a finished control layer — the field is still scattering across task decomposition, retrieval, memory and sandboxing, and memory/persistent context is the layer the speaker is watching hardest.", "Pick models on quality, agentic/tool-calling capability and cost, and plan for a multi-model setup routed by task type; open-weight models are an augmentation to closed ones, and the open-vs-closed debate barely drives real selection.", "Standardize the platform and tooling layer rather than trying to standardize on one model — buy inference and model serving, keep prompts, RAG, eval and product logic in-house, and don't expect fine-tuning to be where your team is yet.", "Budget for the downside of cheap code generation: review burden, skill erosion and codebase understanding are the most-reported negative effects, and 59% already expect today's AI-written code to become a long-term liability.", "If you're building infrastructure, the 96%-with-a-problem scatter is the market map — evals still leads, and the dominant eval method being vibe review is an unsolved opening." ], "topics": [ "evals", "agents", "survey", "cost", "open-weight-models", "multimodal", "build-vs-buy", "ai-engineering" ], "tools": [ "Amplify", "Notion", "Vercel", "Nano Banana", "Nano Banana 2", "ChatGPT Images 2.0", "Meta", "Twitter" ], "quotes": [ { "text": "So it turns out that infinite intelligence still comes with a usage-based bill.", "at": "08:27", "url": "https://www.youtube.com/watch?v=RGe6EjucbzI&t=507s" }, { "text": "The top two are human in the loop approvals and gating permissions which are the right instincts but kind of the same toolkit you'd use to manage an intern.", "at": "11:13", "url": "https://www.youtube.com/watch?v=RGe6EjucbzI&t=673s" }, { "text": "Agents are no longer reading, summarizing, drafting. They're taking actions inside of systems.", "at": "10:56", "url": "https://www.youtube.com/watch?v=RGe6EjucbzI&t=656s" }, { "text": "96% of the people in the survey in this room have a problem with the stack. You just can't agree on which one.", "at": "12:23", "url": "https://www.youtube.com/watch?v=RGe6EjucbzI&t=743s" } ], "words": 3469 }, "summary_url": "/#RGe6EjucbzI", "transcript": { "html": "/transcripts/RGe6EjucbzI.html", "txt": "/transcripts/RGe6EjucbzI.txt", "vtt": "/transcripts/RGe6EjucbzI.vtt" } }