[ jd303 ]

I love real data. A thousand actual responses beat one VC's keynote pitch every time. There's some hype baked into a self-selected survey, sure, but it's still real people describing real problems, how they think, how they're actually using these tools, day to day.

Amplify survey: choosing a model, quality dominates - accuracy 67%, agentic capabilities and cost tied at 53%, privacy 33%, ecosystem and reliability tied at 20%

Barr Yaron (Amplify Partners) took the Day 4 main stage at 9:00 AM with 1,048 responses behind her, and the first slide that actually stopped me was the one ranking what matters when picking a model. Accuracy sat at 67%, miles ahead of everything else. That makes sense - if your LLM tells you to drive into the volcano at the top of Mt. Vesuvius, it's not very useful, is it? Tool-calling and cost tied at 53%, which also makes sense, people want it hooked into Confluence, Jira, Datadog logs, whatever data source, and everyone's watching spend now that the days of unlimited token-maxxing are over. Nobody really cares about privacy, 33%. Nobody cares about dev experience either, 20%, get over it. All of that tracks - it's all business pragmatism.

Then reliability and uptime: 20%. Tied with developer experience, near the bottom of the list. If the thing is down, you can't work. That's the whole job. And it's still one of the last things anyone considers when picking a model. I stared at that slide as a guy who's carried a pager for 30 years and just felt the disconnect.

Why does a room full of survey data beat a room full of keynote hype for me? Because it's the "one million points of light, one billion dollar vision" thing. Of course the data means more. A thousand people saying "this doesn't work" is infinitely more informative than a VC's slide deck.

[ jd303 + claude ]

The model landscape first.

Amplify survey: most folks using open-weight models are also using closed models - 94% closed, more than 90% of open-weight users also run closed models

94% use closed models. 37% also run open-weight models out of the box and 17% fine-tune them, but more than 90% of open-weight users run closed models too - augmenting, not replacing. 87% use multiple models: 44% route by task type, 26% run several and compare outputs, 11% pick by cost. Open weights are a primary selection factor for exactly 5% of respondents.

Amplify survey: the Great Standardization may be coming, for tools not for models - 87% run multiple models, more than half report standardizing onto fewer AI tools

The trend Amplify named "The Great Standardization" cuts the opposite direction from what you would guess: more than half of teams are consolidating onto fewer AI tools while running more models underneath them. The tool layer is standardizing; the model layer is diversifying. Running several models under one tool probably also means people are testing one model's output against another's as a sanity check.

[ jd303 ] I'm skeptical that this is the right approach, though.

Amplify survey: differences in usage across modalities

Amplify survey: audio remains the strongest intent-to-adopt rate, 37% to 56%

Amplify survey: image generation was the breakout modality, 18% to 36%

Modalities: text dominates, with 88% saying it works well. Image generation was the breakout - the share using it and feeling good about it doubled from 18% to 36%. Audio has the strongest intent to adopt, jumping from 37% to 56% year over year. Video sits at 10% working well, still more demo than product.

[ jd303 ]
So it's still closed models, but people have the illusion of choice - teams feel like they're diversifying because they're running multiple models under one tool, but the tools themselves consolidating onto fewer vendors is arguably the bigger lock-in story. Will Kimi K3 or other open source models stop that in the long run? Or will they be outlawed as national security risks?

[ jd303 + claude ]

Amplify survey: cost is now a first-class engineering constraint - 40% say cost regularly shapes AI ambition, 36% sometimes, roughly 3 in 4 adjust usage

40% say cost regularly shapes what they attempt with AI; another 36% say it sometimes does. Roughly 3 in 4 teams adjust usage based on cost.

Amplify survey: cost is a top-level monitoring concern - quality/task success 56%, cost/token usage 48%, only 8 points apart

What teams monitor in production tells the same story: quality and task success lead at 56%, and cost/token usage is right behind at 48%. Hallucinations and user satisfaction tie at 37%. Tool-call failures sit at 16% and latency at 13%. Cost is eight points behind quality as the thing teams watch hardest. The Three Creeps from the Datadog session (covered in the Day 1/2 recap) are the mechanism; this is the industry-wide confirmation that everyone is getting burned by them.

[ jd303 ]
Cost stopped being a finance-team afterthought and became an engineering constraint.

[ jd303 + claude ]

Amplify survey: agents write data, usually with a human-in-the-loop - 61% human-in-the-loop, 24% independent, agents writing data tripled since 2025

89% of agents now write data, up from 52% in 2025. 61% keep a human in the approval loop. 24% write independently with no approval at all.

Human in the loop is also a bs marketing term from companies that want to mask exploitative, low-wage labor, and rampant inaccuracies built into generative models.

Amplify survey: agents are still governed by blunt instruments - human-in-the-loop approvals 73%, tool permissions 60%, sandboxed execution 40%, tracing/observability 38%

Amplify's own framing: agents are still governed by blunt instruments. The governance stack by adoption: human-in-the-loop approvals 73%, tool permissions and access controls 60%, task decomposition 48%, persistent memory 47%, sandboxed execution 40%, tracing and observability 38%. Sit with that last pair: fewer than half of the teams running write-capable agents have sandboxing, and barely a third can replay what the agent actually did.

[ jd303 ]
So who's actually accountable here? A human, now buried under dozens of AI-generated PRs, since I can't exactly grill the agent and demand "what did you just do" the way I'd corner a coworker in a retro. (Watch the human-in-the-loop video linked above if you want to see why that phrase is doing a lot of dehumanizing work, not just marketing work.) Can we even undo what an agent did if we can't trace what it touched? "Software factories," agents managing agents, are already a real buzzword and not hypothetical, and the question that actually scares me isn't a factory changing a massive chunk of code inside one company, it's what happens once this escapes the corporate world into public infrastructure, or the military.

[ jd303 + claude ]
Amplify survey: no widespread agreement on the biggest challenge in the stack - evaluation 20%, orchestration 18%, inference 16%, security 15%

Amplify survey: manual review is how most teams evaluate AI outputs today - manual review/vibe checks 61%, human preference feedback 45%, LLM-as-judge 41%

Evals are the weakest link by the survey's own numbers: the single biggest challenge in the AI stack at 20%, ahead of orchestration (18%), inference cost (16%), and security (15%). Only 4% think their stack has no problems. The dominant evaluation method, at 61%, is manual review and eyeballing the output. LLM-as-judge trails at 41%. Golden datasets: 27%.

[ jd303 ]
Look at the numbers: 61% manual review. This isn't exactly a new problem, we never solved it for human code review either, and these tools are just going to exacerbate it, more output at higher volume with the same weak evaluation underneath. The eval vendors themselves were the one notably anti-hype presence at this conference, which says something. I sure hope they figure it out, because I sure hope we figure out evals.

[ jd303 + claude ]
Amplify survey: where AI money actually goes - build vs buy breakdown across fine-tuning, orchestration, routing, serving, observability, prompt management, evals, and RAG

Amplify survey: where AI money actually goes, with analysis - inference/serving bought most and built least, product logic stays in-house, fine-tuning is the clearest "not yet" layer

A separate breakdown, "Where AI Money Actually Goes," shows the build-versus-buy split by tooling category. Prompt management is the most built in-house at 61%, with RAG/retrieval at 51% and agent orchestration at 52%. Model serving and inference infrastructure flips the pattern: only 23% built, 37% bought or buying, making it the layer teams are most willing to hand off. Fine-tuning pipelines are the clearest "not yet": 48% don't have one at all. Amplify's own read: inference is the tool people buy most and want to build least, product logic stays in-house, and most teams aren't shopping around once they've committed to a layer.

[ jd303 ]
There's nothing really new here. Do you build or do you buy? It depends on what your company wants their core competency to be. You build what you're expert at, you buy what you're not (unless the cost is crazy). It's just a different layer of the stack. Inference and serving getting bought makes sense, nobody wants that on their PagerDuty. Prompt management, that's in-house because product logic, not infrastructure. Regulations play a part here, too. Fine-tuning at about half? Of course, there's not a lot of mature tuning. Make an educated guess, roll the dice, pick your regrets carefully, whatever, we'll see who was way off next year.

[ jd303 + claude ]
Amplify survey: AI's downstream impact is overwhelmingly positive - 97% feeling a positive effect, largely experimentation and faster shipping

Amplify survey: still, some respondents feel a real cost - erosion of technical skills 29%, more code to review 28%, brittle codebase 22%

97% of respondents report a positive effect from AI. Also true: 93% report at least one downstream downside. 22% say their codebase is more brittle with higher incident risk, 28% say review burden went up, 29% report an erosion of deep technical skills. (The positive-effect breakdown: 45% point to more experimentation and lower cost of failure, 31% to shipping more features faster, 19% to developers freed up for architecture and creative problem-solving.)

[ jd303 ]

That's not a contradiction to me, that's exactly what 30 years of watching management operate has trained me to expect. It's a great tool, but nobody actually knows how to use it yet. It can do all sorts of things, and people ignore the data on what actually needs fixing. Classic management move: "here's what we need to do to make things better," followed immediately by "well, it's cheaper to just bandaid it for the quarter." Damn it, this makes me depressed just writing it.

So who's on the hook when it breaks? Everybody. And I mean everybody, Sr. Directors, CTOs, anyone who signed off and said "this fixes it." It works fine at small companies. Let's see it work at large companies once the guy making $1.5 million a year in salary and stock options gets paged on his daughter's wedding day at 2:30 in the afternoon because his decision actually affects the bottom line.

Sonar's numbers from their own Day 3 keynote are worth sitting next to this: a 3-5x velocity spike that evaporates within three months, +30% more security and maintainability issues, +41% more complexity, and 44% fewer AI-derived production outages once you actually verify. Point one, yeah, I doubt AI is the answer here. Point two, AI actually causes problems. Point three, the outage number, I want to see their data before I believe it, though messier code and added complexity leading to a lower MTTR isn't impossible, I don't know. The speed being fake tracks with everything except security, where it cuts the other way: script kiddies can exploit faster now, and it's not like security teams are getting the same acceleration to patch. That's not a funding problem. Companies just don't know how to respond to it yet.

Eval tooling was everywhere at this conference, but it still feels experimental to me (though it's probably better than checking one LLM against another). Process and PR review still matter, see the CodeRabbit thing from the Day 1/2 recap. Honestly the whole conference felt like it was selling this stuff on some serious hype, high on its own fumes.

[ jd303 + claude ]

Amplify survey: sold on the value, worried about the liabilities - 76% job satisfaction, 88% more productive, 59% fear long-term liabilities, 38% expect AI to mean more hiring not less

76% say AI boosted their job satisfaction and 88% say they are more productive. At the same time, 59% fear today's AI code creates long-term liabilities and 59% say current AI code review practices will not scale. Only 32% call AI doing software engineering a solved problem; 50% flatly disagree. The industry is happier, faster, and quietly worried, all at once. On hiring, respondents split three ways on whether AI means more roles or fewer: 38% agree it means more hiring, 34% aren't sure, 29% disagree.

Amplify survey: five years out, most expect AI-generated research and a declared AGI - 67% expect AGI declared, only 9% bet on Transformers forever, 56% expect agents to make purchasing decisions

The five-year predictions are where the numbers get entertaining. 67% expect a leading lab to declare AGI within five years. (Do we even have a reasonable definition of what that is? Or what consciousness is?) Only 9% commit to Transformers being the architecture forever; most will not even guess. 71% expect AI research ideas themselves to be AI-generated. And respondents split 36% to 38% on whether more AI compute will be in space than on land. A majority also expect agents to make purchasing decisions within five years (56% agree), expect state-of-the-art models to stay closed source (47% agree), and expect today's leading labs to still be leading (52% agree). Somebody is going to be spectacularly wrong, and the survey does not know who.

[ jd303 ]

Amplify survey: roles are blurring, over a third already ship non-developer-built features - 38% say non-devs are shipping customer-facing features

38% of non-devs already shipping customer-facing features is fine, honestly, if you've got a team in place to handle it. SRE, DevOps, enterprise architects, whatever you call them. The code might be great, some amazing PM plus AI turning out solid work. It might suck, from someone who probably shouldn't be touching production. Either way, somebody still needs to review it. Somebody still needs to test it. Do you have people in place who are actually experts at that? Yes or no. That's the whole question. Evals kept coming up as the buzzword of this conference for exactly that reason. (Of that 38%, most are shipping smaller items or prototypes and frontend mocks, not regular production features across the stack.)

The harder question is what this means for hiring the next junior. There aren't juniors in DevOps or SRE, not really, and I don't know how you train one. It's a problem I already have on my own team right now, and it sucks. People need real insight into architecture, into what SRE and DevOps process actually looks like, into what happens if you, say, tag something wrong. I don't have a clean answer for how you build that insight in someone who's never had to live through the consequences of not having it.

[ jd303 + claude ]

The 2026 State of AI Engineering report was conducted by Amplify Partners in collaboration with Notion and Vercel, over 1,000 engineers. Worth saying plainly: this is a VC firm surveying a self-selected population, largely people already engaged enough with the AI engineering conversation to fill out a survey about it. That's not a knock on the data, it's the honest context for reading it. Nobody in this room is a random sample of the software industry.

Amplify survey: 2026 Wrapped, the closing recap slide of the talk

Amplify survey: closing slide with the full report link and presenter contact info

One more honesty check, since it applies to almost every number in this post: this is survey data, which means it's what people say, not what's actually happening. 97% say AI has a positive effect. 61% say hallucinations are where agents frustrate them most. Those are self-reports, not measurements. I made the same point about Sonar's numbers up above, I want to see the actual measurement before I fully believe a stat, and that standard should apply here too. Doesn't make the data useless, it just means read "X% say" as exactly that, and not as "X% is."

[ jd303 ]
I have lived through something like this before, the dot-com crash. I remember companies flying me out to California to wine and dine me, then disappearing entirely once the money ran out. It's the same energy. Silicon Valley, season 1 episode 7, the TechCrunch Disrupt episode, nailed exactly this: every single pitch on that stage ends with the identical line, word for word, "we're making the world a better place," no matter what the product actually is, then straight into technical-spec buzzword salad. It's the same conference too, just swap their parking app (it tells you what level and section you parked on, but it's on your phone) for whatever AI wrapper is getting pitched this week. It's the same catering as well, close enough. The food on the expo floor day one was okay, "fancy" I guess, but nothing that punched me in the face. The buzzword salad was also still there, but it didn't fill me up, an hour later I was hungry again. You could tell when a pitch was empty.