AI Engineer World's Fair 2026: Conference Miscellanea
This is a data dump post. This is the post with the most. There is lots of miscellanea, this is what we saved for later. This is the stuff that got cut, this is the overflow glut, for time and space and the recaps, here's what fell through all the gaps. Four keynotes deep and nothing's lost: Fable, Sonar, DeepMind, and cost, Dreaming, DSPy, and the code that's secure, robots still standing, still not quite mature.
This one's got a dump of a lot of the presentation screenshots, with a bit more of a deep dive and a lot less snark. Still some snark. Later posts will go deeper on the analysis side, but I wanted to get this out for everyone to see first. Also, go watch the videos, the actual talks are up on AI Engineer's YouTube channel, so you can see this stuff straight from the people who said it instead of just my slide photos and commentary.
"Using LLMs to Secure Source Code" - Eugene Yan (Anthropic, Project Glasswing), Day 2, 1:30 PM.
This came with the most concrete numbers of the conference. Anthropic's own scanning pipeline, as of May 22, 2026: 23,019 candidate findings, 1,900 reviewed by external firms, 1,726 confirmed valid. That is a 90.8% true positive rate. 467 of those went to maintainers through that reviewed path. But that's not most of what maintainers actually got: another 1,129 findings were reported directly to maintainers by Anthropic, at their own request, a separate branch that skipped external review entirely and may contain false positives. Add those together and you get 1,596 total disclosed, of which 1,451 were acknowledged by the maintainer, 97 patched, 88 advisories published.
Read those numbers in order. Twenty-three thousand findings. Ninety-seven patches. Finding vulnerabilities is no longer the bottleneck; human capacity to triage, report, and patch is. Plan accordingly.
The tooling is public: claude.com/product/claude-security scans your codebase, validates findings, and suggests patches; claude.com/blog/using-llms-to-secure-source-code has field lessons from teams running this in production; the interactive skills and autonomous harness live at github.com/anthropics/defending-code-reference-harness, and github.com/anthropics/claude-code-security-review is a GitHub Action that does AI security review on PRs. The recommended adoption path is sensible: start with open-source dependencies, learn the interactive skills before turning on the automated harness, and budget for verification, triage, patching, and org processes rather than scanning.
The interactive workflow, in a new terminal:
git clone https://github.com/anthropics/defending-code-reference-harness
cd defending-code-reference-harness && claude
/quickstart
/threat-model bootstrap targets/drlibs # build threat model first
/vuln-scan targets/drlibs # scoped scan
/triage targets/drlibs/VULN-FINDINGS.json # verify, dedupe, rank
/patch ./TRIAGE.json --repo targets/drlibs # generate fixes
/customize use ~/code/my-service/{THREAT_MODEL.md,VULN-FINDINGS.json} and ./TRIAGE.md # port to your own codebase
"Claude for Long-Horizon Tasks" - Lance Martin (Anthropic), Day 2, 1:55 PM.
The talk comes down to one design principle: separate the agent doing the work from the agent judging it. From the Anthropic engineering blog, quoted on the slide: "Separating the agent doing the work from the agent judging it proves to be a strong lever to address this issue." The issue being that even on tasks with verifiable outcomes, agents show poor judgment about their own work. A model evaluating its own output is too lenient. A standalone evaluator tuned to be skeptical is far more tractable, and it gives the working agent something concrete to iterate against.
Anthropic ships this pattern two ways:
Claude Code /goal | Managed Agent Outcomes | |
|---|---|---|
| Goal | Measurable end state | Rubric with gradable criteria |
| The judge | Independent grader model (Haiku) | Independent grader sub-agent |
| The loop | Not-met verdict starts the next turn | Iterate, grade, revise |
| Bound | Turn or time clause in the condition | max_iterations |
| Exit | Auto-clear on met, or /goal clear | Rubric passes, or interrupt |
Then Dreaming, the long-horizon memory piece: a periodic batch process takes the transcripts from an agent's daily sessions plus its current memory state and produces an updated memory, new insights, reorganized structure, that feeds the next day's sessions. The pitch is that tomorrow's agent is automatically smarter than today's, without anyone touching a context file. Anthropic's own docs confirm the mechanism: duplicate entries merged, contradictory entries resolved in favor of the most recent value, stale references pruned, explicitly framed as an REM-sleep memory-consolidation metaphor. Worth noting the name shifted between the conference talk and the shipped feature: Martin called it "Dreaming" on stage, the official docs call it "Dreams." Currently a research preview, access by request only.
The headline number from the conference: legal-AI customer Harvey saw roughly a 6x improvement in task completion rates after enabling Dreaming, per Anthropic. The mechanism answers the obvious question, this is user data, not model data, no weights change. Harvey's agents kept forgetting filetype quirks and tool-specific workarounds between sessions, so the same drafting jobs failed the same way repeatedly. Dreaming made those workarounds stick, fewer clarification rounds, fewer repeated error corrections, not a smarter model, just one that stops rediscovering problems it already solved. Worth the caveat too: it is Anthropic's own self-reported number, no external benchmark has replicated it, and it reflects one unusually clean failure mode in Harvey's specific workflow, not a general multiplier to expect elsewhere.
Here's the thing though: that's basically my current workflow already. I sync my actual contexts and memory files between two machines through a git repo, and Claude reads and writes to it every session. I don't know if Dreaming is a formalized version of that same instinct or something genuinely different underneath.
I also don't know if git is the right long-term primitive for this. Just a couple days ago I hit a real example of the problem: an actual merge conflict syncing this exact setup across machines, plus a batch of dates that had drifted a full year out of sync between two sources of truth. Right now I'm the one holding all of that together by hand, with a startup hook and some git flags. It would be nice if this part was just handled by Anthropic or whatever harness I'm running instead of me being the mechanism.
One more thing worth flagging: Dreaming keeps the same marketing pattern as Fable and Claude Tag from the Day 3/4 post. Once again it's not a product, it's a model, a thing, a persona for us to use, never something they're selling us. Names like Dreaming do real work here too, giving the thing a personality and it stops sounding like a feature and starts sounding like something with its own inner life.
"A Field Guide to Fable" - Thariq Shihipar (Anthropic), Day 3, 9:05 AM.
The talk was structured as a literal field guide in four parts: Unhobbling Claude, Finding your Unknowns, Dealing with the Grief, and Being Unreasonable. The names do a lot of work on their own, and the "Being Unreasonable" one already got the marketing-callback treatment in the Day 3/4 recap (twice), so this section leaves that alone and focuses on the other three.
Two details stood out beyond the framework. First, the release context: Fable shipped that morning, after the government pushback situation, announced from the opening keynote of a conference Anthropic was not sponsoring that day. Second, the workflow advice: keep a constant feedback loop running with the model instead of prompting and walking away. His description of the overall experience: like discovering the open world in a video game, when the map finally opens up and you can go anywhere and do anything.
That whole "sad for the old days" thing is romanticized. There's a real feeling in finally fixing a bug, or figuring out how to do something in code, studying your way into it. But there's also just being annoyed digging through logs for a missing semicolon. That's painful, not meaningful. Now you can (almost) tell an AI tool to handle it, and it only sometimes messes up the same way, a missing quote, an unclosed bracket. You do learn the language, or the infrastructure, or the architecture, better by fighting through it yourself. But is that tradeoff, frustration for depth, actually worth it? I don't think so.
Watching pixie_technologist fight her breadboard is the same thing, just connections instead of code. It's not "I missed a semicolon," it's "I put the pin in 3-4 instead of 4-4." It's the same category of annoyance. She hated it. There were tantrums. You already knew what you wanted, you just typed the wrong character or pushed a pin into the wrong hole. That's not learning, that's frustration. This is exactly where AI tools already beat us at code. Maybe one day a robot arm beats us at breadboards too.
It's like Stephen King forgetting a question mark in Pet Sematary and getting the whole book rejected over it. That typo teaches nothing. Same with a missing semicolon, the compiler is technically correct (the best kind of correct), but it doesn't teach you anything either. A race condition is a different animal, figuring one out is a real mental exercise. AI tools are already better than us at avoiding the first kind. Roughly 1 in 5 AI-generated code samples still hallucinate references to libraries or APIs that don't exist, and the errors that are left skew logical and semantic rather than syntactic, north of 60% for models like DeepSeek-Coder and QwenCoder, most broken code still compiles fine, GPT-4 scored 67% on HumanEval's Python coding benchmark.
"In the Land of AI Agents, the Verifiers Are King" - Tariq Shaukat (Sonar), Day 3, 9:25 AM.
The centerpiece stat set: AI-assisted development produces a 3-5x velocity spike that disappears within three months. What stays behind: +30% more security, maintainability, and reliability issues, and +41% more code complexity. The sugar rush ends; the debt compounds. (The accuracy-vs-task-horizon chart and the RAND citation already ran in the Day 3/4 recap, not repeated here.)
Source: He, Miller, Agarwal, Kästner, and Vasilescu, "Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projects", accepted at MSR 2026. Difference-in-Differences methodology, matching the slide's fine-print citation. Kästner is CMU faculty (Software and Societal Systems Department), which is the actual Carnegie Mellon connection.
The frontier models still ship problematic code, per Sonar's own measurements:
| Model | Avg complexity | Bugs/MLOC | Security issues/MLOC |
|---|---|---|---|
| Gemini 3.1 Pro High | 158.4 | 614 | 214 |
| Claude Opus 4.7 Thinking | 171.2 | 803 | 291 |
| GPT-5.5 Medium | 151.4 | 504 | 68 |
Then the two stats that turn code quality from a virtue into a line item. Teams with multi-layered verification in place see 44% fewer AI-derived production outages. And agents consume 8% fewer input and output tokens on cleaner repositories, "Agents care about codebase quality" was the actual slide title. Tech debt now has a per-token price attached to it.
The other lever is context. Agents need to be verification-aware, which means feeding them repository-specific architectural awareness, semantic navigation, dependency guidance, coding standards and guardrails, and the intended architecture, provided dynamically or embedded. Sonar's number: a 30% reduction in tokens consumed when the agent gets targeted context and constraints instead of wandering the repo. This is the direct connective thread to Atlassian's pitch from the Day 1 workshop: their agent platform sells exactly this, context pulled from the Confluence and Jira content you already maintain.
Sonar's proposed architecture, the Agent Centric Development Cycle, is two nested loops. The inner agentic loop: SonarQube MCP supplies codebase context and guardrails, a coding agent generates against explicit specs and eval criteria, and in-loop verification (SonarVortex) scans code in real time as it is produced. Pass, and the PR flow agent takes over. Fail, and you drop into the outer CI verification loop: code review, multi-layered verification, a fix agent, and a quality gate that makes the final pass/fail call before anything deploys.
The durable idea, whatever you think of the product attached to it, is the verification split: algorithmic verification for deterministic properties (syntax and style, data and control flows, structure, dependency analysis, taint analysis, secrets) and agentic verification for the judgment calls (intent, business logic, performance, emergent anomalies, deep semantics, state and environment). Deterministic checks are cheap and repeatable. Save the model-powered judgment for problems that actually need it.
"Research to Reality" - Benoit Schillings (Google DeepMind), Day 3, 10:05 AM.
Schillings opened with the origin story: his code-review tool, Project Pitchfork, grew out of Google X and was originally built just to match reviewer edits against codebase evolutions, well before anyone anticipated LLMs would scale this fast. He framed it as the latest turn of a familiar cycle of skepticism: resisting 68000 assembly, eyeing early compilers with suspicion, dismissing GC and interpreted languages, and now watching that same skepticism turn even the most cynical purist into an avid ML coder.
Then he framed sixty years of software as three eras, each defined by its bottleneck:
| Phase | Core bottleneck | Human role |
|---|---|---|
| Assembly/C++ Era | Silicon and memory constraints | Manual optimization, extreme precision |
| Modern/Cloud Era | Human cognitive load | Designing for modularity and maintainability |
| Frontier AI Era | Verifiability and intent | Inductive thinking, architecture, security guardrails |
The bottleneck was hardware, then it was us, and now it is trust. Schillings broke that Frontier AI Era role into three concrete pieces: active guardrails (secure-by-default architecture instead of post-generation patching), inductive architecture (teaching models to reason about whole systems, not just localized output), and evals refactoring (moving from binary pass/fail unit tests to continuous, nuanced scoring).
Then the slide that should be mandatory viewing in every planning meeting, DeepMind's own assessment of where AI software engineering actually stands: super-human syntax generation, 95%. Local problem solving and tasks, 70%. Multi-step codebase planning, 45%. Architectural system decisions, 25%. Their words on the slide: "Models generate localized code rapidly but struggle to form cohesive systems, design complex architectures, and predict overall security flaws." That is the most honest capability curve any frontier lab showed all week, and it came from the lab with the least to sell in the room.
Why is coding the domain where AI moves fastest? Schillings' answer: because code is the one place models get free, strict, infinite feedback. Coding tasks live in structured, deterministic environments that compile and execute. Compilers and test runs provide continuous verification. That means self-play works, AlphaZero-style: the model proposes, the environment judges, no human labeling required, loop forever. He called it the infinite sandbox.
Prose has no equivalent, nobody's essay compiles. This is why the capability percentages above will keep climbing regardless of whether anything else in AI improves, and it matters more than any demo shown all week: it is a structural argument, not a product roadmap.
This is one of the talks that earned the deeper treatment for a real reason: actual technical content, not hype. Fable was enthusiasm and marketing dressed up as a keynote. This one had something real underneath it. It's the same reason Sonar got its own section above, and the same reason Homa Protocol is getting an entire post of its own instead of a paragraph here.
I have read the assembly output, actual ML code, decoded binary off old systems like a C64 or an Amiga 600, back when that was just how you learned how computers actually worked. Schillings is right that nobody reads it anymore, and mostly that's fine. But sometimes something just doesn't work, and if you never learned the lower level, you don't have the tools to find out why. That skill is going away, justifiably, most people genuinely don't need it. The real question is how far you can abstract things before that stops being true. Look at the ladder so far: binary, then machine language, then assembly, then C, then Python. Whatever comes after Python is probably models and harnesses generating the Python for you, and after that, whole software factories generating the harnesses. For most people, probably pretty far. I don't know exactly where the line is.
The future-engineer disciplines on that slide, correctness, security boundaries, constraints, read like an SRE job description to me, and it's both validating and ominous at once. You need to not trust everything to be right. That's the same instinct as knowing how the lower levels work, a hacker mentality, taking things apart just to see how they actually function instead of trusting the label on the box. SRE and hacker are more related than people give them credit for.
That instinct is exactly what Datadog's Three Creeps talk from the Day 1/2 recap was describing from a different angle, Token Creep, Model Drift, Uncached Calls, three ways an AI-driven system quietly does something other than what you assumed, and you only catch it if you're the kind of person who checks instead of trusts. Tag diligently, monitor against real thresholds, that's the SRE version of taking the thing apart to see how it works. Same discipline DeepMind is describing as the future of the job, just from the cost-and-monitoring side of the stack instead of the code-architecture side.
The abstraction ladder: binary, machine language, assembly, C, Python. Each rung made the one below invisible to most working programmers. The question Schillings put on stage is what the next rung looks like. Human developers need languages optimized for human readability. AI models do not. So: do we still need standard languages, or do we pivot to mathematically precise, highly rigorous specification engines like Rust? Or build something genuinely new, a language optimized for LLM execution and reasoning that humans never need to read at all? (The Day 3/4 recap already teased this via the GibberLink tangent, there's room to actually unpack it here.) He paired it with a second foundation, multimodal design reasoning: software architecture is visual and spatial, not textual, and multimodal models like Gemini are starting to evaluate flowcharts and interface layouts directly against synthesized code.
Schillings' own bridge to what came next: "Universal Execution Engines." Code is becoming a raw material for wider discovery, agents construct software toolboxes on the fly, run complex scientific simulations, and immediately discard the program once a solution is reached, disposable code as scaffolding for something else entirely.
The closing slide read "The gold we cannot see," and for once a keynote closing slide earned it.
Hearing him say it out loud made it feel real instead of theoretical. I'd already been wondering the same thing on my own: are these models eventually going to generate their own languages, not just write in the ones we already have? There's plenty of generative AI already, but would a model-native language actually be an improvement, or just novelty? Do we need to rethink programming languages at some point because of this? I don't know the answer, but I'm glad I'm not the only one who's been thinking about it.
The physical sciences material, computational chemistry, complex biology, was the biggest left turn of the week from what was otherwise a software conference. One thing worth flagging: most of what got shown runs inside harnesses with real safeguards built in. Open-source versions of the same capability down the road might not have those same guardrails. Too early to say what that actually means.
Leftover robotics photos. The Day 3/4 recap gave Dyna, Pathak, Unitree, and the Skydio drone real narrative treatment (not just teasers), so this section is genuinely just the leftover slide/photo material, not additional story. Dyna: 14 more photos beyond the teaser already used. Pathak: 8 more photos beyond the title-slide teaser, including the tradeoff-matrix slides (Simulation vs. Video vs. SUMI vs. Teleop) and the Classical Robotics vs. Skild AI architecture comparison.
"The Unreasonable Effectiveness of Separating the Task from the Model" - DSPy, Day 4, 9:40 AM.
DSPy in three paragraphs, as promised. The pitch: stop hand-crafting prompts. Declare the task as a structured signature and let DSPy optimize the prompts and pipelines against your data, "programming, not prompting." It is open source (MIT license, Stanford NLP origins), runs against any LLM backend, and the current stable line is 2.x with the MIPROv2 optimizer. Production users include Shopify, Dropbox, JetBlue, Moody's, Replit, and AWS.
The commercial subtext: Databricks hired DSPy creator Omar Khattab as a Research Scientist to deepen their investment and pull DSPy users toward their platform, confirmed by Khattab himself. Khattab was not on stage; the presenters were, and the talk played like a pitch because it was one.
The idea survives the delivery. Signatures-over-prompts is a real pattern worth knowing even if you never touch Databricks. pip install dspy and judge for yourself: dspy.ai.