I need to eat (some of) my words
I wrote a post 2 years ago arguing that AI is a Bubble. AI was already adding some value, particularly for coding, but its practical capabilities didn’t justify all the money pouring in. We’d just published research, now licensed by Google & Nvidia, evaluating AI agents on real-world tasks such as booking meetings and sending emails. The results showed they could be trusted with…almost nothing!
Software is built up of long chains of 100% reliable (deterministic) functions. Chain together 100% 1000 times in a row and it’s still 100% reliable. But AI isn’t deterministic and can’t be chained together in the same way, so errors compound through the system. Chain together 99% reliability 1000 times in a row, and you get nothing.
I argued that execs and investors pouring money into AI didn’t understand this. Instead, they were going nuts over flashy demo videos with cherry-picked examples, and a bubble was inflating. I also claimed that big AI companies, which well understood the reality, had a strong incentive to keep inflating the bubble.
Lots of concern lately about Claude agents taking everyone’s jobs has prompted me to revisit my AI = Bubble claim. This post covers:
What I got right (most of it)
What I got wrong (coding agents)
Why coding agents are a special case (the compiler)
How coding agents provide a blueprint for the future (of AI)
What I got right
I argued that people wouldn’t trust AI agents to do most tasks for them. Outside of coding, there are very few real-world AI agents with serious traction. ChatGPT and Claude have grown in popularity for research and writing, but almost all usage still has a human checking the output (i.e. in the loop). Why do they need a human checking? Because they don’t trust the AI.
I predicted that most companies pouring money into AI would get no RoI. This has since been covered widely, most famously in MIT’s damning 2025 report. The pressure to invest in AI hasn’t come from well-thought-out business cases, but instead from boards and CEOs with FOMO. They all know AI is the answer, but they still don’t know what the question is.
The companies that most understand AI’s realistic capabilities have had a huge incentive to inflate the bubble. See the barrage of news articles about financial engineering between tech companies. On our team, we use the AI tools for broadly the same tasks as we did 2 years ago: meeting notes and coding.
What I got wrong - coding agents
Coding agents turned out far better than I expected. In particular, this claim didn’t age well:
“The people who are most excited about Copilot - who think it will replace software engineers - are the people who’ve never used it. Often they’re the people who can’t code at all”
In the last 3 months, coding agents have got so good that many senior engineers no longer write code at all. Instead, they just instruct agents in English to write the code, then review a summary of their output. Even previous sceptics, like Karpathy (and myself), have changed their tune.
Why did I get this wrong? Coding agents still have the problem of compounding errors that I wrote about before. They make mistakes when they write code; they are not 100% reliable. However, they can now correct their mistakes before sending their output to people. I didn’t expect this, and it’s made a huge difference.
When agents make mistakes writing code, they’re caught by the interpreter. This rejects the code, then throws an error along with a step-by-step explanation for what went wrong (the “stacktrace”). The agent can analyse this explanation, then correct itself and try again. Being able to self-correct has proved hugely helpful, both during AI training and in real-world deployment. This process now works so well that engineers rarely need to intervene by writing code themselves.
This trend will continue. While no one knows for sure what will happen, there’s a strong chance that AI will displace many software engineers in the next few years.
Coding agents are a special case…
If coding agents are so good, why do no other AI agents have real traction? Other agents don’t have a compiler and cannot self-correct in the same way. If they send an email to the wrong person, they don’t find out about it when a human complains that it’s messed up. By this point, it’s too late - the damage has been done.
Currently, the only way to mitigate this risk is for agents to ask for permission at an annoyingly high frequency. Here’s a concrete example:
Human: “Book a meeting with Ollie for 9 on Tuesday”
AI: “Which Ollie do you mean out of these 10 options from your contacts…”
Human: “Ollie Styles, he’s the Ollie I last met with”
AI: “Got it! And do you mean 9 am or 9 pm?”
Human: “9 am obviously”
AI: “Ok just checking, sorry! And how long do you want to meet for?”
Human: “Screw it, I’m just going to book the meeting myself…”
You’ve got a lot of context about what you want that’s not easily available to the AI. The time it takes for the AI to collect this context is usually not worth it compared to just doing the job yourself. While AI can learn about your preferences, these are often hard to articulate. Even if you could articulate your preferences perfectly, they change over time. If the AI can’t capture your preferences clearly and quickly enough, it’ll keep making mistakes and you won’t trust it to do the job.
This issue gets even worse for organisation-level AI, which is what most businesses need. For these to work well, many people’s preferences need to be aligned for the AI and kept aligned on an ongoing basis. This is particularly hard when the “right output” is subjective, which applies to most domains outside coding. If people can’t agree on what they want, how can we expect an AI to fulfil their wishes?1
…and a blueprint for the future
We need to make it easier for people to transmit their preferences to AI at high frequency. This will allow AI to spot errors more quickly and self-correct, thereby acting more like a coding agent. In other words, we need to make people more like compilers.
People are pretty far from being compilers right now, and I don’t see a silver bullet. Speeding up the transfer of subjective, hard-to-articulate preferences between humans and AI will be a gradual process. It’ll be even harder in business use cases where multiple people’s preferences need to be coordinated.
This is already reflected in the shifting role of an AI engineer. Breaking a problem down into small steps used to be critical because the models weren’t smart enough to solve bigger problems. But models have improved quickly and now usually follow precise instructions reliably, so the main challenge is making those instructions sufficiently precise. The AI engineer’s most important role now is transforming vague human preferences into sufficiently precise instructions for the AI (i.e. evals).
So, is AI still a bubble?
Yes. But by “bubble”, I don’t mean there’s been no value added. We’re already seeing huge productivity gains among software engineers and it’s having a profound impact on their role. However, coding agents are a special case, so this impact hasn’t spread to most sectors of the economy.
There are still hard questions about reliability and how to communicate what we want from our AI sufficiently clearly. Until we solve these, the productivity gains will be limited. In the meantime, tech companies will try to keep the party going by financial engineering and adding “.ai” to their domains to justify all the money pouring in.
When we crack these problems, AI’s economic value is going to be immense. But as I wrote 2 years ago, it’s likely to take decades, not years, to see a tangible impact across most sectors of the economy. In the meantime, it’ll be ironic if the bulk of the jobs displaced by AI are the engineers who built it.
We’ve written more about building AI when “correct” is subjective at https://artanis.substack.com/p/policy-the-missing-core-of-the-ai
The compiler is a revealing exception because it turns probabilistic output into an environment with cheap, frequent verification. That suggests a broader rule: agents become useful where the work produces its own checkable residue—tests, logs, transactions, reversals, or measured task outcomes. The model may generalize, but the verifier and feedback loop are local. Coding is not merely the first strong agent market; it is a blueprint for where other agent markets can become reliable.
The 'bubble' narrative is too simplistic. It's a massive capital reallocation. Some will be stranded assets, but the winners will define the next decade. Great perspective on the value traps.