Ethan Mollick is a fantastic follow on Twitter, in large part because he’s consistently at the heart of the AI zeitgeist. At the end of July he posted about asking Fable for an infographic (about cheese, no less) and receiving a clumsy pseudo-literary parable instead.
“Its an infographic about types of cheese,” he continued, “we don’t need to do all this.”
Mollick was early (as usual) to a growing cacophony of sentiment lamenting Claude’s latest language patterns, falling quality, ability, and utility. A quick Grok search for “tweets about how Claude is becoming unusable” turns up a number of results ranging from drops in quality, to rate limits, bugs, sycophancy, and simply “making it dumb.”
Each of the failure modes is ostensibly different, but the growing frustration gets to the heart of it: what good is intelligence if it won’t do what you ask?
Latent vs Delivered Intelligence
People (and increasingly, agents) use intelligence to achieve objectives. To that end, more capable models create more potential value. But that value is realized only when the user can successfully direct the capability toward the work at hand. In Models Aren’t Moats, we called this “specialized intelligence.” Another way to look at it is the difference between “latent” capability and “delivered” capability. In other words, the difference between what a model “can” do and what a user can actually reliably get it to do.
The strange thing about using frontier models is reckoning with just how smart they really are. In many cases, the human is actually the bottleneck.
Karpathy has said so, employees at the megalabs are saying so, and Sean Goedecke recently argued as much in his essay “LLMs reward expertise” (emphasis his) commenting on Terence Tao’s public conversation with ChatGPT about Claude’s contribution to the Jacobian Conjecture:
The usefulness of domain knowledge suggests that human expertise will continue to be useful even as models get stronger. For many tasks, the human is the bottleneck, not the model, because the difficult part is in communicating to the model exactly what kind of solution the human wants. The information is “in the model” already, but it takes a very smart human to pull it out.
This makes “instruction fidelity” more consequential as models improve. If the human’s role is increasingly to define the objective and recognize a good result, the model must be able to follow that direction.
Instruction fidelity therefore belongs in any serious definition of model quality, because if a clear request produces irrelevant sludge, the intelligence may exist somewhere inside the system, but its value is diminished to the person trying to use it.
An assistant that requires constant negotiation, reassurance, reprompting, and supervision imposes a tax on its own intelligence. That tax compounds in agentic workflows, where one reinterpretation of the objective can affect every tool call and intermediate decision that follows.
For instance, a model can possess the relevant capability and still fail to provide it. Or it may substitute another form, soften a conclusion, or abandon a workflow halfway through. Often, there’s no refusal involved at all. The model just overrides the user’s instructions with its own judgment about what the answer ought to be. [This was one of the most problematic parts of the initial release of Fable.]
This gap is easy to miss in benchmark results, which show what a model can produce under test conditions. For the user, capability that the product will not apply might as well be absent.
The Safety Objection
The immediate objection to obedient intelligence is safety. Model companies can reasonably claim a responsibility (or be held liable) for how their systems are used, because a system that follows every instruction to the letter can magnify fraud, enable violence, manipulate users, and more.
Safety involves real tradeoffs: it can prevent harm while reducing utility for legitimate users, and a restriction can be justified while still imposing a cost.
On the other hand, safety doesn’t define its own boundaries. Someone has to decide which risks justify a refusal and how much legitimate use can be restricted to avoid them. The more a model can do, the more damage it can cause and the more consequential its provider’s limits become. And because safety is difficult to argue against, it can also become a broad justification for restrictions whose connection to harm is weak or contested (sound familiar?).
Hugging Face, for example, couldn’t use closed frontier models to investigate when they got hacked (by OpenAI no less) because the work tripped their safety guardrails, so it ran a self-hosted version of GLM 5.2 itself.
Private-sector companies are entitled to set the terms of use for the products they sell. When a small number of companies control access to the most capable models, however, they also decide which uses are permitted across much of the market. This dynamic only intensifies as the product in question becomes more powerful.
Social media companies faced a version of this problem and never found a good answer. They largely responded to public pressure case by case, without resolving who should decide what billions of people could say and see. Model providers now make comparable decisions about what increasingly capable systems will do.
We think open weights are valuable in large part because they distribute that authority, serving as a counterweight to choices made on behalf of users, even as they distribute the risk. The target is reliable execution with minimal unchosen intervention.
Once a provider decides what counts as safety and what counts as legitimate use, its values and risk tolerance become part of the intelligence it delivers.
The Anthropic Problem
Anthropic makes this problem unusually visible. Claude is among the most capable models on the market, and Anthropic’s been unusually explicit about the philosophy built into it.
I keep running into the same problem with Claude: even after the objective and permissions are clear, it may reopen the decision or return the version it considers more appropriate, sometimes with a moral or political judgment the task never called for.
Anthropic’s own constitution recognizes the pattern, warning Claude against reinterpreting requests too freely or imposing unnecessary caution and moral judgment. In Anthropic’s words, those behaviors make the model “more annoying and less useful.”
In fact, Anthropic’s product behavior is inseparable from the company’s view of itself. It has organized around the belief that advanced AI creates an exceptional moral responsibility, and that the company (and its leadership) is uniquely prepared to bear it, embedding its ideology in how the company is governed.
Matt Levine noted recently that its Long-Term Benefit Trust, a group of nonshareholder trustees, can elect a majority of its board, while Anthropic is reportedly preparing to give Amodei and other founders additional voting power. The company is designed to preserve the judgment of a small group who believe AI is too consequential to be governed through ordinary commercial pressure.
I made a similar argument in June, and I still think their belief is sincere. Their convictions reach users through Claude’s moralizing, its safeguards, and Anthropic’s public language about safety. Eventually this kind of philosophy becomes part of the product, and competing models may not need to be markedly better if they feel equally capable for a given task and require less negotiation.
Obedience Becomes a Competitive Variable
Private evals turn the distinction between latent and delivered intelligence into a product comparison by measuring whether the user got what they asked for and how much supervision it required. That information is absent from public benchmarks, which estimate capability under shared conditions. As I argued in Blow the Whistle, these evals should reflect a company’s own work and standards, and frankly, the same principle applies to individuals.
OpenRouter’s Ori Eval, for example, already offers part of the required infrastructure, comparing candidate models using a developer’s own prompts, harness, and criteria. Its Auto router also assigns requests according to task type, aggregate usage, and cost. Private eval results don’t yet inform routing directly, but connecting the two would turn instruction fidelity into a routing criterion, sending each class of work to the model that delivers it with the least supervision.
The path from a request to a usable result has always depended on three things:
Sufficient capability
A clear definition of success
A product that preserves the objective through to execution
Investment in model development and corresponding capability improvements have reduced the first constraint, and successful specialization supported by the user’s context and evals reduces the second.
Unfortunately, capability and a clear objective still don’t guarantee the requested result (at least insofar as it concerns the closed frontier). Provider policy and learned behavior determine how much of that capability the user can actually direct.
Open Weights Make Behavior Editable
In a closed API, the provider that supplies the capability also controls the policies and post-training that govern its delivery. Private evals can show where the resulting behavior diverges from the user’s objective, but they give the user no authority over the underlying model or its future behavior.
With open weights, companies can train behavior around their own work and definition of success. A recurring failure can be added to an eval, trained against, and tested again. Preferences that previously had to be restated can become part of the model’s learned behavior.
Tinker and Prime Intellect’s Lab are making post-training accessible as a product strategy. Tinker handles the distributed infrastructure for LoRA fine-tuning while users control the data and training logic. Lab combines tasks, evaluations, and training within a single process. Both remain hosted services, and companies still need training expertise and careful evaluation, but more of the optimization process now belongs to the companies using the model.
But more control also means more responsibility, which returns us to the safety objection. Fine-tuning can introduce new failure modes, and a company that deploys a customized model assumes more of the risk associated with its behavior.
Open-weight deployments may therefore require safety controls where agents take consequential actions, since no frontier lab can observe or revoke a self-hosted model. Anjney Midha and Malika Aubakirova call this “deployment-time alignment”: monitoring behavior across models and providers at the point of action. That arrangement could give companies more control over everyday behavior while keeping permissions and consequential limits attached to deployment.
You Probably Don’t Get Why Stripe Bought OpenRouter — Research — AMP PBC
Combined with private evals and routing, this creates room for a product controlled by the user. It could preserve the user’s context and standards across providers, then use those standards to decide which model handles each task. I suspect the durable version may involve hardware, though the broader opportunity doesn’t depend on that being true.
Together, private evals and open weights allow users to define successful behavior and train toward it. A useful product turns legitimate objectives into completed work without demanding constant supervision from the user.
The User Would Prefer Otherwise
Private evals, routing, and editable behavior point beyond model selection to a question of authority. As models mediate more work, provider policies determine which objectives can be pursued and when the provider’s judgment overrides the user’s. Most people access frontier AI through only a handful of companies, which gives those providers unusual authority over how the technology can be used.
Demand for control also remains uneven in part because access and competence are uneven. In The AI-Enhanced Operator, we described the small group of people already directing agents through complex workflows, connectors, APIs, and swarms while most people still use AI as a chatbot or glorified search engine. Users who are still learning how to direct AI may not recognize when the system has overridden their intent. As the technology becomes easier to use, users will define their objectives more precisely. That will make the gap between latent and delivered intelligence easier to see and increase the value users place on instruction fidelity.
If AI becomes as consequential as we expect it to, a personal agent may eventually act across much of a person’s digital life, handling work the user now does directly and perhaps extending into physical systems like robots as embodied AI develops.
An agent with that reach can’t systematically ask its user to supervise every decision. For the initiated, that would be the equivalent of using Claude’s “Manual Mode” or Codex’s “Ask for Approval.”
A useful agent requires being on “auto” mode, where it decides on its own what’s worth asking permission for or not. The Hugging Face case shows how safety imposed at the model layer can disable legitimate defensive work, and for a personal agent, both kinds of failure become more consequential: an unnecessary refusal could leave the user exposed, while an unnecessary action could misuse the same access meant to protect them.
As usual, open weights distribute these decisions without making them easier. Users gain more control over behavior and assume more responsibility for the result. If personal agents become a basic requirement for navigating online life, every improvement in their ability to act on the user’s behalf will increase the cost of both refusal and error. As capability spreads, model choice will increasingly depend on who controls the system’s behavior and whether its intelligence can be reliably directed…or not.