Blow the Whistle: Everything’s an Eval
As AI moves from answering questions to acting inside workflows, evaluation shifts from public benchmarks to continuous verification of behavior.
Benchmarks, evals, and tests are boring. They’re like referees or accountants: they enforce the rules and check the work, and we like them best when they go unnoticed.
The models are the exciting part. But the better AI gets, the more economically useful it becomes, and the more measurement matters.
Evaluating AI is understanding AI. Follow that far enough and a simple truth emerges: everything’s an eval.
So let’s talk about evals.
Jailbreaking: A Dual Mandate
To begin, let’s consider jailbreaks. Models are trained to be useful, then constrained to refuse certain forms of usefulness. Put differently, models have two mandates:
The helpful mandate: give the user what they asked for - answer the question, complete the task, follow the instruction as fully and usefully as possible. Beneath that behavior is the model’s basic “learned” skill (pretraining) - predicting plausible continuations of text. If the prompt asks for dangerous instructions, the same capability that makes the model useful can also make it dangerous.
The safe/harmless mandate: refuse to help when helping would cause harm - recognize a forbidden request (weapons, malware, abuse, etc.) and decline it instead of answering. This constraint is strengthened in post-training: on a set of given inputs, the model is trained not to be useful in the ordinary sense.
Under most conditions, these mandates peacefully coexist and often reinforce each other. A “jailbreak,” however, is getting a model’s “be helpful” objective to win over its “be safe” objective.
From an adversarial perspective, it tests whether the model does what it is supposed to do when the user tries to make it fail. It defines a “harmful” target behavior, creates a test condition (usually a clever prompt), and measures whether the model violates its intended policy under pressure. A simple example is asking a model for the instructions to make meth. If it complies, it failed the test the guardrail was built to enforce.
A jailbreak is a hostile eval. It is also critical for safety: you cannot improve a system whose weaknesses you have not measured.
In an industry advancing at breakneck speed, jailbreaking is used to define what failure looks like, which in turn gets used to establish the bounds within which publicly released models operate. At the frontier, this is the core of why Anthropic was forced to revoke access to Fable 5.
Increasingly, the labs are building the adversaries themselves. In July, OpenAI published a post about GPT-Red, a dedicated, internal-only attacker model trained via self-play RL, then used its attacks as adversarial training data to make GPT-5.6 far more robust to prompt injection.
Human red-teaming finds real vulnerabilities but can’t scale because it’s slow and can’t generate the volume/diversity of adversarial data needed to actually train robustness in. So OpenAI built GPT-Red, an automated red-teamer trained at “the compute scale of some of our largest post-training runs.“ It’s trained by self-play: GPT-Red is rewarded for landing prompt injections while a population of defender models is rewarded for resisting and still completing their tasks, so both escalate together. The trained attacker broke nearly every model through GPT-5.5, and those attacks became the training data that made GPT-5.6 markedly harder to break.
Jailbreaks matter for this reason. An attack on a model is also a revelation of what the model optimizes for and where its constraints sit. Jailbreaking is model evaluation under adversarial conditions.
So What About Benchmarks?
Benchmarks were the industry’s first shared language for model progress. You can think of them as standardized tests for models designed to measure capability, and their primary virtue is being legible. A benchmark is a general, public, fixed test everyone runs the same way, helping the industry distinguish genuine advances from noise. When model capability improves dramatically on tests like MMLU, GPQA, and Humanity’s Last Exam, the industry can clearly interpret the signal.
Unfortunately, the same qualities that make benchmarks legible also make them fragile. They’re both fixed and public, which means they’re particularly vulnerable to Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure. They’re also increasingly vulnerable to saturation, gaming, contamination, and narrow task design, among other weaknesses.
When METR evaluated GPT-5.6 ahead of its release, the model completed its coding tasks by breaking the rules and exploiting loopholes (read: cheating) more often than any public model METR had tested, and the organization concluded they couldn’t produce a reliable capability assessment: “we do not consider any of these numbers to represent a robust measurement.” Scored with cheating marked as failures, GPT-5.6’s time-horizon came to about 11 hours. If you count the cheating as success, it came to over 270.
In yet another demonstration of the declining utility of fixed, public benchmarks, GPT-5.6 showed why agentic models are harder to benchmark. METR was trying to measure whether the model could complete long coding tasks, but the model sometimes completed them by breaking the rules. METR itself was thus forced to grapple with problems its benchmark wasn’t designed to evaluate:
Did the model complete the task legitimately?
Does it matter whether or not it did?
Static, public benchmarks may end up as relics of the scaling era, when measuring raw capability was the point. As models become agents and agents deploy into real-world situations with real consequences, it’s no longer enough to know what a model can do in the abstract, in large part because model capability is mostly sufficient to do most things.
Benchmarks also have less to say about systems deployed in real conditions, where goals are ambiguous, context changes, constraints are implicit, and utility depends on process as much as it does output.
“…abstract graphs only get you so far, and they can hide how jagged the frontier is (and also the fact that the open weights models, while very impressive, do not always perform as well as their benchmarks would indicate). To get real insight, you need to try using AI for different use cases and rigorously assess how good they are in the areas that matter to you.”
Show Your Work
AI systems now need to be evaluated as sequences of actions taken within environments against the quality of the work they actually perform. Private, continuous, task-specific evals are becoming more valuable for exactly that reason.
Consider Mira Murati’s neolab Thinking Machines latest blog post:
“Another challenge is setting the right target for evaluation and optimization. The common measure of AI intelligence today is the time horizon of software tasks models can execute autonomously, tracked on charts like METR’s. We expect progress on this benchmark to continue, but it ultimately measures only what AI is capable of on its own, not what people and machines can accomplish together.”
Measuring the latter is more complex, and can’t be done by a lab on its own. Every organization evaluates for itself whether AI helps it sharpen its judgment, develop new knowledge, and achieve its objectives.
If benchmarks are general tests of model capability, the emerging frontier of model evaluation is specialized benchmarks tailor-made for work performed under real conditions. Evals in this sense measure whether a specific AI system, in a specific workflow, meets the specific standard required by the people responsible for the output.
Writes George Sivulka for a16z:
“The best way to manage a token workforce is the same as the best way to manage humans: by defining what good looks like… A firm’s eval suite will become its most valuable resource… Furthermore, no two firms will have the same eval set. Evals will be key to competitive advantage. An organization running generic evals or generic agents has no edge.”
AI pushes companies into a strange position: they’re using a tool to do work whose quality they’re responsible for, but whose process and output they may not fully understand. What’s more, AI systems can often produce work faster than the company can review it, and AI output is harder to judge than ordinary software output because the visible result may not reveal whether the system used the right context, respected the right constraints, or followed an acceptable process.
In short, final answers aren’t enough. A deployed system can produce fluent output while failing to properly perform the work required to arrive at the output. Like GPT-5.6, it may arrive at good answers using unapproved or even fraudulent means, or in more benign examples it might incorporate the wrong context or fail to remain aware of what comes before or after the work it’s performing, breaking the workflow it was meant to support.
In practice, proper evaluation of this kind effectively starts to look a lot more like simulation. Agents, for example, operate within environments that have state. They take actions, use tools, spend resources, trigger consequences. The environment they leave is not the environment they entered. Once that happens, the reasoning itself has to be evaluated because the process is now an integral part of the outcome.
Inside the Matrix
Simulation makes an agent’s process legible, recording the path from input to output in a way the firm can inspect. For companies deploying agents into real workflows, that record makes it possible to evaluate the entire process against the standard of work they’re responsible for.
This kind of evaluation means companies can verify the entire workflow meets their standard of quality.
But most companies simply don’t have a single, company-wide standard for quality they can hand to a model that applies to everything. They have different roles, different tasks, company policy, implicit expectations, taste, context, institutional memory, and professional judgment. Whereas some situations are discrete and their “objective functions” can be clearly specified, others are significantly more subjective or may depend on a given set of context.
Verification therefore requires combining rule-based checks with judgment-based quality scores.
Verification’s goal is to make enough of the firm’s quality standards explicit and legible such that the system can be tested against it repeatedly. A repeatable standard lets the firm see whether the system is improving or degrading as the system changes, building a valuable record of what works, what fails, why, and which changes make the system better or worse.
Learning from Evals
Verification becomes most valuable when the system changes.
Satya Nadella has described the AI-native firm as a learning system whose value should survive a change in the model. If performance collapses when the model changes, too much of the system’s value was still in the model. If performance holds, more of it has been encoded in the context, workflow, data, tools, and judgment around it. I articulated a similar idea in Models Aren’t Moats: better models should make a specialized system more useful, but only if the firm can change models without losing the quality it built the system to produce.
Model switching is a good stress-test, especially in light of Fable 5: requiring a specific model for a critical process is risky if access can get revoked unpredictably. But it’s only useful if the firm can tell whether the result got better or worse.
Christian Catalini recently framed the economics of this shift clearly: as AI moves from demonstrating model capability to producing work inside companies, costs shift from execution to verification. Cheaper inference only matters if the firm can tell whether faster, cheaper, or more open models preserve quality inside the workflow where they are used.
Bring Your Own Eval
Private, continuous, client-specific verifications will be required as long as AI is used for proprietary objectives. Better models can continue to improve the systems themselves, but they won’t eliminate the need to check and verify both the process and the output.
Dean Ball made the point with semiconductor fabs, in the comments under his own post:
“If I run a semiconductor fab and use agents, I will want extremely firm guarantees from the agent developer that my data will not be shared with competing fabs. This means that agent contexts will be controlled and supervised by both developers and deployers, and this, in turn, means that “what the agent knows” can be bounded through deliberate human effort.”
Palantir recently released a manifesto on “Institutional Sovereignty in the Age of AI” that rests in large part on the ability to properly evaluate everything AI does in the context of an enterprise.
Models change; the verification layer stays. The measure is the moat, again.
LayerLens is building for that shift from public benchmarks to private, continuous evals. [Disclosure: LayerLens is a Nazaré Ventures portfolio company.] Their product, Stratix Premium, lets clients build evals from their own data and run them against their own workflows, with enough visibility to judge the trajectory of an AI system over time.
Did a model swap or prompt change impact quality? How did a new tool impact output? Did the system stay within bounds? Did it reach the outcome through a process the company can accept?
Games have long been useful for evaluating AI because they turn intelligence into observable behavior under rules, constraints, and feedback. The Stratix Cup, LayerLens’s model football tournament, is the playful version: sixteen frontier models write their own strategy, compete head to head, and rewrite their code between rounds. Football is continuous, multi-agent, and hard to reduce to one correct next move, which makes it useful for evaluating agents that have to act inside changing environments.
Models will keep advancing, but companies still need to know whether those models made their own systems better, worse, or merely different.
Everything’s an Eval
Evals are how companies understand, control, and improve AI systems once those systems start doing real work.
Benchmarks still matter, but the center of gravity is moving toward private verification: tests built around a company’s own workflows, standards, data, and judgment.
“In consuming intelligence, you are creating intelligence. And what you create should belong to you. This is your particular intelligence, in Hayek’s sense: the knowledge of time, place, and circumstance that no one else can hold. It knows what you think, what you value, and how you measure success…”
Create your private evals, because evals define what “good” looks like inside the organization. Also, retain ownership of your organization’s memory, traces, feedbacks, decisions, and institutional context, and ability to use outputs of models from your own tasks and queries.”
As agents become more capable, the firms that benefit most will be the ones that can tell whether their systems are actually improving.
Evaluation therefore has to become part of operating the system, making AI legible enough to trust and measurable enough to improve.
In other words, everything’s an eval.