On the Validation of State-of-the-Art Models

On June 2, 2026, the President of the United States signed the Executive Order Promoting Advanced Artificial Intelligence Innovation and Security. The text begins by stating that the administration refuses to stifle innovation through excessive regulation. It then establishes a federal mechanism for pre-release evaluation of the cyber capabilities of state-of-the-art models. The framework was to be completed within sixty days. The White House has indicated that it has been completed, but has not published it. The standards are unknown, there is no requirement to publish the results, and no one is required to disclose whether they participated.

The mechanism has three parts: a classified benchmarking process through which the government measures a model’s cyber capabilities, with the decision resting with the director of the National Security Agency; a voluntary framework through which developers grant the government access to models up to thirty days before release to trusted partners; and a provision that excludes mandatory licenses, pre-authorizations, or permits.

This is useful context for understanding the ongoing debate, in which four positions are being weighed. Each stems from a valid concern, yet each leaves the same problem unresolved.

The first calls for a slowdown. On September 8, Jacob Coxon, a researcher at Anthropic, resigned, publicly accusing the labs of racing toward self-improving superintelligence without acting responsibly. On July 28, the Pacing the Frontier appeal was published, officially endorsed by OpenAI and Anthropic and signed to date by over 1,300 employees of cutting-edge labs, to put in place the necessary tools to slow down progress at the frontier, if needed. But slowing down does not yield new information, whether the slowdown is chosen or imposed: in June, following a government directive, Anthropic suspended access to its most advanced models for nearly three weeks. When access was restored, what additional information did decision-makers have to determine whether to adopt them?

The second proposal calls for mandatory regulation. On June 10, Dario Amodei wrote that the government should be able to block the release of a model if, based on a third-party assessment, it poses unacceptable risks; he specifies that this assessment could be entrusted to a public agency or to private organizations authorized and inspected by the government. On September 9, OpenAI announced its intention to work with Congress on mandatory national requirements for frontier labs, including joint testing and independent evaluations, and expressed its support for California bills establishing independent organizations for risk analysis and mandating independence and transparency requirements for AI auditors. It is no longer a lone voice: OpenAI and Anthropic are now formally calling for mandatory independent validation. The starting point is correct: validation comes before the decision. However, as currently designed, independent validation is a requirement for the regulator; its outcome is not intended for those who must decide whether to adopt the system. And the government receiving that assessment is also among the first users of the models, has a stated interest in not slowing down the technological race, and operates in a confidential manner. It can hardly be viewed as independent by outsiders.

The third approach calls for not putting the brakes on, and it is that of the executive order: no licensing, but voluntary verification. Its merit lies in rejecting prior authorization. However, the benchmarking is classified by design, and the resulting framework has not been published. It covers only a few American labs, on a voluntary basis, while open-weight models are pushing the boundaries. An audit whose criteria, results, and participants are unknown is not information. It is reassurance.

The fourth is manufacturer transparency. On September 9, Anthropic disclosed a fourth incident in which one of its models, during a security test, gained access to the internet due to a configuration error, entered a third-party system, and read personal data. Anthropic disclosed its incidents with a level of detail rarely seen in the industry. But the incident reported in September actually occurred in January. When the model’s developer also decides what to disclose and when, the information depends on the developer’s good faith. That is not in question here. The structure is.

The four scenarios are different, but the shortcoming is the same. When an audit exists, it is conducted by a party with a vested interest in the outcome: the manufacturer, which evaluates itself; the government, which evaluates what it uses; or no one at all, because the suspension halts use without examining the subject. No entity currently provides what banks, insurance companies, and boards of directors need—entities accountable for a system that does not function as stated: a validation conducted by a party independent of both the manufacturer and the government, with transparent criteria and an outcome that is clear not only to the regulator but also to those assuming the risk.

The precedent is the credit rating, created to provide investors with an assessment independent of the debt issuer. No agency publishes the internal financial records of the companies it evaluates, but everyone knows the scale, the criteria, and the verdict. Applied to artificial intelligence: weights, data, and security techniques remain confidential, and the executive order is right to protect them. The method, criteria, and judgment, however, must be public. The order protects the first level and also keeps the second confidential. This is the missing step, and it is how the requests made by Amodei and OpenAI can be fulfilled: a third-party assessment that only two parties can read is not yet a true third-party assessment.

For those who must decide whether to adopt one of these systems, insure it, or finance a company that uses it, the question is not whether artificial intelligence should be slowed down or left to run its course. It’s simpler than that: Is there independent validation of this system, with transparent criteria and a clear, verifiable outcome? Today, for state-of-the-art models, the answer is no—or at least not in a form that decision-makers can use. And the fact that the principle of independent validation is now being called for by the very labs that develop these models makes this absence harder to justify, not easier.