Anthropic Claims Claude Can Prove Software Safety

Software that runs without error under all conditions could soon become cheaper to produce. For ordinary developers, this approach aims to reduce the time spent reviewing code, particularly when that code is generated by artificial intelligence. The goal is to shift the burden of proof from the person checking the software to the process that creates it.

Anthropic, along with its partner Theorem, reports that its Claude model can now generate code and mathematically prove its correctness. The company frames this capability as a way to make formal verification, a method typically reserved for critical systems, a routine part of development. However, the practical limits of this technology remain a significant hurdle.

Historically, proving software runs error-free under every circumstance has been prohibitively expensive. The effort required to formally verify the seL4 operating system kernel consumed 20 person-years of specialist work. As a result, such verification is usually confined to high-stakes areas like Airbus avionics software.

Anthropic's Frontier Red Team, working with Theorem, delegated the proof generation to Claude. Within weeks, the model produced approximately 180 million lines of proof code. The effort covered 123 open-source libraries, drawing on test programs from Google's OSS-Fuzz security project.

The process involved translating crash tests into mathematical problems and analyzing compiled machine code. Claude then generated extensive proofs, which an independent verification program named Rocq automatically checks. The researchers describe this as 'fully autonomous' proof generation, though the tools and formal foundations were built by humans at Theorem.

During the effort, Claude identified hundreds of bugs that conventional testing had missed. Many of these issues only appear under specific conditions, such as unusually large input files or a particular sequence of access requests. For example, verifying a widely used PDF reader in one week revealed as many security bugs as are typically reported for that software in an entire year. In another application, Claude caught a logical error that had gone undetected for over a decade.

Anthropic acknowledges that 'formally verified' does not mean 'error-free.' The proofs only hold under strict conditions: a specific processor architecture, single-threaded programs, and inputs under 64 MB. In practice, these libraries run on different hardware and in different configurations. The compute cost is steep, with individual proofs running up hundreds of millions of tokens, though Anthropic does not report total project costs.

The project strengthens the case for software automation, but the limitations remain clear. If AI can verify its own code, developers can trust AI-generated software more and spend less time reviewing it. The company expects formal verification with AI will soon become as practical as standard testing, and Theorem's roadmap already targets critical infrastructure like glibc, OpenSSL, and cURL.