Can AI-Written Code Be Mathematically Proven? Practical Formal Verification with Lean

AI coding agents ...

New Demand for 'Proof' Brought by AI Coding

AI coding tools are excellent at converting natural language requirements into plausible code, but there is no guarantee that the output always accurately reflects the requirements. Unit tests and integration tests check only a portion of the input space, and regression tests rarely go beyond scenarios humans anticipated. Especially in areas where failure is costly, such as payments, authentication, and data integrity, merely “passing tests” is insufficient; a mathematical guarantee that “this property holds for all possible inputs” is needed.

Against this backdrop, a notable recent example is OpenAI's internal model research results published in October 2026. A total of 722 results dealing with unsolved mathematical problems, grouped into 372 collections, were uploaded to GitHub, and many of them were released along with proofs formalized in Lean. It is reported that each result required roughly three hours of compute equivalent to ChatGPT Pro, which is also a signal that AI can perform the formalization of mathematical proofs at scale. However, due to concerns that “AI-generated proofs look plausible but are hard to trust,” it is striking that passing Lean verification, not just the announcement of results, was presented as an important trust mechanism.

A Pipeline for Creating 'Code with Proofs' Using Lean

Lean is a proof assistant originally used by mathematicians to formalize and prove theorems. Because its small kernel re-checks every proof step, a proof that passes the kernel is highly reliable with respect to that specification. Bringing this property to software verification allows us to prove theorems alongside code, such as “this function returns a sorted result” or “this data structure maintains its balance condition after every insertion.”

In practice, a plausible pipeline can be structured as follows.

• The developer first defines the specification and invariants of the function to be verified.
• The AI coding agent generates the implementation code along with Lean theorem declarations and proof sketches.
• The developer checks whether the theorem statements truly capture the original requirements and whether any preconditions are missing.
• Once the Lean kernel checks and passes the proof, that property is mechanically guaranteed even after subsequent refactoring.

For example, if verifying a sorting function, you must prove two invariants together: “the result is in non-decreasing order” and “the result is a permutation of the input.” AI is fairly good at generating proofs that satisfy both conditions, but the developer must review whether “this theorem really means the sorting I intended.” If the specification is weak, it may be possible to prove only trivial properties and still appear to pass verification.

Barriers to Adoption in Practice and What AI Can Lower

The biggest barrier to formal verification is the learning curve required for writing specifications and proofs. Because Lean is based on dependent type theory, its syntax and proof strategies are unfamiliar to ordinary developers, and proving even a small function can take considerable time. Moreover, translating specifications themselves into a formal language is very difficult because business requirements must be defined without ambiguity.

AI plays a role in lowering this barrier. It can shorten proof time by suggesting the next proof strategy based on error messages from the proof assistant or automatically finding auxiliary lemmas. Drafting natural language invariant descriptions into Lean theorem form is also an area where AI performs well. However, two cautions are necessary. First, even if an AI-generated proof sketch passes the Lean kernel, it is useless if the specification differs from the original intent. Second, as seen when multiple bugs were discovered this summer in the Lean checker itself that could prove “false,” the trust foundation of verification tools is not perfect. Therefore, formal verification should be understood not as “mathematically absolutely correct” but as “providing very strong guarantees under the premise that the tools and specifications are trustworthy.”

The Role of Invariants and Specifications That Developers Must Review

When reviewing AI-generated proofs and code, the point developers should focus on is not the proof itself but the specifications and invariants. A proof that passes the Lean kernel is formally correct, but a human must check whether the theorem accurately captures business requirements, covers the entire input range, and has no hidden preconditions. For example, when verifying the invariant “array indices do not go out of bounds,” if the AI adds a precondition that only considers empty arrays, it becomes meaningless in a real production environment.

In practice, it is advisable to divide invariants by layer. Managing data structure internal consistency, function preconditions and postconditions, inter-module contracts, and security boundary conditions as separate theorems makes the verification scope clear. AI can quickly generate a list of such theorems and iterate proof attempts, but ultimately the responsibility for judging “is this specification correct?” lies with humans. Leaving a history of specification changes and review rationale in this process makes it easier to trace later why that invariant was chosen, and here human-in-the-loop review archives like md-log are practically useful.

What Formal Verification Can Solve and Its Limitations

Formal verification does not solve all software problems. Non-functional requirements such as UI responsiveness, integration with external APIs, and performance bottlenecks are difficult to express as theorems, and proving the complete correctness of an entire application is not cost-effective. The practical scope of application is critical modules where errors are fatal, such as payment logic, authentication and authorization, consensus algorithms, and database transactions, as well as parts where the core invariants of algorithms are clear.

AI cannot immediately change these limitations, but it is meaningful in that it lowers the cost of application. By having AI handle repetitive proof attempts and auxiliary lemma search, humans can focus on specification design and architectural decisions. However, AI-generated proofs and code must not be accepted uncritically. If Lean checker bugs, specification errors, and the model's overconfidence overlap, the label “verification passed” can actually give false confidence. Therefore, formal verification greatly mitigates the trust problem of vibe coding in core modules, but it still has practical limits when scaling to guarantee the complete correctness of an entire codebase.

To the question of whether AI-generated code can be mathematically proven, the answer is “yes, but only with respect to the specified specs and invariants.” Combining Lean and AI lowers the cost of writing proofs and can guarantee the reliability of core logic far more strongly than existing tests. However, because that trust rests on the accuracy of specifications and the reliability of the tools, a human-in-the-loop approach where people thoroughly review invariants and preconditions is essential.

References

• Formal Verification and Lean: How AI Turns Code Safety from Assertions into Mathematical Proof
• The saying that AI-written proofs look plausible but are hard to trust ...
• When AI translates mathematical proofs into Lean and “verification passed” appears, ...
• Lean proof materials released to examine mathematical model claims; OpenAI internal ...
• 722 AI-Generated Proofs: Who Will Verify Them? The OpenAI ...
• How AI Gets Smarter on Its Own 3-3 (Stanford CS329A made easy ...
• AlphaProof Nexus: A new AI tool to tackle mathematical proofs
• Prakash: “Writing Code Will Be a Dead Art in Two Years” — How Formal Verification Found a Bug Decades of Linux Review Missed — BigGo Finance
• Cheap code, precious trust: Code modernization at scale - Command Line
• &Truth Raises $70M Seed to Build Universal Verification Machine for Code – Unite.AI
• LLMs are making formal software verification feasible at a much larger scale \ Anthropic
• After Reinventing Mathematics, Can A.I. Eradicate Computer Bugs? - The New York Times