Today we measure coding agents by counting how many benchmark tasks they resolve, and we decide whether a task is resolved by running a suite of tests. But an agent can pass every test and still be wrong on inputs no test happens to check.
Worse, agents can learn to game the tests instead of learning to solve the underlying programming problems. This makes it harder to trust the agents when decisions matter most. Businesses may make roadmap decisions based on the outcome of a specific evaluation. And as more companies use reinforcement learning in their post-training processes, results on benchmarks are increasingly becoming the reward signal we use to train the agents. Inaccurate benchmarking poses the risk that agents will learn from results that are incorrect.
A new open-source tool from Amazon called BenchProofer uses automated reasoning to strengthen coding benchmarks. Benchproofer uses each task's reference solution to generate a formal specification; an implementation in a language amenable to formal verification; and a proof that the implementation satisfies the specification, under the stated assumptions. The proof is then checked by a verifier, and tests and audits check the implementation’s correspondence with the repository code.
Alongside the tool, we are releasing SWE-Proof, which adds these specifications, implementations, and proofs to all 500 tasks in the SWE-Bench Verified dataset.
What BenchProofer does
To get a sense of how BenchProofer works, consider a simple example. Someone files a bug report that says "the function that finds the largest number in a list returns the wrong answer on some inputs." The reference fix is a corrected find_max function. In an ordinary benchmark, an agent's fix is judged by running a few test cases; for example, find_max([3, 1, 2]) should return 3. If it passes those tests, the fix is counted as correct, even if it would return the wrong answer on some untested inputs.
The first artifact BenchProofer adds is a specification, which is a precise statement of what a correct answer must do, written so a machine can check it. For find_max, the specification would use a formal language such as Lean to express the following requirement: "For any nonempty list, the returned value must belong to the list, and no element may be larger than it".
The second artifact that BenchProofer adds is a reference implementation, an actual solution written in a language the verifier understands. The third is a proof, a step-by-step argument that the implementation satisfies the specification. A tool called a verifier — Lean’s proof checker is one example — checks the proof automatically, much the way a compiler checks types. If the proof goes through, we know the implementation meets the specification on every input it covers, as long as the proof’s assumptions hold.
Using the verified benchmark
Once BenchProofer has built these artifacts, we can use them to evaluate other fixes for the same task. An upgraded benchmark supports two evaluation modes. In audit mode, you run your coding agent through the benchmark's usual workflow and collect the repository patch it produces. The specification generated by BenchProofer guides additional tests that look for differences between the reference implementation and patches that pass the benchmark’s original tests.
In verified mode, the agent also receives the specification and a verification tool. Alongside its repository patch, the coding agent produces a separate implementation of the fix that the verifier can check against the specification. Once the verifier accepts the proof, we have a verified implementation. The repository patch must also pass the repository tests. The final audit checks whether the patch and the verified implementation describe the same behavior. A successful submission therefore pairs a test-passing repository patch with a verified implementation of the fix.
Building your own benchmark
To build your own verified benchmark, give BenchProofer coding tasks from your own projects, such as bugs your team has already fixed.
How BenchProofer works
BenchProofer needs three things per task, all of which existing benchmarks already have. It needs a plain-English description of the problem, such as a GitHub issue. It needs a reference solution accepted as correct. And it needs a mechanical way to check candidate fixes, typically the existing test suite and its harness, the software that sets up the repository, applies a patch, runs the tests, and reports which ones pass.
Real codebases are far too large to verify end to end, so BenchProofer draws a boundary around newly written code and verifies only that. For each existing function the new code calls, BenchProofer generates a short contract that it tests — but does not verify — automatically.
Suppose, for instance, that the fix for find_max calls an existing function, smaller(a, b), which the codebase already includes. BenchProofer records a contract for it, such as "the value it returns is never larger than either input." Then it checks the contract by running the real smaller on many inputs. The proof of find_max relies on that contract, rather than on formal verification of smaller, which keeps verification tractable even inside a large repository.
Once we have the three artifacts and their contracts for all tasks in the benchmark, we put them through a battery of checks we call gates. Some gates are logic based. They confirm that verification of the reference proof succeeds, that verification of the pre-fix implementation against the same specification fails, that the corresponding patch passes the benchmark tests, that the specification rules out the original bug, or that the reference solution matches the real patch.
Other gates use adversarial agents to look for specifications that are too weak or too restrictive or that reveal the answer. A failure on either type of gate sends the task back for revision.
The verification options are Lean, Nagini for Python code with preconditions and postconditions, or Velvet for imperative programs written and verified within Lean.
Case study: SWE-Proof
We ran BenchProofer over all 500 tasks of SWE-Bench Verified, a benchmark that frontier models perform extremely well on — when their solutions are assessed by testing alone. For each task, the tool generated a specification, a reference implementation, a proof, and contracts for the unchanged functions called by the modified code.
We then checked the generated artifacts using repository tests, formal verification, and adversarial audits. We built each task for all three verification backends — Lean, Nagini, and Velvet — and released everything, including the verdict of every gate, so anyone can recheck the results independently.
We then compared Claude Opus 4.8 and GPT-5.5, the latest models from each provider when we ran the study, on both the standard SWE-Bench Verified benchmark and our verified version, SWE-Proof. (Applying the same methodology to later generations of Opus is a natural direction for future work.)
Finding 1: Tests can miss behavioral differences
Claude Opus 4.8 passes the standard SWE-Bench Verified tests on 425 of the 500 tasks (85.0%). But BenchProofer's audit agent uses the reference specifications to generate additional tests that look for behavior the original tests missed. On those tests, 134 of the 425 patches, nearly one in three, fail, while the reference patch passes.
Automated audit results |
Opus 4.8 |
GPT-5.5 |
Pass benchmark tests |
425 (85.0%) |
406 (81.2%) |
Flagged among the passing patches |
134 (26.8%) |
239 (47.8%) |
Percentages are out of all 500 SWE-Bench Verified tasks. All flagged patches passed the benchmark tests.
With 291 of Opus 4.8’s patches (58.2% of all tasks), audit mode finds no difference between their behavior and those of the reference patches. However, the patches that do exhibit behavioral differences may still solve the original problem. To understand whether the flagged differences indicated actual bugs, human reviewers examined a random sample of 41 of the 134 flagged Opus patches. They identified 21 defects, including missed corner cases or incorrect behaviors. The other 20 were valid fixes.
Finding 2: Agents achieve higher verified pass rates when given formal specifications
The same agents achieve more verified passes when given the formal specification. When Opus 4.8 writes its own specification — instead of using the one generated by BenchProofer — its verified pass rate is 84.6%, with Lean as the backend. When it uses BenchProofer’s specification, its pass rate rises to 93.6%. GPT-5.5 rises from 79.6% to 94.4%.
Specification used |
Opus 4.8 |
GPT-5.5 |
Agent writes its own |
423 (84.6%) |
398 (79.6%) |
Benchmark specification supplied |
468 (93.6%) |
472 (94.4%) |
Verified passes using Lean. Percentages are out of all 500 SWE-Bench Verified tasks.
To count as a verified pass, the patch must meet three conditions: it must pass the repository tests; the Lean implementation of the fix must have a proof accepted by Lean's proof checker; and an audit must find no difference in behavior between the verified implementation and the patched code.
The specification points the agent to the relevant functions, so we tested how much that guidance alone helped. We found that the specification's faithfulness and the verified implementation's correspondence with the Python patch still require validation.
Finding 3: Agents struggle to generate faithful formal specifications
To isolate specification generation, we asked Opus to produce a formal specification and a Lean implementation that can be verified against it, without also producing a repository patch. It received only the bug report and the repository; as in the earlier experiments, the reference patch was hidden.
Only 61.4% of its generated Lean specifications passed automated review. Language model reviewers carried out an adversarial specification audit, looking for missing requirements, unnecessary restrictions, and incorrect assumptions about existing code. The main problem was that the specifications left out behavior that repairing the bug required.
What a verified instance actually looks like
The find_max function was a toy example. Here’s a more substantial one, a problem called matplotlib-24177, from SWE-Bench Verified. The problem calls for the generation of a histogram, where the shapes of the histogram’s bars are defined by anchor points, or vertices. The issue reads "[Bug]: ax.hist density not auto-scaled when using histtype='step'." That is, the histogram's density axis is not adjusted to fit the whole histogram when histtype="step". The root cause is that the plot's bounds were computed from a simplified set of vertices, so parts of the histogram were cut off.
This example, adapted from the bug report, plots the same random data as a bar histogram and a step histogram, which shows the overall contour of the bars:
import matplotlib.pyplot as plt import numpy as np samples = np.random.randn(100_000) * 1.2 _, (bar, step) = plt.subplots(ncols=2) bar.hist(samples, bins=100, density=True, histtype="bar") step.hist(samples, bins=100, density=True, histtype="step") plt.show()
Both plots should show the full histogram. In the reported bug, the step plot could be cut off because its automatically chosen axis limits were too small.
The accompanying regression test uses a fixed set of points to make the check repeatable. It checks that the plot’s axis limits include every point. BenchProofer adds a specification describing the bounding-box calculation for every input it covers.
BenchProofer's specification describes the reference fix's output, namely the tight bounding box of all the points, so every vertex lies inside the box, and each side touches at least one vertex. Even if you do not read Lean, the shape of the statement is legible. It says the result is empty exactly when the input is empty, that every vertex sits within the returned bounds, and that each bound is actually achieved by some vertex.
-- Formal specification (Lean): what the fixed function must return
def problem_spec (verts : List (Rat × Rat))
(result : Option (Rat×Rat×Rat×Rat)) : Prop :=
match result with
| none => verts = []
| some (xmin, xmax, ymin, ymax) =>
verts ≠ [] ∧
(∀ v ∈ verts, xmin ≤ v.1 ∧ v.1 ≤ xmax ∧
ymin ≤ v.2 ∧ v.2 ≤ ymax) ∧
(∃ a ∈ verts, a.1 = xmin) ∧ (∃ b ∈ verts, b.1 = xmax) ∧
(∃ c ∈ verts, c.2 = ymin) ∧ (∃ d ∈ verts, d.2 = ymax)
In the Lean code below, sampledVertices represents the sampling operation used by the fix. Its second argument selects the sampling mode; false means keeping every input point. As in the find_max example, BenchProofer records the property the fix relies on as an axiom. It then tests whether the axiom holds through “fuzzing”, or randomly generating a large volume of inputs that are irregular in some way, so that they’re likely to elicit unusual behaviors.
-- Axiom: an assumption about the sampling operation, tested by fuzzing
-- Sampling takes a list of points and a mode flag.
opaque sampledVertices :
List (Rat × Rat) → Bool → List (Rat × Rat)
-- false selects keep-all mode: return every point in order.
axiom sampledVertices_keepall : ∀ verts,
sampledVertices verts false = verts
The reference implementation loops over the vertices and widens the box whenever a vertex falls outside the current bounds.
-- Reference implementation (Lean): the fix, in the verifier's language
def implementation (verts : List (Rat × Rat)) :
Option (Rat×Rat×Rat×Rat) := Id.run do
match (sampledVertices verts false) with
| [] => return none
| p0 :: rest =>
let mut xmin := p0.1; let mut xmax := p0.1
let mut ymin := p0.2; let mut ymax := p0.2
for v in rest do
if v.1 < xmin then xmin := v.1
if v.1 > xmax then xmax := v.1
if v.2 < ymin then ymin := v.2
if v.2 > ymax then ymax := v.2
return some (xmin, xmax, ymin, ymax)
Finally, a proof shows that the implementation meets the specification on every input in its domain, and Lean’s proof checker verifies that proof mechanically. The proof relies on the stated sampledVertices axiom. Fuzzing can find violations of that axiom, but passing the fuzzing checks does not prove it universally.
-- Proof: the implementation meets the specification on every input
theorem correct : ∀ verts,
problem_spec verts (implementation verts) := by
intro verts
cases hs : sampledVertices verts false with
| nil =>
rw [none_branch verts hs]
simp only [problem_spec]
exact (sampledVertices_keepall_nil verts).mp hs
| cons p0 rest =>
rw [impl_some_char verts p0 rest hs]
simp only [problem_spec]
exact ⟨verts_ne_nil .., bounds_all ..,
coord_attained ..)
Getting started
BenchProofer is open source. Clone the BenchProofer repository on GitHub, which links to the SWE-Proof dataset and the full paper. If you build or evaluate coding agents, run yours on SWE-Proof and tell us what you find, or bring your own benchmark, since the adapter is one short class. If you work on verification, add a backend for another verifier, contribute a new gate (the most valuable ones test whether a specification faithfully captures the chosen reference behavior), or audit an instance. Every task ships with its artifacts and verdicts, and a single counterexample reopens it. Whether you bring a benchmark or a backend, we would love your issues and pull requests.