Your benchmark is leaking. Yes, yours.

The suspicious win

The architecture did not change. The score jumped. A new training dump arrived between the two runs, which makes the dump the first place I would look. Better data can produce real improvements. It can also contain the exam.

Exact copies are the easy case. Evaluation questions can reappear inside tutorials, translations, worked solutions, or lightly rewritten explanations. Removing matching strings will not remove every way the answer can reach the training process.

The mistake is treating contamination as a binary property that one deduplication pass can certify. What matters is whether the evaluation still asks a question the system has to solve, rather than recognize.

Keep a record of where examples came from

Give every evaluation item a source record, a creation date, and a description of how it was transformed. Store the evaluation separately from training-data preparation. A split named "test" is a convention, not an access boundary.

Audit at the document level as well as the question level. The exact question may be absent while its answer and reasoning appear a few paragraphs away. Record near matches for review rather than assuming every similar passage is contamination.

False positives matter too. Two elementary problems can share a structure without either being copied. A similarity score should help a reviewer find evidence; it should not silently rewrite the dataset.

A score audit in four views

The worked benchmark uses 1,000 questions per evaluation and example counts to show how an apparent gain can shrink. The original and expanded-data runs use the same evaluation. The cleaned-data run retrains without suspected contaminated training documents and uses that evaluation again.

The final row evaluates the cleaned model on fresh, reviewed variations with changed names and quantities. That last comparison changes the test, so it cannot isolate contamination by itself. It checks whether the learned behavior transfers to the intended reasoning task.

Which claim survives?

The expanded dump appears to add 12 percentage points over the original run. After removing suspect documents, the remaining gain is two points. Ten of the twelve apparent points disappear. That is enough to pause a broad capability claim and investigate the training change.

It is not enough to say that contamination caused exactly ten points. Removing documents changes more than exposure to answers. It can alter topic coverage, sample count, and training dynamics. A stronger experiment matches those factors and repeats training with several seeds.

The 64% score on fresh variants is another warning, but those questions may differ in difficulty. Have reviewers check equivalence without seeing model outputs. Otherwise a test designed to expose memorization can accidentally become a test of a different skill.

Use paired evidence

Aggregate scores conceal which examples changed. Keep an item-level table showing correctness before and after each intervention. An example that flips from correct to wrong after cleaning deserves inspection, particularly if a near match was removed from training.

For the first three rows, the same questions support paired analysis. For fresh variants, preserve the mapping to the original item and describe the transformation. A renamed variable and a new multi-step dependency should not be treated as equivalent edits.

Report uncertainty appropriate to the experiment. Repeated model runs, training seeds, and sampled test items represent different sources of variation. A narrow interval over test questions does not account for an unstable training procedure. Do not let a precise-looking decimal replace a description of what was repeated.

Keep an inconvenient test

A small evaluation with documented provenance can be more useful than a much larger set whose origin nobody can explain. Protect it, rotate fresh variations, and inspect unexpected improvements before publishing them.

I want the benchmark to make it difficult to fool ourselves. That means keeping the boring records, the failed comparisons, and the version of the score that looked less exciting after the cleanup. The smaller number is often the one worth building on.