Your benchmark is leaking. Yes, yours.

The suspicious win

The architecture did not change. The score jumped. A new training dump arrived between the two runs, which makes the dump the first place I would look. Better data can produce real improvements. It can also contain the exam.

Exact copies are the easy case. Evaluation questions can reappear inside tutorials, translations, worked solutions, or lightly rewritten explanations. Removing matching strings will not remove every way the answer can reach the training process.

The mistake is treating contamination as a binary property that one deduplication pass can certify. What matters is whether the evaluation still asks a question the system has to solve, rather than recognize.

Keep a record of where examples came from

Give every evaluation item a source record, a creation date, and a description of how it was transformed. Store the evaluation separately from training-data preparation. A split named "test" is a convention, not an access boundary.

Audit at the document level as well as the question level. The exact question may be absent while its answer and reasoning appear a few paragraphs away. Record near matches for review rather than assuming every similar passage is contamination.

False positives matter too. Two elementary problems can share a structure without either being copied. A similarity score should help a reviewer find evidence; it should not silently rewrite the dataset.

A score audit in four views

The worked benchmark uses 1,000 questions per evaluation and example counts to show how an apparent gain can shrink. The original and expanded-data runs use the same evaluation. The cleaned-data run retrains without suspected contaminated training documents and uses that evaluation again.

The final row evaluates the cleaned model on fresh, reviewed variations with changed names and quantities. That last comparison changes the test, so it cannot isolate contamination by itself. It checks whether the learned behavior transfers to the intended reasoning task.

Which claim survives?

The expanded dump appears to add 12 percentage points over the original run. After removing suspect documents, the remaining gain is two points. Ten of the twelve apparent points disappear. That is enough to pause a broad capability claim and investigate the training change.

It is not enough to say that contamination caused exactly ten points. Removing documents changes more than exposure to answers. It can alter topic coverage, sample count, and training dynamics. A stronger experiment matches those factors and repeats training with several seeds.

The 64% score on fresh variants is another warning, but those questions may differ in difficulty. Have reviewers check equivalence without seeing model outputs. Otherwise a test designed to expose memorization can accidentally become a test of a different skill.

Use paired evidence

Aggregate scores conceal which examples changed. Keep an item-level table showing correctness before and after each intervention. An example that flips from correct to wrong after cleaning deserves inspection, particularly if a near match was removed from training.

For the first three rows, the same questions support paired analysis. For fresh variants, preserve the mapping to the original item and describe the transformation. A renamed variable and a new multi-step dependency should not be treated as equivalent edits.

Report uncertainty appropriate to the experiment. Repeated model runs, training seeds, and sampled test items represent different sources of variation. A narrow interval over test questions does not account for an unstable training procedure. Do not let a precise-looking decimal replace a description of what was repeated.

Keep an inconvenient test

A small evaluation with documented provenance can be more useful than a much larger set whose origin nobody can explain. Protect it, rotate fresh variations, and inspect unexpected improvements before publishing them.

I want the benchmark to make it difficult to fool ourselves. That means keeping the boring records, the failed comparisons, and the version of the score that looked less exciting after the cleanup. The smaller number is often the one worth building on.

Audit pipeline

The audit begins before similarity search. Freeze the model, training corpus manifest, evaluation revision, prompt template, and scoring code. Recompute the suspicious score from raw outputs. A changed prompt or parser can create an apparent gain without any data effect. Only after reproducing the number do I inspect the new corpus.

Every training document receives a content hash, source group, collection time, and transformation lineage. Evaluation items carry the same fields plus a private family identifier linking variations. Exact matching runs first. Token shingles and embedding retrieval generate candidate near matches. Human reviewers decide whether a candidate exposes the answer, the reasoning path, or only a common topic.

Reviewers work without seeing whether the model answered the item correctly. Showing performance can bias borderline decisions toward the story everyone expects. A second reviewer checks all answer-exposing matches and a random sample of negatives. Agreement is reported with the counts, not compressed into “manually verified.”

The cleaned corpus removes confirmed exposures and matched documents from the same source group. Retraining uses a replacement sample matched by topic and length, so cleaning does not simply shrink the corpus.

def candidates(eval_item, corpus, k=100):
    exact = corpus.lookup_hash(normalize(eval_item.text))
    lexical = corpus.bm25(eval_item.text, limit=k)
    semantic = corpus.vector_search(embed(eval_item.text), limit=k)
    merged = deduplicate(exact + lexical + semantic)
    return sorted(merged, key=lambda x: x.max_similarity, reverse=True)

queue = [candidates(item, corpus) for item in evaluation]

Exposure rate and score sensitivity

The first additional figure is a cumulative audit curve. Exact matching identifies 1.1% of evaluation items as exposed. Lexical near-match review raises that to 4.7%. Semantic retrieval and source-family expansion reach 8.6%. Each stage uses the union of previous findings, so the curve cannot decrease.

The second figure groups the worked evaluation by exposure label. On confirmed exposed items, the expanded-data model scores 96.5%. On reviewed clean items it scores 71%. The 25.5-point gap does not prove memorization for every item, but it explains why a modest exposed subset can move the aggregate score.

The denominator in the exposure curve is 1,000 evaluation items. The subgroup score reports exact counts because the groups differ sharply in size. A 96.5% result on 86 items should not be visually compared with 71% on 914 as if both estimates had equal precision.

Statistical model

Items are not independent when several come from one source family. Treating all 1,000 as unrelated makes uncertainty look smaller than it is. I would bootstrap by family, resampling whole groups, and repeat the complete scoring calculation. The unit of resampling should match the unit through which leakage could travel.

Training adds another source of variation. Run at least three seeds for the original, expanded, and cleaned corpus conditions. Pair evaluation items across models, but do not pair independent training runs as if they shared optimization noise. Report every seed and the chosen summary before inspecting which one supports the preferred conclusion.

The causal comparison is expanded versus cleaned data under a matched training recipe. The fresh-variant score answers a different question about transfer. Presenting both is valuable. Subtracting one from the other and calling the result “contamination points” is not justified without stronger assumptions.

Release and reproduction record

A public artifact can release evaluation hashes, transformation recipes, match categories, model outputs, aggregate counts, and code without releasing a sealed test verbatim. Authorized auditors need access to the source documents and reviewer decisions. The public report should state what remains private and why.

Save the retrieval index revision, embedding model, search parameters, reviewer rubric, conflicts, and adjudications. Near-match audits change when any of those components changes. A bare list of suspected URLs cannot reproduce the process.

The release claim should follow the evidence. In this worked study, the apparent twelve-point gain falls to two after matched cleaning, confirmed exposures score much higher than reviewed-clean items, and performance drops again on fresh variants. Together those results make the broad capability claim unsafe. They also give the team a concrete path to a better evaluation.

An inconvenient benchmark is one whose provenance is documented, whose variants are controlled, and whose result can get worse without anyone changing the slide deck. Keep that version.