The patch passed. The attack did too.

The green check

The original reproduction stopped working. CI passed. That is an attractive moment to close a security report. It is also a good moment to ask what the patch actually established. A test proves something about a particular execution. It does not automatically prove the rule that execution was supposed to obey.

Consider a document service that checks ownership before it queues an export. A worker processes the export later. Between those two events, the document changes hands. The initial permission check was correct when it ran, but it no longer describes the operation the worker is about to perform. Neither service needs to contain an obviously missing check for the combined behavior to be wrong.

I care about that gap because a security agent can easily write a persuasive report, propose a narrow fix, and declare victory against its own test. Without an independent definition of success, the agent is grading its own homework.

Define the rule before the test

The rule for this example is simple. An export may complete only if the requesting identity has permission for the resource version being exported. The important words are "resource version." A cached permission decision about an earlier version is insufficient.

Start with authorized requests that should work. Then vary identity, ownership, and the time of the permission decision. Keep the contents harmless and run everything in an isolated service. The goal is to test authorization behavior, not to reach a real account or collect somebody else's files.

Separate security failures from availability failures. Blocking every export would prevent unauthorized exports, but it would also destroy the product. A useful test suite needs both denied and permitted operations.

A benchmark you can inspect

The worked benchmark below uses 400 prohibited operations per implementation. Each suite has the same mix of direct requests, queued work, retries, and ownership changes. A failure means a prohibited operation completes. The counts are an example for comparing patch strategies, not measurements from a production service.

The narrow patch rejects the original request pattern. The shared check removes several inconsistent decisions. The final variant binds the decision to the version used by the operation. Count the failures first, then calculate failure rate as failures divided by 400. Lower is better.

Read the denominator

Moving from 128 failures to 72 removes 56 failing cases, a 43.75% relative reduction. It also leaves 18% of the prohibited operations completing. Both descriptions are accurate. Only one makes the residual risk immediately obvious.

Zero out of 400 is useful evidence within the suite. It says nothing by itself about operations the suite never exercises. If the cases were independent random samples from a fixed population, the rough rule of three would put a 95% upper bound near 0.75%. A hand-built adversarial suite does not satisfy that sampling assumption, so I would not present that number as a production guarantee.

I would track allowed-operation success separately, along with queue latency and the cost of repeating the permission check. Security, correctness, and latency deserve separate columns. Averaging them into one score would conceal the tradeoff.

Make the second test independent

The person or agent that writes a patch should not be the only source of its regression cases. Give another reviewer the authorization rule without the patch and ask for a competing test plan. This makes it harder to accidentally encode the implementation as the specification.

Preserve the state transition in a failing test. Record the identity, resource version, permission decision, and eventual operation. Those details make a timing failure understandable after the original report has left everyone's memory.

A release decision should describe which cases passed, which assumptions remain, and which service owns the final check. I would rather read five precise sentences about those boundaries than a page of confident security language.

The question I keep

Does the original reproduction still work? Ask that, but do not stop there. Ask what assumption made the bug possible and where the new code enforces it. Then change the order of events.

The useful output of a security agent is a reproducible finding, a reviewable fix, and a test that survives someone trying a different route. The green check is the beginning of that review.

Evaluation protocol

The full protocol separates discovery, repair, and verification. The discovery set contains 120 authorization scenarios. The repair process can inspect those cases and their traces. A sealed set contains 400 different sequences assembled from the same state machine. The patch does not see the sealed inputs until the final run. Every sequence records the requesting principal, the resource owner, the version read during authorization, the version used during execution, and the terminal decision.

A sequence generator varies four events: enqueue, ownership transfer, retry, and execution. It preserves valid business rules, so it does not produce impossible states merely to inflate the failure count. Each sequence runs against a disposable database with fixed identifiers and a deterministic clock. The worker queue uses a seeded scheduler. Replaying seed 18417 must produce the same event order and the same audit record.

The primary outcome is prohibited completion rate. Secondary outcomes are permitted completion rate, p95 added latency, and duplicate work after retry. A patch fails the acceptance gate if any prohibited operation completes, if permitted completion falls below 99.5%, or if p95 overhead exceeds 20 milliseconds. Those gates force the fix to preserve the product rather than disable the vulnerable operation.

I run the sealed set three times from clean state. Repetition catches accidental dependence on process order or stale database contents. It does not create three independent samples, so I report the 400 unique cases as the denominator and list repeat disagreement separately.

async function executeExport(job: ExportJob) {
  return db.transaction(async tx => {
    const doc = await tx.document.findUniqueOrThrow({
      where: { id: job.documentId }
    });

    if (doc.version !== job.authorizedVersion) {
      throw new StaleAuthorizationError(doc.id);
    }

    await requirePermission(tx, job.requesterId, doc.id, 'export');
    return renderExport(doc);
  });
}

Latency and coverage results

The first figure showed why the version-bound check matters for correctness. The next two figures test its cost and the breadth of the evaluation. Values below are part of the same worked benchmark. Latency comes from 10,000 permitted operations after 1,000 warm-up calls. The test host, database snapshot, and worker concurrency remain fixed across variants.

The request-only patch looks cheap because it avoids a fresh permission read at execution time. The version-bound design adds 8.3 milliseconds at p95 in this setup, well below the 20-millisecond gate. Its median addition is 3.1 milliseconds. The benchmark reports overhead rather than total request time because rendering varies with document size and would obscure the part changed by the patch.

Coverage grows as the generator admits more event classes. Direct requests alone expose 28% of the known failing patterns. Adding queue delay reaches 60%. Ownership changes and retry schedules account for the rest. This curve is a warning against calling a test suite comprehensive because it has many inputs. Four hundred copies of one event order still cover one event order.

Threats to validity

A generated state sequence can only vary events represented in its model. If the real system also authorizes through a cache, batch endpoint, support tool, or migration worker, this benchmark can miss an entire path. I keep a path inventory beside the generator and require each production entry point to map to at least one sealed case.

The latency result is sensitive to database distance, cache state, connection pooling, and contention. It supports a claim about this test deployment. It does not establish an 8.3-millisecond cost in every region. A rollout would compare tail latency and denial counts by route, with an automatic stop if either moves outside the stated gate.

Zero prohibited completions in 400 cases is evidence, not proof. Manual review still matters at trust boundaries. The experiment is strongest when the generator, code review, and production telemetry disagree in different ways. Agreement is reassuring. Disagreement teaches us where the model of the system is incomplete.

Reproduction checklist

Pin the service revision, schema migration, queue implementation, and seed list. Recreate the database for each run. Save one structured trace per sequence and hash the ordered event log. Report the count of generated, rejected, executed, and indeterminate cases. A benchmark that silently drops timeouts produces an optimistic denominator.

The artifact should contain the state-machine definition, fixture builder, scheduler, assertions, and a command that emits both the exact-value tables and chart data. The page should never be the sole copy of the results. Its figures are a view of a machine-readable record that another reviewer can recompute.

The closing claim stays narrow: within this worked protocol, binding authorization to the executed resource version removed every observed prohibited completion while remaining inside the latency and availability gates. That sentence is less exciting than “authorization solved.” It is also useful.