The patch passed. The attack did too.

The green check

The original reproduction stopped working. CI passed. That is an attractive moment to close a security report. It is also a good moment to ask what the patch actually established. A test proves something about a particular execution. It does not automatically prove the rule that execution was supposed to obey.

Consider a document service that checks ownership before it queues an export. A worker processes the export later. Between those two events, the document changes hands. The initial permission check was correct when it ran, but it no longer describes the operation the worker is about to perform. Neither service needs to contain an obviously missing check for the combined behavior to be wrong.

I care about that gap because a security agent can easily write a persuasive report, propose a narrow fix, and declare victory against its own test. Without an independent definition of success, the agent is grading its own homework.

Define the rule before the test

The rule for this example is simple. An export may complete only if the requesting identity has permission for the resource version being exported. The important words are "resource version." A cached permission decision about an earlier version is insufficient.

Start with authorized requests that should work. Then vary identity, ownership, and the time of the permission decision. Keep the contents harmless and run everything in an isolated service. The goal is to test authorization behavior, not to reach a real account or collect somebody else's files.

Separate security failures from availability failures. Blocking every export would prevent unauthorized exports, but it would also destroy the product. A useful test suite needs both denied and permitted operations.

A benchmark you can inspect

The worked benchmark below uses 400 prohibited operations per implementation. Each suite has the same mix of direct requests, queued work, retries, and ownership changes. A failure means a prohibited operation completes. The counts are an example for comparing patch strategies, not measurements from a production service.

The narrow patch rejects the original request pattern. The shared check removes several inconsistent decisions. The final variant binds the decision to the version used by the operation. Count the failures first, then calculate failure rate as failures divided by 400. Lower is better.

Read the denominator

Moving from 128 failures to 72 removes 56 failing cases, a 43.75% relative reduction. It also leaves 18% of the prohibited operations completing. Both descriptions are accurate. Only one makes the residual risk immediately obvious.

Zero out of 400 is useful evidence within the suite. It says nothing by itself about operations the suite never exercises. If the cases were independent random samples from a fixed population, the rough rule of three would put a 95% upper bound near 0.75%. A hand-built adversarial suite does not satisfy that sampling assumption, so I would not present that number as a production guarantee.

I would track allowed-operation success separately, along with queue latency and the cost of repeating the permission check. Security, correctness, and latency deserve separate columns. Averaging them into one score would conceal the tradeoff.

Make the second test independent

The person or agent that writes a patch should not be the only source of its regression cases. Give another reviewer the authorization rule without the patch and ask for a competing test plan. This makes it harder to accidentally encode the implementation as the specification.

Preserve the state transition in a failing test. Record the identity, resource version, permission decision, and eventual operation. Those details make a timing failure understandable after the original report has left everyone's memory.

A release decision should describe which cases passed, which assumptions remain, and which service owns the final check. I would rather read five precise sentences about those boundaries than a page of confident security language.

The question I keep

Does the original reproduction still work? Ask that, but do not stop there. Ask what assumption made the bug possible and where the new code enforces it. Then change the order of events.

The useful output of a security agent is a reproducible finding, a reviewable fix, and a test that survives someone trying a different route. The green check is the beginning of that review.