The model found a bug in our physics

An excellent score

A robot moves a block through a wall and receives an excellent reward. From the optimizer's perspective, the experiment has worked. The block reached the target. The fact that a real wall would object is a concern we forgot to encode in the environment.

This is the failure I want to see before I spend another week training. The policy has learned a property of the simulator, but the dashboard calls it task performance. A clean learning curve can make that category error look like progress.

The interesting question is not whether the agent was being clever. It is whether the measurement distinguishes useful behavior from behavior that only succeeds because the test environment is incomplete.

Two definitions of success

Suppose the original evaluator checks the block's final distance from the target. A second evaluator also checks whether the trajectory crosses an obstacle or exceeds an allowed contact tolerance. The first metric measures arrival. The second measures arrival through an admissible path.

Record both. If a policy reaches the target in 980 of 1,000 episodes but only 310 trajectories satisfy the physical checks, the success rate is either 98% or 31%, depending on the question. The 67-point gap is the result worth investigating.

The stricter evaluator still needs review. Contact tolerances, object geometry, and sampling frequency can all create their own blind spots. An independent evaluator means a different failure mode, not an infallible judge.

Stress the environment

Here is a worked 1,000-episode benchmark for a fixed policy under four evaluation conditions. Hold the start-state list constant. Change the timestep, collision checks, or simulator implementation without retraining the policy. Score target arrival under each condition.

These example counts show the signature of an environment-dependent result. They do not report a hardware experiment. The comparison is useful because the policy stays fixed while the assumptions underneath it change.

What the drop tells you

The fall from 98% to 29% is a 69-percentage-point difference. It is evidence that the original success metric does not transfer across these evaluation conditions. It is not enough to identify a particular collision bug. The second simulator may also differ in friction, actuator limits, or object geometry.

Change one factor at a time before drawing a causal conclusion. First align the obvious parameters. Then inspect paired trajectories from the same initial state. Look for the first timestep where their behavior diverges, rather than comparing only the final frames.

An average reward hides rare spectacular failures. I would keep counts of obstacle crossings, penetration depth, invalid contacts, and valid arrivals. A policy that improves average reward while doubling severe violations has not become safer just because the curve points upward.

Build a less comfortable evaluation

Reserve a small collection of scenes with thin obstacles, awkward angles, and contacts near timestep boundaries. Keep those scenes outside policy training. They should be annoying enough that a human wants to inspect the replay when the score changes.

Randomizing physical parameters during training may reduce reliance on one precise configuration, but it cannot repair an invalid reward definition. The objective still needs to reject trajectories that solve the wrong problem. More variation in a broken test is still a broken test.

For a larger run, use several training seeds and report the variation between trained policies. Replaying one policy a thousand times measures a different source of uncertainty from training ten policies. The experiment needs both if the claim is about the training method.

Before the next training run

I would spend the first hour watching the highest-reward trajectories, not the average ones. Optimizers tend to find the edges of what we permit. Those edges are where the mismatch between the task and the measurement becomes visible.

The right fix may make the score worse. If the wall stops being optional and the success rate falls, the experiment has become more honest. That is a better starting point for the next model.