One GPU, one weekend, one world model

The constraint

One GPU. Forty-eight hours. A fixed camera looking at a hand and a few objects on a desk. The goal is a model that predicts what happens after a proposed action, with a budget small enough to force a clear experiment.

The temptation is to make a good-looking video. That is a different objective from building a useful world model. A video can look reasonable while ignoring the action entirely. If the hand usually moves right in the dataset, a model that always predicts rightward movement may appear competent.

I would spend the weekend testing whether the action has the intended effect on the prediction. Texture quality can wait. A blurry block moving in the correct direction is more useful to a planner than a beautiful block doing the wrong thing.

Budget the experiment

A practical 48-hour plan might reserve eight hours for data checks, 24 for training, eight for evaluation, and eight for reruns. This is a proposed allocation, not a hardware timing result. Protect the evaluation time before the first training job starts.

Split data by complete recording session. Nearby frames from one recording should not land on opposite sides of the train-test boundary. Otherwise the model can benefit from nearly identical scenes without learning the behavior you care about.

Use short prediction horizons first. Save the same checkpoints and evaluation inputs for every variant. When the budget is tight, a clear comparison between a weak baseline and one candidate is worth more than six incompletely evaluated models.

Give the model an inconvenient action

The worked benchmark below uses 500 paired tests. Each pair starts with the same image but asks for opposing actions. A pair passes only when both predicted object displacements follow their respective requested directions. A rightward prediction for both members fails.

The example counts compare a copied-frame baseline, a model without action input, an action-conditioned model, and an action-conditioned model trained with balanced action sampling. The chart measures directional control, not photorealism or general physical understanding.

What 81% does not mean

The balanced variant succeeds on 405 pairs and fails on 95. Its 23-point improvement over ordinary action conditioning is meaningful within this example. It does not mean the model understands 81% of physics.

Directional correctness ignores distance, contact timing, object identity, and what happens behind the hand. A prediction can pass this test and still be useless for placing a fragile object near the edge of a table. The metric is deliberately narrow because a narrow result is easier to debug.

Add separate checks for displacement error, object persistence, and collisions. Keep results by horizon rather than averaging a one-step prediction with a long rollout. Repeated prediction feeds the model its own mistakes, so a stable short-horizon score can coexist with rapid long-horizon drift.

Ablations that fit on one GPU

Remove the action input. Shuffle actions between clips. Rebalance the training sampler. Each change tests a particular explanation for success. If shuffled actions leave the score unchanged, the conditioning mechanism deserves attention before the model grows.

Match the training-step budget across comparisons. More data passes or a longer run can otherwise masquerade as an architectural improvement. For a weekend experiment, I would keep the architecture fixed and change one ingredient at a time.

Keep an error gallery indexed by test case. The same 20 examples, rendered after every checkpoint, can expose a model that improves the average while forgetting an awkward object. Use the gallery to generate new questions, not as a replacement for the complete test set.

What earns another weekend

A small model earns more compute when it beats a baseline on an intervention the baseline cannot solve. That result does not need to look impressive in a video montage. It needs to survive a different action.

At the end of the run I want a checkpoint, a fixed evaluation set, the commands needed to rerun it, and a list of failures. Those are enough to decide what Monday should look like.