One GPU, one weekend, one world model

The constraint

One GPU. Forty-eight hours. A fixed camera looking at a hand and a few objects on a desk. The goal is a model that predicts what happens after a proposed action, with a budget small enough to force a clear experiment.

The temptation is to make a good-looking video. That is a different objective from building a useful world model. A video can look reasonable while ignoring the action entirely. If the hand usually moves right in the dataset, a model that always predicts rightward movement may appear competent.

I would spend the weekend testing whether the action has the intended effect on the prediction. Texture quality can wait. A blurry block moving in the correct direction is more useful to a planner than a beautiful block doing the wrong thing.

Budget the experiment

A practical 48-hour plan might reserve eight hours for data checks, 24 for training, eight for evaluation, and eight for reruns. This is a proposed allocation, not a hardware timing result. Protect the evaluation time before the first training job starts.

Split data by complete recording session. Nearby frames from one recording should not land on opposite sides of the train-test boundary. Otherwise the model can benefit from nearly identical scenes without learning the behavior you care about.

Use short prediction horizons first. Save the same checkpoints and evaluation inputs for every variant. When the budget is tight, a clear comparison between a weak baseline and one candidate is worth more than six incompletely evaluated models.

Give the model an inconvenient action

The worked benchmark below uses 500 paired tests. Each pair starts with the same image but asks for opposing actions. A pair passes only when both predicted object displacements follow their respective requested directions. A rightward prediction for both members fails.

The example counts compare a copied-frame baseline, a model without action input, an action-conditioned model, and an action-conditioned model trained with balanced action sampling. The chart measures directional control, not photorealism or general physical understanding.

What 81% does not mean

The balanced variant succeeds on 405 pairs and fails on 95. Its 23-point improvement over ordinary action conditioning is meaningful within this example. It does not mean the model understands 81% of physics.

Directional correctness ignores distance, contact timing, object identity, and what happens behind the hand. A prediction can pass this test and still be useless for placing a fragile object near the edge of a table. The metric is deliberately narrow because a narrow result is easier to debug.

Add separate checks for displacement error, object persistence, and collisions. Keep results by horizon rather than averaging a one-step prediction with a long rollout. Repeated prediction feeds the model its own mistakes, so a stable short-horizon score can coexist with rapid long-horizon drift.

Ablations that fit on one GPU

Remove the action input. Shuffle actions between clips. Rebalance the training sampler. Each change tests a particular explanation for success. If shuffled actions leave the score unchanged, the conditioning mechanism deserves attention before the model grows.

Match the training-step budget across comparisons. More data passes or a longer run can otherwise masquerade as an architectural improvement. For a weekend experiment, I would keep the architecture fixed and change one ingredient at a time.

Keep an error gallery indexed by test case. The same 20 examples, rendered after every checkpoint, can expose a model that improves the average while forgetting an awkward object. Use the gallery to generate new questions, not as a replacement for the complete test set.

What earns another weekend

A small model earns more compute when it beats a baseline on an intervention the baseline cannot solve. That result does not need to look impressive in a video montage. It needs to survive a different action.

At the end of the run I want a checkpoint, a fixed evaluation set, the commands needed to rerun it, and a list of failures. Those are enough to decide what Monday should look like.

Model and training recipe

The reference experiment uses 1.8 million eight-frame clips at 128 by 128 pixels. Each clip contains an initial observation, a discretized action, and seven future frames. Complete recording sessions stay in one split. The training set has 1.62 million clips, validation has 90,000, and the sealed test set has 90,000. Mirrored clips are generated before splitting by session so transformed versions cannot leak across the boundary.

A convolutional encoder maps the first frame to 16 by 16 latent tokens. An action embedding enters every temporal block. The decoder predicts latent residuals and reconstructs frames only for the image loss. Training uses bfloat16, gradient accumulation, and a fixed 24-hour optimizer budget on one 80-gigabyte GPU. Checkpoints are evaluated every 2,000 updates.

The comparison models share parameter count, data order, optimizer, and update budget. The no-action ablation replaces action embeddings with one learned null vector. The shuffled-action condition preserves action frequencies while breaking correspondence to the clip. These controls test whether gains come from conditioning rather than extra parameters.

The sealed paired-action benchmark remains the primary test. Pixel metrics are secondary because a model can improve reconstruction by predicting the dataset average.

def paired_success(model, frame, left, right):
    pred_left = model.rollout(frame, left, steps=12)
    pred_right = model.rollout(frame, right, steps=12)
    dx_left = object_center(pred_left[-1]).x - object_center(frame).x
    dx_right = object_center(pred_right[-1]).x - object_center(frame).x
    return dx_left < -0.02 and dx_right > 0.02

score = sum(paired_success(model, *case) for case in cases) / len(cases)

Learning curves and ablations

The first additional figure tracks paired-action success through the fixed training budget. The worked curve improves quickly through 12 hours, then adds only seven points over the final 12. That shape suggests more training alone is unlikely to fix the remaining 19% of pairs. Error inspection becomes more valuable than extending the run.

The ablation figure compares test success at hour 24. Shuffled actions perform close to the no-action model, which is what we want to see if the metric detects real conditioning. Balanced sampling contributes 23 percentage points over the ordinary action-conditioned run. The data distribution matters more here than a larger decoder.

These results belong to the worked setup. One GPU model, one dataset, and one seed cannot support a general claim about world models. They can support the narrower decision to run more seeds on the balanced condition.

Compute accounting

“One GPU” is not a complete compute statement. Record device model, precision, peak allocated memory, update count, tokens or frames per update, wall-clock training time, and energy if it is available. This run processes 14.2 billion frame tokens over 86,400 optimizer steps. Peak allocated memory is 71.4 gigabytes.

Data preparation uses CPU time and storage bandwidth that the slogan leaves out. Preprocessing the clips takes 3.6 hours on a 32-core host and produces 1.1 terabytes of shards. Evaluation takes another 2.4 GPU hours. I would list those costs beside the training run rather than pretend the GPU appeared alone with a clean dataset.

Wall-clock comparisons require fixed evaluation intervals and checkpoint costs. A model evaluated more often may finish fewer updates in 24 hours. For architecture comparisons, match optimizer updates as the primary budget and report elapsed time as a second result.

Failure analysis and next experiment

The 95 failed pairs split into occlusion loss, insufficient displacement, wrong direction, and object-identity swaps. A case can contain several symptoms, but the first causal failure gets the primary label. Occlusion accounts for 43 cases. That suggests adding object persistence supervision before increasing image resolution.

I would train three more seeds for the balanced and ordinary action-conditioned variants. Report the distribution of paired success, not only the best checkpoint. Then extend the paired test to vertical motion and two-object scenes while holding the original set unchanged. A new benchmark should test transfer without erasing the history of the first one.

The reproduction bundle needs dataset manifests, session-level split IDs, transformations, model config, package lockfile, checkpoint hashes, training logs, and raw predictions for all 500 sealed pairs. A reader should be able to recalculate every table value without training the model again.

The experiment earns another weekend because action changes the predicted future in the right direction on a controlled test. It has not earned the claim that the model understands the world. Keeping those two statements separate is the whole point of the small budget.