Can a video model tell which of its futures to trust?

Noise-Play: using a diffusion model's own denoising dynamics to pick better samples, before they finish generating.

Give a video model the same first frame ten times and you get ten different futures. Some are physically sensible. Others quietly break the scene: objects duplicate, a pipe morphs into something that was never there. Choosing among them usually means finishing every video and asking an outside judge. This post is about an alternative we explored this summer: asking the model itself, partway through generation, which of its candidate futures it can reconstruct most reliably, and betting on those.

The short version is that the model's own denoising process does carry some information about how a sample will turn out, well before the video is done. We found a way to read it, turned it into a small inference-time algorithm called Noise-Play, and tested it on two video generators. The effects are modest and I'll try to be upfront about their size. But the idea is cheap, needs no training and no external verifier, and I think it points somewhere interesting.

I'll assume you know roughly how diffusion and flow-matching models generate by iteratively denoising. Below is an overview of the sections; click to jump.

Same input, different futures

Let's start with a concrete example. Physics-IQ[1] is a benchmark built from real physical scenarios, designed to test whether generative video models understand physical principles. In one episode, a tennis ball rolls out of a pipe onto a table where two other balls are already resting. We give a frozen LTX-2.3 model the opening of that episode plus a prompt, and sample eight videos from eight different noise seeds. Four of them are below.

In the top row, no ball ever leaves the pipe and the rest of the scene stays static. In the second, a ball does appear, but then duplicates. In the bottom two, the pipe and the environment morph in ways that no physical table would allow, and those get the lowest scores by a wide margin: 0.168 and 0.032, against 0.577 and 0.486.

Note that the best-scoring video isn't the one that tells the right story, since nothing happens in it. It's the one that breaks the least. The conditioning signal is identical across all four, and the spread in quality is enormous.

That matters because of where we want these models to go. Video models are increasingly used as world models: generated futures stand in for the consequences of a robot's actions, to train or evaluate policies without touching the real world[2][3][4]. A useful world model has to do more than produce a convincing video. It has to preserve objects, respect geometry and physics, and stay consistent with what it was told. And if the quality varies this much from seed to seed, then deciding which sample to trust is as important as being able to generate one at all.

Best-of-K, and why we wanted something else

The standard answer is best-of-K: draw K independent samples, score each completed video with an external reward model or verifier (often a video-language model acting as a judge), and keep the winner[5]. It works, but it has two problems that bothered us.

1. The signal belongs to someone else. Whatever biases and blind spots the judge has are inherited by the selection, and they needn't match those of the generator[6][7].
2. You pay for every candidate in full. Best-of-K needs all K videos completed and decoded before it can choose. Most of that compute goes to samples you are about to throw away.

A handful of recent works ask whether the generator can supply a useful signal itself[8][9][10]. Proprio[8], for example, uses the model's own flow residual to score samples. That's the direction we wanted to push on, and it led to a pointed question:

Before generation is complete, does a video model's own denoising process reveal which of its stochastic futures will ultimately be more reliable?

A first clue

Here is the observation that got us started. Take the four trajectories from Figure 1 and stop them 40% of the way through denoising. At that point the model's current guess of the final video is a blurry smudge, nothing you'd trust a human or a VLM to grade. Now do something odd to each trajectory: keep its current guess of the clean video fixed, but disturb it using noise directions that come from the other trajectories, and measure how much the model's prediction changes. Call the result the noise value. (I'll define it properly in a moment.)

The ordering lines up perfectly: 2.888 < 3.006 < 3.126 < 3.247 against final scores of 0.577 > 0.486 > 0.168 > 0.032. The differences are small, but they are ordered the right way around. Of course, this is one episode and four seeds; by itself it's an anecdote. The rest of this post is about turning it into something we can measure at scale.

Noise-value: how stable is a future?

The intuition is a simple stress test. Suppose a model is partway through denoising and has a guess y for the final video. Now ask it a slightly rude question: if I keep your guess but swap in a different noise, do you come back to the same answer? If y is something the model represents robustly, it should. If y was a fragile coincidence of one particular noise draw, a different noise should shake it loose.

Setting up

Take a flow-style model where the noisy state is an interpolation between a noise endpoint z and a clean video latent y:

and let f_θ(x_t, t) be the model's prediction of the clean endpoint. The question is where the "different noise" comes from. We found a convenient source: the other samples. If we are running K trajectories in parallel from different seeds, then at time t trajectory k has a state x_t^(k) and proposes a clean endpoint y_k = f_θ(x_t^(k), t). Rearranging the interpolation gives the noise that trajectory implies,

Those are already in memory: no sampling and no extra forward pass. To challenge candidate k with trajectory j, we keep y_k and replace its noise with ẑ_j, then let the model denoise once more:

Averaged over the noise directions we could have chosen, that drift is the quantity we care about:

A small R_t means the model tends to return to roughly the same clean future when approached from other noise directions. The peers' noise gives us a free, on-distribution choice for q: these are noise draws the model itself is currently working with, not arbitrary Gaussian samples. One detail worth stating plainly is that the challenge changes only the noise direction. By construction, the proposed clean endpoint stays fixed.

Does it predict anything?

The test we cared about most was simple: does noise-value, measured early, tell us anything about how good the finished video will be? For each of the 198 Physics-IQ scenarios we start K = 8 independent trajectories from the same conditioning. At checkpoints 20%, 40%, 60% and 80% of the way through denoising, we compute an internal score for every trajectory, using both Noise-Play and Proprio[8]. Then we finish all eight videos and score them with Physics-IQ, which serves purely as the oracle: neither method ever sees it.

We compare each internal ranking against the oracle ranking in two ways: the Spearman rank correlation, and the pairwise accuracy, i.e. the fraction of candidate pairs where the internal score correctly says which video will score higher. A random scorer should land at 0 and 50%.

On LTX-2.3, the signal is strongest around 40% of the way through denoising. There Noise-Play reaches a Spearman correlation of about 0.20 with the eventual Physics-IQ score and ranks a pair correctly 57.8% of the time, while Proprio sits near 52%. After that the signal stays positive but weakens. On HunyuanVideo-1.5 the picture is different in shape: Noise-Play stays above Proprio at every checkpoint, with a flatter profile. The random scorer hovers around zero in both, as it should.

Let me be careful about what this does and doesn't say. A correlation of 0.20 is weak: the oracle outcome is noisy and most of its variance is not explained by this signal. What it does establish is that the signal is real, that it shows up on two different generators, and that it is present substantially before generation is complete. One reading is that there is an intermediate regime where candidate futures have already acquired enough structure to differ in stability, but haven't yet fully committed. That's a hypothesis that fits the data; we haven't directly tested it.

From diagnostic to algorithm

If stability measured mid-generation is informative, the obvious move is to use it to prune. That is Noise-Play: run K trajectories in parallel; at a few chosen noise levels let them challenge each other, drop the ones the model defends worst, and spend the remaining compute on the survivors.

For each candidate k in a round, we define its defense error as the average distance between its original proposal and its challenged reconstructions, over all K_r − 1 peers:

In our LTX-2.3 runs we used K = 4 with two rounds: a first at step 16 of the model's 30-step schedule that cuts four trajectories to two, and a second at step 23 that picks the final one. (These step indices are hyperparameters tied to LTX-2.3's schedule, not constants; more on porting below.) The picture:

The core of it is small. Here is a simplified version of the challenge step, following src/counterfactual_debate_ltx2.py:

# at a debate step we already hold, per active trajectory k:
#   z[k] its noisy state    x0_hat[k] its clean proposal    sigma the noise level
noise = (z - (1 - sigma) * x0_hat) / sigma       # implied noise per trajectory (free)

for k in active:
    for j in active:
        if j == k: continue
        z_swap = (1 - sigma) * x0_hat[k] + sigma * noise[j]  # k's proposal, j's noise
        y_kj = denoiser(z_swap, sigma)                       # one unguided forward pass
        err[k] += dist(pool(x0_hat[k]), pool(y_kj)) / (len(active) - 1)

survivors = argsort(err)[:next_K]                # smallest defense error survives

Two implementation notes. First, dist compares the two latents after a light 1×3×3 average pool (spatial only, no temporal blur) and takes an L2 norm per token. Second, the challenge costs one unguided forward pass per ordered pair of candidates, so the number of extra passes grows like K(K−1). For K = 4 that is 12 passes in round 1 and 2 in round 2. This is cheap for small K, but it is quadratic, and the code even has to chunk the batch to avoid running out of memory at K = 8.

Does it help?

We evaluated Noise-Play on three notions of sample quality: physical plausibility on Physics-IQ[1], temporal motion consistency on WorldScore-Dynamic[11], and 3D geometric consistency on WorldScore-Static. The comparison is always the same: the Noise-Play-selected rollout against a single-seed baseline. We use two generators: LTX-2.3 in a video-to-video setting and HunyuanVideo-1.5 in an image-to-video setting.

Table 1. Physics-IQ. Noise-Play uses only the frozen generator's own denoising dynamics for selection.

The Physics-IQ gains are small in absolute terms, around one point on a scale where the models sit at 26 to 43, but they point the same way for both generators and both versions of the benchmark. WorldScore tells a similar story, with larger relative gains on the geometric metric:

Table 2. WorldScore. Pairwise preference is the fraction of scenes where the Noise-Play-selected rollout beats the single-seed baseline; its Δ is in percentage points above the 50% chance level.

Pairwise preference over the baseline is 54.7% (dynamic) and 54.4% (static) for LTX-2.3: a mild edge. For HunyuanVideo-1.5 it is 68.1% and 66.4%, a far clearer one. Static-scene reprojection error, which measures how well the video's geometry holds together across views, drops by 12.2% on LTX-2.3 and 23.5% on HunyuanVideo-1.5. Motion smoothness improves only modestly for both. Since the selection criterion is identical across generators and across these quite different notions of quality, we read this as evidence that the signal is picking up something about coherence in general, rather than something specific to Physics-IQ.

What the geometry gains look like

Numbers hide what is actually changing, so here are two WorldScore-Static episodes, in which the camera orbits a scene that should otherwise stay still. These are individual examples chosen to illustrate the effect, not averages. Notice how the single-seed LTX-2.3 library drifts and rearranges itself as the camera moves, and how much steadier the Noise-Play pick is.

A controlled comparison

The previous section compares Noise-Play against a single seed. That leaves open how much of the gain is "any selection helps" versus "this particular signal helps". So we ran a cleaner test. For each of the 198 scenarios of Physics-IQ Verified[12], three selectors receive the exact same K = 4 noise initializations and differ only in how they choose the survivor: Random (no quality signal), Proprio (the model's own flow residual) and Noise-Play (noise-value across the seeds). Only the chosen trajectory is continued and decoded.

Random selection gets 40.3; both Noise-Play and Proprio get 44.8. A paired bootstrap with 20,000 resamples and a paired sign-flip permutation test put the gain of Noise-Play over Random at +4.49 (95% CI [+2.72, +6.33], p < 5×10⁻⁵). Against Proprio the difference is −0.03 (95% CI [−1.28, +1.26], p = 0.96): a tie. So the internal signal clearly beats an uninformed choice from the same pool, and the two internal signals are indistinguishable on average.

When they disagree, who is right?

The tie hides something interesting. Noise-Play and Random pick different trajectories in 159 of the 198 scenarios. In those cases, the Noise-Play pick averages 44.5 against Random's 38.8. Excluding two ties, Noise-Play's pick scores higher in 106 of the 157 decisive disagreements, a 67.5% rate. So when the signal actually changes which seed gets chosen, the change more often than not helps.

Noise-Play and Proprio disagree in 101 scenarios, 99 of them decisive. Noise-Play wins 45 of those: 45.5%, with p = 0.42. That's a coin flip.

What it costs

Proprio probes consistency by sampling ε several times and evaluating its flow residual at those perturbed states, for each candidate. Noise-Play can instead stop a sample mid-generation at the challenge steps. End to end, Random takes 280.1 s per scenario, Proprio 349.6 s, and Noise-Play 301.7 s. That's a 13.7% reduction relative to Proprio at the same quality, and a small premium over Random (about 8%) for +4.49 points. Random is still the cheapest, but it uses no quality signal at all.

What we took away

Stepping back from the tables, here is what I think the summer taught us, roughly in order of how confident I am.

1. The model already knows more than it says. A frozen video generator's own denoising dynamics rank its stochastic futures better than chance, with no extra training and no external judge. This held on two different generators and across three different quality measures.
2. The middle of denoising is where it shows up. On LTX-2.3 the signal peaks around 40% progress and is nearly absent at 20%. Our guess is that early on there isn't yet much structure to be stable or unstable about, and later a candidate has mostly committed. We haven't tested that explanation directly.
3. Selecting early is what makes it cheap. Because the signal is usable mid-generation, the trajectories we discard never finish. That is where the cost saving over finishing-then-scoring comes from.
4. Different internal signals can tie while choosing differently. Noise-Play and Proprio reach the same average quality, but pick different trajectories in about half the scenarios, and neither wins those disagreements. I would not conclude they are interchangeable. It's at least suggestive that they are looking at partly different things, and I'd be curious whether combining them helps. We haven't tried.
5. Letting samples talk to each other is a useful idea. The usual picture of parallel sampling is K isolated runs. Here, each trajectory's noise gives the others something to be tested against, The noise itself is free, since it is already computed, and each challenge then costs a single forward pass.
6. The effects are modest, and I'd rather say so. The Physics-IQ gains are around one point; correlations are around 0.2; the pairwise preference on LTX-2.3 is about 54%. The cleanest statistical result is the controlled comparison against random choice. It's real, but this is a selector, so it can only be as good as the best candidate in its pool.

Open questions

One premise of the algorithm is that the challenge noise should come from peer trajectories. We have a variant that draws fresh noise instead, but a clean, matched comparison of the two at full scale hasn't been run yet, so I don't want to claim that peers matter. Two others: how the cost and benefit scale with K, given the quadratic number of challenge passes; and whether the approach carries over to model families that predict velocity rather than the clean endpoint, where one first has to derive the endpoint estimate.

Trying it yourself

The code is at github.com/richardhoff88/denoising-debate. Start with NOISE_PLAY.md, which maps the report's algorithms to the code. The reference implementation is src/counterfactual_debate_ltx2.py (run_counterfactual_debate_schedule_ltx2 is the cleaner R-round version if you are porting). The validated LTX-2.3 configuration on Physics-IQ looks like this:

python scripts/run_counterfactual_debate_ltx2.py \
    --checkpoint-path "$MODELS_DIR/ltx-2.3/ltx-2.3-22b-dev.safetensors" \
    --gemma-root "$MODELS_DIR/gemma-3-12b" \
    --video <conditioning_video.mp4> --prompt-file <prompt.txt> --out-dir <out_dir> \
    --K 4 --debate-step 16 --debate-step-2 23 --finalists 2 \
    --challenger-noise-source peer_residual --aggregation-mode mean_uniform \
    --num-frames 193 --cond-num-frames 73 --frame-rate 24.0 --offload cpu

If you want to try it on another model, the notes in the repo flag what to re-derive. The debate steps are not portable: what matters is the noise level at which the challenge happens, so match by noise level (or fraction of the schedule), not by raw step index. You also need the model's own relationship between its output and a clean-endpoint estimate, and the peer-noise formula (z − (1 − σ)·x̂₀) / σ assumes a linear interpolation between noise and data, so check your model's convention.

Closing thoughts

It's easy to treat a diffusion model's randomness as a nuisance: something to average away or filter by an outside judge after the fact. What we've seen is that the randomness comes with a structure the model itself can read. Independently initialized trajectories turn out to carry information about each other, and about which futures the model represents most stably.

Whether this grows into a general recipe for trustworthy generation depends on questions we haven't answered: does it compose with other internal signals, does it scale to larger candidate pools without the quadratic cost, and does it keep helping as base models improve. But as a training-free way to get a modest, measurable improvement in physical plausibility, motion consistency and 3D geometry from the model you already have, I think it's a promising place to dig. If you try it, or break it, I'd like to hear about it.

About this post. This is joint work with Sarah Liaw, Ruojin Cai and Yilun Du, based on our report Video Reasoning via Denoising-Time Stability. The experimental numbers are all from that report; implementation details (step indices, pass counts, the command below) come from the repo. Nothing here is new analysis.

References

1. Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini, Robert Geirhos. Do generative video models understand physical principles? 2026.
2. Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, Dale Schuurmans. Foundation models for decision making: Problems, methods, and opportunities. 2023. https://arxiv.org/abs/2303.04129
3. Boyuan Chen et al. Large video planner enables generalizable robot control. 2026. https://arxiv.org/abs/2512.15840
4. Gemini Robotics Team et al. Evaluating Gemini robotics policies in a Veo world simulator. 2026. https://arxiv.org/abs/2512.10675
5. Fangfu Liu et al. Video-T1: Test-time scaling for video generation. 2025. https://arxiv.org/abs/2503.18942
6. Audrey Huang et al. Is best-of-n the best of them? Coverage, scaling, and optimality in inference-time alignment. 2025. https://arxiv.org/abs/2503.21878
7. Ming Liu, Wensheng Zhang. Is your video language model a reliable judge? 2025. https://arxiv.org/abs/2503.05977
8. Mariam Hassan, Kaouther Messaoud, Wuyang Li, Alexandre Alahi. Proprio: Latent self-scoring and inference-time refinement for physically plausible video generation. 2026. https://arxiv.org/abs/2605.28230
9. Sangwon Jang et al. Self-refining video sampling. 2026. https://arxiv.org/abs/2601.18577
10. Haoxuan Wu et al. Through the prism: Preference representation in intermediate states of video diffusion models. 2026. https://arxiv.org/abs/2606.20310
11. Haoyi Duan, Hong-Xing Yu, Sirui Chen, Li Fei-Fei, Jiajun Wu. WorldScore: A unified evaluation benchmark for world generation. 2025. https://arxiv.org/abs/2504.00983
12. Tim Rädsch et al. Physics-IQ verified. arXiv:2606.18943, 2026. https://arxiv.org/abs/2606.18943