This research investigates why state-of-the-art video diffusion models frequently generate content that violates physical laws, identifying root causes within internal attention mechanisms rather than data deficiencies. Through an interpretability study of the "motion planning" process during early denoising stages, the authors discovered a "first shape, then details" phenomenon. By analyzing cross-attention trajectory patterns and causal head contributions, they isolated a specific subset of attention heads responsible for driving motion trajectories. Crucially, the study reveals that Rotary Position Embedding (RoPE) induces excessive spatial attention decay. This decay causes early candidate regions to prematurely lock into physically implausible positions, which suppresses viable trajectories in subsequent frames and triggers motion failures. To mitigate this, the researchers propose a lightweight architectural modification that scales the frequency of RoPE across various denoising steps. This adjustment reduces excessive attention decay, allowing the model to explore a broader range of candidate regions and establish more coherent, physically plausible motion. The effectiveness of this approach in enhancing physical commonsense is validated through both training-free and training-based experimental frameworks.

Read original