Predicting and Preventing Hallucinations in Generative World Models

Researchers propose a novel hypothesis that hallucinations in world models are concentrated in low-coverage regions of the state-action space, introducing MMBench2 to detect and mitigate these drifts in visually fluent but dynamically inaccurate rollouts.

The Challenge of Visual Fluency vs. Dynamic Accuracy

Modern generative world models have achieved a high level of proficiency in rendering action-controllable futures that are visually convincing. However, a critical gap remains between visual fluency and physical grounding. These models frequently suffer from "hallucinations," where the generated sequences appear realistic to the human eye but gradually drift away from the ground-truth dynamics of the environment.

The Low-Coverage Hypothesis

The authors, Nicklas Hansen and Xiaolong Wang, hypothesize that these hallucinations are not random but are concentrated in specific low-coverage regions of the state-action space. In these under-represented areas of the training distribution, the model lacks sufficient data to accurately predict the physical consequences of specific actions, leading to a divergence between the predicted rollout and the actual environmental dynamics.

Introducing MMBench2

To validate this hypothesis and develop mitigation strategies, the researchers introduced MMBench2. This new benchmark is designed specifically for visual world modeling and consists of:

• 427 hours of high-quality visual data.
• 210 distinct tasks.
• Comprehensive ground-truth action sequences to enable precise measurement of dynamic drift.

By utilizing lightweight, data-centric signals, the researchers aim to detect when a model enters these low-coverage regions, thereby providing a mechanism to prevent hallucinations before they compromise the integrity of the rollout.

Note: The provided source text was truncated; further details regarding the specific mitigation techniques and final experimental results are not available in the provided snippet.