SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

SimFoundry introduces a modular framework for automated zero-shot real-to-sim scene construction, enabling the creation of high-fidelity digital twins and "digital cousins" to scale the training and evaluation of robotic policies.

Overcoming the Real-World Training Bottleneck

Training and evaluating robot policies within real-world environments presents significant challenges, primarily due to the high costs associated with physical hardware, the risk of damage, and the inherent difficulty of scaling data collection. To address these limitations, researchers have proposed SimFoundry, a system designed to bridge the gap between real-world observations and simulation-ready environments.

Zero-Shot Real-to-Sim Construction

SimFoundry leverages a modular and automated pipeline to perform zero-shot scene construction directly from video input. By transforming a video of a real-world setting into a simulation-ready digital twin, the system allows developers to replicate complex environments without manual 3D modeling or tedious environment configuration.

Digital Twins and Digital Cousins

Beyond simple reconstruction, SimFoundry supports extensive editing of objects, scenes, and tasks. This capability enables the generation of "digital cousins"—affordance-preserving variations of the reconstructed real-world scenes. By creating these variations, the system provides a diverse set of training environments that maintain the functional properties of the original scene while introducing the variability necessary for robust policy generalization.

Impact on Policy Transfer

The primary objective of the SimFoundry framework is to facilitate more effective policy learning. The researchers demonstrate that policies trained on the diverse data generated by SimFoundry can transfer zero-shot to challenging real-world scenarios, reducing the reliance on extensive real-world fine-tuning and accelerating the deployment cycle for robotic agents.

Note: The provided source text is a summary; detailed architectural specifications and specific quantitative performance benchmarks were not included in the input.