Carnegie Mellon University researchers developed REST3D, a framework that converts casual images into physically stable 3D scenes. Existing single-image reconstruction methods create geometrically plausible but physically inconsistent results, with objects floating or penetrating. REST3D integrates physical scene understanding with physics-constrained refinement to create simulation-ready digital assets.

The method uses an agentic technique that constructs a scene-tree representation capturing object physical states and inter-object relationships from a gravity-support perspective. This structure guides reconstruction through image-to-3D models and physics-constrained optimization, resolving physical violations while preserving visual consistency with the input image.

Experiments show REST3D significantly reduces physical errors and improves simulation stability on both synthetic and real-world datasets. The researchers demonstrated the technology with VR-based human-object interaction, enabling users to naturally interact with stable virtual objects through hand-based interactions in reconstructed scenes.