PIN-WM

Learning Physics-INformed World Models for Non-Prehensile Manipulation

Robotics: Science and Systems (RSS) 2025

PIN-WM : Learning Physics-INformed World Models for Non-Prehensile Manipulation

Wenxuan Li1,∗ , Hang Zhao2,∗ , Zhiyuan Yu2 , Yu Du1 , Qin Zou2,4 , Ruizhen Hu3,†, Kai Xu1,† ,
1National University of Defense Technology     2Wuhan University    3Shenzhen University 4Guangdong Laboratory of Artificial Intelligence and Digital Economy
*Equal contributions    †Corresponding author
icon RSS 2025
Paper arXiv Code

Push object on a slippery plane

PIN-WM
Baseline
PIN-WM
Baseline

Push object on a rough plane

PIN-WM
Baseline

Flip object on a rough plane

PIN-WM
Baseline

Other results

PIN-WM

Abstract

While non-prehensile manipulation (e.g., controlled pushing/poking) constitutes a foundational robotic skill, its learning remains challenging due to the high sensitivity to complex physical interactions involving friction and restitution. To achieve robust policy learning and generalization, we opt to learn a world model of the 3D rigid body dynamics involved in non- prehensile manipulations and use it for model-based reinforcement learning.

We propose PIN-WM, a Physics-INformed World Model that enables efficient end-to-end identification of a 3D rigid body dynamical system from visual observations. Adopting differentiable physics simulation, PIN-WM can be learned with only few-shot and task-agnostic physical interaction trajectories. Further, PIN-WM is learned with observational loss induced by Gaussian Splatting without needing state estimation. To bridge Sim2Real gaps, we turn the learned PIN-WM into a group of Digital Cousins via physics-aware perturbations which perturb physics and rendering parameters to generate diverse and meaningful variations of the PIN-WM.

Extensive evaluations on both simulation and real-world tests demonstrate that PIN- WM, enhanced with physics-aware digital cousins, facilitates learning robust non-prehensile manipulation skills with Sim2Real transfer, surpassing the Real2Sim2Real state-of-the-arts.

Method

Banner Image

A.Real2Sim System Identification:

(a) Rendering alignment : The robot in the target domain moves around the object, capturing multi-view observations to estimate the rendering parameters 𝜶 of 2D Gaussian Splats. Once optimized, 𝜶 is frozen.
(b) Identification of physics parameters : Both source and target domain apply the same task-agnostic physical interactions 𝒂t. In the source domain, dynamics are computed via LCP with physical parameters θ to update the rendering. θ is then optimized with the rendering loss between two domains.

B.Sim2Real Policy Transfer:

(c) Policy learning with physics-aware digital cousins : The identified world model is then used for policy learning. Physics-aware perturbations are introduced to 𝜶 and θ to mitigate the remained discrepancies from inaccurate observations.
(d) Transfer to the target domain without fine-tuning : This ensemble of perturbed world models enhances the Sim2Real transferability of learned policies.

BibTeX

@article{li2025pin,
        title={PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulation},
        author={Li, Wenxuan and Zhao, Hang and Yu, Zhiyuan and Du, Yu and Zou, Qin and Hu, Ruizhen and Xu, Kai},
        journal={arXiv preprint arXiv:2504.16693},
        year={2025}
      }