Abstract
We introduce GPWM, a Generative Particle World Model that predicts how 3D scenes evolve by generating the motion of particles and rigid bodies over time.
Unlike video world models that generate scene appearance, GPWM operates directly in 3D physical space. A single representation and architecture cover rigid, deformable, cloth, and granular materials, in both passive physics and robotic interaction. GPWM is an object-aware multi-resolution transformer that combines flow matching with diffusion forcing to generate coherent physical trajectories far beyond its training horizon.
We train and evaluate GPWM on ParticleWorlds, a 25.5-million-frame corpus spanning diverse geometries, materials, interactions, and robot embodiments. GPWM outperforms the best prior learned-dynamics baseline in each of six physical domains, reducing prediction error by 31% on average. Breaking from the domain-specific setting of prior methods, a single Unified GPWM models these heterogeneous regimes with one set of weights, largely outperforming per-domain baselines and even surpassing our own specialist models in two domains. GPWM further generalizes to unseen geometries, physical properties, object counts, and rollout horizons.
Beyond prediction, GPWM enables contact-rich planning by learning a generative prior over joint actuator–object motion. Instead of searching over unconstrained actions, planning searches the model’s latent noise for interactions that achieve a desired object-space goal. With the same world model, noise-space planning substantially outperforms action-space search on pushing and picking, where complex contact sequences are hard to discover. Executed open-loop in simulation, the resulting plans achieve goal-directed manipulation without reference action trajectories or goal-specific policies.
Method
An object-aware, multi-resolution spatio-temporal transformer
Given the observed frames, GPWM models a distribution over the next frames, optionally conditioned on material properties and prescribed actuator motion.
The network progressively compresses dense particles into a small set of anchor tokens, performs most global interaction modeling at coarse resolution, and propagates the resulting features back to the original particles.
Dataset
ParticleWorlds: 25.5 million frames in six domains
Simulated trajectories spanning rigid objects, elastic and elastoplastic solids, granular materials, and cloth, covering passive dynamics and robotic manipulation with rods, parallel-jaw grippers, and articulated hands.
Streaming
Streaming long-horizon generation
Near-future frames are progressively resolved, while farther-future frames remain at higher noise levels and continue to be revised. Once the earliest frames are fully denoised, they are committed, the prediction window advances, and fresh Gaussian noise is appended at the far end of the horizon.
Comparison
Comparison with learned 3D dynamics models
GNS, a graph-based particle simulator; PGND, a hybrid particle–grid model; PointWorld, a point-cloud world model; and PhysCtrl, a generative model of physical dynamics. All methods use the same evaluation trajectories and available actuator conditioning.
Prediction error, in numbers
Planning
Planning with a generative interaction prior
GPWM generates actuator and object motion jointly, so planning can search its latent noise instead of raw actions. Latent noise optimization backpropagates a reward on the desired final object configuration through the generated trajectory, and the actuator part of the result is executed open-loop in simulation, with no reference action trajectory or goal-specific policy. Using the same world model, noise-space planning substantially outperforms action-space search on pushing and picking.
Task progress, in numbers
Real world
Predictions from a single image
Previously unseen object geometries reconstructed from single RGB images, with their dynamics predicted by the Free Fall specialist. The resulting trajectories are rendered back into the scene.
Try it: drop a ball anywhere on the desk
Click the table to drop an elastic or a sand ball.
Try the full model
Drop and drag objects reconstructed from photos, and GPWM predicts the next two seconds. The demo runs on Hugging Face and may take a moment to start.
Open it on Hugging Face for full screen, or if the embedded demo reports a GPU quota limit.
BibTeX
@article{ren2026gpwm,
title = {{GPWM}: Modeling the World as {3D} Particle Motion},
author = {Ren, Robert and Bronars, Matthew and Zhu, He and Tang, Yikai and Zhang, Jianing
and Zakharov, Sergey and Tokmakov, Pavel and Fragkiadaki, Katerina},
journal = {arXiv preprint arXiv:2611.08338},
eprint = {2611.08338},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
year = {2026}
}