Issue on real-world datasets collection

[zengxu-he]

Thanks for the incredible work.

I notice that all the experiments are conducted on simulation platform. Could you please show more details on real-world data collection? Such as, should real-world dynamic datasets are necessary to fintune the model? Or, can static real-world datasets be used to make the finetuned model get the same perfomance?

If real-world dynamic datasets are necessary, could you show more information about the robot trajectory design principles?

[FangHeng]

Thanks for your kind words and the great question!

Real-world study is on the way. We are currently consolidating a set of real-world experiments and will update both the repository and paper in the near future — please stay tuned.

We carried out real-world experiments in a setup with strictly calibrated object assets aligned with the RoboTwin asset library, and the policy trained purely in simulation on our clean + random data already shows encouraging performance when directly deployed on the real robot

On the role of real-world data. Based on our experiments, our current view is:

• Large-scale simulation pretraining provides a strong prior. The pretrained policy already captures a lot of the spatiotemporal structure needed for dynamic manipulation.
• A modest amount of real-world fine-tuning may still necessary to close the residual sim-to-real gap.
• Whether static-only real-world data is sufficient to match the performance of dynamic real-world fine-tuning remains an open question. We do not yet have a definitive answer. We believe it deserves further systematic study

Trajectory design principles. For real-world dynamic data collection, we would recommend following the same design logic as in our simulation setup. Specifically, it may be helpful to organize motions into hierarchical levels of complexity and keeping the motion speed within a range that is reasonable for both the robot controller and the policy's reaction bandwidth. Aligning the real-world trajectory distribution with the simulation data in this way is likely to make the policy benefit more directly from DOMINO pre-training. The reference implementation of our simulation trajectory generator can be found in envs/utils/trajectory_generator.py, which can serve as a starting point for designing your own real-world collection scheme.

Thanks again for the interest.