Problem formulation

Deep learning-based computation of the rigid 6D transformation from the object to the camera, also known as object pose estimation, is crucial for a variety of applications, such as robotic manipulation or augmented reality. Classic methods are known as instance-level since they only work on the specific object instance determined at training time. For more complicated cases such as occlusions, changed object textures or environmental conditions may greatly influence the estimation accuracy.

Task definition

We consider real2sim transfer through the recovery of real-world objects into our simulation platform. Powered by the highly realistic simulator, we’re already capable of generating synthetic sensor data with high fidelity, while, a direct real2sim alignment still face the challenges as mentioned above. We aim to study how to improve our pose estimation framework that alleviates these obstacles, such as multi-view uncertainty and correspondences to improve the pose estimation results. In addition, we’re studying to augment our robot simulation data through randomizations of the manipulation environment. Existing robotic grasping algorithms will be trained on our generated new data to be compared by their performance.

You shall offer

• Solid knowledge and experience in computer vision, and deep learning.

• Coding skills in Python and Linux. Experience in simulation (IsaacSim) is a plus.

We will offer

• Robot & powerful sensors for data collection.

• Powerful GPU server for training your AI.

References

• Wen, Bowen, et al. "Foundationpose: Unified 6d pose estimation and tracking of novel objects." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

• Liu, Jian, et al. "Deep Learning-Based Object Pose Estimation: A Comprehensive Survey." arXiv preprint arXiv:2405.07801 (2024).