Problem formulation

Vision-based deep learning policies for robotic manipulation suffer from information bottlenecks given a single viewpoint, which becomes extremely severe under strong occlusions of objects. Some strategies leverage multi-view observations to alleviate the problem by point reconstruction explicitly (e.g. Neural Radiance Field (NeRF)) or implicitly in the feature field. Before that, an active agent can plan “where to see” or the so-called Next Best View (NBV) that aims to maximize the information gained from the other viewpoints. One of the directions is uncertainty quantification, which may guide your agent to observe the scene out of curiosity.

Task definition

In this thesis, the primary objective is to investigate methodologies for Next Best View (NBV) planning and how uncertainty quantification can be utilized to guide an agent's exploratory behavior for pointcloud-based deep policy for robotic grasping. We will deploy the developed policy on a physical robotic platform and validate its effectiveness in real-world scenarios, focusing on its ability to handle occlusions or dynamic environments. On the other hand, we may expect the network to be pre-trained by reconstruction tasks of novel viewpoints so it may enhance the scene understanding even if the data from a single viewpoint is given.

You shall offer

• Solid knowledge and experience in computer vision, deep learning.

• Coding skills in Python and Linux.

• Experience in simulation is a plus.

We will offer

• Powerful robot for experiments.

• Powerful GPU server for training your AI.

References

• Dai, Qiyu, et al. "Graspnerf: Multiview-based 6-dof grasp detection for transparent and specular objects using generalizable nerf." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.

• NeU-NBV: Next Best View Planning Using Uncertainty Estimation in ImageBased Neural Rendering.