3D Object Localization With 2D Object Detector and 2D Localization

In this research, we deal with the problem of estimating 3D object positions based on 2D localization data and 2D object bounding boxes determined on the RGB images by a CNN object detector. We use a mobile-manipulating robot equipped with an RGB-D camera and a 2D laser scanner. In contrast to other methods, which are based either on end-to-end neural networks or machine learning solutions, we propose an approach that allows estimating 3D object positions from a sequence of images and 2D robot poses obtained from an onboard 2D localization system. We determine a set of lines from the robot localization data and the object detections, which define the observation directions. The closest point to a set of lines determines an approximate object location. We define the 3D object position estimation as an optimization problem and solve using efficient GPU implementation. Finally, we present results based on data collected on the real robot in an unstructured environment.