MS-VRO: A Multistage Visual-Millimeter Wave Radar Fusion Odometry

Monocular visual odometry (VO) has extensive applications in mobile robots and computer vision. However, current applications of monocular VO systems in complex environments still have limitations. Accurate, robust, and easy-to-use VO is still an unsolved problem to some extent. In recent years, the single-chip millimeter-wave (mmWave) radar has been increasingly used in various types of mobile robots due to its advantages of small size, low cost, and robustness in harsh weather conditions. In this article, we apply the mmWave radar to a VO system and propose a multistage visual-radar fusion odometry framework, MS-VRO. The framework is based on a typical monocular VO system. By merging mmWave radar data in different stages, the proposed odometry improves the accuracy, robustness, and generalization ability of VO. The framework contains a new visual-radar initialization method, a visual-radar joint optimization method, and a radar-aided visual feature selection and processing method that can remove dynamic object features and bad map points. Through these, the proposed method solves the problems of monocular VO, including scale ambiguity, scale drift, and performance degradation in dynamic environments. We build a dataset that can be used for research on visual-radar fusion odometry and test the proposed method on the new dataset and other public datasets. The result shows that the proposed odometry achieves significantly better performance than VO methods and is more accurate and robust compared to some typical visual-inertial odometry methods.