Multi-sensor Fusion Network for Unstructured Scene Segmentation with Surface Normal Incorporated

LiDAR (Light Detection and Ranging) and camera are two commonly used sensors to acquire data of different modalities in environmental perception. For autonomous vehicles operating in unstructured scenes, it is particularly important to fuse these two different data for semantic segmentation. Most existing methods rely solely on data from a single sensor, which partially limits segmentation performance. Some fusion-based methods, however, are unable to trade off the contribution of point cloud features and image features, resulting in mediocre performance. To address these issues, we propose a multi-sensor fusion network for unstructured scene segmentation with surface normal incorporated, called MF-SN Net. For the first factor, we effectively use complementary features to highlight the characteristics of unstructured scenes. Point represented and RV (range view) represented LiDAR information are combined as baseline to fully utilize 3D information while ensuring efficiency. RGB images from camera are re-weighted and fused with RV represented LiDAR features at different scales. Second, we propose a novel method to re-weight image features before fusing them into LiDAR features using SN (surface normal) information, which can effectively reduce the negative impact of inaccurate image features. Third, we design a cross-layer attention module to enhance more semantic information from high-level features to different layers, which can optimize features extraction from original point clouds. What’s more, we make a synthesized dataset using CARLA simulator to enrich the experimental scenes, so that the network’s performance can be evaluated in various conditions. Experimental results on different datasets demonstrate the effectiveness and robustness of our network, showing its competitiveness with state-of-the-art methods.