Multimodal Sensor Fusion with Deep Learning

This report documents the efforts over the three year 6.1 base program titled Multimodal Sensor Fusion with Deep Learning. Herein a novel framework for fusing hyperspectral imagery (HSI) and LiDAR data for urban land use and land cover classification is detailed. In this approach multimodal sensor fusion was enhanced by utilizing deep learning and fuzzy logic to amalgamate information between spatial and spectral domains. Deep learning techniques enabled exploitation of mid and high level correlations between contrasting domains that have weak correlation between low level representations. The introduction of fuzzy logic further improved sensor fusion by providing advantages with difficult samples and the opportunity to gain network explainability.