MAPNet: Multi-modal attentive pooling network for RGB-D indoor scene classification.

•Orderless pooling can maintain spatial invariance in local information aggregation for indoor scene classification.•Intra-modality Attentive Pooling mines and pools discriminative local semantic cues in each modality.•Cross-modality Attentive Pooling learns to attend on different modalities in terms of different local cues to fuse the selected discriminative semantic cues across modalities.•The attention weights in the model are interpretable for understanding both scene classification and RGB-D fusion.•State-of-the-art results are achieved on both challenging SUN RGB-D Dataset and NYU Depth V2 Dataset.