Salient Object Detection From Arbitrary Modalities
CoRR(2024)
Abstract
Toward desirable saliency prediction, the types and numbers of inputs for a
salient object detection (SOD) algorithm may dynamically change in many
real-life applications. However, existing SOD algorithms are mainly designed or
trained for one particular type of inputs, failing to be generalized to other
types of inputs. Consequentially, more types of SOD algorithms need to be
prepared in advance for handling different types of inputs, raising huge
hardware and research costs. Differently, in this paper, we propose a new type
of SOD task, termed Arbitrary Modality SOD (AM SOD). The most prominent
characteristics of AM SOD are that the modality types and modality numbers will
be arbitrary or dynamically changed. The former means that the inputs to the AM
SOD algorithm may be arbitrary modalities such as RGB, depths, or even any
combination of them. While, the latter indicates that the inputs may have
arbitrary modality numbers as the input type is changed, e.g. single-modality
RGB image, dual-modality RGB-Depth (RGB-D) images or triple-modality
RGB-Depth-Thermal (RGB-D-T) images. Accordingly, a preliminary solution to the
above challenges, ı.e. a modality switch network (MSN), is proposed in this
paper. In particular, a modality switch feature extractor (MSFE) is first
designed to extract discriminative features from each modality effectively by
introducing some modality indicators, which will generate some weights for
modality switching. Subsequently, a dynamic fusion module (DFM) is proposed to
adaptively fuse features from a variable number of modalities based on a novel
Transformer structure. Finally, a new dataset, named AM-XD, is constructed to
facilitate research on AM SOD. Extensive experiments demonstrate that our AM
SOD method can effectively cope with changes in the type and number of input
modalities for robust salient object detection.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined