ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictions
CVPR 2024(2024)
Abstract
Although Vision Transformer (ViT) has achieved significant success in
computer vision, it does not perform well in dense prediction tasks due to the
lack of inner-patch information interaction and the limited diversity of
feature scale. Most existing studies are devoted to designing vision-specific
transformers to solve the above problems, which introduce additional
pre-training costs. Therefore, we present a plain, pre-training-free, and
feature-enhanced ViT backbone with Convolutional Multi-scale feature
interaction, named ViT-CoMer, which facilitates bidirectional interaction
between CNN and transformer. Compared to the state-of-the-art, ViT-CoMer has
the following advantages: (1) We inject spatial pyramid multi-receptive field
convolutional features into the ViT architecture, which effectively alleviates
the problems of limited local information interaction and single-feature
representation in ViT. (2) We propose a simple and efficient CNN-Transformer
bidirectional fusion interaction module that performs multi-scale fusion across
hierarchical features, which is beneficial for handling dense prediction tasks.
(3) We evaluate the performance of ViT-CoMer across various dense prediction
tasks, different frameworks, and multiple advanced pre-training. Notably, our
ViT-CoMer-L achieves 64.3
62.1
methods. We hope ViT-CoMer can serve as a new backbone for dense prediction
tasks to facilitate future research. The code will be released at
https://github.com/Traffic-X/ViT-CoMer.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined