Clustering-based Image-Text Graph Matching for Domain Generalization
arxiv(2023)
Abstract
Learning domain-invariant visual representations is important to train a
model that can generalize well to unseen target task domains. Recent works
demonstrate that text descriptions contain high-level class-discriminative
information and such auxiliary semantic cues can be used as effective pivot
embedding for domain generalization problem. However, they use pivot embedding
in global manner (i.e., aligning an image embedding with sentence-level text
embedding), not fully utilizing the semantic cues of given text description. In
this work, we advocate for the use of local alignment between image regions and
corresponding textual descriptions. To this end, we first represent image and
text inputs with graphs. We subsequently cluster nodes in those graphs and
match the graph-based image node features into textual graphs. This matching
process is conducted globally and locally, tightly aligning visual and textual
semantic sub-structures. We experiment with large-scale public datasets, such
as CUB-DG and DomainBed, and our model achieves matched or better
state-of-the-art performance on these datasets. Our code will be publicly
available upon publication.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined