GOMAA-Geo: GOal Modality Agnostic Active Geo-localization
CoRR(2024)
摘要
We consider the task of active geo-localization (AGL) in which an agent uses
a sequence of visual cues observed during aerial navigation to find a target
specified through multiple possible modalities. This could emulate a UAV
involved in a search-and-rescue operation navigating through an area, observing
a stream of aerial images as it goes. The AGL task is associated with two
important challenges. Firstly, an agent must deal with a goal specification in
one of multiple modalities (e.g., through a natural language description) while
the search cues are provided in other modalities (aerial imagery). The second
challenge is limited localization time (e.g., limited battery life, urgency) so
that the goal must be localized as efficiently as possible, i.e. the agent must
effectively leverage its sequentially observed aerial views when searching for
the goal. To address these challenges, we propose GOMAA-Geo - a goal modality
agnostic active geo-localization agent - for zero-shot generalization between
different goal modalities. Our approach combines cross-modality contrastive
learning to align representations across modalities with supervised foundation
model pretraining and reinforcement learning to obtain highly effective
navigation and localization policies. Through extensive evaluations, we show
that GOMAA-Geo outperforms alternative learnable approaches and that it
generalizes across datasets - e.g., to disaster-hit areas without seeing a
single disaster scenario during training - and goal modalities - e.g., to
ground-level imagery or textual descriptions, despite only being trained with
goals specified as aerial views. Code and models are publicly available at
https://github.com/mvrl/GOMAA-Geo/tree/main.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要