Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understanding
CoRR(2023)
摘要
Power-law scaling indicates that large-scale training with uniform sampling
is prohibitively slow. Active learning methods aim to increase data efficiency
by prioritizing learning on the most relevant examples. Despite their appeal,
these methods have yet to be widely adopted since no one algorithm has been
shown to a) generalize across models and tasks b) scale to large datasets and
c) yield overall FLOP savings when accounting for the overhead of data
selection. In this work we propose a method which satisfies these three
properties, leveraging small, cheap proxy models to estimate "learnability"
scores for datapoints, which are used to prioritize data for the training of
much larger models. As a result, our models require 46
updates and up to 25
uniformly trained visual classifiers on JFT and multimodal models on ALIGN.
Finally, we find our data-prioritization scheme to be complementary with recent
data-curation and learning objectives, yielding a new state-of-the-art in
several multimodal transfer tasks.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要