Scaling Laws for Imitation Learning in Single-Agent Games
arxiv(2023)
摘要
Imitation Learning (IL) is one of the most widely used methods in machine
learning. Yet, many works find it is often unable to fully recover the
underlying expert behavior, even in constrained environments like single-agent
games. However, none of these works deeply investigate the role of scaling up
the model and data size. Inspired by recent work in Natural Language Processing
(NLP) where "scaling up" has resulted in increasingly more capable LLMs, we
investigate whether carefully scaling up model and data size can bring similar
improvements in the imitation learning setting for single-agent games. We first
demonstrate our findings on a variety of Atari games, and thereafter focus on
the extremely challenging game of NetHack. In all games, we find that IL loss
and mean return scale smoothly with the compute budget (FLOPs) and are strongly
correlated, resulting in power laws for training compute-optimal IL agents.
Finally, we forecast and train several NetHack agents with IL and find they
outperform prior state-of-the-art by 1.5x in all settings. Our work both
demonstrates the scaling behavior of imitation learning in a variety of
single-agent games, as well as the viability of scaling up current approaches
for increasingly capable agents in NetHack, a game that remains elusively hard
for current AI systems.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要