Bridging State and History Representations: Understanding Self-Predictive RL
CoRR(2024)
摘要
Representations are at the core of all deep reinforcement learning (RL)
methods for both Markov decision processes (MDPs) and partially observable
Markov decision processes (POMDPs). Many representation learning methods and
theoretical frameworks have been developed to understand what constitutes an
effective representation. However, the relationships between these methods and
the shared properties among them remain unclear. In this paper, we show that
many of these seemingly distinct methods and frameworks for state and history
abstractions are, in fact, based on a common idea of self-predictive
abstraction. Furthermore, we provide theoretical insights into the widely
adopted objectives and optimization, such as the stop-gradient technique, in
learning self-predictive representations. These findings together yield a
minimalist algorithm to learn self-predictive representations for states and
histories. We validate our theories by applying our algorithm to standard MDPs,
MDPs with distractors, and POMDPs with sparse rewards. These findings culminate
in a set of practical guidelines for RL practitioners.
更多查看译文
关键词
Reinforcement Learning,Representation Learning,POMDPs,Information States,Self-supervised Learning
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要