Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning
arxiv(2024)
摘要
Reinforcement learning with multiple, potentially conflicting objectives is
pervasive in real-world applications, while this problem remains theoretically
under-explored. This paper tackles the multi-objective reinforcement learning
(MORL) problem and introduces an innovative actor-critic algorithm named MOAC
which finds a policy by iteratively making trade-offs among conflicting reward
signals. Notably, we provide the first analysis of finite-time
Pareto-stationary convergence and corresponding sample complexity in both
discounted and average reward settings. Our approach has two salient features:
(a) MOAC mitigates the cumulative estimation bias resulting from finding an
optimal common gradient descent direction out of stochastic samples. This
enables provable convergence rate and sample complexity guarantees independent
of the number of objectives; (b) With proper momentum coefficient, MOAC
initializes the weights of individual policy gradients using samples from the
environment, instead of manual initialization. This enhances the practicality
and robustness of our algorithm. Finally, experiments conducted on a real-world
dataset validate the effectiveness of our proposed method.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要