RLingua: Improving Reinforcement Learning Sample Efficiency in Robotic Manipulations With Large Language Models
IEEE Robotics and Automation Letters(2024)
Abstract
Reinforcement learning (RL) has demonstrated its capability in solving
various tasks but is notorious for its low sample efficiency. In this paper, we
propose RLingua, a framework that can leverage the internal knowledge of large
language models (LLMs) to reduce the sample complexity of RL in robotic
manipulations. To this end, we first present how to extract the prior knowledge
of LLMs by prompt engineering so that a preliminary rule-based robot controller
for a specific task can be generated. Despite being imperfect, the
LLM-generated robot controller is utilized to produce action samples during
rollouts with a decaying probability, thereby improving RL's sample efficiency.
We employ the actor-critic framework and modify the actor loss to regularize
the policy learning towards the LLM-generated controller. RLingua also provides
a novel method of improving the imperfect LLM-generated robot controllers by
RL. We demonstrated that RLingua can significantly reduce the sample complexity
of TD3 in the robot tasks of panda_gym and achieve high success rates in
sparsely rewarded robot tasks in RLBench, where the standard TD3 fails.
Additionally, We validated RLingua's effectiveness in real-world robot
experiments through Sim2Real, demonstrating that the learned policies are
effectively transferable to real robot tasks. Further details and videos about
our work are available at our project website https://rlingua.github.io.
MoreTranslated text
Key words
Reinforcement learning (RL),large language models (LLMs),robotic manipulations,sample complexity
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined