Predictive Knowledge in Robots: An Empirical Comparison of Learning Algorithms

Knowledge is central to intelligence. Intelligence can be thought of as the ability to acquire knowledge and apply it effectively. Despite being a subject of intense interest in artificial intelligence, it is not yet clear what the best approach is for an intelligent system to acquire and maintain a large body of knowledge. One interesting approach that we pursue in this thesis is based on the view that much of world knowledge is predictive. For example, to know that a box is heavy, is to predict that we need lots of effort to lift it. We call this predictive approach to maintaining and acquiring knowledge, the predictive knowledge approach. In this thesis, we implement an instance of this approach in order to explore and assess it further. To do so, we build upon the techniques and ideas of reinforcement learning. In particular, we use the idea of value functions. In conventional RL, value functions capture predictions about reward. Recently, value functions have been extended to capture more general predictions which can constitute knowledge. A value function in the extended form is called a general value function (GVF). GVFs provide a language to talk and think about predictions. More generally, we can think of GVFs as a language for representing predictive knowledge. In this thesis, we develop the predictive knowledge view using the language of GVFs and apply it to several robot domains. Our work has three main contributions. First, we contribute to the idea of predictive knowledge by providing several new examples of it on robot domains, gaining a more substantive understanding of knowledge as predictions. Second, we perform ii empirical comparisons of many off-policy temporal-difference (TD) learning algorithms including gradient-TD and emphatic-TD families of methods on robot data. Third, we systematically study the learning process on robots. Such studies provide insights about how to effectively evaluate and compare algorithms on real-world systems.