MathBot: A Personalized Conversational Agent for Learning Math

Online math education often lacks key features of in-person instruction, such as personalized feedback. To emulate such interactivity, we developed MathBot, a rule-based chatbot that explains math concepts, provides practice questions, and offers tailored feedback. We evaluated MathBot through three Amazon Mechanical Turk studies in which participants learned about arithmetic sequences. In the first study, we found that more than 40% of our participants indicated a preference for learning with MathBot over videos and written tutorials from Khan Academy that covered similar material. Although more participants preferred the Khan Academy materials, our results point to demand for alternative forms of online education. The second study measured learning gains, and found that MathBot produced comparable gains to the Khan Academy videos and tutorials. Combined with the findings of our first study, these results indicate that conversational agents can appeal to a substantial proportion of the population without sacrificing learning. Finally, in the third study, we integrated a contextual bandit algorithm into MathBot to experiment with different personalization strategies. Compared to a randomized A/B experiment, we found that the contextual bandit learned a similarly effective pedagogical policy at a lower cost. Our findings suggest that personalized conversational agents are Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. © 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM. ISBN 123-4567-24-567/08/06. . . $15.00 DOI: http://dx.doi.org/10.475/123_4 promising tools to complement existing online resources for math education. ACM Classification