Reinforcement Learning and Apprenticeship Learning for Robotic Control (invited lecture for ALT and DS 2006) Author: Andrew Ng Affiliation: Computer Science Department, Stanford University, Stanford, U.S.A. Abstract. Many control problems, such as autonomous helicopter flight, legged robot locomotion, and autonomous driving are difficult because (i) It is hard to write down, in closed form, a formal specification of the control task (for example, what is the cost function for "driving well"?), (ii) It is difficult to learn good models of the robot's dynamics, and (iii) It is expensive to find closed-loop controllers for high dimensional, highly stochastic domains. Using apprenticeship learning—in which we learn from a human demonstration of a task—as a unifying theme, I will present formal results showing how many control problems can be efficiently addressed given access to a demonstration. In presenting these ideas, I will also draw from a number of case studies, including applications in autonomous helicopter flight, quadruped obstacle negotiation, snake robot locomotion, and high-speed off-road navigation. Finally, I will also describe the application of these ideas to the STAIR (STanford AI Robot) project, which has the long term goal of integrating methods from all major areas of AI—including spoken dialog/NLP, manipulation, vision, navigation, and planning—to build a general-purpose, "intelligent" home/office robotic assistant. Joint work with Pieter Abbeel, Adam Coates, Ashutosh Saxena, Jeremy Kolter, Honglak Lee, Yirong Shen, Justin Driemeyer, Justin Kearns, and Chioma Osondu. ©Copyright 2006 Author