Many robotics tasks, even seemingly simple procedural tasks like assembly and cleaning, require a continuous cycle of planning, learning, adapting and executing of diverse skills and sub-tasks. It is thus hard to scale and generalize learning agents and hard-programed agents to long-horizon complex tasks. To this end, my research is about enabling autonomous agents to perform long-horizon, complex physical tasks. More specifically, I propose a hybrid paradigm that augments classic rule-based methods (program) with the flexibility of learning-based approaches. The key insights are that the program representation enforces an explicit split between the hierarchical subtask structure of any long-horizon task and the exact execution of each of the subskills, and hence learning methods can focus on elemental components, such as skill acquisition, program inference, and task execution, rather than learning everything end-to-end. When combined with recent learning methods, such as deep reinforcement learning and meta-learning, our approach can generate flexible and interpretable long-horizon plans, and adaptively follow these plans using a set of learned subskills. In this talk, I will talk about three layers on my work: (1) learning to infer a program, (2) developing learning methods for skill acquisition and generalization, and (3) learning to execute a program-guided task.

Joseph Lim is an assistant professor in the computer science department at the University of Southern California. Previously, he was a postdoctoral scholar at Stanford Artificial Intelligence Laboratory with the Computer Vision group led by Professor Fei-Fei Li. Before that, he completed my PhD at Massachusetts Institute of Technology under the guidance of Professor Antonio Torralba, and also had a half-year long postdoc under Professor William Freeman. He received his bachelor degree at the University of California - Berkeley, where he worked in the Computer Vision lab under the guidance of Professor Jitendra Malik. He also have spent time at Microsoft Research, Adobe Creative Technologies Lab, and Google.