Recently, the computer vision community has made impressive progress on object recognition with deep learning approaches. However, for any visual system to interact with objects, it needs to understand much more than simply recognizing where the objects are. The goal of my research is to explore and solve physical understanding tasks for interaction -- finding an object's pose in 3D, interpreting its physical interactions, and understanding its various states and transformations. Unfortunately, obtaining extensive annotated data for such tasks is often intractable, yet required by recent popular learning techniques.

In this talk, I take a step away from expensive, manually labeled datasets. Instead, I develop learning algorithms that are supervised through physical constraints combined with structured priors. I will first talk about how to build learning algorithms, including a deep learning framework (e.g., convolutional neural networks), that can utilize geometric information from 3D CAD models in combination with real-world statistics from photographs. Then, I will show how to use differentiable physics simulators to learn object properties simply by watching videos.

Joseph Lim will be an assistant professor at the University of Southern California starting Spring 2017, and is currently a postdoctoral researcher working with Professor Fei-Fei Li at Stanford University. He completed his PhD at Massachusetts Institute of Technology under the guidance of Professor Antonio Torralba, and also had a half-year long postdoc under Professor William Freeman. His research interests are in artificial intelligence, computer vision, and machine learning. He is particularly interested in deep learning, structure learning, and multi-domain data. Joseph graduated with BA in Computer Science from UC Berkeley, where he worked under Professor Jitendra Malik.