Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image. In this talk, I will first investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. Using a training set of aligned image pairs, these networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Second, I will present an approach for learning to translate an image from a source domain to a target domain in the absence of paired examples. We exploit the property that translation should be “cycle consistent”, in the sense that if we translate, e.g., an sentence from English to French, and then translate it back from French to English, we should arrive back at the original sentence. Qualitative results are presented on several tasks where paired training data does not exist, including collection style transfer, object transfiguration, season transfer, photo enhancement, etc. Quantitative comparisons against several prior methods demonstrate the superiority of our approach.

Jun-Yan Zhu is a Ph.D. student at the Berkeley AI Research (BAIR) Lab, working on computer vision, graphics and machine learning with Professor Alexei A. Efros. He received his B.E. from Tsinghua University in 2012 and was a Ph.D. student at CMU from 2012-13. His research goal is to build machines capable of recreating the visual world. Jun-Yan is currently supported by the Facebook Graduate Fellowship.