High-performance DNN inference and training is essential for the ongoing ML revolution. Training of DNNs requires massive memory capacity and bandwidth, and is generally a huge pain, especially for researchers. While significant research effort has been dedicated to inference accelerator, less work has been done on training, especially work that crosses the algorithmic and implementation layers. The result is high-cost accelerators, in particular, with very expensive high-bandwidth memories. I will motivate and discuss some of our recent work on accelerating training (of CNNs) that combines understanding of and changes to the algorithm with matching hardware architecture modifications. Specifically, I will present a technique that reduces the bandwidth requirements of training modern CNNs like ResNet and Inception by 4x and a second technique that can speed up training of pruned networks by 2x or more even on current hardware. The algorithmic changes further motivate a new architecture that achieves the efficiency of a large systolic array without suffering low utilization when using our pruning while training approach.

Mattan Erez is a Professor at the Department of Electrical and Computer Engineering at the University of Texas at Austin. His research focuses on improving the performance, efficiency, and scalability of computing systems through advances in memory systems, hardware architecture, software systems, and programming models. His current focus areas are architectures for machine learning, large-scale and high-performance computing, and memory systems. His work aims to improve cooperation across system layers and develop flexible and adaptive mechanisms for proportional resource usage. Mattan received a BSc in Electrical Engineering and a BA in Physics from the Technion, Israel Institute of Technology and his MS and PhD. in Electrical Engineering from Stanford University. He was awarded a Presidential Early Career Award for Scientists and Engineers from President Obama and received an Early Career Research Award from the Department of Energy and an NSF CAREER Award.