Facebook's infrastructure serves millions requests per second,providing reliable personalized experience to more than a billionpeople from all over the world. It is comprised of hundreds ofdistributed, interconnected internal services which rapidly evolve andchange in a decentralized manner. This infrastructure is deployedamongst many geographically distributed datacenters.

This presents non-trivial capacity management challenges: we need areliable tool for understanding capacity bottlenecks and analyzingperformance of individual services so that we can appropriatelyallocate resources amongst these services. Understanding the capacityof each of these datacenters is critical to guarantee optimal userexperience during planned and unplanned datacenter outages.

To meet these challenges, we developed Keanu, a family of automatedcontinuous load testing tools running at different levels ofinfrastructure hierarchy. Keanu provides essential information foroptimizing resource allocation, it identifies capacity regressions andis used as an A/B testing framework for improving performance ofindividual services. In this talk I’ll describe and motivate theapproach for continuous load testing of large-scale systems. I’ll alsopresent the common patterns that cause problems in our infrastructureand talk about the ways load testing is essential to unveil theseissues.

Wonho Kim is a software engineer in the infrastructure group atFacebook, where he works on datacenter capacity management anddisaster recovery. He received his Ph.D. in Computer Science fromPrinceton University (2012), an M.A. from Princeton University (2010),and a B.S. from Seoul National University (2006). He is a recipient ofDoctoral Study Fellowship from Korean Foundation for Advanced Studies(KFAS).