Honey, I Blew Up the Cloud
Over the past couple of years, Bloomberg engineers have rapidly scaled several single-cell, private OpenStack clouds from inception to tens of thousands of VMs and Ceph-backed volumes governed by thousands of security groups and associated rule sets. While this spectacular growth has come with its fair share of aches and pains, we have rejiggered our control plane topology on the fly, tailored and distributed the metadata service to stave off database query load, patched Neutron ORM code to address API brownouts, and more in order to manage our continued rapid growth. With most of our growing pains now behind us, we would love to share with the community our journey and many of the optimizations we found necessary to implement along the way to get us to where we are today.
