Over last year Kubernetes has become similar platform for containers as OpenStack in virtual machines. We spent last 6 month of running Kubernetes for microservices applications as well as for OpenStack itself. We containerazed whole OpenStack including libvirt for more than 100 compute nodes.
We discover that networking is one of the most challenging aspects to running Kubernetes. Therefore over the last month, tcp cloud has run extensive performance and diagnostics tests across multiple overlay providers (OpenContrail, Calico, Romana, etc.) as well as against the base Kubernetes Flannel configurations and have come to several insights into CPU penalties. Network design issues at scale as well as performance comparisons use different encapsulation techniques. We deployed Kubernetes cluster on 300 physical servers!
We will explain what our architectural design was, the results of our testing as well as our design insights into how the real production will be affected by the outcome of the evaluation.