Kubernetes is rapidly becoming the standard orchestration tool for declaratively managing open infrastructure. Over the last two years, we have been running baremetal Kubernetes clusters in production that are running challenging containerized workloads including OpenStack itself. We have upgraded these workloads and the Kubernetes infrastructure itself while maintaining these mission critical environments powering our 5G infrastructure. In this talk we will revisit some of the lessons learned in dealing various challenges along the way from upgrading Kubernetes and the unexpected fallouts that can occur when running complex workloads; docker stability and upgrades; CPU time stealing issues with real time workloads; CNI upgrades in running environments; debugging containerized neutron agents; and issues when workloads like OpenStack tap into functionality like hugepages, cpu pinning, and others that Kubernetes may not account for cleanly from release to release.