Queens release was the first one which addressed OpenStack problem of upgrades between relatively frequent releases.
Fast Forward Upgrade (FFU), Skip Upgrade or (N+3) upgrade are the terms describe the same thing, upgrade from OpenStack Newton to OpenStack Queens.
We are now a year after Queens release and it may be a good time to look again on the N+3 upgrade,
when Telcos are scheduling to perform upgrades on their own infrastructure and understand the complexity of these upgrades.
In Telco world, OpenStack upgrade can't be isolated from other components and triggers other elements upgrade:
- operating system upgrade
- kernel drivers upgrades
- HW firmware upgrades
- and finally, VNFs workload running on top of the cluster which often need to be upgraded as well
On this session, I am going to discuss the most common problems we've seen so far?
Share some lessons learned. And compare TripleO (N+3) upgrade with other ways of doing the upgrades.
OpenStack upgrades are difficult and (N+3) Fast Forward Upgrade is even more challenging. This is a complex procedure which is hard to fully automate as it's rather unreliable and depend on customer-specific configuration.
There are multiple projects out there which shows how to do an upgrade in a few minutes, but is it really a full upgrade? Docker images swap is NOT a full upgrade.
The full upgrade includes also upgrade of underlying infrastructure:
- VNFs workload upgrade
- Ceph upgrade
- operating system upgrade
- kernel drivers upgrades
- firmware upgrades
Full cluster reboot after upgrade with VNFs running on top of OpenStack cluster is slightly more complex than a reboot of a cluster running web type workload.
Ceph upgrade is a separate and complex beast on its own especially when you want to migrate Ceph data from FileStore to Blue store during your upgrade.
There is a lot of corner cases which you should know and be prepared before your upgrade. The main goal of that session is to share experiences we gained doing real production OpenStack upgrades and FFU with TripleO.