Storage upgrade mechanism

[lavalamp]

Before we remove an API object version, we need to be certain that all stored objects have been upgraded to a version that will be readable in the future. The old plan for that was to run the cluster/upgrade-storage-objects.sh script after each upgrade. Unfortunately, there are a number of problems with this:

• The script is old and incomplete, as api authors haven't been consistently adding their API objects to it (no doubt due to a lack of documentation in the right places).
• Instructions to run the script in the release notes have been spotty.
• We believe not all distributions have been running the script after upgrades.

Therefore, we need to design a robust solution for this problem, which works in the face of HA installations, user-provided apiservers, rollbacks, patch version releases, etc. Probably this won't look much like a script that you run after an upgrade, instead it may look more like a system where apiservers come to consensus on the desired storage version, plus a controller that does the job that script was supposed to do after a consensus change.

In the mean time, it is not safe to remove API object versions, so we are instating a moratorium on API object version removal until this system is in place. (Deprecations are still fine.)

[thockin]

Thanks, Daniel. Good summary. To echo:

it is not safe to remove API object versions, so we are instating a moratorium on API object version removal

Emphasis on removal. It sucks to carry old APIs forward, but this is a liability I think we can't ignore. We've gotten lucky this far.

@kubernetes/sig-network-api-reviews @cmluciano @dcbw @danwinship

@kubernetes/api-approvers @kubernetes/api-reviewers

[smarterclayton]

For context the openshift approach is we force a migration before every upgrade starts, and the migration has to complete. Migration is a kubectl cmd that reads all objects and writes a no-op change (which forces storage turn over). We use this for protobuf migration, field defaulting, new versions, storage versions conversion, self link fixup, etc.

It's basically a production version of upgrade-storage-objects.sh - depending on how complicated we want to get, i'd say it's the minimum viable path, while the cluster consensus is probably fine but more complicated.

https://github.com/openshift/origin/blob/master/pkg/oc/admin/migrate/storage/storage.go

[thockin]

Is this blocking the removal of Alpha APIs, too?

[lavalamp]

@smarterclayton Is that an upstream command?

[lavalamp]

@thockin I think it depends on how the alpha API is storing its objects.

[smarterclayton]

Downstream, it is a bunch of utility code for a migration framework (we have other migrators for things like RBAC, alpha annotations to beta fields, image references) and then a set of simple commands. I don't know that I think of this as kubectl as much as kubeadm or a separate command for admins (in openshift, oc adm is all admin focused commands and oc is all end user focused, but we don't quite have the equiv in kube)

[lavalamp]

@smarterclayton Gotcha. I will take a look at it but that still doesn't sound like an ideal way to solve the problem. Like, it's good if you're going to run one or two of these things with super-well-trained humans doing the upgrade, but Kubernetes isn't installed like that :)

[smarterclayton]

It's just part of general cluster upgrade operations. Someone has to roll the masters. Someone has to lay down new config. I agree there can be magic involved. But magic may have higher cost in some scenarios. On Sep 8, 2017, at 4:28 PM, Daniel Smith <notifications@github.com> wrote: @smarterclayton <https://github.com/smarterclayton> Gotcha. I will take a look at it but that still doesn't sound like an ideal way to solve the problem. Like, it's good if you're going to run one or two of these things with super-well-trained humans doing the upgrade, but Kubernetes isn't installed like that :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#52185 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p9i4xB-URdFNh97YstqxPPuQg8_cks5sgaN8gaJpZM4PReU1> .

[enj]

I have worked closely with OpenShift's installer/upgrade team on their use of the oc adm migrate storage command. I will say running it automatically pre and post upgrade is the trivial part. The command is surprisingly good at finding random edge cases. It could be turned into a controller of sort, though I am not sure what state the controller would try to drive the system from/to.

[timothysc]

Per conversation on sig-api-machinery, there was general consensus that it makes more sense to have this as a controllers responsibility with some set of strategies.