CRD Lifecycle Management

[robscott]

As the project has reached GA, we've started to get some questions about if cluster provisioning tools and providers should start including Gateway API CRDs. As a project, we need to develop some guidelines for this approach. Note that even if we suggest that cluster provisioning tools include Gateway API CRDs, that will never be the exclusive way for these CRDs to be managed. I'll add my own proposal below to avoid confusing the overall topic with my opinion.

[robscott]

I think that it would be beneficial for cluster provisioning tools and providers to include optional Gateway API CRD management with the following guidelines:

• Only standard channel CRDs should be installed
• The latest version of standard channel CRDs should be included as soon as they are available and rolled out to all Kubernetes versions supported by Gateway API
• This automation layer should never uninstall CRDs

One of the major pain points in adopting Gateway API is CRD management. I think that providing more managed options will help with this. Of course we need to be clear that this will never be the only way to install Gateway API, and that experimentation in non-prod environments with Experimental Channel CRDs is very much encouraged.

From the perspective of cluster provisioning tools and providers, this would mean that several times a year (3-6), they would need to pick up the latest Gateway API CRD definitions and make the latest version available via upgrade.

[tao12345666333]

Since CRD is cluster-level, does this mean that the Gateway API also needs to have periodic releases similar to Kubernetes?

In addition, do we need to provide tools like conftest to assist users in CRD upgrades? (Check whether some changed fields are used in the cluster, and prompt the user whether it can be safely upgraded.

[robscott]

Although we are trying to work towards a more predictable release cadence, that does not need to be aligned with Kubernetes. An important goal of Gateway API is that each release will be compatible with the 5 most recent Kubernetes minor versions. When combined with a recommendation of only automating the installation of "standard channel" (GA) CRDs, this would mean that each new set of CRDs in a standard channel release would also come with stringent backwards compatibility guarantees and the same API version (v1). I believe that means a tool like conftest would not be necessary.

Adding all of that up, the goal is for everyone to be able to safely install the latest version of standard channel Gateway API CRDs as soon as they're released, as long as they're running on one of the 5 most recent versions of Kubernetes.

[Rycieos]

Maybe this is a naive approach, but what if functionality was added to Kubernetes where CRD based APIs could be managed by a resource. For example:

apiVersion: extensions.k8s.io/v1
kind: CrdExtension
metadata:
  name: gateway-api
spec:
  source: https://github.com/kubernetes-sigs/gateway-api
  channel: experimental
  version: v1.0.0

Then the api server would handle installing the CRDs, and if needed, updating them.

Getting this merged into the api server would probably be difficult, so maybe it would need to be an extension server that managed Kubernetes providers would need to offer.

[shaneutt]

Some time ago I remember something vaguely similar to this being asked, where CRDs would have a hub and could be installed as a reaction to use. Honestly to me it sounds like it would have a large number of gotchas, but it would be entirely reasonable to bring it up in an api machinery sync to get feedback.

[youngnick]

The problems here come from trying to reconcile multiple versions. Because different implementations can require different versions, we basically need something where implementations can register the version they need, and then some sort of solver that can figure out what version can satisfy the requirements of all of the implementations.

This is basically the same problem as package managers or language module version management, and requires a similar level of thought - which is why it isn't solved yet. A registry is a good idea, but the hard part is in determining what the correct version is. Probably we want something like Go's minimum compatible version or something, but it's a hard problem, and our use of experimental and standard, while important and necessary, complicates this even further.

[aojea]

Shane did a great presentation about this problem today in SIG network and it really caught my attention.

I was investigating more about this topic and it is interesting you discarded the canonical path of working with CRDs https://gateway-api.sigs.k8s.io/geps/gep-922/#2-reuse-alpha-api-versions and implemented a new path, that it seems has some different problems #3075 .

It seems the overhead of dealing server side with groups and versions and conversions, now is translated to the clients (implementers not users) that need to (client side) try to understand the groups versions and this third dimension of channels, with the different that the GVK between channels are the same, with also different fields.

I really think that is a topic that requires advice from experts, let me tag @jpbetz

[youngnick]

We ended up not doing that because running a webhook for CRDs for a shared API like this is too complex. That was before CEL was a thing, so maybe that might be easier today.

I'd be interested to hear more about where this canonical path is defined though, when we started 5 years ago, there was certainly nothing I could find talking about how to do CRDs, which is why we made up our own thing.

[jpbetz]

This is basically the same problem as package managers or language module version management

This resonates with me.

There appear to be some intersecting problems (experimental versions and feature gating concepts), but the version conflict resolution problem seems central here? I'd like to understand this better. My understanding is that helm does not upgrade CRDs due to risks of the CRD upgrade causing data loss. Are the problems here something that could be solved if CRDs were to be somehow made safe for upgrade and helm adopted a mode where it would perform upgrades automatically based on a version conflict resolution scheme?

[aojea]

when we started 5 years ago, there was certainly nothing I could find talking about how to do CRDs,

That is the part I'm a bit surprised about so I'm now thinking if I'm misunderstanding something, but for CRDs multiversioning you need to implement conversion, same as we do with internal types
https://book.kubebuilder.io/multiversion-tutorial/tutorial
https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/

and this was always this way AFAIK,

because running a webhook for CRDs for a shared API like this is too complex.

heh, that is why the bar in the tree is so high and there is an API review process https://github.com/kubernetes/community/blob/master/sig-architecture/api-review-process.md

There is definitively good docs about this https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md , no easy to discover though

EDIT

Ok, I misunderstood the problem . @liggitt explained me better the problem, I was assuming the problem was only with Types and did not realize this is for fields too

My apologies

[shaneutt]

Are the problems here something that could be solved if CRDs were to be somehow made safe for upgrade and helm adopted a mode where it would perform upgrades automatically based on a version conflict resolution scheme?

Potentially 🤔

Another dimension to the problem we may need to think about is that Gateway API is in a sort of a "uniquely important dependency" position by nature being one of the first "official APIs via CRDs". Gateway API CRDs are increasingly hard dependencies of many different ingress solutions (with some implementations in fact already ONLY supporting Gateway API) across many different implementations on any particular cluster. For myself at least, it's reminiscent of cert-manager.

This dependency problem naturally inclines platforms to provide the CRDs out-of-the-box. GKE (for instance) deploys and manages the lifecycle of Gateway API CRDs today, and other platforms are currently considering doing similar things. This approach may conflict with individual products packaging the CRDs themselves (with Helm or something else) and therefore is at least an issue of having documentation (if not tooling) to provide some guidance and standards on how we (Kubernetes) recommend platforms and integrators manage the lifecycle of these CRDs and how to navigate versioning (see also a mailing-list topic on this subject, which eventually led to a best practices guide we're currently working on).

[shaneutt]

(a little bit of a side note, but as it relates to this: some of you might find this problem relating to CRDs with Gateway API interesting: linkerd/linkerd2#13032)

[jpbetz]

On the feature gate / experimental topic:

I could imagine an alternative to having experimental and stable CRDs.. you had a single set of CRDs with all possible fields, then you use ValidatingAdmissionPolicy to control if the CRDs are in experimental or stable mode. A ValidatingAdmissionPolicy could be configured to know which feature gates are on or off via params. This approximates how builtins's handle feature gated fields, and would allow for more sophisticated checks. For example, a ValidatingAdmissionPolicy could allow the use of a feature, even if the feature gate is off, when it detects that a resource is already using that feature. We do this today in buildin types.

[youngnick]

Thanks @jpbetz, this is pretty interesting. What's the minimum Kubernetes version for ValidatingAdmissionPolicy to be available? I like the sound of this solution, but it would be good to get an idea of what version everyone will need to be running before we can look at moving.

I like the sound of this because it will remove the issue of not being able to safely upgrade from experimental -> stable at the moment (because we can't guarantee that experimental fields won't have breaking changes between experimental and stable).

[jpbetz]

Thanks @jpbetz, this is pretty interesting. What's the minimum Kubernetes version for ValidatingAdmissionPolicy to be available? I like the sound of this solution, but it would be good to get an idea of what version everyone will need to be running before we can look at moving.

I like the sound of this because it will remove the issue of not being able to safely upgrade from experimental -> stable at the moment (because we can't guarantee that experimental fields won't have breaking changes between experimental and stable).

ValidatungAdmissionPolicy is GA in 1.31.

[mlavacca]

This approach sounds very interesting, thanks for bringing it up, @jpbetz! So, to understand better, you set the enabled FGs (or in our case the API channel) in the validatingAdmissionPolicy, and the API validation uses such FGs as conditions to perform validation on the objects' fields?

As Nick said, this approach would help with the many headaches we have today with the double channel.

[youngnick]

@robscott and I have a talk at KCS Salt Lake City in a week and a bit, let's talk some more about this there and come back with updates.