H4HIP: Helm Sequencing Proposal

[Jeb135]

proposal to add resource sequencing support to Helm for v4

[scottrigby]

Nice proposal 👏 Definitely has come up in conversations a lot. Added some suggestions, and questions.

[scottrigby]

Does this proposal also include sequencing for uninstall, or only install/upgrade? For example, waiting to uninstall a resource until all that depends on it have completely uninstalled?

[Jeb135]

Good point I didn't think about that. At least with uninstall it can just be done backwards to the installation, but upgrade would add some complexity that I'm not sure how to tackle.

[Jeb135]

I added a section to clarify that this would apply to all 3, installation/uninstall/upgrade

[joejulian]

and rollback, I assume

[scottrigby]

To be clear, helm.sh/depends-on/chart refers to some chart that is not listed as a dependency of the current chart? Is that right?

• assuming this because the wait flag should already check that dependent charts are fully installed before installing the current chart
• is the idea that this is limited to other charts that have already been deployed to the same namespace, or is there an idea to check charts in other namespaces (assuming the user has correct RBAC permissions)?

My other question here is, do we want to allow depending on other named k8s resources that are not defined by a chart? Eg, helm.sh/depends-on/resource? Or is the idea that users could annotate other resources not defined by a chart with helm.sh/layer, declaring that they are considered part of the same layer of a chart?

[Jeb135]

helm.sh/depends-on/chart is referring to any direct chart dependencies. I was unsure if Helm waits to install the current chart until all dependent charts are installed and assumed not. If Helm is already waiting for dependent charts to be installed, then this section can probably be removed?

[banjoh]

assuming this because the wait flag should already check that dependent charts are fully installed before installing the current chart

helm.sh/depends-on/chart meant for both dependencies and subcharts, will wait regardless of --wait/--wait-for-jobs CLI flags being present. Unless we want to have a way to opt-out from sequencing, which we can then assume if --wait/--wait-for-jobs are absent, not sequencing. Today, --wait will wait for certain resources only. In the proposal we have helm.sh/resource-ready which a chart developer can use to define what they mean by "readiness".

I think we need to add a section explaining how --wait/--wait-for-jobs will work with sequencing annotations. I added it to Open Issues

My other question here is, do we want to allow depending on other named k8s resources that are not defined by a chart? Eg, helm.sh/depends-on/resource? Or is the idea that users could annotate other resources not defined by a chart with helm.sh/layer, declaring that they are considered part of the same layer of a chart?

I think using helm.sh/depends-on/resource to specify a resource makes sense. Helm would eventually interpret layers as a group of resources to wait for

[banjoh]

A suggestion came my way regarding reasoning around sequencing chart dependencies/subcharts. Chart authors would want to place such annotations on a chart-level definition rather than deep inside chart templates. Defining once also makes sense because only one reference of the chart dependency is needed. Any other references are repetition since helm will wait for the chart before deploying the parent chart. We came up with something like below. We'd move helm.sh/depends-on/charts from chart templates

In the parent Chart.yaml

name: MyChart
annotations:
  # List of charts that need to be installed before MyChart
  helm.sh/depends-on/charts: "subchart, minio, rabbitmq"
dependencies:
- name: rabbitmq
  version: "1.2.3"
- name: minio
  version: "5.0.0"

[scottrigby]

Is the idea that this would be a check that Helm could use to override—kstatus ready condition (as an example implementation)? https://github.com/kubernetes-sigs/cli-utils/blob/master/pkg/kstatus/README.md#the-ready-condition

[Jeb135]

Yeah

[scottrigby]

Personally, I quite like the general direction of proposal 1—as a replacement of numeric weights entirely—over proposal 2, which extends that. My point of view is I would like to move the numeric weighting into the "alternatives considered" section. But would like to hear other maintainers and community feedback on this.

I have wanted to introduce a depends_on functionality for helm resources—as a dynamic alternative to weights—to allow users to specify a specific order they are applied (a similar concept is used by used by projects like terraform, flux, and docker-compose).

My main reasoning for this is numbered weights are static—you must know the entire list of how all resources should be ordered, and this often takes a lot of finagling and fussing to get it right.

While named dependencies are dynamic—creating a DAG on the fly, based only on when and if certain resources have an actual dependency on one or more other resources.

It might be good to note some of this in the proposal. I could sketch up some suggested text for this if you prefer, but would also be happy for you to take a stab at it in your HIP. What do you think?

[Jeb135]

Go ahead! I'll add you as a contributor, I appreciate the assistance.

[Jeb135]

I moved the weights proposal to rejected ideas and tried to capture some of the points you made here in why.

[mattfarina]

This is making me wonder if we need a new version of charts to handle the compatibility issues.

[mattfarina]

When it comes to sequencing, I'm not sure what you mean is clear in the document. I could come to two different conclusions:

1. The ordering of resources when they are uploaded to the k8s API server in one push
2. An ordered set of communications to the k8s API server where set it sent and completed prior to the next one being sent

Can you please provide some more clarity on what you mean.

[Jeb135]

2 is what we're going for here. I'll make a note to try and make that more clear.

[Jeb135]

Clarified what "resource sequencing" meant in the HIP

[mattfarina]

The # at the front denotes a comment. I'm not sure the k8s yaml parser, that we currently use and is supported by the k8s project, provides a means to parse and use data in comments. This needs to be investigated before this possibility could be considered. See https://github.com/kubernetes-sigs/yaml

[banjoh]

For reference, the ## Layer: <name> would be above/below # Source:. Maybe there is no need for 2 #. We can have it be # Layer: <name>.

helm template from an example chart

---
# Source: foo/templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:

[banjoh]

I made some changes to allow a chart author to build a dependency of subcharts. Its based on #373 (comment) post I had made earlier

[banjoh]

This HIP will depend on #374 HIP

[joejulian]

What happens when it breaks? How will partial successes be rolled back? What if a rollback fails a stage?

[joejulian]

What happens when it breaks? How will partial successes be rolled back? What if a rollback fails a stage?

Which also makes me think about storage. Will each stage need to be a release in order to manage failures and rollbacks? What will that do to storage requirements?

[scottrigby]

What happens when it breaks? How will partial successes be rolled back? What if a rollback fails a stage?

I think this HIP should not change the way rollbacks work today, only the order in which resources within a chart and subcharts are applied, and how waiting for readiness works if --wait is passed when sequencing is specified.

Which also makes me think about storage. Will each stage need to be a release in order to manage failures and rollbacks? What will that do to storage requirements?

This HIP should not change storage either. A chart's resources and and its subcharts should continue to be stored the same way for a release as if it didn't have any sequencing of those specified.

To clarify for some of the questions above, @banjoh / @Jeb135 do you think this is a fair summary of sequencing, scope, and waiting for readiness in this HIP?

1. Sequencing scope:

   • order of RESOURCES can only be specified for resources within the same chart
   • order of SUBCHARTS can only be specified for subcharts of the same parent chart (this can of course be recursive as subcharts work in Helm 3)

2. Sequencing order during upgrade/install/uninstall actions:

   • Helm will build a "Subcharts DAG" for a parent chart being acted upon, using the order specified by the parent and it's subchart siblings. This will start with the parent chart and recursively include subcharts, and any of their subcharts.
   • Helm will also build a "Resources DAG" for each individual subchart (and the top level parent chart) within the Subcharts DAG
   • Install and upgrade actions will start with the lowest level of the Charts DOG, and handle one subchart at a time moving to the top level parent chart. Uninstall action will follow the reverse order.
   • When each individual subchart (and top-level chart) is being handled following the correct Charts DAG order, the resource groups ("layers") within that individual subchart/chart will be handled following the Resources DAG for that chart/subchart until complete. The Resources DAG order within an individual subchart/chart will follow the same ASC/DESC logic above depending on the install/upgrade/uninstall action.
   • After the resources within each individual subchart/chart are handled for it's Resources DAG, Helm will move onto the next subchart/chart in the Charts DAG until complete.

3. Waiting for readiness during Install/Upgrade:

   • the --wait flag for each action allows the user to opt-in to waiting for readiness of each layer in the Resources DAG, and each subchart/chart in the Charts DAG, before moving onto the next. If the --wait flag is not set, Helm will not wait for readiness of any chart-specified sequencing.
   • As Helm moves through the Charts DAG, the --wait flag will wait for all the resources in each subchart/chart to be ready before moving onto then next. Similar to how this works for all resources in a chart in Helm 3, but now with stages in a specified order.
   • At each subchart/chart in the Charts DAG above—as Helm moves through the Resources DAG for that subchart/chart—the --wait flag will wait for all resources within each group/layer of the Resources DAG to be Ready before moving onto the next. When all layers are complete, this signals that all resources within that subchart/chart are Ready, so the next chart/subchart in the Charts DAG can take its turn.

[joejulian]

What happens when it breaks? How will partial successes be rolled back? What if a rollback fails a stage?

I think this HIP should not change the way rollbacks work today, only the order in which resources within a chart and subcharts are applied, and how waiting for readiness works if --wait is passed when sequencing is specified.

If staging charts is important, isn't staging rollbacks important? Can a staged upgrade, where subchart a is installed but subchart b fails, just have the whole previous manifest be reapplied?

Which also makes me think about storage. Will each stage need to be a release in order to manage failures and rollbacks? What will that do to storage requirements?

This HIP should not change storage either. A chart's resources and and its subcharts should continue to be stored the same way for a release as if it didn't have any sequencing of those specified.

Perhaps, but what about these stages? If subchart a is installed, should that count as a successful upgrade? If subchart b fails, can you just roll back to where subchart a succeeded?

Personally, I don't understand the concept of imperative operations in kubernetes. I need feedback from people that do understand the reasons to know what steps are needed to restore to a successful release.

[scottrigby]

@joejulian

If staging charts is important, isn't staging rollbacks important?

Yes it is important, both the layers within a chart/subchart, and the subcharts themselves if sequenced.

1. For resources that don't already exist in the previous revision being rolled back to (let's call them NEW resources), Helm currently deletes them.
   • This HIP proposes that—if the current release installed these in a defined order–these should be uninstalled in the reverse-order in which they were installed (this info will be stored in the current release object).
2. For resources that have been modified from the rollback revision to the current release (let's call these EXISTING resources), and for resources that were in the rollback revision but removed from the current release (let's call these OLD resources), Helm currently updates/installs these all at once.
   • This HIP proposes that—if the previous revision installed these in a defined order—this order is followed again on rollback (this info would be stored in the rollback revision release object).

👍 This is a good call out, and this should be spelled out explicitly in this HIP. @banjoh do you want to add this?

Can a staged upgrade, where subchart a is installed but subchart b fails, just have the whole previous manifest be reapplied?

• If any stage fails, the installation/upgrade stops and the release is marked as a failed release. This also applies to a failure during helm rollback command if called explicitly.
• If installation/upgrade is atomic, the above rollback logic will be applied to the failed release.

Perhaps, but what about these stages? If subchart a is installed, should that count as a successful upgrade? If subchart b fails, can you just roll back to where subchart a succeeded?

No, this HIP does not propose allowing subcharts to be installed/upgraded/uninstalled/rolled back separately from the release. If a user wants this, they should do what we currently recommend—to make these all separate releases, and use a separate orchestration tool (eg, Helmfile) to sequence these. @mattfarina covered this—I think well—in #375.

Personally, I don't understand the concept of imperative operations in kubernetes. I need feedback from people that do understand the reasons to know what steps are needed to restore to a successful release.

Yes, that is the entire conceit of this HIP. It is for applications that don't work well with Kubernetes recommended eventual consistency.

[scottrigby]

LGTM 👍 All feedback has been addressed from Helm dev meetings and in comments on this PR, and the HIP is very clear and actionable. Can’t wait to see the progress unfold 🙂

[sabre1041]

Very promising proposal. A few comments and changes requested

[sabre1041]

layer is not very descriptive into how its being used. End users may not understand how it can/should be used

[banjoh]

layer is not very descriptive into how its being used. End users may not understand how it can/should be used

I'll rename this to resource-group. I think this is a clearer name. Its come up before.

[scottrigby]

I think resource-group is much more clear. It's self-explanatory, unlike "layer" 👍

[sabre1041]

How will the dependency of subcharts be handled as it relates to tags and name? is there a concern if there is an overlap between the two? should a distinction be made?

[sabre1041]

Does there need to be some commentary/discussion regarding how to construct readiness-success and/or readiness-failure?

[banjoh]

Does there need to be some commentary/discussion regarding how to construct readiness-success and/or readiness-failure?

@sabre1041 I updated the doc to better explain how readiness checks work. https://github.com/Jeb135/helm-community/blob/hip-sequencing/hips/hip-00xx-sequencing.md#readiness