Flux for temporary review environments

[asoee]

Vi are planning to use flux for our dev/stage/prod where it seems a perfect fit.
Now I am looking at providing temporary environments, where our developers
can deploy a feature branch, to preview the new features.
So that would be multiple instances of the same application in the cluster,
each in their own namespace.
These deployment would have a typical lifespan of hours to weeks.

Would flux still be a good fit, for this type of workflow, or should I look at more
conventional "push" deployments with helm?

[stefanprodan]

There are two types of ephemeral environments that we wish to address in Flux v2:

• preview namespaces (Kustomize+Helm)
• preview clusters (ClusterAPI)

To deploy applications from feature branches onto ephemeral namespaces, we could introduce a new toolkit component (e.g. preview-controller) that reacts to events such as PR/branch create/push/delete.

The preview-controller could manage ephemeral namespaces and app deployments by generating GitRepository, Kustomization, and HelmRelease objects on the fly. When the PR is merged and the branch deleted, the controller will react by removing the generated objects and thus triggering garbage collection of the whole preview environment.

The preview-controller could leverage the kustomize-controller var substitutions feature and helm-controller values override to inject values for vars like: PR_ID and GIT_BRANCH. These variables could be used to prefix/suffix the ephemeral namespace, ingress sub-domain/url, certs, configmap data, etc.

[goldsam]

This would be a phenomenal addition to the GOTK suite!

[jmcruz1983]

Any update? Thanks!

[navpreet-securitas]

what could be the timeline for this ? considering flux GA is available now

[kalihman]

Any update on this feature?

[stefanprodan]

Yes we are going to release this feature under flux-operator, check out controlplaneio-fluxcd/flux-operator#140

[paprickar]

Just wanted to share that we are also in need of a more dynamic environment.
We currently use fluxcd v1 / helm-operator and for static deployments this works well.

What we want:
Having environments spawned for each Pull Request.

• fresh deployment, ingress etc.
• fresh database maybe including test data (maybe something which could be done with an additional controller)
• get feedback in the CI pipeline when the new system are up and running (we currently use fluxctl to ask for the deployment status)

We now have to see if we can keep the current GitOps approach or if we have to find something new.

[naushadmohammed]

@paprickar did you guys find any solution with fluxcd? or have you guys moved to a different solution altogether. We have implemented fluxcd for our static environments, and currently looking for solutions to implement dynamic environments with fluxcd, but this seems like its going to take some time.. Appreciate your comments on how you decided to proceed.

[mrknmc]

FWIW we use Flux for all static deployments but implemented this outside of Flux using GitHub Actions and Helm upgrade/uninstall on PR open/close events. It works pretty well, you could also use comments on PRs if you wanted to get fancy.

[paprickar]

Nothing new on that yet was to busy with other topics.
I am looking into ArgoCD - it provide an cli where we can spawn ad hoc environments and then remove them (in the end very similar to what mrknmc proposed)

[naushadmohammed]

@mrknmc thank you for the response, even we are now looking at Github Actions for our preview requirements. I found one solution which is available at Github Actions Marketplace : https://github.com/marketplace/actions/preview-pull-requests-in-azure-kubernetes-service
I might try it out as well. Also, if it is possible for you to give a glimpse of how you guys have configured preview environments, that would be helpful.

[naushadmohammed]

@paprickar yes there are different solution, but what I was focusing on is using fluxcd itself instead of going with two different solution for static environments and dynamic environment.

[stealthybox]

I've done feature branch releases like this before:

This script acts as a simple branch controller that produces GitRepositories, Kustomizations, and a target Namespace.

You can put this into a CronJob or sleep-loop within a Deployment like so.

You'll need to have an image that has git + kubectl.
The Pod should have a ServiceAccount RoleBinding to create GitRepositories, Kustomizations, and Namespaces.
This script doesn't set up the clone -- you'll need to add that part. (Re-use the secretRef from the GitRepository)

parse_branches() {
  tr '/' ' ' | awk '{print $3}'
}

sanitize_branches() {
  # "--branch_TEST-@@@.^-name--" --> "branch-test-name"
  tr '[:upper:]' '[:lower:]' \
    | tr -c '[:alnum:]' - \
    | tr -s - \
    | sed -e 's/^-*//' -e 's/-*$//' -e 's/--*/-/'
}

git fetch origin

git branch -a --list 'origin/*feature*' | parse_branches | sort -u > ./branches
kubectl get gitrepository -oname | grep feature | cut -d/ -f2 | sort -u > ./in_cluster
comm -2 <(cat ./branches | sanitize_branches) ./in_cluster > ./to_delete

for raw_br in $(cat ./branches); do
  kube_br="$(echo "${raw_br}" | sanitize_branches)"
  kubectl apply -f- << EOF
---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: ${kube_br}
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: ${raw_br}
  url: ssh://git@github.com/myorg/myrepo
  secretRef: flux-system
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: ${kube_br}
  namespace: flux-system
spec:
  interval: 1m0s
  path: ./
  prune: true
  sourceRef:
    kind: GitRepository
    name: ${kube_br}
  targetNamespace: ${kube_br}
---
apiVersion: v1
kind: Namespace
metadata:
  name: ${kube_br}
EOF
done

for kube_br in $(cat ./to_delete); do
  kubectl delete --wait=false gitrepository "${kube_br}"
  kubectl delete --wait=false kustomization "${kube_br}"
  kubectl delete --wait=false namespace "${kube_br}"
done 

[hobbsh]

I would also like to add that this would be an extremely valuable feature to have and while @stealthybox's solution could provide an interim solution, I am looking forward to a preview-controller.

[alam0rt]

definitely keen on this feature - I might try to PoC this out using @stealthybox 's suggestion.

[alam0rt]

have started spiking this out as I am interested in this pattern for a platform I am building for multiple product teams https://github.com/alam0rt/ephemeral-environment-controller

[stealthybox]

@alam0rt thanks for publishing that! It's looking really cool.

[jcputter]

any status on this?

[Boes-man]

Hope to see this on the roadmap soon.

[GregoireW]

Here, I will focus on ephemeral environment == "I got an application, I want to deploy multiple preview of my app on a single cluster."

Ephemeral environment is a challenge by itself. The first step is to know if it is a valid approach on your app. On some case, I think we can assume this can be a good and an easy thing to manage (front application for instance). On some other case, I do not really see a value to ephemeral environment vs a testcontainer based test on CI ( app like data streaming service for instance). For application in between, an ephemeral environment may be complex to generate ( duplicate data storage ? use a single data storage but then how to ensure no conflict between data definition ? What if I need a configuration change because of my PR ? So yes, not impossible, but can be really tricky) and may have more or less value to build it.

All in all, having a process to do that is something I will vote for.

Earlier comment suggest an approach to create a "preview-controller" driven by PR webhook. I'm not so found of this approach.
Today Flux work as a passive component inside the cluster. Adding a listener make it harder to setup and raise some security issues.

If the trigger is webhook it also means the controller will have to create an update policy because it will be triggered before the image is built (or the failure to build one :D ). Doing that raises the question as why do we need this controller? if we need to call something (with a secret) to create something in a cluster, why not simplify the process and call directly (with a secret) a job on the repository to create the needed file. Ok it will be provider dependent, but the controller would also need support on each provider for simplicity.
On GitHub, providing a GitHub action to do the process may be simpler and may also solve some other use case where we would want configuration change.

The other solution I have in mind is to extends the image automation process to duplicate environment. From image tag, we can extract multiple data (for instance PR number) and from that, duplicate some kubernetes objects.
It is quite a simple solution, easy to implements, but the drawback is it simplicity. We can only duplicate object (and change some value by extracting data from tag) but not make a big change configuration change between PR for instance.

[GregoireW]

Writing this make me think a solution inside flux is perhaps not the best answer. On GitHub creating a well crafted GitHub action may be beneficial for instance. Will try to think a little bit more about this.

[sbernheim]

@GregoireW -

I'd expect that the webook would only notify the preview-controller or source-controller that the source repo has changed somehow and the controller would then go look at the source repo to pull the latest or look at the repo's PRs to take some action based on their contents. In that case, if you don't configure your GitHub Action to poke the webhook, the controller would eventually poll and pick up the change or PR.

Your GitHub action shouldn't need a secret key or auth token in order to poke the webhook, since the poke itself doesn't result in any changes applied to your cluster if there are no corresponding changes in the source repo.

But I'm speculating a bit here and don't know for sure if @stefanprodan might be thinking differently.

[GregoireW]

@sbernheim This hook partially already exist. It listens gitrepository event : https://fluxcd.io/docs/guides/webhook-receivers/

What is difficult here is ephemeral / temporary environment may not be a copy/pasted from something. You may need to update something.

[maniSbindra]

Hi, I have created a very basic pr-ephemeral-env-controller controller. This currently works with Github and creates (updates/ deletes) new Flux HelmReleases for each PRs, I have based it on the Argo CD ApplicationSet Pull Request Generator. The Flux HelmRelease refers a Helm Chart through a Flux Source Repo. The values passed to the chart by the controller are the PR Number and PR Head SHA. You can design your Helm Chart to create a new environment in separate Namespace for each PR, or totally isolated (including isolated cloud services) environments for each PR.

In the sample linked on the Controller Repository, for each PR to the sample application repository (The sample app is a simple todo API which needs a backend postgres database), a new isolated environment is created for each PR, with several isolated resources (The Helm Chart uses Crossplane Azure Jet Provider and Crossplane Helm Provider resources), including a new resource group, a new AKS cluster to which the application deployment and service (corresponding to the PR SHA commit of the application) are applied , and a new Azure Postgres backend database to which the application points to read and persist data. The Helm chart repository which the Flux HelmRelease points to is ephemeral-env-infra

Thought I will share it here. Thanks.

[rpuserh]

Any update on adding new resources like "preview namespaces (Kustomize+Helm)" "preview clusters (ClusterAPI)" to devops toolkit to support ephemeral envs?

[codablock]

The Template Controller might be a viable solution for some of you. It allows to create arbitrary templated objects (e.g. HelmRelease or Kustomizatin) from "matrix inputs", which in turn can be arbitrary objects as well. The controller also provides a GitProjector CRD and ListGitlabMergeRequests and ListGithubPullRequests CRDs, which can be used as matrix inputs for the templates, allowing you to create preview environments based on git branches or PRs.

See the Introducing the Template Controller and building GitOps Preview Environments blog post for an introduction and examples. The example are based on Kluctl, but can easily be changed to work with Kustomization/HelmRelease.

[danielloader]

Just thought I'd ask, though no pressure to answer - is there a desire to get this functionality or workflow formalised before flux v2 hits GA/1.0.0 status across the board? Would be good to know if there's a timeline/priority for it, or if it's just a loosely aspirational goal to have as and when the time is right.

[makkes]

is there a desire to get this functionality or workflow formalised before flux v2 hits GA/1.0.0 status across the board

The answer is 99% likely to be "no" because we're all committed to bringing Flux v2 to GA first and foremost.

[Ph0tonic]

Hi thanks for your answer 👍 . Now that Flux v2 has reached GA, is there a chance that this issue will be considered? This feature seems to be the most requested feature for flux (upvotes in discussion), which would be awesome.

[tplass-ias]

our developers have simply written the commit automation to place the manifests into the directories/branches for dev deploys. Basically a really convoluted way to run kubectl apply from the CI system, but it's what they need.

We just push temp deploys into a shared namespace because apps don't typically need custom ServiceAccounts and it would be commit automation into an admin repo to get ephemeral SA's+Namespaces, which is a no-go for us.

[tplass-ias]

looking at the Template Controller and various "create HR's from pull requests" mentioned above, I wonder how those will stitch together fleets of microservices in different repos? e.g. repo-a's service makes calls to repo-b's service, how do we declare the intent to connect these 2 ephemeral/temporary releases? In the end, repo-a would need to know repo-b's ephemeral service name, so do we just put that data in the PR where it will most likely get merged into master?

[6uellerBpanda]

not sure if this helps but here is an example how we achieved it in combination with gitlab ci.

this might not be a replacement for a more "controller" like approach but that's what we/devs needed atm -> https://6uellerbpanda.gitlab.io/posts/Feature-branch-deployments-flux-gitlab-ci/

[Smana]

Hey everyone! Are there any plans for a future controller? Just want to know if I need to build my own workflow based on the examples above :) . I'll also have a look at the Template controller.

[stefanprodan]

Starting with Flux Operator v0.14, the operator has the ability to create, update and delete application instances on-demand based on the ResourceSet definitions and GitHub Pull Requests / GitLab Merge Requests state.

For more details on how to use the ephemeral environments feature, see the following guides:

• Ephemeral Environments for GitHub Pull Requests
• Ephemeral Environments for GitLab Merge Requests