Can k3s please support ingress-nginx on HA clusters?

[victor-sudakov]

Is your feature request related to a problem? Please describe.
Yes and no. I suppose ingress-nginx is not officially supported on K3S, and it does not work for me in a HA configuration.

Describe the solution you'd like
It would be good to have an up-to-date documentation on how to run ingress-nginx on K3S, especially in HA configurations. Or a clear statement that such a configuration is not supported and never will be.

Describe alternatives you've considered
I have studied several articles on the Web explaining how to install ingress-nginx in K3S. Some of them suggest the baremetal version of the manifest which obviously does not work on HA clusters as it uses hostNetwork. Others suggest the cloud manifest which would not even install cleanly. Of course I have used --disable traefik when installing K3S nodes.

Additional context
If anyone has a working setup of a HA K3S cluster with ingress-nginx could you please share your configuration or better yet, document your practice on https://rancher.com/docs/k3s/latest/en/

K3S version
v1.23.6. I'm using the PostgreSQL datastore for HA.

[brandond]

The upstream helm chart should work fine, which bit of it are you running into trouble with? We're never going to support it in terms of it being a packaged component, but there's no reason it shouldn't work if you deploy the chart properly.

[victor-sudakov]

The upstream helm chart should work fine, which bit of it are you running into trouble with? We're never going to support it in terms of it being a packaged component, but there's no reason it shouldn't work if you deploy the chart properly.

Which exactly chart are you implying? I've tried several, and I'm willing to post the result of running any chart you name.

[brandond]

The official one maintained by the Kubernetes project... which one have you been trying?
https://github.com/kubernetes/ingress-nginx

[victor-sudakov]

The chart from the ingress-nginx/ingress-nginx repo installs without errors, but the simple curl localhost test works only on one of the HA nodes, on the other node the request just times out.

If it's of any importance:

NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
ingress-nginx   ingress-nginx   1               2022-06-01 16:25:55.022850922 -0700 MST deployed        ingress-nginx-4.1.3     1.2.1     

[brandond]

It sounds like you might want to configure the chart to deploy as a daemonset, or change the external traffic policy if running it behind a LoadBalancer service?
https://kubernetes.io/docs/concepts/services-networking/service/#traffic-policies

None of these things are k3s related though, they are just decisions about how you want to run and expose the ingress.

[victor-sudakov]

Nope, helm install --set controller.kind=DaemonSet ... ingress-nginx/ingress-nginx does not fix the timeout on one of the nodes. That's why I was asking for a tested, documented best practice.

[victor-sudakov]

if running it behind a LoadBalancer service?

I actually was not planning to. Maybe just a haproxy choosing one of the K3S nodes which is finds alive. My purpose is to create a cheap yet HA cluster on two EC2 instances which would survive the downtime of one of the instances.

[victor-sudakov]

The way I created my nodes:

curl -sfL https://get.k3s.io | sh -s - server --docker --disable traefik \
   --token=XXXXXX --datastore-endpoint="postgres://postgres:XXX@YYY/kubernetes"

[brandond]

Deploying to k3s shouldn't be different than deploying to any other mutlinode cluster. I believe there are a couple guides out there if you look around. For example: https://www.suse.com/support/kb/doc/?id=000020082

[victor-sudakov]

Deploying to k3s shouldn't be different than deploying to any other mutlinode cluster.

And yet there exist environment-specific installation instructions for many popular Kubernetes implementations. Even separate manifests for different clouds and implementations are provided in the link above.

I believe there are a couple guides out there if you look around.

And many of them are outdated or misguided or unsuitable for a multinode K3S cluster. I've checked a number of them, believe me.

For example: https://www.suse.com/support/kb/doc/?id=000020082

It basically boils down to installing from the standard https://kubernetes.github.io/ingress-nginx repo (as you advised above) with the following non-default values:

 controller:
      kind: DaemonSet
      hostNetwork: true
      hostPort:
        enabled: true
      service:
        enabled: false
      publishService:
        enabled: false
      metrics:
        enabled: true
        serviceMonitor:
          enabled: true
      config:
        use-forwarded-headers: "true"

I'll check it out today and report here

[brandond]

Yep, this basically flips it over to a HostPort/HostNetwork daemonset, instead of a deployment+service, as I suggested above. It can be made to work in either scenario, or even daemonset + service, but this is a common model for folks that don't need to use LB services and expect to be able to hit the ingress on all nodes.

Deciding how to do it should be driven by your requirements or personal preference. Neither mode is required by anything specific to K3s. The chart is very configurable, since lots of folks have strong opinions on how they want their ingress controller to operate. If you'd like to see the upstream docs contain k3s-specific suggestions, you might open an issue on their repo.

[victor-sudakov]

For example: https://www.suse.com/support/kb/doc/?id=000020082

If you use the configuration from the article above, as is, installation will fail with a

Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: resource mapping not found for name: "ingress-nginx-controller" namespace: "" from "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
ensure CRDs are installed first

If you however remove the metrics section and leave only this:

controller:
  kind: DaemonSet
  hostNetwork: true
  hostPort:
    enabled: true
  service:
    enabled: false
  publishService:
    enabled: false
  config:
    use-forwarded-headers: "true"

Then it looks like the ingress installation is a success and curl localhost responds on all the 3 nodes.

It would be great if K3S new users did not have to go through trial and error like I've had to.

[brandond]

It's not a packaged component, and we don't document things that we don't package. If you think the upstream ingress-nginx docs should include some k3s-specific instructions, please open an issue at https://github.com/kubernetes/ingress-nginx/issues