RKE2 installation sets fs.file-max to 65535

[zchenyu]

Environmental Info:
RKE2 Version: v1.30.5+rke2r1

Node(s) CPU architecture, OS, and Version:

Linux 64-181-207-149 6.8.0-49-generic #49~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Nov  6 17:42:15 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
N/A

Describe the bug:

After installing RKE2 (see below), fs.file-max gets set to 65k, which is much too low, especially for control plane nodes.

$ cat /proc/sys/fs/file-max 
9223372036854775807

$ curl -sfL https://get.rke2.io | sh -
$ systemctl enable rke2-agent.service
$ systemctl start rke2-agent.service

$ cat /proc/sys/fs/file-max 
65535

Steps To Reproduce:

• Installed RKE2:

$ cat /proc/sys/fs/file-max 
9223372036854775807

$ curl -sfL https://get.rke2.io | sh -
$ systemctl enable rke2-agent.service
$ systemctl start rke2-agent.service

$ cat /proc/sys/fs/file-max 
65535

$ cat /etc/rancher/rke2/config.yaml
tls-san:
  - XXX
node-label:
  - XXX
server: XXX
token: XXX

Expected behavior:
fs.file-max to stay the same (2^64)

Actual behavior:
fs.file-max gets reduced to 2^16

Additional context / logs:

[brandond]

much too low, especially for control plane nodes.

Can you explain what specific problem this is causing for you? We have yet to receive any other complaints of this.

I believe there must be a privileged pod (probably one of the CNI pods, or perhaps niginx) that is doing this, as there is nothing in RKE2, and nothing I am aware of in core Kubernetes, that sets this at a system level.

For example, RKE2's systemd unit sets a higher limit than what you're reporting:

rke2/bundle/lib/systemd/system/rke2-server.service

Line 18 in 649521a

LimitNOFILE=1048576

[zchenyu]

After the node runs for a while, it the node basically becomes unusable. e.g.

• Cannot SSH into the node
• If I luckily do SSH, I can't do anything on the file system (e.g. vim, cat, ls)
• Lot's of k8s issues pop up, e.g. Node becomes NotReady, pods start failing, etc

[brandond]

We've certainly never run into that. What makes you think that this is related to the max file descriptors? What are the resources (CPU/memory/disk) of the nodes in question?

[zchenyu]

Because adding fs.file-max=9223372036854775807 to sysctl.conf and then running sudo sysctl -p immediately resolves the issue.
These are beefy 8xH100 nodes (~100 vCPU, ~1 TB RAM, ~20 TB of SSD)

[zchenyu]

but even if it weren't an issue, I think it's a bit unexpected that RKE (or some of its dependencies) messes with the system defaults unless there's a specifically good reason that should be documented. Worth getting to the bottom of it?

[brandond]

TBH I'm not sure we've ever tested it on a node that large. General our users tend to scale out not up.

Are you running anything on this node or is it literally just sitting there idle? Something must be consuming all these file descriptors if you are running out and need an increased limit. Any idea what that might be?

[zchenyu]

Yeah we have large GPU workloads running on them. The machine's default file-max is reasonably big (2^64), and in fact it matches my tiny thinkpad laptops' default configuration as well (afaict, it's Ubuntu default). The issue is that the process of installing RKE somehow lowers it to something much much lower.

[dereknola]

On a ubuntu VM, I see no change when enabling rke2-server with your configuration:

root@server-0:~$ cat /proc/sys/fs/file-max 
9223372036854775807
root@server-0:/home/vagrant# curl -sfL https://get.rke2.io | sh -
[INFO]  finding release for channel stable
[INFO]  using v1.31.4+rke2r1 as release
[INFO]  downloading checksums at https://github.com/rancher/rke2/releases/download/v1.31.4+rke2r1/sha256sum-amd64.txt
[INFO]  downloading tarball at https://github.com/rancher/rke2/releases/download/v1.31.4+rke2r1/rke2.linux-amd64.tar.gz
[INFO]  verifying tarball
[INFO]  unpacking tarball file to /usr/local
root@server-0:/home/vagrant# cat /proc/sys/fs/file-max 
9223372036854775807
root@server-0:/home/vagrant# systemctl enable rke2-server.service
Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → /usr/local/lib/systemd/system/rke2-server.service.
root@server-0:/home/vagrant# systemctl start rke2-server.service
root@server-0:/home/vagrant# cat /proc/sys/fs/file-max
9223372036854775807

same with agent as well.

[brandond]

@zchenyu I also have tried to reproduce this locally to figure out what pod might be doing this, and I do not see this behavior either.

Are you sure that it is just the installation of RKE2 that does this for you? You don't have some other automation, or a privileged pod unique to your environment, that is doing it?

[zchenyu]

I'm not 100% sure, but my initial post shows the sequence of steps I took that leads me to believe it was RKE2.

Happy to jump on a call to demo.
We're using v1.30.5+rke2r1. Regarding your test with v1.31.4+rke2r1, I have confirmation from a LambdaLabs engineer that they cannot repro with a newer version. I personally haven't tested with a new version (yet). I can try when I have a free minute

[zchenyu]

I just tested with v1.32.1+rke2r1 and it seems to no longer exhibit this behavior.

[brandond]

🤷🏻 I still have no idea what was doing it.