Buildkit in rootless mode - or: using feature gates with hetzner-k3s

[jampy]

what I am ultimately trying to do

I'd like to run Buildkit in rootless mode in a hetzner-k3s cluster.

However, I also want it to run without --oci-worker-no-process-sandbox to get a better isolation of the build process.

The reason is that with the --oci-worker-no-process-sandbox option my Dockerfile builds apparently can start daemon processes that continue running even if the Dockerfile build completes - which to me is highly problematic.

Running Buildkit in privileged mode would allow better build isolation, but has it's own security implications.

the difficulty

According to this document the --oci-worker-no-process-sandbox option also requires securityContext.procMount to be set to Unmasked and also requires spec.hostUsers to be set to false.

As of Kubernetes v1.32, both options must be enabled using "feature gates": ProcMountType=true and UserNamespacesSupport=true.

how I've tried to solve this

In my cluster_config.yaml I've set all *_args options to include said feature gates. Here are the relevant parts of my cluster_config.yaml:

k3s_version: v1.32.1+k3s1

# ...

kube_api_server_args:
  - feature-gates=ProcMountType=true,UserNamespacesSupport=true

kube_scheduler_args:
  - feature-gates=ProcMountType=true,UserNamespacesSupport=true

kube_controller_manager_args:
  - feature-gates=ProcMountType=true,UserNamespacesSupport=true

kube_cloud_controller_manager_args:
  - feature-gates=ProcMountType=true,UserNamespacesSupport=true

kubelet_args:
  - feature-gates=ProcMountType=true,UserNamespacesSupport=true

kube_proxy_args:
  - feature-gates=ProcMountType=true,UserNamespacesSupport=true

I re-created the whole cluster after changing the config.

My Buildkit deployment manifest (managed using ArgoCD):

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/instance: buildkitd
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: buildkitd
    app.kubernetes.io/version: v0.19.0
    argocd.argoproj.io/instance: buildkit
  name: buildkitd
  namespace: buildkit
spec:
  replicas: 3
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: buildkitd
  template:
    metadata:
      annotations:
        container.apparmor.security.beta.kubernetes.io/buildkitd: unconfined
      labels:
        app: buildkitd
    spec:
      containers:
        - args:
            - '--addr'
            - unix:///run/user/1000/buildkit/buildkitd.sock
            - '--addr'
            - tcp://0.0.0.0:1234
          image: moby/buildkit:v0.19.0-rootless
          imagePullPolicy: IfNotPresent
          livenessProbe:
            exec:
              command:
                - buildctl
                - debug
                - workers
            initialDelaySeconds: 5
            periodSeconds: 30
          name: buildkitd
          ports:
            - containerPort: 1234
          readinessProbe:
            exec:
              command:
                - buildctl
                - debug
                - workers
            initialDelaySeconds: 5
            periodSeconds: 30
          securityContext:
            procMount: Unmasked  # <--------
            runAsGroup: 1000
            runAsUser: 1000
            seccompProfile:
              type: Unconfined
          volumeMounts:
            - mountPath: /home/user/.local/share/buildkit
              name: buildkitd
      hostUsers: false  # <--------
      volumes:
        - emptyDir: {}
          name: buildkitd

the problem

The buildkit pods fail to start with reason FailedCreatePodSandBox: Failed to create pod sandbox: the handler "" is not known.

Other pods (without procMount / hostUsers) run fine.

I am now stuck because I have no idea what this message is trying to tell me.

I'm afraid that I've already messed up too much anyway. I'm not that experienced with Kubernetes and until today I didn't even know that "feature gates" even existed in Kubernetes. I'm not sure if this is simply a configuration problem, something that's not supported by hetzner-k3s or a bug somewhere.

Any advice that gets me closer to the goal of safely running Buildkit in the cluster would be greatly appreciated.

I have started this as a GitHub discussion in the hope that it may somehow be useful to others.

[vitobotta]

Feature gates can be risky, and there's no guarantee they'll work, depending on the type of feature and other factors. I've only used them for testing, to be honest.