Best option to sync local-path PVs within multi-node k3s cluster?

[pschiffe]

Hello, I have a multi node k3s cluster with Rancher's local-path volumes. I would like to distribute the content of some volumes to all nodes, but tools like Longhorn are overkill for me. Syncing the volumes, for example, once per day would be enough.

I found this tool, but not sure what would be the best option to choose. Is it possible to use Rclone replication without intermediate S3 storage? As I'm syncing just within the cluster. Also copyMethod direct is fine as well.

Or maybe Rsync replication (via TLS)? But I don't need TLS and I need 1 to many replication, not just 1 to 1. Syncthing seems to be too complex. ReplicationDestination Volume Populator feature also looks cool.

What would be the best way to proceed in my case? Thanks.

[tesshuflower]

Hi @pschiffe ,

It sounds like rclone would normally be a good pick for you, but does require some intermediate location to store the files.

The advantage is you can have 1 replicationsource to store the data and then run multiple replicationdestinations to pull the data down, to fan it out.

I haven't looked into all the available storage providers for rclone, but you could look at running something locally in your cluster (for example we use minio for our dev/testing which provides an s3 compatible store you can run within the cluster).

Another option is to use rsync - and we would recommend rsync-tls as this is the newer version of the mover that supports running unprivileged. If you do use rsync you would need a 1-1 replicationsource-to-replicationdestination however. Essentially you'd be pointing each replicationsource to a replicationdestination within your cluster, so you could use a clusterIP service type (this is the default) to do the communication locally.

Lastly, syncthing could still be an option, but would normally be something you keep running to keep files in sync live. This would depend on your application itself and how you want to replicate the data.

[pschiffe]

Thank you for the overview. This is what I ended up with for small 3 node cluster to distribute mariadb backups from one to all nodes. For bigger clusters I'll probably use rclone.

apiVersion: volsync.backube/v1alpha1
kind: ReplicationDestination
metadata:
  name: flux-mariadb-backup-2
spec:
  trigger:
    schedule: "0 3 * * *"
  rsyncTLS:
    copyMethod: Direct
    capacity: 1Gi
    accessModes:
      - ReadWriteOnce
    storageClassName: local-path-retain
    moverSecurityContext:
      runAsUser: 999
      runAsGroup: 999
      fsGroup: 999
    moverAffinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: node-index
                  operator: In
                  values:
                    - "2"

---
apiVersion: volsync.backube/v1alpha1
kind: ReplicationDestination
metadata:
  name: flux-mariadb-backup-3
spec:
  trigger:
    schedule: "0 3 * * *"
  rsyncTLS:
    copyMethod: Direct
    capacity: 1Gi
    accessModes:
      - ReadWriteOnce
    storageClassName: local-path-retain
    moverSecurityContext:
      runAsUser: 999
      runAsGroup: 999
      fsGroup: 999
    moverAffinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: node-index
                  operator: In
                  values:
                    - "3"

---
apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
  name: flux-mariadb-backup-2
spec:
  sourcePVC: flux-mariadb-backup-1
  trigger:
    schedule: "5 3 * * *"
  rsyncTLS:
    keySecret: volsync-rsync-tls-flux-mariadb-backup-2
    address: volsync-rsync-tls-dst-flux-mariadb-backup-2
    copyMethod: Direct
    moverSecurityContext:
      runAsUser: 999
      runAsGroup: 999
      fsGroup: 999
    moverAffinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: node-index
                  operator: In
                  values:
                    - "1"

---
apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
  name: flux-mariadb-backup-3
spec:
  sourcePVC: flux-mariadb-backup-1
  trigger:
    schedule: "25 3 * * *"
  rsyncTLS:
    keySecret: volsync-rsync-tls-flux-mariadb-backup-3
    address: volsync-rsync-tls-dst-flux-mariadb-backup-3
    copyMethod: Direct
    moverSecurityContext:
      runAsUser: 999
      runAsGroup: 999
      fsGroup: 999
    moverAffinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: node-index
                  operator: In
                  values:
                    - "1"

[pschiffe]

I have one more question. I've noticed that ReplicationSource jobs are created just when they are scheduled and they disappear after the job is done, which I think is correct. However ReplicationDestination jobs are running all the time and now I'm getting alerts KubeJobNotCompleted from k8s mixin - https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/alerts/apps_alerts.libsonnet#L283

Is the ReplicationDestination job supposed to be running all the time, or do I have some misconfiguration? Could it be just a pod instead of a job, if it has to be running all the time?

[tesshuflower]

You've setup the replicationdestination as we normally recommend for rsync/rsync-tls - that is, there is no schedule defined, which means it'll run all the time.

What happens is you get a job, and that job will complete after a synchronization has completed, but then it'll immediately startup a new replication with a new job to wait for the next synchronization, which won't happen until the replicationsource starts up and tries to communicate.

If you don't want the job to be running on these in-between times you can still add a schedule (spec.trigger.schedule) to your replicationdestination, but you'll want to make sure it starts up a little ahead of your scheduled replications from the source side ideally.

[pschiffe]

Oh, this is awesome. I've added spec.trigger.schedule to the destinations and it's perfect now. I've also updated my original configuration above. Thank you for maintaining this great tool!

[pschiffe]

Another question 😅
Is rsync configured to delete files in destination if the files are removed from source? Where can I see the rsync options used? Are they configurable?

[tesshuflower]

Yes, using rsync or rsync-tls will try to keep the destination exactly in-sync with the source by deleting files on the destination that no longer exist on the source.

You can look at exactly what it does here: https://github.com/backube/volsync/blob/main/mover-rsync-tls/client.sh#L152-L167

[pschiffe]

Thank you 🙏