Automating setup of a simple bucket backend

[hglynn]

How can I automate the mounting of a cloud storage bucket and a few other config tweaks?

Right now, I can install and run, and do basic setup including admin for SFTPGO on Google Cloud.
I do this via terraform and cloud-init.

I can setup the bucket on the webgui correctly, it uses instance creds, i can sftp files inbound and they appear 👍

When I export / backup the config, the config files have different sections that are not mentioned in the sftpgo.json file/docs.

Can i add a fragment to the sftpgo.json file or can i feed in something else that lands in the config DB?

[hglynn]

OK I got a chance to spend a bit more time looking at this and found a solution

1. Spin up the sftpgo server and do a backup of the config sessions to get a largely empty skeleton

2. I wrote the skeleton file during "runcmd" of cloud-init to a temp file, replacing hardcodes with terraform vars

3. Also in your runcmd cloud-init script, set SFTPGO_LOADDATA_FROM and SFTPGO_LOADDATA_MODE env vars

The equivalent command line if doing it manually is :
sftpgo initprovider --loaddata-mode 0 --loaddata-from /root/sftpgo-initial-config.json

I'm still having a bit of trouble with the default admin password as a result of the (re-)init provider but I am sure I will figure that out

[DinoChiesa]

How can I automate the mounting of a cloud storage bucket and a few other config tweaks

You can use gcsfuse to mount a GCS bucket into the filesystem. Does sftpgo care that an upload directory is mounted from GCS? I would guess it does not but I haven't tried it.

When you do it this way, the reads/writes to the GCS bucket are authenticated based on the machine identity. So you don't need to create and download a .JSON key file and provision that in sftpgo. This will work if and only if the machine actually HAS a Google Cloud identity, in other words, only if sftpgo is running in GCE. If you are running sftpgo in some other host, then you still need to use the JSON key file.

This is how I set up gcsfuse on a GCE-hosted VM running Debian, for use with the internal-sftp server.

    gcloud compute ssh "${VM_INSTANCE_NAME}" --project="$GCP_PROJECT" --zone="${GCE_VM_ZONE}" \
        --command="bash -s" <<EOF
sudo mkdir /path/to/mount/point
sudo chown ${SFTP_USER}:sftp_grp /path/to/mount/point
echo "deb https://packages.cloud.google.com/apt gcsfuse-$(lsb_release -c -s) main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo apt-get update
sudo apt install -y fuse gcsfuse
EOF

    printf "mounting gcs via gcsfuse...\n"
    gcloud compute ssh "${VM_INSTANCE_NAME}" --project="$GCP_PROJECT" --zone="${GCE_VM_ZONE}" \
        --command "sudo gcsfuse -o allow_other  --file-mode=555 --dir-mode=777  ${BUCKET_NAME} /path/to/mount/point"

That would need to be slightly different for use with sftpgo (a different user I suppose) but I don't know the specifics.