Profile test, singularity Error on HPC - could not extract squashfs data, unsquashfs not found

[djbradshaw2]

Description of the bug

Dear Nextflow Mag Pipeline developers,

Thank you for such a great tool! While trying to run the test mag dataset using singularity, the pipeline fails very quickly with an Exit Status 255 and an error regarding not being able to extract squashfs data due to unsquashfs not being found. This is despite modules of nextflow, singularity, and unsquashfs being loaded and unsquashfs command being recognized and functional when run on the command line. This is being run on an HPC. Please see the attached nextflow.log

Please let me know if you have any questions or need any other information. Thank you for your time and help.

Sincerely,

David

Command used and terminal output

module load nextflow
module load singularity
module load squashfs-4.4-gcc-9.2.0-7d3vq45

nextflow run nf-core/mag -r 3.1.0 -profile test,singularity --outdir test_output.out

Pulling Singularity image https://depot.galaxyproject.org/singularity/bowtie2:2.4.2--py38h1c8e9b9_1 [cache /mnt/beegfs/home/dbradshaw2015/work/singularity/depot.galaxyproject.org-singularity-bowtie2-2.4.2--py38h1c8e9b9_1.img]
Pulling Singularity image https://depot.galaxyproject.org/singularity/krona:2.7.1--pl526_5 [cache /mnt/beegfs/home/dbradshaw2015/work/singularity/depot.galaxyproject.org-singularity-krona-2.7.1--pl526_5.img]
Pulling Singularity image https://depot.galaxyproject.org/singularity/ubuntu:20.04 [cache /mnt/beegfs/home/dbradshaw2015/work/singularity/depot.galaxyproject.org-singularity-ubuntu-20.04.img]
Pulling Singularity image https://depot.galaxyproject.org/singularity/fastp:0.23.4--h5f740d0_0 [cache /mnt/beegfs/home/dbradshaw2015/work/singularity/depot.galaxyproject.org-singularity-fastp-0.23.4--h5f740d0_0.img]
Pulling Singularity image https://depot.galaxyproject.org/singularity/fastqc:0.12.1--hdfd78af_0 [cache /mnt/beegfs/home/dbradshaw2015/work/singularity/depot.galaxyproject.org-singularity-fastqc-0.12.1--hdfd78af_0.img]
Staging foreign file: https://busco-data.ezlab.org/v5/data/lineages/bacteria_odb10.2024-01-08.tar.gz
Execution cancelled -- Finishing pending tasks before exit
Pulling Singularity image https://depot.galaxyproject.org/singularity/pandas:1.4.3 [cache /mnt/beegfs/home/dbradshaw2015/work/singularity/depot.galaxyproject.org-singularity-pandas-1.4.3.img]
-[nf-core/mag] Pipeline completed with errors-
WARN: Singularity cache directory has not been defined -- Remote image will be stored in the path: /mnt/beegfs/home/dbradshaw2015/work/singularity -- Use the environment variable NXF_SINGULARITY_CACHEDIR to specify a different location
ERROR ~ Error executing process > 'NFCORE_MAG:MAG:BUSCO_QC:BUSCO_DB_PREPARATION (bacteria_odb10.2024-01-08.tar)'

Caused by:
Process NFCORE_MAG:MAG:BUSCO_QC:BUSCO_DB_PREPARATION (bacteria_odb10.2024-01-08.tar) terminated with an error exit status (255)

Command executed:

mkdir buscodb
tar -xf bacteria_odb10.2024-01-08.tar.gz -C buscodb

cat <<-END_VERSIONS > versions.yml
"NFCORE_MAG:MAG:BUSCO_QC:BUSCO_DB_PREPARATION":
tar: $(tar --version 2>&1 | sed -n 1p | sed 's/tar (GNU tar) //')
END_VERSIONS

Command exit status:
255

Command output:
(empty)

Command error:
INFO: Convert SIF file to sandbox...
FATAL: while extracting /mnt/beegfs/home/dbradshaw2015/work/singularity/depot.galaxyproject.org-singularity-ubuntu-20.04.img: root filesystem extraction failed: could not extract squashfs data, unsquashfs not found

Work dir:
/mnt/beegfs/home/dbradshaw2015/work/ef/1e6db689ccf7adc9357eaaee2efef5

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh

-- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

-- Check '.nextflow.log' file for details

Relevant files

No response

System information

Nextflow version: 24.04.4
Hardware: HPC
Executor: local
Container: Singularity (3.4.1)
OS: Linux
nf-core/mag: 3.1.0

[jfy133]

Hi @djbradshaw2

Unfortunately this appears to be an issue with your singularity installation or configuration, as it comes up a lot with singularity itself rather than Nextflow nor the pipeline, e.g. apptainer/singularity#5711.

You might need to check the installation again that it's working.
But it also looks like you're running a very old version of Singularity which may also be causing an issue - you should update/request an update to this, and/or installation of the more modern and maintained 'Apptainer' fork of SIngularity

[djbradshaw2]

Hi @jfy133,

Thank you very much for the information and guidance! Will request an update of singularity/installation of apptainer, pending my OIT's preference.

Sincerely,

David

[kw-cn]

Have you solved the problem?I ran into the same problem, but my Singularity version is 3.8.6.When I tried to create a sandbox, Singularity reported an error.
singularity build test m.sif
INFO: Starting build...
INFO: Verifying bootstrap image m.sif
ERROR: unpackSIF failed: root filesystem extraction failed: could not extract squashfs data, unsquashfs not found
FATAL: While performing build: packer failed to pack: root filesystem extraction failed: could not extract squashfs data, unsquashfs not found

[djbradshaw2]

Hi @kw-cn,

Sorry for the delayed response! And also, sorry I have not, but that is mainly because OIT has not had the time to solve it yet. They were able to get the mag pipeline to work on the HPC using an account with full permissions and updated singularity but have not translated that to regular users yet.

Not sure I can offer any advice besides checking to see if squashfs-tools is in your environment (conda, python, or whatever you are using) and installing it if it is not. Sorry if this is something you have already tried though...

Good luck!

Sincerely,

David