MBCS mapping step not running with snappy 0.3.0

[steinbrecht]

Describe the bug
I can't run the mbcs mapping step in the ngs_mapping pipeline using snappy 0.3.0. First, the ngs_mapping step starts a snakemake workflow, which works. Then, the mbcs wrapper starts a nested snakemake workflow, which fails. In the snakemake logs, there are the following two errors:

Traceback (most recent call last):
  File "/data/cephfs-1/work/groups/cubi/users/steinbrd_c/projects/test_calling_default/ngs_mapping/.snakemake/scripts/tmpfyx9oe4g.wrapper.py", line 341, in <module>
    run_snakemake(
  File "/data/cephfs-1/work/groups/cubi/users/steinbrd_c/snappy-pipeline/snappy_wrappers/wrapper_parallel.py", line 289, in run_snakemake
    raise SnakemakeExecutionFailed("Could not perform nested Snakemake call")

and

RuleException:
CalledProcessError in file /data/cephfs-1/work/groups/cubi/users/steinbrd_c/snappy-pipeline/snappy_pipeline/workflows/ngs_mapping/Snakefile, line 140:
Command 'source /data/cephfs-1/work/groups/cubi/users/steinbrd_c/miniforge3/bin/activate '/data/cephfs-1/work/groups/cubi/users/steinbrd_c/projects/test_calling_default/ngs_mapping/.snakemake/conda/edd526f7592cd0e87e0707a4ead87410_'; set -euo pipefail;  python /data/cephfs-1/work/groups/cubi/users/steinbrd_c/projects/test_calling_default/ngs_mapping/.snakemake/scripts/tmpfyx9oe4g.wrapper.py' returned non-zero exit status 1.
  File "/data/cephfs-1/work/groups/cubi/users/steinbrd_c/snappy-pipeline/snappy_pipeline/workflows/ngs_mapping/Snakefile", line 140, in __rule_ngs_mapping_mbcs_run
  File "/data/cephfs-1/work/groups/cubi/users/steinbrd_c/miniforge3/envs/snappy_env/lib/python3.12/concurrent/futures/thread.py", line 59, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message

To Reproduce
Define a workflow with

  ngs_mapping:
    tools:
      dna: [mbcs]

in the config file and run snappy version 0.3.0 with snappy-snake --snappy-pipeline-use-profile cubi-v1.

Additional context
The problem might be something like the newer python version (3.12) being used in snappy 0.3.0 which collides with the mbcs wrapper.

[steinbrecht]

The problem can be solved by using sbatch pipeline_job.sh and adding my conda env for snappy in the pipeline_job.sh script in lines 98 and 102.

[tedil]

Yes, the way these parallel wrappers are implemented right now is not ideal, exactly because of issues like this. We're planning to replace this with snakemake's scattergather, see #538 for an example.

Regarding adding the conda env: I think snappy start-step (and similar commands) also allow specifying a conda env which is then inserted into the pipeline_job.sh; if that is not already documented properly, we should do that.