Merge pull request #178 from Clinical-Genomics/rc400 (major)

### Added 

- Release candidate microSALT 4.0.0

.github/pull_request_template.md

@@ -20,9 +20,9 @@ _If not self-evident, mention what prompted the change._
 _If the update is a hotfix, it is sufficient to rely on the development testing along with the Travis self-test automatically applied to the PR._
 
 _Test routine to verify the stability of the PR:_
-- _`bash /home/proj/production/servers/resources/hasta.scilifelab.se/update-microsalt-stage.sh BRANCHNAME`_
+- _`bash /home/proj/production/servers/resources/hasta.scilifelab.se/install-microsalt-stage.sh BRANCHNAME`_
 - _`us`_
-- _`source activate S_microSALT`_
+- _`conda activate S_microSALT`_
 - _(SITUATIONAL) `export MICROSALT_CONFIG=/home/proj/dropbox/microSALT.json`_
 - _Select a relevant subset of the following:_
 - _`microSALT analyse project MIC3109`_

.github/workflows/run_tests.yml

@@ -0,0 +1,44 @@
+name: Tests
+
+on: [push]
+
+
+env:
+  CONDA_PREFIX: /home/runner/work/microSALT/microSALT/tests
+  COVERALLS_REPO_TOKEN: ${{ secrets.COVERALLS_REPO_TOKEN }}
+
+jobs:
+  build-linux:
+    runs-on: ubuntu-20.04
+    strategy:
+      max-parallel: 5
+
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python 3.6
+      uses: actions/setup-python@v3
+      with:
+        python-version: '3.6.15'
+    - name: Create conda/mamba environment using micromamba
+      uses: mamba-org/setup-micromamba@v1
+      with:
+        environment-file: environment.yml
+        cache-downloads: true
+    - name: Install microSALT
+      run: |
+        mkdir ~/.microSALT && cp configExample.json ~/.microSALT/config.json
+        GITHUB_BRANCH=${GITHUB_REF##*/}
+        pip install -r requirements.txt -r requirements-dev.txt
+        pip install -U .
+    - name: Lint with flake8
+      run: |
+        # stop the build if there are Python syntax errors or undefined names
+        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
+        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
+        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+    - name: Test with pytest
+      run: |
+        pytest --cov=microSALT tests
+    - name: Run Coveralls
+      run: |
+        coveralls

.gitignore

@@ -13,4 +13,5 @@ unique_references/*.n*
 .idea/
 default_sample_info.json
 tmp.json
-
+.venv/
+.vscode*

README.md

@@ -1,4 +1,4 @@
-![Build status](https://travis-ci.com/Clinical-Genomics/microSALT.svg?branch=master) 
+[![Build status](https://github.com/clinical-genomics/microsalt/actions/workflows/run_tests.yml/badge.svg)](https://github.com/clinical-genomics/microsalt/actions/workflows/run_tests.yml)
 [![Coverage Status](https://coveralls.io/repos/github/Clinical-Genomics/microSALT/badge.svg?branch=master)](https://coveralls.io/github/Clinical-Genomics/microSALT?branch=master)
 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.4026043.svg)](https://doi.org/10.5281/zenodo.4026043)
 
@@ -10,22 +10,33 @@
 
 __Microbial Sequence Analysis and Loci-based Typing pipeline__
 
-_The microbial sequence analysis and loci-based typing pipeline (microSALT) is used to analyse microbial samples.
-It produces a quality control of the sample, determines a sample's organism specific sequence type, and its resistance pattern. microSALT also provides a database storage solution and report generation of these results._
+_The microbial sequence analysis and loci-based typing pipeline (microSALT) is
+used to analyse microbial samples.  It produces a quality control of the
+sample, determines a sample's organism specific sequence type, and its
+resistance pattern. microSALT also provides a database storage solution and
+report generation of these results._
 
-_microSALT uses a combination of python, sqLite and flask. Python is used for the majority of functionality, the database is handled through sqLite and the front-end is handled through flask. All analysis activity by microSALT requires a SLURM cluster._
+_microSALT uses a combination of python, sqLite and flask. Python is used for
+the majority of functionality, the database is handled through sqLite and the
+front-end is handled through flask. All analysis activity by microSALT requires
+a SLURM cluster._
 
 ## Quick installation
-* `yes | bash <(curl https://raw.githubusercontent.com/Clinical-Genomics/microSALT/master/install.sh)`
-* `cp configExample.json $HOME/.microSALT/config.json`
-* `vim $HOME/.microSALT/config.json`
+
+1. `yes | bash <(curl https://raw.githubusercontent.com/Clinical-Genomics/microSALT/master/install.sh)`
+2. `cp configExample.json $HOME/.microSALT/config.json`
+3. `vim $HOME/.microSALT/config.json`
 
 ## Configuration
-Copy the configuration file to microSALTs hidden home directory, _or_ copy the configuration file anywhere and direct the envvar MICROSALT_CONFIG to it. See example: 
+
+Copy the configuration file to microSALTs hidden home directory, _or_ copy the
+configuration file anywhere and direct the envvar MICROSALT_CONFIG to it. See
+example:
 
 `cp configExample.json $HOME/.microSALT/config.json`
 
 _or_
+
 ```
 cp configExample.json /MY/FAV/FOLDER/config.json
 export MICROSALT_CONFIG=/MY/FAV/FOLDER/config.json
@@ -34,35 +45,49 @@ export MICROSALT_CONFIG=/MY/FAV/FOLDER/config.json
 __Then edit the fields to match your environment__.
 
 ## Usage
-* `microSALT analyse` contains functions to start sbatch job(s) & produce output to `folders['results']`. Afterwards the parsed results  are uploaded to the SQL back-end and produce reports (HTML), which are then automatically e-mailed to the user.
-* `microSALT utils` contains various functionality, including generating the sample description json, manually adding new reference organisms and re-generating reports.
+
+- `microSALT analyse` contains functions to start sbatch job(s) & produce
+  output to `folders['results']`. Afterwards the parsed results  are uploaded
+  to the SQL back-end and produce reports (HTML), which are then automatically
+  e-mailed to the user.
+- `microSALT utils` contains various functionality, including generating the
+  sample description json, manually adding new reference organisms and
+  re-generating reports.
 
 ## Databases
+
 ### MLST Definitions
-microSALT will automatically download & use the MLST definitions for any organism on pubMLST (https://pubmlst.org/databases/).
-Other definitions may be used, as long as they retain the same format. 
+
+microSALT will automatically download & use the MLST definitions for any
+organism on [pubMLST](https://pubmlst.org/databases). Other definitions may be
+used, as long as they retain the same format.
 
 ### Resistance genes
-microSALT will automatically download & use the resistance genes of resFinder (https://cge.cbs.dtu.dk/services/data.php).
+
+microSALT will automatically download & use the resistance genes of [ResFinder](https://bitbucket.org/genomicepidemiology/resfinder).
 Any definitions will work, as long as they retain the same formatting.
 
 ## Requirements
+
 ### Hardware
-* A slurm enabled HPC
-* A (clarity) LIMS server
 
+- A [SLURM](https://slurm.schedmd.com) enabled HPC system
+- A (clarity) LIMS server
 
 ### Software
-* Conda ( https://www.anaconda.com/distribution/ )
-* Python 3.6
-* sqLite Service ( https://www.sqlite.org/download.html )
+
+- [Conda](https://conda.io)
+- Python 3.6
+- [SQLite](https://www.sqlite.org)
 
 ## Contributing to this repo
+
 This repository follows the Github flow approach to adding updates.
 For more information, see https://guides.github.com/introduction/flow/
 
 ## Credits
-* Isak Sylvin - Lead developer
-* Emma Sernstad - Accreditation ready reports
-* Tanja Normark - Various issues
-* Maya Brandi - Various issues
+
+- Isak Sylvin - Lead developer
+- Emma Sernstad - Accreditation ready reports
+- Tanja Normark - Various issues
+- Maya Brandi - Various issues

environment.yml

@@ -8,11 +8,12 @@ dependencies:
   - blast=2.12.0
   - bwa=0.7.17
   - picard=2.20.3
-  - pigz=2.4
+  - pigz>=2.4
   - quast=5.0.2
   - samtools=1.13
-  - spades=3.13.1
   - trimmomatic=0.39
   - r-base=4.1.1
   - openjdk=11.0.9.1
-
+  - pytest=6.2.5
+  - pytest-cov=2.12.1
+  - skesa=2.5.1

install.sh

@@ -1,6 +1,7 @@
 #!/usr/bin/env bash
 
 set -e
+set -x
 shopt -s nullglob
 
 #Suggests provided branch. Else suggests master
@@ -51,12 +52,18 @@ while true; do
 done
 echo "Thank you, setting up environment $cname!"
 
+#Only use micromamba if inside GitHub Actions
+if [[ -z "${GITHUB_ACTIONS}" ]]; then
+    conda_cmd="micromamba"
+else
+    conda_cmd="conda"
+fi
 #Unload environment
-conda info | tac | tac | grep -q $cname && source deactivate || :
+$conda_cmd info | tac | tac | grep -q $cname && source deactivate || :
 #Remove environment if already present
-conda remove -y -n $cname --all || :
+$conda_cmd remove -y -n $cname --all || :
 
-conda env create -n $cname -f https://raw.githubusercontent.com/Clinical-Genomics/microSALT/$branch/environment.yml
+$conda_cmd env create -n $cname -f https://raw.githubusercontent.com/Clinical-Genomics/microSALT/$branch/environment.yml
 source activate $cname
 
 if [[ $type == "release" ]]; then

microSALT/__init__.py

@@ -10,7 +10,7 @@
 from flask import Flask
 from distutils.sysconfig import get_python_lib
 
-__version__ = "3.3.7"
+__version__ = "4.0.0"
 
 app = Flask(__name__, template_folder="server/templates")
 app.config.setdefault("SQLALCHEMY_DATABASE_URI", "sqlite:///:memory:")
@@ -76,7 +76,10 @@
         logger.addHandler(ch)
 
         # Create paths mentioned in config
-        db_file = re.search("sqlite:///(.+)",preset_config["database"]["SQLALCHEMY_DATABASE_URI"],).group(1)
+        db_file = re.search(
+            "sqlite:///(.+)",
+            preset_config["database"]["SQLALCHEMY_DATABASE_URI"],
+        ).group(1)
         for entry in preset_config.keys():
             if entry != "_comment":
                 if (
@@ -119,8 +122,10 @@
                                 )
                                 output, error = proc.communicate()
                                 if proc.returncode != 0:
-                                  logger.error("Database writing failed! Invalid user access detected!")
-                                  sys.exit(-1)
+                                    logger.error(
+                                        "Database writing failed! Invalid user access detected!"
+                                    )
+                                    sys.exit(-1)
                             else:
                                 unmade_fldr = preset_config[entry][thing]
                             if not pathlib.Path(unmade_fldr).exists():
@@ -135,13 +140,13 @@
         )
         logger.addHandler(fh)
 
-        #Integrity check database 
+        # Integrity check database
         cmd = "sqlite3 {0}".format(db_file)
         cmd = cmd.split()
         cmd.append("pragma integrity_check;")
         proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
         output, error = proc.communicate()
-        if not 'ok' in str(output):
+        if not "ok" in str(output):
             logger.error("Database integrity failed! Lock-state detected!")
             sys.exit(-1)