Move CI to GitHub Actions

.github/pull_request_template.md

@@ -20,9 +20,9 @@ _If not self-evident, mention what prompted the change._
 _If the update is a hotfix, it is sufficient to rely on the development testing along with the Travis self-test automatically applied to the PR._
 
 _Test routine to verify the stability of the PR:_
-- _`bash /home/proj/production/servers/resources/hasta.scilifelab.se/update-microsalt-stage.sh BRANCHNAME`_
+- _`bash /home/proj/production/servers/resources/hasta.scilifelab.se/install-microsalt-stage.sh BRANCHNAME`_
 - _`us`_
-- _`source activate S_microSALT`_
+- _`conda activate S_microSALT`_
 - _(SITUATIONAL) `export MICROSALT_CONFIG=/home/proj/dropbox/microSALT.json`_
 - _Select a relevant subset of the following:_
 - _`microSALT analyse project MIC3109`_

.github/workflows/run_tests.yml

@@ -0,0 +1,44 @@
+name: Tests
+
+on: [push]
+
+
+env:
+  CONDA_PREFIX: /home/runner/work/microSALT/microSALT/tests
+  COVERALLS_REPO_TOKEN: ${{ secrets.COVERALLS_REPO_TOKEN }}
+
+jobs:
+  build-linux:
+    runs-on: ubuntu-20.04
+    strategy:
+      max-parallel: 5
+
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python 3.6
+      uses: actions/setup-python@v3
+      with:
+        python-version: '3.6.15'
+    - name: Create conda/mamba environment using micromamba
+      uses: mamba-org/setup-micromamba@v1
+      with:
+        environment-file: environment.yml
+        cache-downloads: true
+    - name: Install microSALT
+      run: |
+        mkdir ~/.microSALT && cp configExample.json ~/.microSALT/config.json
+        GITHUB_BRANCH=${GITHUB_REF##*/}
+        pip install -r requirements.txt -r requirements-dev.txt
+        pip install -U .
+    - name: Lint with flake8
+      run: |
+        # stop the build if there are Python syntax errors or undefined names
+        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
+        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
+        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+    - name: Test with pytest
+      run: |
+        pytest --cov=microSALT tests
+    - name: Run Coveralls
+      run: |
+        coveralls

README.md

@@ -1,4 +1,4 @@
-![Build status](https://travis-ci.com/Clinical-Genomics/microSALT.svg?branch=master) 
+[![Build status](https://github.com/clinical-genomics/microsalt/actions/workflows/run_tests.yml/badge.svg)](https://github.com/clinical-genomics/microsalt/actions/workflows/run_tests.yml)
 [![Coverage Status](https://coveralls.io/repos/github/Clinical-Genomics/microSALT/badge.svg?branch=master)](https://coveralls.io/github/Clinical-Genomics/microSALT?branch=master)
 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.4026043.svg)](https://doi.org/10.5281/zenodo.4026043)
 
@@ -10,22 +10,33 @@
 
 __Microbial Sequence Analysis and Loci-based Typing pipeline__
 
-_The microbial sequence analysis and loci-based typing pipeline (microSALT) is used to analyse microbial samples.
-It produces a quality control of the sample, determines a sample's organism specific sequence type, and its resistance pattern. microSALT also provides a database storage solution and report generation of these results._
+_The microbial sequence analysis and loci-based typing pipeline (microSALT) is
+used to analyse microbial samples.  It produces a quality control of the
+sample, determines a sample's organism specific sequence type, and its
+resistance pattern. microSALT also provides a database storage solution and
+report generation of these results._
 
-_microSALT uses a combination of python, sqLite and flask. Python is used for the majority of functionality, the database is handled through sqLite and the front-end is handled through flask. All analysis activity by microSALT requires a SLURM cluster._
+_microSALT uses a combination of python, sqLite and flask. Python is used for
+the majority of functionality, the database is handled through sqLite and the
+front-end is handled through flask. All analysis activity by microSALT requires
+a SLURM cluster._
 
 ## Quick installation
-* `yes | bash <(curl https://raw.githubusercontent.com/Clinical-Genomics/microSALT/master/install.sh)`
-* `cp configExample.json $HOME/.microSALT/config.json`
-* `vim $HOME/.microSALT/config.json`
+
+1. `yes | bash <(curl https://raw.githubusercontent.com/Clinical-Genomics/microSALT/master/install.sh)`
+2. `cp configExample.json $HOME/.microSALT/config.json`
+3. `vim $HOME/.microSALT/config.json`
 
 ## Configuration
-Copy the configuration file to microSALTs hidden home directory, _or_ copy the configuration file anywhere and direct the envvar MICROSALT_CONFIG to it. See example: 
+
+Copy the configuration file to microSALTs hidden home directory, _or_ copy the
+configuration file anywhere and direct the envvar MICROSALT_CONFIG to it. See
+example:
 
 `cp configExample.json $HOME/.microSALT/config.json`
 
 _or_
+
 ```
 cp configExample.json /MY/FAV/FOLDER/config.json
 export MICROSALT_CONFIG=/MY/FAV/FOLDER/config.json
@@ -34,35 +45,49 @@ export MICROSALT_CONFIG=/MY/FAV/FOLDER/config.json
 __Then edit the fields to match your environment__.
 
 ## Usage
-* `microSALT analyse` contains functions to start sbatch job(s) & produce output to `folders['results']`. Afterwards the parsed results  are uploaded to the SQL back-end and produce reports (HTML), which are then automatically e-mailed to the user.
-* `microSALT utils` contains various functionality, including generating the sample description json, manually adding new reference organisms and re-generating reports.
+
+- `microSALT analyse` contains functions to start sbatch job(s) & produce
+  output to `folders['results']`. Afterwards the parsed results  are uploaded
+  to the SQL back-end and produce reports (HTML), which are then automatically
+  e-mailed to the user.
+- `microSALT utils` contains various functionality, including generating the
+  sample description json, manually adding new reference organisms and
+  re-generating reports.
 
 ## Databases
+
 ### MLST Definitions
-microSALT will automatically download & use the MLST definitions for any organism on pubMLST (https://pubmlst.org/databases/).
-Other definitions may be used, as long as they retain the same format. 
+
+microSALT will automatically download & use the MLST definitions for any
+organism on [pubMLST](https://pubmlst.org/databases). Other definitions may be
+used, as long as they retain the same format.
 
 ### Resistance genes
-microSALT will automatically download & use the resistance genes of resFinder (https://cge.cbs.dtu.dk/services/data.php).
+
+microSALT will automatically download & use the resistance genes of [ResFinder](https://bitbucket.org/genomicepidemiology/resfinder).
 Any definitions will work, as long as they retain the same formatting.
 
 ## Requirements
+
 ### Hardware
-* A slurm enabled HPC
-* A (clarity) LIMS server
 
+- A [SLURM](https://slurm.schedmd.com) enabled HPC system
+- A (clarity) LIMS server
 
 ### Software
-* Conda ( https://www.anaconda.com/distribution/ )
-* Python 3.6
-* sqLite Service ( https://www.sqlite.org/download.html )
+
+- [Conda](https://conda.io)
+- Python 3.6
+- [SQLite](https://www.sqlite.org)
 
 ## Contributing to this repo
+
 This repository follows the Github flow approach to adding updates.
 For more information, see https://guides.github.com/introduction/flow/
 
 ## Credits
-* Isak Sylvin - Lead developer
-* Emma Sernstad - Accreditation ready reports
-* Tanja Normark - Various issues
-* Maya Brandi - Various issues
+
+- Isak Sylvin - Lead developer
+- Emma Sernstad - Accreditation ready reports
+- Tanja Normark - Various issues
+- Maya Brandi - Various issues

environment.yml

@@ -8,11 +8,12 @@ dependencies:
   - blast=2.12.0
   - bwa=0.7.17
   - picard=2.20.3
-  - pigz=2.4
+  - pigz>=2.4
   - quast=5.0.2
   - samtools=1.13
   - spades=3.13.1
   - trimmomatic=0.39
   - r-base=4.1.1
   - openjdk=11.0.9.1
-
+  - pytest=6.2.5
+  - pytest-cov=2.12.1

install.sh

@@ -1,6 +1,7 @@
 #!/usr/bin/env bash
 
 set -e
+set -x
 shopt -s nullglob
 
 #Suggests provided branch. Else suggests master
@@ -51,12 +52,18 @@ while true; do
 done
 echo "Thank you, setting up environment $cname!"
 
+#Only use micromamba if inside GitHub Actions
+if [[ -z "${GITHUB_ACTIONS}" ]]; then
+    conda_cmd="micromamba"
+else
+    conda_cmd="conda"
+fi
 #Unload environment
-conda info | tac | tac | grep -q $cname && source deactivate || :
+$conda_cmd info | tac | tac | grep -q $cname && source deactivate || :
 #Remove environment if already present
-conda remove -y -n $cname --all || :
+$conda_cmd remove -y -n $cname --all || :
 
-conda env create -n $cname -f https://raw.githubusercontent.com/Clinical-Genomics/microSALT/$branch/environment.yml
+$conda_cmd env create -n $cname -f https://raw.githubusercontent.com/Clinical-Genomics/microSALT/$branch/environment.yml
 source activate $cname
 
 if [[ $type == "release" ]]; then

microSALT/cli.py

@@ -43,7 +43,8 @@
     sys.exit(-1)
 
 
-def set_cli_config(config):
+@click.pass_context
+def set_cli_config(ctx, config):
     if config != "":
         if os.path.exists(config):
             try:

requirements-dev.txt

@@ -1,9 +1,10 @@
 black==21.9b0
-coverage==5.5
+coverage<5
+flake8>=4.0.1,<4.1
 gitlint==0.15.1
 pytest==6.2.5
-pytest-cov==2.12.1
-python-coveralls==2.9.3
+pytest-cov
+python-coveralls
 pylint==2.10.2
 mock==4.0.3
 yamllint==1.26.3

setup.py

@@ -24,10 +24,10 @@
     data_files=[('expec', ['unique_references/ExPEC.fsa']),
                 ('logos', ['artwork/microsalt.jpg', 'artwork/swedac.jpg']),
                 ('testdata', ['tests/testdata/sampleinfo_samples.json','tests/testdata/sampleinfo_mlst.json','tests/testdata/sampleinfo_projects.json','tests/testdata/sampleinfo_reports.json','tests/testdata/sampleinfo_expec.json','tests/testdata/sampleinfo_resistance.json','tests/testdata/quast_results.tsv', 'tests/testdata/blast_single_resistance.txt','tests/testdata/blast_single_loci.txt','tests/testdata/alignment.stats.ref','tests/testdata/alignment.stats.raw','tests/testdata/alignment.stats.map','tests/testdata/alignment.stats.ins','tests/testdata/alignment.stats.dup','tests/testdata/alignment.stats.cov','configExample.json']),
-                ('testproject/AAA1234_2000.1.2_3.4.5/AAA1234A1/', ['tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A1/dummysequences_1.fastq.gz', 'tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A1/dummysequences_2.fastq.gz']), 
-                ('testproject/AAA1234_2000.1.2_3.4.5/AAA1234A2/', ['tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A2/dummysequences_1.fastq.gz', 'tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A2/dummysequences_2.fastq.gz']),
-                ('testproject/AAA1234_2000.1.2_3.4.5/AAA1234A3/', ['tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A3/dummysequences_1.fastq.gz', 'tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A3/dummysequences_2.fastq.gz']),
-                ('testproject/AAA1234_2000.1.2_3.4.5/AAA1234A4/', ['tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A4/dummysequences_1.fastq.gz', 'tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A4/dummysequences_2.fastq.gz'])],
+                ('tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A1/', ['tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A1/dummysequences_1.fastq.gz', 'tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A1/dummysequences_2.fastq.gz']),
+                ('tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A2/', ['tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A2/dummysequences_1.fastq.gz', 'tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A2/dummysequences_2.fastq.gz']),
+                ('tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A3/', ['tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A3/dummysequences_1.fastq.gz', 'tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A3/dummysequences_2.fastq.gz']),
+                ('tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A4/', ['tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A4/dummysequences_1.fastq.gz', 'tests/testdata/AAA1234_2000.1.2_3.4.5/AAA1234A4/dummysequences_2.fastq.gz'])],
     entry_points={
         'console_scripts': ['microSALT=microSALT.cli:root'],
     },

tests/test_commands.py

@@ -38,8 +38,8 @@ def dbm():
   dbm = DB_Manipulator(config=preset_config,log=logger)
   dbm.create_tables()
 
-  for antry in unpack_db_json('sampleinfo_projects.json'):
-    dbm.add_rec(antry, 'Projects')
+  for entry in unpack_db_json('sampleinfo_projects.json'):
+    dbm.add_rec(entry, 'Projects')
   for entry in unpack_db_json('sampleinfo_mlst.json'):
     dbm.add_rec(entry, 'Seq_types')
   for bentry in unpack_db_json('sampleinfo_resistance.json'):
@@ -85,7 +85,7 @@ def path_testproject():
   #Check if release install exists
   for entry in os.listdir(get_python_lib()):
     if 'microSALT-' in entry:
-      testproject = os.path.abspath(os.path.join(os.path.expandvars('$CONDA_PREFIX'), 'testproject/AAA1234_2000.1.2_3.4.5'))
+      testproject = os.path.abspath(os.path.join(os.path.expandvars('$CONDA_PREFIX'), 'testdata/AAA1234_2000.1.2_3.4.5'))
   return testproject
 
 

tests/test_config.py

@@ -70,7 +70,7 @@ def test_paths(exp_config):
   for entry in preset_config.keys():
     if entry != '_comment':
       if isinstance(preset_config[entry], str) and '/' in preset_config[entry] and entry not in ['database', 'genologics']:
-        unmade_fldr = preset_config[entry][thing]
+        unmade_fldr = preset_config[entry]
         assert (pathlib.Path(unmade_fldr).exists())
 
       #level two