Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Candidate for a Nextflow-based MVP #528

Draft
wants to merge 64 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
226bfda
Multiple changes
GallVp Nov 25, 2024
69c1264
Updated test CI
GallVp Nov 25, 2024
2bfe387
Added missing modules, profiles and manifest fields
GallVp Nov 25, 2024
3c9b2f8
Now using EDTA's test genome for the small test
GallVp Nov 25, 2024
4b7bc1b
Updated PR template
GallVp Nov 26, 2024
9838bf9
Merge pull request #1 from GallVp/update/ci
GallVp Nov 26, 2024
0195b02
Added fasta_helitronscanner_scan_draw
GallVp Nov 26, 2024
3453410
Updated snapshots
GallVp Nov 26, 2024
73ba207
Merge pull request #2 from GallVp/add/heli
GallVp Nov 26, 2024
29dbed0
Added CLEANUP_TANDEM
GallVp Nov 28, 2024
f297c7f
Merge pull request #3 from GallVp/cleanup_tandem
GallVp Nov 28, 2024
d43a72d
Initial try on format_helitronscanner
jguhlin Nov 20, 2024
c65812d
Make changes suggested by @GallVp
jguhlin Nov 27, 2024
0492d4a
Added some tests - not yet working, need the nf-test.config from the …
jguhlin Nov 27, 2024
28d08e0
Typo
jguhlin Nov 27, 2024
5726e39
Incorporated FORMAT_HELITRONSCANNER_OUT
GallVp Dec 3, 2024
2665f7e
Merge pull request #4 from GallVp/format_helitronscanner
GallVp Dec 3, 2024
104fac3
Renamed test to tests to avoid meta.id conflicts in nf-test and added…
GallVp Dec 5, 2024
427f3db
Fixed singularity container
GallVp Dec 5, 2024
f68a619
Fixed nextflow-setup version
GallVp Dec 5, 2024
3446362
Merge pull request #5 from GallVp/add/ltr_retriever_postprocess
GallVp Dec 5, 2024
f0491de
Multiple changes
GallVp Nov 25, 2024
74edd95
Updated test CI
GallVp Nov 25, 2024
cbd0d2f
Added missing modules, profiles and manifest fields
GallVp Nov 25, 2024
b555145
Now using EDTA's test genome for the small test
GallVp Nov 25, 2024
bdfe5ff
Updated PR template
GallVp Nov 26, 2024
aaae5e8
Added fasta_helitronscanner_scan_draw
GallVp Nov 26, 2024
5a874e7
Updated snapshots
GallVp Nov 26, 2024
d2feab0
Added CLEANUP_TANDEM
GallVp Nov 28, 2024
1d99e2a
Initial try on format_helitronscanner
jguhlin Nov 20, 2024
30c2fd6
Make changes suggested by @GallVp
jguhlin Nov 27, 2024
1f35229
Added some tests - not yet working, need the nf-test.config from the …
jguhlin Nov 27, 2024
1fb9dd2
Typo
jguhlin Nov 27, 2024
fde5ef3
Incorporated FORMAT_HELITRONSCANNER_OUT
GallVp Dec 3, 2024
41683a7
Renamed test to tests to avoid meta.id conflicts in nf-test and added…
GallVp Dec 5, 2024
82b34ac
Fixed singularity container
GallVp Dec 5, 2024
c941496
Fixed nextflow-setup version
GallVp Dec 5, 2024
9816823
Updated master from upstream
GallVp Dec 11, 2024
49a750f
Merge branch 'master' into update/master
GallVp Dec 11, 2024
fbfe3ed
Merge pull request #6 from GallVp/update/master
GallVp Dec 11, 2024
aa5abbb
Added TIR_LEARNER_POSTPROCESS
GallVp Dec 12, 2024
2dc08de
Merge pull request #7 from GallVp/add/tl_pp
GallVp Dec 12, 2024
295db65
Added REPEATMODELER_POSTPROCESS
GallVp Dec 12, 2024
d36deb0
Updated snapshots
GallVp Dec 12, 2024
ad367c3
Merge pull request #8 from GallVp/add/rm2_pp
GallVp Dec 12, 2024
f30f663
Added HELITRONSCANNER_POSTPROCESS
GallVp Dec 12, 2024
18cd782
Merge pull request #9 from GallVp/add/hs_pp
GallVp Dec 12, 2024
81207b3
Added ANNOSINE_POSTPROCESS
GallVp Dec 12, 2024
e31fd6b
Merge pull request #10 from GallVp/add/anno_pp
GallVp Dec 12, 2024
ba29b09
Added PROCESS_K
GallVp Dec 13, 2024
b71316f
Fixed conda version
GallVp Dec 16, 2024
96daead
One more attempt to match conda vers
GallVp Dec 16, 2024
6b9f76d
Fixed syntax
GallVp Dec 16, 2024
a9d9738
Removed blast from versions
GallVp Dec 16, 2024
28cd23b
Merge pull request #11 from GallVp/add/processk
GallVp Dec 16, 2024
09c7118
Added final_filter
GallVp Dec 17, 2024
4e9fbc5
Merge pull request #12 from GallVp/add/final_filter
GallVp Dec 17, 2024
ecdaccf
Updated the sanitize process
GallVp Dec 17, 2024
c0c34c2
Merge pull request #13 from GallVp/update/sanitize
GallVp Dec 17, 2024
438b78f
Added parameter validation by nf-schema
GallVp Dec 17, 2024
ac93561
Added pre-commit hook for version check
GallVp Dec 17, 2024
8e48bd5
Updated snapshots
GallVp Dec 17, 2024
c509827
Updated README
GallVp Dec 18, 2024
54210ac
Merge pull request #14 from GallVp/feat/nfcore
GallVp Dec 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 4 additions & 11 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,14 @@
<!--
# jguhlin/EDTA pull request

Many thanks for contributing to jguhlin/EDTA!
Many thanks for contributing to EDTA!

Please fill in the appropriate checklist below (delete whatever is not relevant).
These are the most common things requested on pull requests (PRs).

PRs should be made against the nextflow_reboot branch.
-->

## PR checklist

- [ ] PR to `nextflow_reboot` branch
- [ ] `conda` and `container` directives.
- [ ] Docker container + singularity container (optional)
- [ ] Flow `meta.id` with each data channel
- [ ] Use nf-core resource labels such as `process_high`
- [ ] Used nf-core module
- [ ] Use `versions.yml` or versions topic
- [ ] No process in the `main.nf`. We can have a process in a sub-workflow file
- [ ] This comment contains a description of changes (with reason).
- [ ] If you've fixed a bug or added code that should be tested, add tests!
- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker --outdir <OUTDIR>`).
30 changes: 30 additions & 0 deletions .github/check_binaries.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
#!/usr/bin/env bash

set -euo pipefail

EDTA_raw_pl_version=$(md5sum EDTA_raw.pl | cut -f1 -d' ')
bin_EDTA_raw_pl_version=$(md5sum bin/EDTA_raw.pl | cut -f1 -d' ')

if [[ $EDTA_raw_pl_version != $bin_EDTA_raw_pl_version ]]; then
echo 'EDTA_raw.pl != bin/EDTA_raw.pl'
echo 'Please synchronize bin/EDTA_raw.pl with EDTA_raw.pl'
exit 1
fi

EDTA_processK_pl_version=$(md5sum EDTA_processK.pl | cut -f1 -d' ')
bin_EDTA_processK_pl_version=$(md5sum bin/EDTA_processK.pl | cut -f1 -d' ')

if [[ $EDTA_processK_pl_version != $bin_EDTA_processK_pl_version ]]; then
echo 'EDTA_processK.pl != bin/EDTA_processK.pl'
echo 'Please synchronize bin/EDTA_processK.pl with EDTA_processK.pl'
exit 1
fi

EDTA_pl_version=$(md5sum EDTA.pl | cut -f1 -d' ')
bin_EDTA_pl_version=$(md5sum bin/EDTA.pl | cut -f1 -d' ')

if [[ $EDTA_pl_version != $bin_EDTA_pl_version ]]; then
echo 'EDTA.pl != bin/EDTA.pl'
echo 'Please synchronize bin/EDTA.pl with EDTA.pl'
exit 1
fi
10 changes: 10 additions & 0 deletions .github/include.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
".":
- ./.github/workflows/**
- ./nf-test.config
tests:
- ./assets/*
- ./bin/*
- ./conf/*
- ./main.nf
- ./nextflow_schema.json
- ./nextflow.config
11 changes: 11 additions & 0 deletions .github/version_checks.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#!/usr/bin/env bash

set -euo pipefail

config_version=$(sed -n "/^\s*version\s*=\s*'/s/version//p" nextflow.config | tr -d "=[:space:]'")
perl_version=$(sed -n 's|^my $version = "\(.*\)";|\1|p' EDTA.pl | tr -d '[:space:]')

if [[ "v$config_version" != $perl_version ]]; then
echo "config_version (v$config_version) != perl_version ($perl_version)"
exit 1
fi
109 changes: 69 additions & 40 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,14 +1,12 @@
name: EDTA Nextflow CI
on:
push:
branches:
- nextflow_reboot
pull_request:
branches:
- nextflow_reboot

env:
NXF_ANSI_LOG: false
NFT_WORKDIR: "~"
NFT_DIFF: "pdiff"
NFT_DIFF_ARGS: "--line-numbers --expand-tabs=2"
NXF_SINGULARITY_CACHEDIR: ${{ github.workspace }}/.singularity
NXF_SINGULARITY_LIBRARYDIR: ${{ github.workspace }}/.singularity

Expand All @@ -17,62 +15,93 @@ concurrency:
cancel-in-progress: true

jobs:
nf-test-changes:
name: Check for changes
runs-on: ubuntu-latest
outputs:
nf_test_files: ${{ steps.list.outputs.components }}
steps:
- uses: actions/checkout@v4.2.1
with:
fetch-depth: 0

- name: List nf-test files
id: list
uses: adamrtalbot/detect-nf-test-changes@v0.0.4
with:
head: ${{ github.sha }}
base: origin/${{ github.base_ref }}
include: .github/include.yaml

- name: print list of nf-test files
run: |
echo ${{ steps.list.outputs.components }}

test:
name: Run pipeline with test data
# Only run on push if this is the jguhlin nextflow_reboot branch (merged PRs)
if: "${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'jguhlin/EDTA') }}"
name: ${{ matrix.nf_test_files }} ${{ matrix.profile }} NF-${{ matrix.NXF_VER }}
needs: [nf-test-changes]
if: needs.nf-test-changes.outputs.nf_test_files != '[]'
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
nextflow:
- '24.04.4'
profile:
- docker
- singularity
- conda
NXF_VER:
- "24.10.3"

nf_test_files: ["${{ fromJson(needs.nf-test-changes.outputs.nf_test_files) }}"]
profile: [conda, docker, singularity]

steps:
- name: Check out pipeline code
uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4
uses: actions/checkout@v4.2.1

- name: Install Nextflow
uses: nf-core/setup-nextflow@v2
uses: nf-core/setup-nextflow@v2.0.0
with:
version: ${{ matrix.nextflow }}
version: "${{ matrix.NXF_VER }}"

# - name: Disk space cleanup
# uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1
# Will be needed with large data sets. Can take long to run
- uses: actions/setup-python@v5.2.0
with:
python-version: "3.11"
architecture: "x64"

- name: Install pdiff to see diff between nf-test snapshots
run: |
python -m pip install --upgrade pip
pip install pdiff

- uses: nf-core/setup-nf-test@v1.1.2
with:
version: 0.9.2

- name: Setup apptainer
if: matrix.profile == 'singularity'
uses: eWaterCycle/setup-apptainer@main

- name: Set up Singularity
if: matrix.profile == 'singularity'
run: |
mkdir -p $NXF_SINGULARITY_CACHEDIR
mkdir -p $NXF_SINGULARITY_LIBRARYDIR

- name: Set up miniconda
if: matrix.profile == 'conda'
uses: conda-incubator/setup-miniconda@a4260408e20b96e80095f42ff7f1a15b27dd94ca # v3
with:
miniconda-version: "latest"
auto-update-conda: true
channels: conda-forge,bioconda

- name: Conda setup
if: matrix.profile == 'conda'
- name: Run nf-test
run: |
conda clean -a
conda install -n base conda-libmamba-solver
conda config --set solver libmamba
echo $(realpath $CONDA)/condabin >> $GITHUB_PATH
echo $(realpath python) >> $GITHUB_PATH
nf-test test --verbose ${{ matrix.nf_test_files }} --profile "+${{ matrix.profile }}"

confirm-pass:
runs-on: ubuntu-latest
needs: [test]
if: always()
steps:
- name: All tests ok
if: ${{ !contains(needs.*.result, 'failure') }}
run: exit 0
- name: One or more tests failed
if: ${{ contains(needs.*.result, 'failure') }}
run: exit 1

- name: Run pipeline with test data
- name: debug-print
if: always()
run: |
nextflow run \
${GITHUB_WORKSPACE} \
-profile ${{ matrix.profile }},test
echo "toJSON(needs) = ${{ toJSON(needs) }}"
echo "toJSON(needs.*.result) = ${{ toJSON(needs.*.result) }}"
20 changes: 20 additions & 0 deletions .github/workflows/linting.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
name: linting
on:
pull_request:

jobs:
pre-commit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4

- name: Set up Python 3.12
uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5
with:
python-version: "3.12"

- name: Install pre-commit
run: pip install pre-commit

- name: Run pre-commit
run: pre-commit run --all-files
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,6 @@ bin/TIR-Learner*/Module3/Maize_model.sav
work/
.nextflow.log*
.nextflow/*
results/
.nf-test*
null/
19 changes: 19 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
repos:
- repo: local
hooks:
- id: check_binaries
name: Check binaries
language: system
entry: >
.github/check_binaries.sh
always_run: true
fail_fast: true
pass_filenames: false
- id: version_checks
name: Version checks
language: system
entry: >
.github/version_checks.sh
always_run: true
fail_fast: true
pass_filenames: false
Loading
Loading