Addition of a new sub-module 'status' to assembly module. #334

ens-LCampbell · 2024-03-25T15:56:18Z

This submodule allows for users to:

track the assembly 'status' of INSDC accessions.
Status results obtained from assembly summary reports (JSON)
Users can request status checks by providing either A) list of core(s) dbs; or B) list of GCA/GCF accessions
Status tracking is achieved via use of containerised ncbi 'datasets' and integration of singularity to python (via spthon)
ncbi datasets is set up to perform batched searches, can be customised. Currently set to batches of n=100.
The results of said status check is stored for user inspection as TSV file.
Status of given INSDC accessions also retrieves (where it exists) the associated 'paired' assembly

This PR comprises the first iteration of assembly status tracking (i.e. version 1.0). With expanded functionality planned in later versions.

For other details on functionality see Jira ticket ENSMETAZOA-167

MatBarba

Works well and fast, nice one!

Lots of comments but mostly nitpicking and suggestions.

The main issue I had is that you need singularity installed, is there a way to run the system datasets instead?

containers/ncbi_datasets_v16.10.0.def

src/python/ensembl/io/genomio/assembly/status.py

containers/ncbi_datasets_v16.10.0.def

src/python/ensembl/io/genomio/assembly/status.py

containers/ncbi_datasets_v16.10.0.def

Code refinement, superflous condition Co-authored-by: Matthieu Barba <mbarba@ebi.ac.uk>

refinement on condition Co-authored-by: Matthieu Barba <mbarba@ebi.ac.uk>

Fix typos Co-authored-by: Matthieu Barba <mbarba@ebi.ac.uk>

typo fix Co-authored-by: Matthieu Barba <mbarba@ebi.ac.uk>

src/python/ensembl/io/genomio/assembly/status.py

Fix for metaconf script

JAlvarezJarreta

This is a extremely useful functionality added, nice work.

I may want to revisit at some point in the future if we want to preserve this as part of our library or as a separate item for the single reason that this will not work inside a container (container within container). But for now, welcome to GenomIO!

containers/ncbi_datasets_v16.10.0.def

pyproject.toml

src/python/ensembl/io/genomio/assembly/status.py

@jalvarez

Code changes/fixes from @jalvarez Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

JAlvarezJarreta

Just a few minor edits and suggestions.

src/python/ensembl/io/genomio/assembly/status.py

Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

add newline Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

change shorthand qry -> query Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

qry -> query Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

qry->query Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

capitalisation to formated string Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

Warning message improvement Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

Remove batch size comment, not needed default set Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

shorthand wording fix Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

wording change Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

fix to formatted string and list parentheses Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

runtime error instead of critical logging Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

src/python/ensembl/io/genomio/assembly/status.py

Improvement to print_json call() Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

shorthand wording fix Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

allow empty dict and check in TSV creation instead Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

py3.8 dictionary dict -> Dict declaration Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

punctuation Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

src/python/ensembl/io/genomio/assembly/status.py

ens-LCampbell added 6 commits March 25, 2024 15:23

Add container definition file datasets v16.10

2cf1db9

New assembly tracking sub module v1.0

ad03293

Black formatting

33646d5

Fix pylint issues on line lengths

42b8d26

Code improvements, list-compre, unneeded else

9e26774

Update assembly module __init__.py

f11d4c8

ens-LCampbell requested review from JAlvarezJarreta, vsitnik, Dishalodha and MatBarba March 25, 2024 15:56

ens-LCampbell added 4 commits March 25, 2024 16:27

Re-Blacked

2b53e73

Fixes and new function on input file param parsing

e710518

Refactor - Too many nested blocks

a8ccc13

Black formatting

27f4c4b

MatBarba approved these changes Mar 26, 2024

View reviewed changes

vsitnik reviewed Mar 26, 2024

View reviewed changes

containers/ncbi_datasets_v16.10.0.def Show resolved Hide resolved

ens-LCampbell and others added 4 commits March 26, 2024 13:51

Update src/python/ensembl/io/genomio/assembly/status.py

4df0c5f

Code refinement, superflous condition Co-authored-by: Matthieu Barba <mbarba@ebi.ac.uk>

Update src/python/ensembl/io/genomio/assembly/status.py

557ecc9

refinement on condition Co-authored-by: Matthieu Barba <mbarba@ebi.ac.uk>

Update src/python/ensembl/io/genomio/assembly/status.py

734db6d

Fix typos Co-authored-by: Matthieu Barba <mbarba@ebi.ac.uk>

Update src/python/ensembl/io/genomio/assembly/status.py

2d3e377

typo fix Co-authored-by: Matthieu Barba <mbarba@ebi.ac.uk>

MatBarba reviewed Mar 26, 2024

View reviewed changes

src/python/ensembl/io/genomio/assembly/status.py Outdated Show resolved Hide resolved

ens-LCampbell and others added 5 commits March 26, 2024 16:21

Vasily suggestion linebreaks on def file

7437350

@MatBarba: Various code improvements

3e5c85d

Merge pull request #335 from Ensembl/main

a0dea4a

Fix for metaconf script

Change accession query to accn, revert extract_assembly_metadata

331029e

Simplified strain/isolate logic + report dict

e818957

JAlvarezJarreta requested changes Mar 28, 2024

View reviewed changes

ens-LCampbell and others added 3 commits April 2, 2024 10:49

Apply suggestions from code review

e4a9416

Code changes/fixes from @jalvarez Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

Clean up commented code, add successful exit

c06ad7d

Add license header to datasets singularity def file

7173a54

Replace personal email with metazoa email

1c528e7

JAlvarezJarreta assigned ens-LCampbell Apr 8, 2024

JAlvarezJarreta requested changes Apr 9, 2024

View reviewed changes

Jorge round of suggesstions/tweaks

71e49e3

ens-LCampbell requested a review from JAlvarezJarreta April 10, 2024 08:29

ens-LCampbell and others added 13 commits April 10, 2024 10:20

black and pylint

1ac417a

Simplify parameterisation check

df3a254

Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

Update src/python/ensembl/io/genomio/assembly/status.py

d4bb430

add newline Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

Update src/python/ensembl/io/genomio/assembly/status.py

c35b5d1

change shorthand qry -> query Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

Update src/python/ensembl/io/genomio/assembly/status.py

2202d60

qry -> query Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

Update src/python/ensembl/io/genomio/assembly/status.py

9e127c3

qry->query Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

Update src/python/ensembl/io/genomio/assembly/status.py

a58ebe3

capitalisation to formated string Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

Update src/python/ensembl/io/genomio/assembly/status.py

d93dc71

Warning message improvement Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

Update src/python/ensembl/io/genomio/assembly/status.py

1312ff2

Remove batch size comment, not needed default set Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

Update src/python/ensembl/io/genomio/assembly/status.py

255f080

shorthand wording fix Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

Update src/python/ensembl/io/genomio/assembly/status.py

9da00a7

wording change Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

Update src/python/ensembl/io/genomio/assembly/status.py

437a674

fix to formatted string and list parentheses Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

Update src/python/ensembl/io/genomio/assembly/status.py

1a7d88a

runtime error instead of critical logging Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

JAlvarezJarreta reviewed Apr 10, 2024

View reviewed changes

src/python/ensembl/io/genomio/assembly/status.py Outdated Show resolved Hide resolved

ens-LCampbell and others added 6 commits April 10, 2024 11:45

Update src/python/ensembl/io/genomio/assembly/status.py

711a684

Improvement to print_json call() Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

Update src/python/ensembl/io/genomio/assembly/status.py

1a6617e

shorthand wording fix Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

Update src/python/ensembl/io/genomio/assembly/status.py

70dfed0

allow empty dict and check in TSV creation instead Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

Update src/python/ensembl/io/genomio/assembly/status.py

3e947f6

py3.8 dictionary dict -> Dict declaration Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

Update src/python/ensembl/io/genomio/assembly/status.py

dce58eb

punctuation Co-authored-by: J. Alvarez-Jarreta <jalvarez@ebi.ac.uk>

edits to declaration and doc string

8f8b8f2

JAlvarezJarreta reviewed Apr 10, 2024

View reviewed changes

src/python/ensembl/io/genomio/assembly/status.py Outdated Show resolved Hide resolved

Update src/python/ensembl/io/genomio/assembly/status.py

be90eed

JAlvarezJarreta approved these changes Apr 10, 2024

View reviewed changes

Fix to Path declaration and docs string tweaks

af6d341

ens-LCampbell merged commit b483b5b into main Apr 10, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Addition of a new sub-module 'status' to assembly module. #334

Addition of a new sub-module 'status' to assembly module. #334

ens-LCampbell commented Mar 25, 2024

MatBarba left a comment

JAlvarezJarreta left a comment

JAlvarezJarreta left a comment

Addition of a new sub-module 'status' to assembly module. #334

Addition of a new sub-module 'status' to assembly module. #334

Conversation

ens-LCampbell commented Mar 25, 2024

MatBarba left a comment

Choose a reason for hiding this comment

JAlvarezJarreta left a comment

Choose a reason for hiding this comment

JAlvarezJarreta left a comment

Choose a reason for hiding this comment