Faster transform-gisaid #90
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Processing in transform-gisaid is represented as a pipeline of steps. Each step either manipulates the data (`Transform`) or filters the data (`Filter`). At the start of each pipeline is a `DataSource`. To test this, I processed the same gisaid dataset with both the old script and the new script. The key differences in the output are: 1. The new script does _not_, by default, sort the sequences before outputting. It does perform the same deduplication process however. To sort the sequence data, use the option `--sorted-fasta`. 2. The new script interprets errors in `source-data/gisaid_annotations.tsv` differently than the old script. It ignores all rows that do not have four columns. As a result, some annotations are not processed. Additionally, there is the option `--output-unix-newline`, which forces all output (fasta files and metadata csv files) to use unix newlines. Fixes #77
|
Rebased onto master. |
The latest `transform-gisaid` script prints a warning to stdout when an annotation could not be applied to the metadata. Send the output of `transform-gisaid` to Slack to alert nCoV build maintainers of malformed annotation entries.
Closed
kairstenfay
commented
Aug 29, 2020
|
|
||
| ./bin/flag-metadata data/gisaid/metadata.tsv > data/gisaid/flagged_metadata.txt | ||
| ./bin/check-locations data/gisaid/metadata.tsv \ | ||
| data/gisaid/location_hierarchy.tsv \ | ||
| gisaid_epi_isl | ||
|
|
||
| if [[ "$branch" == master ]]; then | ||
| ./bin/notify-slack --upload "flagged-annotations" < "$flagged_annotations" |
There was a problem hiding this comment.
The printed warnings in Tony's transform-gisaid are now sent to Slack, but only when on the master branch. I just wanted to check that this is the desirable behavior. @eharkins
kairstenfay
added a commit
that referenced
this pull request
Aug 31, 2020
PR #90 recently introduced a major refactoring and optimization of `transform-gisaid` that has resulted in producing Windows (CRLF) line endings in the output TSVs. As a temporary fix to keep the ingest operational, always invoke the `--output-unix-newline` option in `ingest-gisaid` when invoking `transform-gisaid`.
kairstenfay
added a commit
that referenced
this pull request
Aug 31, 2020
PR #90 recently introduced a major refactoring and optimization of `transform-gisaid` that has resulted in producing Windows (CRLF) line endings in the metadata and additional info output TSVs. This was somehow not detected in local testing where output files were shown to be identical between the original and new transform scripts. As a temporary fix to keep the ingest operational, always use the `--output-unix-newline` option when invoking `transform-gisaid` in `ingest-gisaid`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Processing in transform-gisaid is represented as a pipeline of steps. Each step either manipulates the data (
Transform) or filters the data (Filter). At the start of each pipeline is aDataSource.To test this, I processed the same gisaid dataset with both the old script and the new script. The key differences in the output are:
--sorted-fasta.source-data/gisaid_annotations.tsvdifferently than the old script. It ignores all rows that do not have four columns. As a result, some annotations are not processed.Additionally, there is the option
--output-unix-newline, which forces all output (fasta files and metadata csv files) to use unix newlines.Fixes #77
originally opened as #88 by @ttung