feat: finemapping template and DAG for UKB PPP #10
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
project-defiant
approved these changes
Jul 11, 2024
tskir
force-pushed
the
tskir-finemapping
branch
from
99e0990 to
300425b
Compare
|
@project-defiant This has been quite substantially rewritten compared to the first draft. Could you do another round of reviews, please? |
project-defiant
approved these changes
Sep 18, 2024
| **common.shared_dag_kwargs, | ||
| ) as dag: | ||
| ( | ||
| FinemappingBatchOperator.partial( |
There was a problem hiding this comment.
The partial will not work beyond the threshold, and I have tested it on local airflow DAG, this breaks on around ~5k partial tasks even with the threshold increase.
There was a problem hiding this comment.
Indeed, but just to clarify, in this PR the partial/expand routine iterates not on individual loci (of which there are potentially 100,000s in the worst case), but on chunks of the manifest, of which there are <10 in either case
This was referenced Sep 19, 2024
project-defiant
pushed a commit
that referenced
this pull request
Sep 23, 2024
* feat: template for creating finemapping jobs * feat: example DAG for creating finemapping jobs * fix: quote parameters containing = for Hydra * chore: add GENTROPY_DOCKER_IMAGE to common layer * feat: always use a list of jobs in the DAG * refactor: use manifest as input * feat: implement generate_manifests_for_finemapping * refactor: rewrite the DAG to use new functions * fix: import errors in DAG * fix: multiple fixes following test runs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The idea is to have a common finemapping template, which specific DAGs can reuse and modify according to their needs.