Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Convert workflow to use Pegasus WMS #5

Open
wants to merge 161 commits into
base: base
Choose a base branch
from
Open

Convert workflow to use Pegasus WMS #5

wants to merge 161 commits into from

Conversation

spigo900
Copy link
Collaborator

No description provided.

@spigo900
Copy link
Collaborator Author

spigo900 commented Jul 27, 2020 •
edited
Loading

Possible "future work"/room to improve on this:

  • Currently there are hardcoded (yet simple) exceptions re: AlphaNLI and HellaSwag training parameters.
    • It might be handy to be able to specify simple exceptions like these in the Pegasus parameters file somehow.
  • The parameters files that the training script gets are "polluted" with whatever extra stuff is floating around in the Pegasus script's parameters file.
    • For example, the training script gets the number of bootstrapping samples to take when calculating accuracy in ensemble.py.
    • This is because of how the model parameters are set up.
    • It might be a good idea to put the model parameters in their own space so that they can be passed to the training script without all the other junk.

@spigo900 spigo900 marked this pull request as ready for review July 30, 2020 17:34
@spigo900
Copy link
Collaborator Author

spigo900 commented Jul 30, 2020 •
edited
Loading

@denizbeser This should be ready for review now.

I haven't tested the rebased version, but the original worked correctly.

I've implemented my "future work" ideas in a separate branch plus a few other improvements, but I'm going to hold off on merging those until I've tested that they work with the full workflow and not just the dev parameters. That may end up being a separate pull request.

@spigo900
Copy link
Collaborator Author

spigo900 commented Aug 4, 2020 •
edited
Loading

I've merged the various improvements (mentioned above) into this branch. The updated version works on the smaller development config (pegasus-dev.params), however I haven't finished testing it with the bigger config (pegasus-dev-full.params). (I ran it yesterday, but about eight of the jobs didn't finish, I think because they hit the job time limit.)

Training takes long enough for the failed models (training on AlphaNLI and HellaSwag) that I'm going to hold off on re-launching the full test run until tomorrow. For now I'm going to work on job throttling. At some point I will also add some better documentation on how to use the Pegasus script and workflows.

@spigo900
Copy link
Collaborator Author

This is confirmed to work with the full workflow.

Joseph Cecil added 21 commits August 18, 2020 13:57
@spigo900
Add Pegasus wrapper requirement
da52c7c
@spigo900
Add parameter files
d7ca092
@spigo900
Add vistautils requirement
b2e862b
@spigo900
Initial Pegasus skeleton
e1adb5a
@spigo900
Update root parameters file
b01a1db
@spigo900
Pass real job parameters to Pegasus
57fe8b2
@spigo900
Clean up some
f9cd332
@spigo900
Store more job info
bae796e
@spigo900
Fix _includes in runner.params
6fcd5d1
@spigo900
More pegasus stuff
feb5167
@spigo900
Use simple loop to read in job parameters for Pegasus runner
ed3b77c
@spigo900
Create parameters files mirroring Hydra configs
57d8c85
@spigo900
Clean up SAGA code
1963323
@spigo900
Modify train.py to take parameters files as input
0a23da0
@spigo900
Don't overwrite actual parameters with combinations parameters.
fc03fb3
@spigo900
Fix style issues
917c595
@spigo900
Move Slurm configuration into configuration file
57750a8
@spigo900
Remove old TODO
5beadd0
@spigo900
Fix usage of parameters_only_entry_point()
c5bdf99
@spigo900
Fix runner params.
55ef547
@spigo900
Fix resource request creation.
b9e4cc4
@spigo900
Ensemble: Fix try_without
69ce3b1
@spigo900
Fix dev ensembling params
1d1814d
@spigo900
Fix how eval.py gets model name
6b77a09
@spigo900
Fix how eval.py loads the model configuration
cbe7e6e
@spigo900
eval.py: Include other parameters in model configuration
b869efc 
Previously, the model could not find the task data.
@spigo900
Fix deprecation warning in eval.py
c1814e0 
np.random.random_integers is deprecated, apparently
@spigo900
Override testing: Change nonsense parameter instead of real one
0482a81 
This should speed up testing in the future, although I think there's no
more need for testing right now.
@spigo900
Rename nonsense parameter
4e35777
@spigo900
Fix typo
3d9128d
@spigo900
Use ephemeral in pegasus-dev configuration
c04cf32
@spigo900
Fix partition setup in Pegasus workflow
56a39b9
@spigo900
Add full-workflow development parameters
3d7c7b4 
These still run the full workflow, but don't run anything on MICS.
@spigo900
Shorter time limit for AlphaNLI in pegasus-dev-full.params
976009e 
This will probably not work.

Once fan-out is working I can set it to use MICS without disrupting
other users.
@spigo900
Fix how gold labels are found and frontload parameter-getting
9183aef 
This should make the script fail faster when it doesn't get the right
parameters.
@spigo900
Catch only FileNotFoundError in "couldn't find preds" exception block
d1e9e4a
@spigo900
Fix how model_without_seed is constructed
4b8f34a
@spigo900
Shorten line
7e20de8
@spigo900
Pegasus: Pass gold labels to ensemble script in new way
da14165
@spigo900
Ensemble: Rename variable to match parameter task_to_threshold
e7e8f39
@spigo900
Pegasus: Remove unnecessary line
24983ad
@spigo900
Ensemble: Include parameters in the dict of successful models
5a7582c
@spigo900
Pegasus: Fix how iteration over ensembling tasks is done
1c69984
@spigo900
Pegasus: Actually add task_to_gold parameters to ensembling parameters
9b192bd
@spigo900
Pegasus: Do job throttling
0ed048e
@spigo900
Pegasus: Pass imported modules instead of strings to run
bf5665f 
This should prevent typo-related problems and ensure that the references
get updated if the module names ever change again.
@spigo900
Check the list of tasks against
95510e2 
Thinking about it, probably the smarter thing to do would be to simply
not specify it in parameters_combinations but just use the keys from
tasks_to_thresholds. But I don't want to attempt that right now, since
the workflow is working now.
@spigo900
Rearrange comment
0ab6443
@spigo900
Document how to run the ensembling workflow
8f36dc3
@spigo900
Change AlphaNLI, HellaSwag, and SIQA to use internal dev sets
0eb9806
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@denizbeser denizbeser denizbeser approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@spigo900 @denizbeser