add best metric support checkpoint #679
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This pull request was exported from Phabricator. Differential Revision: D52739597 |
JKSenthil
added a commit
to JKSenthil/tnt
that referenced
this pull request
Jan 17, 2024
Summary: # Context This diff stack is aimed to implement best metric checkpoint support. # This Diff This diff adds the best metric support checkpoint functionality to the base checkpointer. 1. Adds `BestCheckpointConfig` struct to configure how to checkpoint best metric. It takes a metric name and mode (ie min or max) which indicates which is used to determine best value 2. In the BaseCheckpointer constructor: * Adds `save_every_n_eval_epochs` to BaseCheckpointer, which can be used to save at end of eval epoch during fit * If best checkpoint config is used, will sort existing checkpoint paths by metric value, as opposed to recency 3. Adds `self._should_save_checkpoint()` to check if checkpoint needs to be saved or not (is used for both normal and best checkpoint configs) 4. Augments `self._generate_checkpoint_and_upkeep()` to consider best checkpoint config and calls appropriate logic # Next Diff Forwards these args for best metric checkpoint to the derivative checkpointers (TSS and DCPSaver) # Future Can consider case of passing lambda function instead of reading the metric value from the unit Reviewed By: galrotem Differential Revision: D52739597
JKSenthil
force-pushed
the
export-D52739597
branch
from
fb01159 to
9e38c4c
Compare
|
This pull request was exported from Phabricator. Differential Revision: D52739597 |
JKSenthil
force-pushed
the
export-D52739597
branch
from
9e38c4c to
6ad3164
Compare
JKSenthil
added a commit
to JKSenthil/tnt
that referenced
this pull request
Jan 17, 2024
Summary: # Context This diff stack is aimed to implement best metric checkpoint support. # This Diff This diff adds the best metric support checkpoint functionality to the base checkpointer. 1. Adds `BestCheckpointConfig` struct to configure how to checkpoint best metric. It takes a metric name and mode (ie min or max) which indicates which is used to determine best value 2. In the BaseCheckpointer constructor: * Adds `save_every_n_eval_epochs` to BaseCheckpointer, which can be used to save at end of eval epoch during fit * If best checkpoint config is used, will sort existing checkpoint paths by metric value, as opposed to recency 3. Adds `self._should_save_checkpoint()` to check if checkpoint needs to be saved or not (is used for both normal and best checkpoint configs) 4. Augments `self._generate_checkpoint_and_upkeep()` to consider best checkpoint config and calls appropriate logic # Next Diff Forwards these args for best metric checkpoint to the derivative checkpointers (TSS and DCPSaver) # Future Can consider case of passing lambda function instead of reading the metric value from the unit Reviewed By: galrotem Differential Revision: D52739597
|
This pull request was exported from Phabricator. Differential Revision: D52739597 |
JKSenthil
added a commit
to JKSenthil/tnt
that referenced
this pull request
Jan 17, 2024
Summary: # Context This diff stack is aimed to implement best metric checkpoint support. # This Diff This diff adds the best metric support checkpoint functionality to the base checkpointer. 1. Adds `BestCheckpointConfig` struct to configure how to checkpoint best metric. It takes a metric name and mode (ie min or max) which indicates which is used to determine best value 2. In the BaseCheckpointer constructor: * Adds `save_every_n_eval_epochs` to BaseCheckpointer, which can be used to save at end of eval epoch during fit * If best checkpoint config is used, will sort existing checkpoint paths by metric value, as opposed to recency 3. Adds `self._should_save_checkpoint()` to check if checkpoint needs to be saved or not (is used for both normal and best checkpoint configs) 4. Augments `self._generate_checkpoint_and_upkeep()` to consider best checkpoint config and calls appropriate logic # Next Diff Forwards these args for best metric checkpoint to the derivative checkpointers (TSS and DCPSaver) # Future Can consider case of passing lambda function instead of reading the metric value from the unit Reviewed By: galrotem Differential Revision: D52739597
JKSenthil
added a commit
to JKSenthil/tnt
that referenced
this pull request
Jan 18, 2024
Summary: # Context This diff stack is aimed to implement best metric checkpoint support. # This Diff This diff adds the best metric support checkpoint functionality to the base checkpointer. 1. Adds `BestCheckpointConfig` struct to configure how to checkpoint best metric. It takes a metric name and mode (ie min or max) which indicates which is used to determine best value 2. In the BaseCheckpointer constructor: * Adds `save_every_n_eval_epochs` to BaseCheckpointer, which can be used to save at end of eval epoch during fit * If best checkpoint config is used, will sort existing checkpoint paths by metric value, as opposed to recency 3. Adds `self._should_save_checkpoint()` to check if checkpoint needs to be saved or not (is used for both normal and best checkpoint configs) 4. Augments `self._generate_checkpoint_and_upkeep()` to consider best checkpoint config and calls appropriate logic # Next Diff Forwards these args for best metric checkpoint to the derivative checkpointers (TSS and DCPSaver) # Future Can consider case of passing lambda function instead of reading the metric value from the unit Reviewed By: galrotem Differential Revision: D52739597
JKSenthil
force-pushed
the
export-D52739597
branch
from
6ad3164 to
c7ca97e
Compare
|
This pull request was exported from Phabricator. Differential Revision: D52739597 |
Summary: # Context This diff stack is aimed to implement best metric checkpoint support. # This Diff This diff adds the best metric support checkpoint functionality to the base checkpointer. 1. Adds `BestCheckpointConfig` struct to configure how to checkpoint best metric. It takes a metric name and mode (ie min or max) which indicates which is used to determine best value 2. In the BaseCheckpointer constructor: * Adds `save_every_n_eval_epochs` to BaseCheckpointer, which can be used to save at end of eval epoch during fit * If best checkpoint config is used, will sort existing checkpoint paths by metric value, as opposed to recency 3. Adds `self._should_save_checkpoint()` to check if checkpoint needs to be saved or not (is used for both normal and best checkpoint configs) 4. Augments `self._generate_checkpoint_and_upkeep()` to consider best checkpoint config and calls appropriate logic # Next Diff Forwards these args for best metric checkpoint to the derivative checkpointers (TSS and DCPSaver) # Future Can consider case of passing lambda function instead of reading the metric value from the unit Reviewed By: galrotem Differential Revision: D52739597
JKSenthil
force-pushed
the
export-D52739597
branch
from
c7ca97e to
74bdab9
Compare
|
This pull request was exported from Phabricator. Differential Revision: D52739597 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Context
This diff stack is aimed to implement best metric checkpoint support.
This Diff
This diff adds the best metric support checkpoint functionality to the base checkpointer.
BestCheckpointConfigstruct to configure how to checkpoint best metric. It takes a metric name and mode (ie min or max) which indicates which is used to determine best valuesave_every_n_eval_epochsto BaseCheckpointer, which can be used to save at end of eval epoch during fitself._should_save_checkpoint()to check if checkpoint needs to be saved or not (is used for both normal and best checkpoint configs)self._generate_checkpoint_and_upkeep()to consider best checkpoint config and calls appropriate logicNext Diff
Forwards these args for best metric checkpoint to the derivative checkpointers (TSS and DCPSaver)
Future
Can consider case of passing lambda function instead of reading the metric value from the unit
Differential Revision: D52739597