Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stateful dataloader #1241

Closed
wants to merge 30 commits into from
Closed

Stateful dataloader #1241

wants to merge 30 commits into from

Conversation

andrewkho
Copy link
Contributor

@andrewkho andrewkho commented Mar 29, 2024 •
edited
Loading

Adds a StatefulDataLoader - drop in replacement for torch.utils.data.DataLoader - that offers state_dict/load_state_dict for mid-epoch checkpointing.

please see [RFC] Stateful DataLoader (TODO: link)

Please read through our contribution guide prior to
creating your pull request.

  • Note that there is a section on requirements related to adding a new DataPipe.

Fixes #{issue number}

Changes

  • Add torchdata.stateful_dataloader.StatefulDataLoader class
  • copy test_dataloader unit-tests from pytorch/torch/utils/data for regressions
  • adds test_state_dict unit-tests testing stateful functionality.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 29, 2024
@andrewkho andrewkho marked this pull request as ready for review April 5, 2024 00:08
@andrewkho andrewkho requested a review from gokulavasan April 5, 2024 00:14
@facebook-github-bot
Copy link
Contributor

@andrewkho has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

1 similar comment
@facebook-github-bot
Copy link
Contributor

@andrewkho has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@gokulavasan gokulavasan force-pushed the stateful_dataloader branch 3 times, most recently from e68f6d2 to a5b7695 Compare April 9, 2024 05:29
@facebook-github-bot
Copy link
Contributor

@gokulavasan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@gokulavasan gokulavasan force-pushed the stateful_dataloader branch from a5b7695 to 761a7cf Compare April 11, 2024 13:33
@facebook-github-bot
Copy link
Contributor

@gokulavasan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@gokulavasan
Capture base_seed as part of main snapshot and initialize worker loop…
0c862b1 
…, add test for it

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@gokulavasan gokulavasan force-pushed the stateful_dataloader branch from 435a496 to 0c862b1 Compare April 13, 2024 02:38
@facebook-github-bot
Copy link
Contributor

@gokulavasan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@gokulavasan
Add unit test to assert error if worker count changes on resumption
82a7872 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@gokulavasan gokulavasan force-pushed the stateful_dataloader branch from 5de543d to 82a7872 Compare April 13, 2024 08:41
@facebook-github-bot
Copy link
Contributor

@gokulavasan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

1 similar comment
@facebook-github-bot
Copy link
Contributor

@gokulavasan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@gokulavasan
Copyright and minor linter
a2101ca 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@gokulavasan gokulavasan force-pushed the stateful_dataloader branch from c244ad7 to a2101ca Compare April 13, 2024 09:39
@facebook-github-bot
Copy link
Contributor

@gokulavasan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@gokulavasan merged this pull request in 0c98f9c.

@andrewkho andrewkho deleted the stateful_dataloader branch April 23, 2024 23:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gokulavasan gokulavasan Awaiting requested review from gokulavasan

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Merged
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@andrewkho @facebook-github-bot @gokulavasan