Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle crashed runs on wandb #20

Open
euanong opened this issue Jan 21, 2024 · 1 comment
Open

Handle crashed runs on wandb #20

euanong opened this issue Jan 21, 2024 · 1 comment
Assignees
@euanong
Labels
engineering priority: medium Non-blocking, but might become blocking

Comments

@euanong
Copy link
Collaborator

euanong commented Jan 21, 2024

When you KeyboardInterrupt, wandb seems to upload some extra data... note-to-self to do something about this.
@euanong euanong added engineering priority: medium Non-blocking, but might become blocking labels Jan 21, 2024
@euanong euanong self-assigned this Jan 21, 2024
@JasonGross
Copy link
Owner

Related: if we checkpoint the model so much that the data exceeds ~115 MB, the uploader seems to hang forever after ~115 MB, I wonder if there's a way to manually resume

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@euanong euanong

Labels
engineering priority: medium Non-blocking, but might become blocking
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@JasonGross @euanong