Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Usage of Dictionaries will cause a "Memory Leak" in Multi-Node Environment #2448

Open
MathiasBaumgartinger opened this issue Dec 5, 2024 · 0 comments
Labels
datasets Geospatial or benchmark datasets
Milestone

Comments

@MathiasBaumgartinger
Copy link
Contributor

Description

I am working on a SLURM multi-node architecture where each node has two GPUs. In such a multi-process environment, the usage of a dictionaries (i.e. samples provided by torchgeo-datamodules) will cause -- quote -- "copy-on-access problem of forked python processes". See pytorch/pytorch#13246 (comment) for more information. This refcounting problem leads to an ever increasing usage of memory, which will sooner or later cause the process to crash.

Steps to reproduce

Use any datamodule providing dictionaries as samples on multi-node environment

Version

0.6.0

@MathiasBaumgartinger MathiasBaumgartinger changed the title Multiple Workers will cause a "Memory Leak" in Multi-Node Environment [BUG] Usage of Dictionaries will cause a "Memory Leak" in Multi-Node Environment Dec 5, 2024
@adamjstewart adamjstewart added this to the 0.6.3 milestone Dec 5, 2024
@adamjstewart adamjstewart added the datasets Geospatial or benchmark datasets label Dec 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datasets Geospatial or benchmark datasets
Projects
None yet
Development

No branches or pull requests

2 participants