You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am working on a SLURM multi-node architecture where each node has two GPUs. In such a multi-process environment, the usage of a dictionaries (i.e. samples provided by torchgeo-datamodules) will cause -- quote -- "copy-on-access problem of forked python processes". See pytorch/pytorch#13246 (comment) for more information. This refcounting problem leads to an ever increasing usage of memory, which will sooner or later cause the process to crash.
Steps to reproduce
Use any datamodule providing dictionaries as samples on multi-node environment
Version
0.6.0
The text was updated successfully, but these errors were encountered:
MathiasBaumgartinger
changed the title
Multiple Workers will cause a "Memory Leak" in Multi-Node Environment
[BUG] Usage of Dictionaries will cause a "Memory Leak" in Multi-Node Environment
Dec 5, 2024
Description
I am working on a SLURM multi-node architecture where each node has two GPUs. In such a multi-process environment, the usage of a dictionaries (i.e. samples provided by torchgeo-datamodules) will cause -- quote -- "copy-on-access problem of forked python processes". See pytorch/pytorch#13246 (comment) for more information. This refcounting problem leads to an ever increasing usage of memory, which will sooner or later cause the process to crash.
Steps to reproduce
Use any datamodule providing dictionaries as samples on multi-node environment
Version
0.6.0
The text was updated successfully, but these errors were encountered: