Error Accept failed No space left on device when test 2 * 8 H800 test_internode

[Huixxi]

proxy.cc:1587 NCCL WARN [Service thread] Accept failed No space left on device
misc/shmutils.cc:87 NCCL WARN Error: failed to extend /dev/shm/nccl-nyj24T to 10486116 bytes, error: No space left on device (28)
terminate called after throwing an instance of 'terminate called after throwing an instance of 'terminate called after throwing an instance of 'EPExceptionEPExceptionEPException'
'
'
what(): what(): Failed: CUDA error /home/logs/DeepEP/csrc/deep_ep.cpp:89 'an illegal memory access was encountered'Failed: CUDA error /home/logs/DeepEP/csrc/deep_ep.cpp:89 'an illegal memory access was encountered' what():

Failed: CUDA error /home/logs/DeepEP/csrc/deep_ep.cpp:89 'an illegal memory access was encountered'
terminate called after throwing an instance of 'EPException'
what(): Failed: CUDA error /home/logs/DeepEP/csrc/deep_ep.cpp:89 'an illegal memory access was encountered'

[Huixxi]

sudo mount -o remount,size=8G /dev/shm # 调整为更大值如16G