You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
torch.Size([80362])
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1544174967633/work/aten/src/THC/THCCachingHostAllocator.cpp line=265 error=59 : device-side assert triggered
Traceback (most recent call last):
File "/data1/home/ideatmp/sigir21/hgcn/models/base_models.py", line 124, in compute_metrics
loss = F.binary_cross_entropy(pos_scores, torch.ones_like(pos_scores))
File "/data1/home/ideatmp/miniconda3/envs/HGN/lib/python3.6/site-packages/torch/nn/functional.py", line 2027, in > > binary_cross_entropy
input, target, weight, reduction_enum)
RuntimeError: reduce failed to synchronize: device-side assert triggered
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data1/home/ideatmp/.pycharm_helpers/pydev/pydevd.py", line 1668, in
main()
File "/data1/home/ideatmp/.pycharm_helpers/pydev/pydevd.py", line 1662, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/data1/home/ideatmp/.pycharm_helpers/pydev/pydevd.py", line 1072, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/data1/home/ideatmp/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/data1/home/ideatmp/sigir21/hgcn/train.py", line 213, in
train(args, feature)
File "/data1/home/ideatmp/sigir21/hgcn/train.py", line 129, in train
train_metrics = model.compute_metrics(embeddings, data, 'train')
File "/data1/home/ideatmp/sigir21/hgcn/models/base_models.py", line 127, in compute_metrics
print(pos_scores)
File "/data1/home/ideatmp/miniconda3/envs/HGN/lib/python3.6/site-packages/torch/tensor.py", line 66, in repr
return torch._tensor_str._str(self)
File "/data1/home/ideatmp/miniconda3/envs/HGN/lib/python3.6/site-packages/torch/_tensor_str.py", line 277, in _str
tensor_str = _tensor_str(self, indent)
File "/data1/home/ideatmp/miniconda3/envs/HGN/lib/python3.6/site-packages/torch/_tensor_str.py", line 195, in _tensor_str
formatter = _Formatter(get_summarized_data(self) if summarize else self)
File "/data1/home/ideatmp/miniconda3/envs/HGN/lib/python3.6/site-packages/torch/_tensor_str.py", line 221, in > get_summarized_data
return torch.cat((self[:PRINT_OPTS.edgeitems], self[-PRINT_OPTS.edgeitems:]))
RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-> > bld/pytorch_1544174967633/work/aten/src/THC/THCCachingHostAllocator.cpp:265
I debug the error, it shows when training, the curvature is nan. How to solve this problem?
The text was updated successfully, but these errors were encountered:
I debug the error, it shows when training, the curvature is nan. How to solve this problem?
The text was updated successfully, but these errors were encountered: