CUDNN_STATUS_EXECUTION_FAILED

[aravindchaluvadi]

I amfacing this issue while running the train.py command

I am using CUDA=10.0 and GPU= GeForce GTX 1660 Ti

COMMAND : python train.py --task=QG --model=NQG --load_glove=True --feature_rich --data=squad --rnn=GRU --dec_hidden_size=512 --dropout=0.5 --batch_size=64 --eval_batch_size=64 --use_focus=True --n_mixture=3 --decoding=greedy

#==== Weight Initialization ====#

Total Parameters: 12295050

Loaded word embedding!
Traceback (most recent call last):
File "train.py", line 110, in
model.to(device)
File "/home/xxx/FocusSeq2Seq/focus/lib/python3.6/site-packages/torch/nn/modules/module.py", line 386, in to
return self._apply(convert)
File "/home/xxxx/FocusSeq2Seq/focus/lib/python3.6/site-packages/torch/nn/modules/module.py", line 193, in _apply
module._apply(fn)
File "/home/xxxx/FocusSeq2Seq/focus/lib/python3.6/site-packages/torch/nn/modules/module.py", line 193, in _apply
module._apply(fn)
File "/home/xxxx/FocusSeq2Seq/focus/lib/python3.6/site-packages/torch/nn/modules/module.py", line 193, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/xxxx/FocusSeq2Seq/focus/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 127, in _apply
self.flatten_parameters()
File "/home/xxx/FocusSeq2Seq/focus/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 123, in flatten_parameters
self.batch_first, bool(self.bidirectional))
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

[j-min]

I have not seen that cuDNN error.
This might be an cuDNN compatibility issue for GTX 1660 Ti discussed here.
Could you run other PyTorch code in your computer? Could you run this code in different hardward environment?