coref

[zz-hitnlper]

RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle)

[zz-hitnlper]

Traceback (most recent call last):
File "/home/zhangzhi/code/asp1190613/util/runner.py", line 147, in train
loss = wrapper(**example_gpu) / grad_accum
File "/home/zhangzhi/anaconda3/envs/asp2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zhangzhi/code/asp1190613/models/model_coref.py", line 145, in forward
seq2seq_output = self.model(
File "/home/zhangzhi/anaconda3/envs/asp2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zhangzhi/code/asp1190613/models/t5_coref.py", line 168, in forward
outputs = self.t5(
File "/home/zhangzhi/anaconda3/envs/asp2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zhangzhi/anaconda3/envs/asp2/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1485, in forward
encoder_outputs = self.encoder(
File "/home/zhangzhi/anaconda3/envs/asp2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zhangzhi/anaconda3/envs/asp2/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1107, in forward
layer_outputs = layer_module(
File "/home/zhangzhi/anaconda3/envs/asp2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
python-BaseException
return forward_call(*input, **kwargs)
File "/home/zhangzhi/anaconda3/envs/asp2/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward
self_attention_outputs = self.layer[0](
File "/home/zhangzhi/anaconda3/envs/asp2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zhangzhi/anaconda3/envs/asp2/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 594, in forward
attention_output = self.SelfAttention(
File "/home/zhangzhi/anaconda3/envs/asp2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zhangzhi/anaconda3/envs/asp2/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward
query_states = shape(self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head)
File "/home/zhangzhi/anaconda3/envs/asp2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zhangzhi/anaconda3/envs/asp2/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle)

[zz-hitnlper]

这是什么原因啊

[zz-hitnlper]

谁有能跑通的共治消解的代码啊，