spaCy trf model is slow due to torch

[kanayer]

I have a spacy trf model running on a docker container (no GPU) and in production, it seems to be very slow. It takes up to 17s to process a single sentence. When I did Cprofile to check which step is taking the most time, the result was this:

ncalls  tottime  percall  cumtime  percall filename:lineno(function)

 73    5.838    0.080    5.838    0.080 {built-in method torch._C._nn.linear}

I tried to increase the number of torch threads by doing:

>>> torch.get_num_threads()
8
>>> torch.set_num_threads(16)
>>> torch.get_num_threads()
16

but it didn't help. Is there any way to speed it up?

[polm]

Transformer models are intended to be run on GPU and will be very slow without one. We consider it important that the code works, from a correctness perspective, but it will be impractical for most use cases.

If you do not have a GPU you should be fine with one of the CNN (non-trf) pipelines.