GPU cuda implementation

[123LiVo321]

I'm very sorry for my ignorance, but apparently I'm not doing well with a very basic matter. I can't setup gpu-cuda. I tried different writing options in marker/settings.py:

TORCH_DEVICE=cuda
# TORCH_DEVICE: Optional[str] = None

or
TORCH_DEVICE: Optional[str] = cuda

together with

INFERENCE_RAM=16
# INFERENCE_RAM: int = 40

or
INFERENCE_RAM: int = 16

all 4 combinations, but i still get:

Loaded detection model vikp/surya_det2 on device cpu with dtype torch.float32
Loaded detection model vikp/surya_layout2 on device cpu with dtype torch.float32
Loaded reading order model vikp/surya_order on device cpu with dtype torch.float32
Loaded recognition model vikp/surya_rec on device cpu with dtype torch.float32
Loaded texify model to cpu with torch.float32 dtype

cpu on 100% , nothing on the side of GPU usage or memmory. I do have 4060ti / 16gb and using torch with gpu in other applications...

Would someone be kind enough to explain to the noob where he is making a mistake?

[VikParuchuri]

Just set an environment variable, like TORCH_DEVICE="cuda" python convert.py .... . It should auto-detect a CUDA device if it is available.

[123LiVo321]

So... I didn't follow the exact suggested order of installation and installed marker-pdf first.
Then see in pip list that 'torch' is already installed and thought everything is set&done... there was a torch, but.... yeah...but....
...so...
When I finally properly read the install instructions, visit the https://pytorch.org/get-started/locally/ , deleted the whole virtual and make brand new with install torch !!WITH CUDA!! properly, voila :

(marker) d:\marker>echo %TORCH_DEVICE%
cuda

(marker) d:\marker>echo %INFERENCE_RAM%
16

(marker) d:\marker>marker 1-input 2-output
Loaded detection model vikp/surya_det2 on device cuda with dtype torch.float16
Loaded detection model vikp/surya_layout2 on device cuda with dtype torch.float16
Loaded reading order model vikp/surya_order on device cuda with dtype torch.float16

• and I was thinking thats the WIN. But...but...there is always but(s) when I'm learning new things....

the log continues :

Loaded recognition model vikp/surya_rec on device cuda with dtype torch.float16
Loaded texify model to cuda with torch.float16 dtype
Converting 1 pdfs in chunk 1/1 with 1 processes, and storing in d:\marker\2-output
Detecting bboxes: 100%|██████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00, 1.17s/it]
C:\Users_\anaconda3\envs\marker\Lib\site-packages\surya\postprocessing\affinity.py:28: RuntimeWarning: invalid value encountered in divide
scaled_sobel = np.uint8(255 * abs_sobelx / np.max(abs_sobelx))
C:\Users_\anaconda3\envs\marker\Lib\site-packages\surya\postprocessing\affinity.py:28: RuntimeWarning: invalid value encountered in cast
scaled_sobel = np.uint8(255 * abs_sobelx / np.max(abs_sobelx))
Could not extract any text blocks for d:\marker\1-input\Contributing.pdf
Empty file: d:\marker\1-input\Contributing.pdf. Could not convert.
Processing PDFs: 100%|██████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00, 1.73s/pdf]
[W CudaIPCTypes.cpp:96] Producer process tried to deallocate over 1000 memory blocks referred by consumer processes. Deallocation might be significantly slowed down. We assume it will never going to be the case, but if it is, please file but to https://github.com/pytorch/pytorch
[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

When discussing it with the brand new chatGPT it suggests me tampering with the code, which seems a bit strange and tbh foreign to me.
I'm always able to run the project via the CPU if necessary, but it would be nice to use it as intended.

Would you mind pushing me one more time?

p.s.: I also noticed that after re-install the project completely [git clone, new env] and first run - the models were not downloaded again. If they aren't in the project repo or in the conda env, where they are?
p.p.s: Should I make a new 'Issue' instead of this off topic continuation?

[FlypigW]

pytorch/pytorch#118859