How to use GPU, the default is the CPU used

[Nicolas-XF]

https://github.com/thewh1teagle/kokoro-onnx/blob/main/examples/with_gpu.py
I installed PIP Install Kokoro-ONNX [GPU] soundDevice

But how should I operate in the CPU that is actually called?

My graphics card is RTX2070

[muduolin]

I have the same issue. it never uses GPU. I followed some instructions online but it did not work.

pip uninstall onnxruntime
pip install onnxruntime-gpu

I also tried:
pip uninstall kokoro-onnx
pip install kokoro-onnx[gpu]

None of this work. I don't think it is project related, it must be something with my OS or python setup.

[goldenboynicohd]

Same issue with everything installed as per muduolin on a GTX 1080

DEBUG [init.py:35] koko-onnx version 0.4.0 on Linux-6.12.10-arch1-1-x86_64-with-glibc2.40 #1 SMP PREEMPT_DYNAMIC Sat, 18 Jan 2025 02:26:57 +0000
DEBUG [init.py:54] Providers: ['CPUExecutionProvider']

pyproject.toml :

[project]
name = "test-gpu-kokoro"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
"kokoro-onnx[gpu]>=0.4.0",
"onnxruntime-gpu>=1.20.1",
"soundfile>=0.13.1",
]

[muduolin]

I found the solution on Windows 11. Need to downgrade CUDA to 11.8.

1. Install CUDA 11.8 (along side 12.8), Because I install 11.8 on top of 12.8, The system variable CUDA_PATH is set to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8

Make sure the bin directory (C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin) is ahead of 12.8 in the PATH variable

2. Install cuDNN 9.7.1 into the CUDA directory

3. pip install onnxruntime-gpu
   Add system variable ONNX_PROVIDER to CUDAExecutionProvider

4. python -c "import onnxruntime as ort;print(ort.get_device())"
   should return GPU

5. python -c "import onnxruntime as ort;print(ort.get_available_providers())"
   return ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']

6. "python hello.py" works with GPU
   but "uv run hello.py" does not work. someone mentioned uv virtual environment does not know how to load GPU

[goldenboynicohd]

Got it working too on linux thanks to muduolin.

Solution is to
export ONNX_PROVIDER=CUDAExecutionProvider
then it succeeds with uv run

uv run with_log.py
DEBUG [init.py:35] koko-onnx version 0.4.0 on Linux-6.13.1-arch1-1-x86_64-with-glibc2.41 #1 SMP PREEMPT_DYNAMIC Sun, 02 Feb 2025 01:02:29 +0000
DEBUG [init.py:54] Providers: ['CUDAExecutionProvider'

Thanks a lot

[Nicolas-XF]

Specify 11.8 or use cpu

[lhr-30]

python podcast.py
2025-02-18 08:49:26.337076107 [W:onnxruntime:, transformer_memcpy.cc:74 ApplyImpl] 39 Memcpy nodes are added to the graph main_graph for CUDAExecutionProvider. It might have negative impact on performance (including unable to run CUDA graph). Set session_options.log_severity_level=1 to see the detail logs before this message.
2025-02-18 08:49:26.353295781 [W:onnxruntime:, session_state.cc:1168 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2025-02-18 08:49:26.353328411 [W:onnxruntime:, session_state.cc:1170 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
2025-02-18 08:49:26.655019882 [W:onnxruntime:Default, scatter_nd.h:51 ScatterNDWithAtomicReduction] ScatterND with reduction=='none' only guarantees to be correct if indices are not duplicated.

Does anyone meet this issue? Is it ok to have this?