C lib loading: add fallback with sensible error msg #1615

Titus-von-Koeller · 2025-04-28T16:14:04Z

fixes #1548
related to #693 #853 #1189
#1210
#1260
#1281
#1427
#1467
#1607

github-actions · 2025-04-28T16:17:46Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Titus-von-Koeller · 2025-05-05T09:42:03Z

CASE 1 - missing deps (reproduced by deleting some linked libs and installing torch version with a different packaged cuda major version):

❯ python -c 'from bitsandbytes.cextension import lib; lib.cquantize_blockwise_fp16_nf4()'
WARNING: BNB_CUDA_VERSION=124 environment variable detected; loading libbitsandbytes_cuda124.so.
This can be used to load a bitsandbytes version that is different from the PyTorch CUDA version.
If this was unintended set the BNB_CUDA_VERSION variable to an empty string: export BNB_CUDA_VERSION=
If you use the manual override make sure the right libcudart.so is in your LD_LIBRARY_PATH
For example by adding the following to your .bashrc: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<path_to_cuda_dir/lib64

bitsandbytes library load error: libcudart.so.12: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "/home/ubuntu/src/bnb/bitsandbytes/cextension.py", line 328, in <module>
    lib = get_native_library()
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/src/bnb/bitsandbytes/cextension.py", line 315, in get_native_library
    dll = ct.cdll.LoadLibrary(str(binary_path))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/ctypes/__init__.py", line 454, in LoadLibrary
    return self._dlltype(name)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libcudart.so.12: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/ubuntu/src/bnb/bitsandbytes/cextension.py", line 294, in __getattr__
    raise RuntimeError(f"{self.formatted_error}\n\nNative code method attempted to access: {name}")

RuntimeError: 🚨 CUDA SETUP ERROR: Missing dependency: libcudart.so.12 🚨

CUDA 12.x runtime libraries were not found in the LD_LIBRARY_PATH.

To fix this, make sure that:
1. You have installed CUDA 12.x toolkit on your system
2. The CUDA runtime libraries are in your LD_LIBRARY_PATH

You can add them with (and persist the change by adding the line to your .bashrc):
   export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/cuda-12.x/lib64

Original error: libcudart.so.12: cannot open shared object file: No such file or directory

🔍 Run this command for detailed diagnostics:
python -m bitsandbytes

If you've tried everything and still have issues:
1. Include ALL version info (operating system, bitsandbytes, pytorch, cuda, python)
2. Describe what you've tried in detail
3. Open an issue with this information:
   https://github.com/bitsandbytes-foundation/bitsandbytes/issues

Native code method attempted to access: cquantize_blockwise_fp16_nf4

Titus-von-Koeller · 2025-05-05T13:51:02Z

case 2 - custom configured CUDA version (other than PyTorch CUDA version):

❯ python -c 'from bitsandbytes.cextension import lib; lib.cquantize_blockwise_fp16_nf4()'
WARNING: BNB_CUDA_VERSION=125 environment variable detected; loading libbitsandbytes_cuda125.so.
This can be used to load a bitsandbytes version that is different from the PyTorch CUDA version.
If this was unintended set the BNB_CUDA_VERSION variable to an empty string: export BNB_CUDA_VERSION=
If you use the manual override make sure the right libcudart.so is in your LD_LIBRARY_PATH
For example by adding the following to your .bashrc: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<path_to_cuda_dir/lib64

bitsandbytes library load error: Configured CUDA binary not found at /home/ubuntu/src/bnb/bitsandbytes/libbitsandbytes_cuda125.so
Traceback (most recent call last):
  File "/home/ubuntu/src/bnb/bitsandbytes/cextension.py", line 335, in <module>
    lib = get_native_library()
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/src/bnb/bitsandbytes/cextension.py", line 315, in get_native_library
    raise RuntimeError(f"Configured CUDA binary not found at {cuda_binary_path}")
RuntimeError: Configured CUDA binary not found at /home/ubuntu/src/bnb/bitsandbytes/libbitsandbytes_cuda125.so
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/ubuntu/src/bnb/bitsandbytes/cextension.py", line 298, in __getattr__
    raise RuntimeError(f"{self.formatted_error}Native code method attempted to access: lib.{name}()")
RuntimeError: 
🚨 CUDA VERSION MISMATCH 🚨
Requested CUDA version:          12.5
Detected PyTorch CUDA version:   11.8
Available pre-compiled versions: 
  - 12.3
  - 12.4

This means:
The version you're trying to use is NOT distributed with this package

Attempted to use bitsandbytes native library functionality but it's not available.

This typically happens when:
1. bitsandbytes doesn't ship with a pre-compiled binary for your CUDA version
2. The library wasn't compiled properly during installation from source

To make bitsandbytes work, the compiled library version MUST exactly match the linked CUDA version.
If your CUDA version doesn't have a pre-compiled binary, you MUST compile from source.

You have two options:
1. COMPILE FROM SOURCE (required if no binary exists):
   https://huggingface.co/docs/bitsandbytes/main/en/installation#cuda-compile
2. Use BNB_CUDA_VERSION to specify a DIFFERENT CUDA version from the detected one, which is installed on your machine and matching an available pre-compiled version listed above

Original error: Configured CUDA binary not found at /home/ubuntu/src/bnb/bitsandbytes/libbitsandbytes_cuda125.so

🔍 Run this command for detailed diagnostics:
python -m bitsandbytes

If you've tried everything and still have issues:
1. Include ALL version info (operating system, bitsandbytes, pytorch, cuda, python)
2. Describe what you've tried in detail
3. Open an issue with this information:
   https://github.com/bitsandbytes-foundation/bitsandbytes/issues

Native code method attempted to access: lib.cquantize_blockwise_fp16_nf4()

Titus-von-Koeller · 2025-05-05T15:19:06Z

case 3 - no BNB CUDA native lib but CUDA detected (through PyTorch):

❯ python -c 'from bitsandbytes.cextension import lib; lib.cquantize_blockwise_fp16_nf4()'
WARNING: BNB_CUDA_VERSION=124 environment variable detected; loading libbitsandbytes_cuda124.so.
This can be used to load a bitsandbytes version that is different from the PyTorch CUDA version.
If this was unintended set the BNB_CUDA_VERSION variable to an empty string: export BNB_CUDA_VERSION=
If you use the manual override make sure the right libcudart.so is in your LD_LIBRARY_PATH
For example by adding the following to your .bashrc: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<path_to_cuda_dir/lib64

bitsandbytes library load error: Configured CUDA binary not found at /home/ubuntu/src/bnb/bitsandbytes/libbitsandbytes_cuda124.so
Traceback (most recent call last):
  File "/home/ubuntu/src/bnb/bitsandbytes/cextension.py", line 262, in <module>
    lib = get_native_library()
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/src/bnb/bitsandbytes/cextension.py", line 242, in get_native_library
    raise RuntimeError(f"Configured CUDA binary not found at {cuda_binary_path}")
RuntimeError: Configured CUDA binary not found at /home/ubuntu/src/bnb/bitsandbytes/libbitsandbytes_cuda124.so
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/ubuntu/src/bnb/bitsandbytes/cextension.py", line 225, in __getattr__
    raise RuntimeError(f"{self.formatted_error}Native code method attempted to access: lib.{name}()")
RuntimeError: 
🚨 Forgot to compile the bitsandbytes library? 🚨
1. You're not using the package but checked-out the source code
2. You MUST compile from source

Attempted to use bitsandbytes native library functionality but it's not available.

This typically happens when:
1. bitsandbytes doesn't ship with a pre-compiled binary for your CUDA version
2. The library wasn't compiled properly during installation from source

To make bitsandbytes work, the compiled library version MUST exactly match the linked CUDA version.
If your CUDA version doesn't have a pre-compiled binary, you MUST compile from source.

You have two options:
1. COMPILE FROM SOURCE (required if no binary exists):
   https://huggingface.co/docs/bitsandbytes/main/en/installation#cuda-compile
2. Use BNB_CUDA_VERSION to specify a DIFFERENT CUDA version from the detected one, which is installed on your machine and matching an available pre-compiled version listed above

Original error: Configured CUDA binary not found at /home/ubuntu/src/bnb/bitsandbytes/libbitsandbytes_cuda124.so

🔍 Run this command for detailed diagnostics:
python -m bitsandbytes

If you've tried everything and still have issues:
1. Include ALL version info (operating system, bitsandbytes, pytorch, cuda, python)
2. Describe what you've tried in detail
3. Open an issue with this information:
   https://github.com/bitsandbytes-foundation/bitsandbytes/issues

Native code method attempted to access: lib.cquantize_blockwise_fp16_nf4()

Titus-von-Koeller · 2025-05-05T15:41:37Z

case 4a - no fitting CUDA lib relative to PyTorch-detected CUDA installation:

❯ python -c 'from bitsandbytes.cextension import lib; lib.cquantize_blockwise_fp16_nf4()'
bitsandbytes library load error: Configured CUDA binary not found at /home/ubuntu/src/bnb/bitsandbytes/libbitsandbytes_cuda118.so
Traceback (most recent call last):
  File "/home/ubuntu/src/bnb/bitsandbytes/cextension.py", line 262, in <module>
    lib = get_native_library()
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/src/bnb/bitsandbytes/cextension.py", line 242, in get_native_library
    raise RuntimeError(f"Configured CUDA binary not found at {cuda_binary_path}")
RuntimeError: Configured CUDA binary not found at /home/ubuntu/src/bnb/bitsandbytes/libbitsandbytes_cuda118.so
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/ubuntu/src/bnb/bitsandbytes/cextension.py", line 225, in __getattr__
    raise RuntimeError(f"{self.formatted_error}Native code method attempted to access: lib.{name}()")
RuntimeError: 
🚨 CUDA VERSION MISMATCH 🚨
Requested CUDA version:          11.8
Detected PyTorch CUDA version:   11.8
Available pre-compiled versions: 
  - 12.4

This means:
The version you're trying to use is NOT distributed with this package

Attempted to use bitsandbytes native library functionality but it's not available.

This typically happens when:
1. bitsandbytes doesn't ship with a pre-compiled binary for your CUDA version
2. The library wasn't compiled properly during installation from source

To make bitsandbytes work, the compiled library version MUST exactly match the linked CUDA version.
If your CUDA version doesn't have a pre-compiled binary, you MUST compile from source.

You have two options:
1. COMPILE FROM SOURCE (required if no binary exists):
   https://huggingface.co/docs/bitsandbytes/main/en/installation#cuda-compile
2. Use BNB_CUDA_VERSION to specify a DIFFERENT CUDA version from the detected one, which is installed on your machine and matching an available pre-compiled version listed above

Original error: Configured CUDA binary not found at /home/ubuntu/src/bnb/bitsandbytes/libbitsandbytes_cuda118.so

🔍 Run this command for detailed diagnostics:
python -m bitsandbytes

If you've tried everything and still have issues:
1. Include ALL version info (operating system, bitsandbytes, pytorch, cuda, python)
2. Describe what you've tried in detail
3. Open an issue with this information:
   https://github.com/bitsandbytes-foundation/bitsandbytes/issues

Native code method attempted to access: lib.cquantize_blockwise_fp16_nf4()

case 4b - custom BNB_CUDA_VERSION=124 and cuda detected

only difference is in this part

🚨 CUDA VERSION MISMATCH 🚨
Requested CUDA version:          12.4
Detected PyTorch CUDA version:   11.8
Available pre-compiled versions: 
  - 12.3

C lib loading: add fallback with sensible error msg

1f2a970

Titus-von-Koeller mentioned this pull request Apr 28, 2025

Fix AttributeError in 4-bit quantization when CUDA library is not loaded #1607

Closed

matthewdouglas added the CUDA Setup label Apr 28, 2025

matthewdouglas added this to the v0.46.0 milestone Apr 28, 2025

Titus-von-Koeller added 3 commits April 29, 2025 15:03

further improvements to C lib fallback

c2480e3

further tweaks to reporting

1c1b257

cleanup existing code

9274fb1

matthewdouglas previously approved these changes Apr 30, 2025

View reviewed changes

further cleanup

43ee60f

Titus-von-Koeller dismissed matthewdouglas’s stale review via 43ee60f April 30, 2025 16:24

Titus-von-Koeller added 2 commits May 5, 2025 14:01

validated case1 missing dep + case2 custom cuda, yet missing lib

bc2c2a8

delete dead code

71a9ce4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

C lib loading: add fallback with sensible error msg #1615

C lib loading: add fallback with sensible error msg #1615

Titus-von-Koeller commented Apr 28, 2025

github-actions bot commented Apr 28, 2025

Titus-von-Koeller commented May 5, 2025 •

edited

Loading

Titus-von-Koeller commented May 5, 2025

Titus-von-Koeller commented May 5, 2025

Titus-von-Koeller commented May 5, 2025

C lib loading: add fallback with sensible error msg #1615

Are you sure you want to change the base?

C lib loading: add fallback with sensible error msg #1615

Conversation

Titus-von-Koeller commented Apr 28, 2025

github-actions bot commented Apr 28, 2025

Titus-von-Koeller commented May 5, 2025 • edited Loading

Titus-von-Koeller commented May 5, 2025

Titus-von-Koeller commented May 5, 2025

Titus-von-Koeller commented May 5, 2025

Titus-von-Koeller commented May 5, 2025 •

edited

Loading