Installation in Docker : error in compilation of extlib/mat_mul.cu #269

fmagera · 2022-08-04T16:13:13Z

fmagera
Aug 4, 2022

Hi !
Would it be possible to have a Dockerfile with the right config ?
I've tried many compatible versions of pytorch and CUDA, but I always get the same error when building theseus-ai..
In my last trial, I started from nvidia ngc pytorch container nvcr.io/nvidia/pytorch:21.06-py3, which is an Ubuntu 20.04 with cuda 11.3, python 3.8 and I reinstalled torch==1.10.1+cu113 version.

Here's the full error :

    /home/dir/theseus/theseus/extlib/mat_mult.cu(74): error: no instance of overloaded function "atomicAdd" matches the argument list
                argument types are: (double *, double)
    /home/dir/theseus/theseus/extlib/mat_mult.cu(239): error: no instance of overloaded function "atomicAdd" matches the argument list
                argument types are: (double *, double)
    2 errors detected in the compilation of "/home/dir/theseus/theseus/extlib/mat_mult.cu".
    ninja: build stopped: subcommand failed.
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1717, in _run_ninja_build
        subprocess.run(
      File "/opt/conda/lib/python3.8/subprocess.py", line 516, in run
        raise CalledProcessError(retcode, process.args,
    subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
    The above exception was the direct cause of the following exception:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/dir/theseus/setup.py", line 60, in <module>
        setuptools.setup(
      File "/opt/conda/lib/python3.8/site-packages/setuptools/__init__.py", line 163, in setup
        return distutils.core.setup(**attrs)
      File "/opt/conda/lib/python3.8/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/opt/conda/lib/python3.8/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/opt/conda/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/opt/conda/lib/python3.8/site-packages/setuptools/command/develop.py", line 38, in run
        self.install_for_development()
      File "/opt/conda/lib/python3.8/site-packages/setuptools/command/develop.py", line 140, in install_for_development
        self.run_command('build_ext')
      File "/opt/conda/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/opt/conda/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/opt/conda/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 87, in run
        _build_ext.run(self)
      File "/opt/conda/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
        _build_ext.build_ext.run(self)
      File "/opt/conda/lib/python3.8/distutils/command/build_ext.py", line 340, in run
        self.build_extensions()
      File "/opt/conda/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 735, in build_extensions
        build_ext.build_extensions(self)
      File "/opt/conda/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 194, in build_extensions
        self.build_extension(ext)
      File "/opt/conda/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 208, in build_extension
        _build_ext.build_extension(self, ext)
      File "/opt/conda/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension
        objects = self.compiler.compile(sources,
      File "/opt/conda/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 556, in unix_wrap_ninja_compile
        _write_ninja_file_and_compile_objects(
      File "/opt/conda/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1399, in _write_ninja_file_and_compile_objects
        _run_ninja_build(
      File "/opt/conda/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1733, in _run_ninja_build
        raise RuntimeError(message) from e
    RuntimeError: Error compiling objects for extension
    ----------------------------------------
ERROR: Command errored out with exit status 1: /opt/conda/bin/python3.8 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/dir/theseus/setup.py'"'"'; __file__='"'"'/home/dir/theseus/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

Answered by luisenp

Aug 4, 2022

HI @fmagera, sorry about the install issues. Can you take a look at #257 and see if it helps? It adds a bash script that can be used to generate a Dockerfile for compiling wheels using pytorch's manylinux containers. Usage is ./build_scripts/build_wheel.sh <OUTPUT_DIR>. If the wheels don't work for you maybe you can use it as a starting point?

View full answer

luisenp · 2022-08-04T17:02:13Z

luisenp
Aug 4, 2022
Collaborator

HI @fmagera, sorry about the install issues. Can you take a look at #257 and see if it helps? It adds a bash script that can be used to generate a Dockerfile for compiling wheels using pytorch's manylinux containers. Usage is ./build_scripts/build_wheel.sh <OUTPUT_DIR>. If the wheels don't work for you maybe you can use it as a starting point?

1 reply

mhmukadam Aug 22, 2022

See this reply below for the detailed steps to follow.

fanyangr · 2022-08-04T20:05:24Z

fanyangr
Aug 4, 2022

Hi @fmagera , I had a similar issue. What I did was to use conda to install ninja again first. Then I completely remove the theseus file and clone it again, because there might be some cmake files from previous installlation that could lead to additional error. Hope that also works for you.

0 replies

fmagera · 2022-08-05T09:20:34Z

fmagera
Aug 5, 2022
Author

The script worked, thanks 👍

0 replies

MickShen7558 · 2022-08-11T16:17:13Z

MickShen7558
Aug 11, 2022

Hi @fmagera,

Thank you for your scripts. I had the same issue when installing Theseus in my container. Now I built the wheel and successfully installed it to my conda env in my container. But when I run pytest, I failed with the following message:

======================================================================================================= short test summary info ========================================================================================================
FAILED theseus/extlib/tests/test_cusolver_lu_solver.py::test_lu_solver_1 - ModuleNotFoundError: No module named 'theseus.extlib.cusolver_lu_solver'
FAILED theseus/extlib/tests/test_cusolver_lu_solver.py::test_lu_solver_2 - ModuleNotFoundError: No module named 'theseus.extlib.cusolver_lu_solver'
FAILED theseus/extlib/tests/test_cusolver_lu_solver.py::test_lu_solver_3 - ModuleNotFoundError: No module named 'theseus.extlib.cusolver_lu_solver'
FAILED theseus/extlib/tests/test_cusolver_lu_solver.py::test_lu_solver_4 - ModuleNotFoundError: No module named 'theseus.extlib.cusolver_lu_solver'
FAILED theseus/extlib/tests/test_cusolver_lu_solver.py::test_lu_solver_5 - ModuleNotFoundError: No module named 'theseus.extlib.cusolver_lu_solver'
FAILED theseus/extlib/tests/test_cusolver_lu_solver.py::test_lu_solver_6 - ModuleNotFoundError: No module named 'theseus.extlib.cusolver_lu_solver'
FAILED theseus/extlib/tests/test_mat_mult.py::test_mat_mult_1 - ModuleNotFoundError: No module named 'theseus.extlib.mat_mult'
FAILED theseus/extlib/tests/test_mat_mult.py::test_mat_mult_2 - ModuleNotFoundError: No module named 'theseus.extlib.mat_mult'
FAILED theseus/extlib/tests/test_mat_mult.py::test_mat_mult_3 - ModuleNotFoundError: No module named 'theseus.extlib.mat_mult'
FAILED theseus/extlib/tests/test_mat_mult.py::test_mat_mult_4 - ModuleNotFoundError: No module named 'theseus.extlib.mat_mult'
FAILED theseus/extlib/tests/test_mat_mult.py::test_mat_mult_5 - ModuleNotFoundError: No module named 'theseus.extlib.mat_mult'
FAILED theseus/extlib/tests/test_mat_mult.py::test_mat_mult_6 - ModuleNotFoundError: No module named 'theseus.extlib.mat_mult'
FAILED theseus/optimizer/autograd/tests/test_lu_cuda_sparse_backward.py::test_sparse_backward_step - RuntimeError: Theseus C++/Cuda extension cannot be loaded
FAILED theseus/optimizer/linear/tests/test_lu_cuda_sparse_solver.py::test_sparse_solver - RuntimeError: Theseus C++/Cuda extension cannot be loaded
FAILED theseus/optimizer/linear/tests/test_lu_cuda_sparse_solver.py::test_sparse_solver_multistep_gradient - RuntimeError: Theseus C++/Cuda extension cannot be loaded
FAILED theseus/optimizer/linear/tests/test_lu_cuda_sparse_solver.py::test_sparse_solver_multistep_exception - RuntimeError: Theseus C++/Cuda extension cannot be loaded
FAILED theseus/optimizer/nonlinear/tests/test_levenberg_marquardt.py::test_ellipsoidal_damping_compatibility_cuda - RuntimeError: Theseus C++/Cuda extension cannot be loaded

Seems like cusolver_lu_solver and mat_mult are not correctly installed. Besides, there are issues when loading the Theseus C++/Cuda extension. Do you have any ideas on how to fix those?

0 replies

fmagera · 2022-08-11T16:56:38Z

fmagera
Aug 11, 2022
Author

Hi @MickShen7558,

It's not my scripts, I'm not the one to thank :)
Yes I had the same issue, I was running the tests from my copy of theseus. I guess it's just an import confusion problem so I worked around it by copying the installed .so libraries in the local repo to run the tests, and they all passed.

14 replies

luisenp Aug 18, 2022
Collaborator

Heads up, in the newest version usage is ./build_scripts/build_wheel.sh <ROOT_DIR> <THESEUS_TAG> <CUDA_VERSION>. I have tested it with cuda 10.2, 11.3, and 11.6. You can also pass "cpu" instead of the number to compile w/o the C++ CUDA extensions.

MickShen7558 Aug 21, 2022

Hi @luisenp, many many thanks to you! Now theseus is successfully installed on my docker with CUDA supports. I cannot thank you enough for your timely help!

For the record to the others facing the same issue, I installed theseus with the following steps:

./build_scripts/build_wheel.sh ~/theseus_build 0.1.0 11.3
# copy the .whl into the container and enter it
conda create --name theseus_whl python=3.9
source activate theseus_whl
export CUDA_HOME=/usr/local/cuda-11.3
pip install torch --extra-index-url https://download.pytorch.org/whl/cu113
conda install -c conda-forge suitesparse
pip install ...../theseus_ai-0.1.0+cu113-cp39-cp39-manylinux_2_17_x86_64.whl

saxenam06 Oct 5, 2022

Hello, Many Thanks for the detailed steps. I am able to build the wheel files for the only CPU case and successfully install Theseus following the above steps. However, I am facing problem in building wheel files for CUDA 11.3 using ./build_scripts/build_wheel.sh ~/theseus_build 0.1.0 11.3.
Here are the details.
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_May__3_19:15:13_PDT_2021
Cuda compilation tools, release 11.3, V11.3.109
Build cuda_11.3.r11.3/compiler.29920130_0

I am in a base conda environement with python version 3.9 when I run ./build_scripts/build_wheel.sh ~/theseus_build 0.1.0 11.3.
I get the error in step 16 below. The Docker is verified to run the hello world. Request your kind view on this. Thanks!

Step 12/16 : RUN git clone https://github.com/facebookresearch/theseus.git
---> Running in 1447a388340b
Cloning into 'theseus'...
Removing intermediate container 1447a388340b
---> c376c7768bae
Step 13/16 : WORKDIR theseus
---> Running in 2f328539a7d5
Removing intermediate container 2f328539a7d5
---> 9b146f337a47
Step 14/16 : RUN git fetch --all --tags
---> Running in 87407da2364a
Fetching origin
Removing intermediate container 87407da2364a
---> bf82b09e8300
Step 15/16 : RUN git checkout tags/0.1.0 -b tmp_build
---> Running in 37b0a4efafb2
Switched to a new branch 'tmp_build'
Removing intermediate container 37b0a4efafb2
---> c3303c01ebac
Step 16/16 : CMD python3 -m build --no-isolation
---> Running in 02937d5d2946
Removing intermediate container 02937d5d2946
---> 0df41dc2fcbc
Successfully built 0df41dc2fcbc
Successfully tagged theseus_3.9_img:latest
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown.
ERRO[0001] error waiting for container: context canceled
Error: No such container:path: theseus_3.9:theseus/dist/theseus-ai-0.1.0.tar.gz
Error: No such container:path: theseus_3.9:theseus/dist/theseus_ai-0.1.0-cp39-cp39-linux_x86_64.whl
theseus_3.9
Untagged: theseus_3.9_img:latest
Deleted: sha256:0df41dc2fcbcfeb52eb775b5aa550e261d51ccde2c5a18f1f1aacc8cc89b6f6b
Deleted: sha256:c3303c01ebacf4749a15bdf411ebff877411d046e41e2d0006fd9fb060d76352
Deleted: sha256:208a85281b6641c02a22b9075efc61beca6438f39a31a928f4c106d7e0d1f417
Deleted: sha256:bf82b09e830068fd04210c6fd0274d11941281d05a85d023b21a01029324a647
Deleted: sha256:9a61eb623370f307f42c69a8deda69ba3a0454e223cbf8cd13f499d0e216b59b
Deleted: sha256:9b146f337a47b779af8c88d94617873be6408005fa9ef8745aa42166a2e479cf
Deleted: sha256:c376c7768baed70e9bf9f087fbe1d9da2b1718092d9b4628109f9c53d2b781c7
Deleted: sha256:dd160790436fb09b05d351c30c2e4b82a2101cb85aa618e6d1c730207ff10605
Deleted: sha256:20f7e817336d40fe05125c6e7a58d5b383dcf2e3dc11131f79b64340686ceff3
Deleted: sha256:6ae34abfa31410b45c987f16a5aae29cb2a112edfafe1632d89d7ee5da2424bf
Deleted: sha256:bfa8b6779dd12d4325005b31b215312e1ae3cad641659d1e17c9900d05759e7c
Deleted: sha256:e3e80867fdc25150d336b552a835aeac6d5c5c0f18e6ba1a5447417dde7dfc75
Deleted: sha256:5c9d00b2292e53d72fc2643c1f9a6c2ebf6d9a0224ecb4a29fab398776790c02
Deleted: sha256:cd3111afe2e5822be75f6faedf249ed7b5b6941751c522745840716671cd2b47
Deleted: sha256:fde92e4b46363c98474ca5d25c7527e2624d623e09db88c2cf88346a5abce6c5
Deleted: sha256:fcdd4f6b7229491388e49f015902fffcbc0f467bcf00732dae807d5826eaf7ad
Deleted: sha256:44ab229b9c4654ec14b52dee092c9b914b607f925efe28db047b163eaa572f09
Deleted: sha256:fe20bff5b246a44dc8a445945ba9ea9a792959abc393368528eb7c0a79f8b326
Deleted: sha256:32fa92a6022c4b11f30284a5933d5b6994fa6859eac730de143a75e7af2f8369
Deleted: sha256:816d77e00b3267bfde5b0bc75f7fb1d76a52093ae1c4996947be76e666825daf
Deleted: sha256:7b7909b8e6bdcabc24caf2e73e95cf366730e3f3a6f7318768351266c129c882
Deleted: sha256:192a15313150d24014b07517f202ee6b98a356adfe32e5960ae69cee479b5bfe
Deleted: sha256:ab7120c8c795617be2234324dfee95157ee6392dd9301aaf5b9950cce225aaea
Deleted: sha256:9914800af87a12e684d5a9bebaf2a40ec4b6ce5f2fc13455dde6e48ad12ac721

luisenp Oct 5, 2022
Collaborator

Hi @saxenam06, I'm not familiar with this error, but it seems to be related with some issue with nvidia-docker. Coincidentally, in #304 we are introducing a way to compile the wheel even when CUDA is not on the machine (precisely because of a case where nvidia-docker was not working). Could you try with the version of the build script in that PR? It's still preliminary so you may need to tweak it a bit. Or, if you are willing to wait a bit, we are planning to merge this in the next week or so.

PS: BTW, the latest version is 0.1.1 so you can use that tag now when compiling the wheel.

saxenam06 Oct 5, 2022

Many thanks for your quick response @luisenp. I will look into the PR. This is very interesting and helpful as if I understand correctly then we can use this to compile the wheel from a system without CUDA and then use it to install Theseus elsewhere.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Installation in Docker : error in compilation of extlib/mat_mul.cu #269

{{title}}

Replies: 5 comments 15 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Installation in Docker : error in compilation of extlib/mat_mul.cu #269

fmagera Aug 4, 2022

Replies: 5 comments · 15 replies

luisenp Aug 4, 2022 Collaborator

mhmukadam Aug 22, 2022

fanyangr Aug 4, 2022

fmagera Aug 5, 2022 Author

MickShen7558 Aug 11, 2022

fmagera Aug 11, 2022 Author

luisenp Aug 18, 2022 Collaborator

MickShen7558 Aug 21, 2022

saxenam06 Oct 5, 2022

luisenp Oct 5, 2022 Collaborator

saxenam06 Oct 5, 2022

fmagera
Aug 4, 2022

Replies: 5 comments 15 replies

luisenp
Aug 4, 2022
Collaborator

fanyangr
Aug 4, 2022

fmagera
Aug 5, 2022
Author

MickShen7558
Aug 11, 2022

fmagera
Aug 11, 2022
Author

luisenp Aug 18, 2022
Collaborator

luisenp Oct 5, 2022
Collaborator