Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

setuptools version requirement breaking Docker-based AutoTuner builds #2888

Open
jeffng-or opened this issue Feb 28, 2025 · 1 comment
Open

setuptools version requirement breaking Docker-based AutoTuner builds #2888

jeffng-or opened this issue Feb 28, 2025 · 1 comment
Assignees
@vvbandeira @luarss
Labels
autotuner Flow autotuner

Comments

@jeffng-or
Copy link
Contributor

jeffng-or commented Feb 28, 2025 •
edited
Loading

Subject

[Build] for compilation-related issues.

Describe the bug

An update to the version requirement for Python3 setuptools is breaking Docker-based builds:

      /tmp/pip-build-env-1trg79qq/overlay/local/lib/python3.10/dist-packages/setuptools_scm/_integration/setuptools.py:31: RuntimeWarning:
      ERROR: setuptools==59.6.0 is used in combination with setuptools-scm>=8.x
      
      Your build configuration is incomplete and previously worked by accident!
      setuptools-scm requires setuptools>=61

I've tried a couple of workarounds to uninstall either python3-setuptools or python3-pip and nothing is working:

  1. pip3 install --upgrade setuptools==61.0.0 installs setuptools 61.0.0 in /usr/local/lib/python3.10/dist-packages, but leaves version 59.6.0 in /usr/lib/python3/dist-packages/setuptools. The 59.6.0 version is then used during AT requirements installation and fails
  2. Uninstalling setuptools and then re-installing pip3 ends up reinstalling setuptools 59.6.0, so back to same situation as bullet 1 above.

@vvbandeira / @luarss , any ideas?

Expected Behavior

Functioning docker build like before

Environment

commit ee66b1911acd536828c34c490e6cf62193dada3b

To Reproduce

Here's the Dockerfile that was working previously

# syntax=docker/dockerfile:1
#
# Installs ORFS from docker image 
#

FROM openroad/orfs:v3.0-2380-g20d2f7bf

# install AT required packages
RUN pip3 install -U -r /OpenROAD-flow-scripts/tools/AutoTuner/requirements.txt

# ORFS installation dir
WORKDIR /OpenROAD-flow-scripts/tools/AutoTuner/src

To reproduce:
docker build -t autotuner:1.0 -f Dockerfile .

Relevant log output

#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile.at
#1 transferring dockerfile: 338B done
#1 DONE 0.0s

#2 resolve image config for docker-image://docker.io/docker/dockerfile:1
#2 DONE 0.2s

#1 [internal] load build definition from Dockerfile.at
#1 transferring dockerfile: 338B done
#1 DONE 0.0s

#3 docker-image://docker.io/docker/dockerfile:1@sha256:93bfd3b68c109427185cd78b4779fc82b484b0b7618e36d0f104d4d801e66d25
#3 CACHED

#4 [internal] load build definition from Dockerfile.at
#4 DONE 0.0s

#5 [internal] load metadata for docker.io/openroad/orfs:v3.0-2380-g20d2f7bf
#5 DONE 0.1s

#6 [internal] load .dockerignore
#6 transferring context: 2B done
#6 DONE 0.0s

#7 [1/3] FROM docker.io/openroad/orfs:v3.0-2380-g20d2f7bf@sha256:c47f99593497fa0757c3a0adfd5bb36728099421dcff21a98bdfba21d776eeb9
#7 CACHED

#8 [2/3] RUN pip3 install -U -r /OpenROAD-flow-scripts/tools/AutoTuner/requirements.txt
#8 0.774 Collecting ray[default,tune]==2.9.3
#8 0.873   Downloading ray-2.9.3-cp310-cp310-manylinux2014_x86_64.whl (64.9 MB)
#8 2.030      ???????????????????????????????????????? 64.9/64.9 MB 38.4 MB/s eta 0:00:00
#8 2.160 Collecting ax-platform<=0.3.7,>=0.3.3
#8 2.174   Downloading ax_platform-0.3.7-py3-none-any.whl (1.2 MB)
#8 2.197      ???????????????????????????????????????? 1.2/1.2 MB 58.6 MB/s eta 0:00:00
#8 2.242 Collecting hyperopt==0.2.7
#8 2.256   Downloading hyperopt-0.2.7-py2.py3-none-any.whl (1.6 MB)
#8 2.269      ???????????????????????????????????????? 1.6/1.6 MB 131.2 MB/s eta 0:00:00
#8 2.309 Collecting optuna==3.6.0
#8 2.323   Downloading optuna-3.6.0-py3-none-any.whl (379 kB)
#8 2.329      ?????????????????????????????????????? 379.9/379.9 KB 86.7 MB/s eta 0:00:00
#8 2.565 Collecting pandas<=2.2.1,>=2.0
#8 2.578   Downloading pandas-2.2.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.0 MB)
#8 2.667      ??????????????????????????????????????? 13.0/13.0 MB 135.0 MB/s eta 0:00:00
#8 2.715 Collecting bayesian-optimization==1.4.0
#8 2.728   Downloading bayesian_optimization-1.4.0-py3-none-any.whl (17 kB)
#8 2.761 Collecting colorama==0.4.6
#8 2.774   Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)
#8 2.817 Collecting tensorboard<=2.16.2,>=2.14.0
#8 2.835   Downloading tensorboard-2.16.2-py3-none-any.whl (5.5 MB)
#8 2.877      ???????????????????????????????????????? 5.5/5.5 MB 138.1 MB/s eta 0:00:00
#8 3.206 Collecting protobuf==3.20.3
#8 3.219   Downloading protobuf-3.20.3-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
#8 3.228      ???????????????????????????????????????? 1.1/1.1 MB 133.4 MB/s eta 0:00:00
#8 3.812 Collecting SQLAlchemy==1.4.17
#8 3.827   Downloading SQLAlchemy-1.4.17.tar.gz (7.6 MB)
#8 3.878      ???????????????????????????????????????? 7.6/7.6 MB 155.7 MB/s eta 0:00:00
#8 4.396   Preparing metadata (setup.py): started
#8 4.631   Preparing metadata (setup.py): finished with status 'done'
#8 4.689 Collecting urllib3<=1.26.15
#8 4.702   Downloading urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
#8 4.707      ?????????????????????????????????????? 140.9/140.9 KB 44.7 MB/s eta 0:00:00
#8 4.970 Collecting matplotlib==3.10.0
#8 4.985   Downloading matplotlib-3.10.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.6 MB)
#8 5.040      ???????????????????????????????????????? 8.6/8.6 MB 161.4 MB/s eta 0:00:00
#8 5.197 Collecting jsonschema
#8 5.210   Downloading jsonschema-4.23.0-py3-none-any.whl (88 kB)
#8 5.214      ???????????????????????????????????????? 88.5/88.5 KB 32.5 MB/s eta 0:00:00
#8 5.216 Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.10/dist-packages (from ray[default,tune]==2.9.3->-r /OpenROAD-flow-scripts/tools/AutoTuner/requirements.txt (line 1)) (8.1.8)
#8 5.320 Collecting frozenlist
#8 5.333   Downloading frozenlist-1.5.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (241 kB)
#8 5.339      ?????????????????????????????????????? 241.9/241.9 KB 65.2 MB/s eta 0:00:00
#8 5.358 Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from ray[default,tune]==2.9.3->-r /OpenROAD-flow-scripts/tools/AutoTuner/requirements.txt (line 1)) (1.1.0)
#8 5.358 Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from ray[default,tune]==2.9.3->-r /OpenROAD-flow-scripts/tools/AutoTuner/requirements.txt (line 1)) (6.0.2)
#8 5.381 Collecting aiosignal
#8 5.394   Downloading aiosignal-1.3.2-py2.py3-none-any.whl (7.6 kB)
#8 5.435 Collecting filelock
#8 5.449   Downloading filelock-3.17.0-py3-none-any.whl (16 kB)
#8 5.487 Collecting packaging
#8 5.500   Downloading packaging-24.2-py3-none-any.whl (65 kB)
#8 5.504      ???????????????????????????????????????? 65.5/65.5 KB 25.6 MB/s eta 0:00:00
#8 5.505 Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from ray[default,tune]==2.9.3->-r /OpenROAD-flow-scripts/tools/AutoTuner/requirements.txt (line 1)) (2.32.3)
#8 5.532 Collecting tensorboardX>=1.9
#8 5.546   Downloading tensorboardX-2.6.2.2-py2.py3-none-any.whl (101 kB)
#8 5.550      ?????????????????????????????????????? 101.7/101.7 KB 37.0 MB/s eta 0:00:00
#8 5.616 Collecting fsspec
#8 5.629   Downloading fsspec-2025.2.0-py3-none-any.whl (184 kB)
#8 5.634      ?????????????????????????????????????? 184.5/184.5 KB 55.4 MB/s eta 0:00:00
#8 5.802 Collecting pyarrow>=6.0.1
#8 5.815   Downloading pyarrow-19.0.1-cp310-cp310-manylinux_2_28_x86_64.whl (42.1 MB)
#8 6.127      ???????????????????????????????????????? 42.1/42.1 MB 59.0 MB/s eta 0:00:00
#8 6.175 Requirement already satisfied: pydantic!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,<3 in /usr/local/lib/python3.10/dist-packages (from ray[default,tune]==2.9.3->-r /OpenROAD-flow-scripts/tools/AutoTuner/requirements.txt (line 1)) (2.10.6)
#8 6.176 Requirement already satisfied: grpcio>=1.42.0 in /usr/local/lib/python3.10/dist-packages (from ray[default,tune]==2.9.3->-r /OpenROAD-flow-scripts/tools/AutoTuner/requirements.txt (line 1)) (1.70.0)
#8 6.204 Collecting opencensus
#8 6.218   Downloading opencensus-0.11.4-py2.py3-none-any.whl (128 kB)
#8 6.222      ?????????????????????????????????????? 128.2/128.2 KB 44.8 MB/s eta 0:00:00
#8 6.327 Collecting virtualenv!=20.21.1,>=20.0.24
#8 6.341   Downloading virtualenv-20.29.2-py3-none-any.whl (4.3 MB)
#8 6.368      ???????????????????????????????????????? 4.3/4.3 MB 166.3 MB/s eta 0:00:00
#8 6.403 Collecting prometheus-client>=0.7.1
#8 6.416   Downloading prometheus_client-0.21.1-py3-none-any.whl (54 kB)
#8 6.420      ???????????????????????????????????????? 54.7/54.7 KB 19.1 MB/s eta 0:00:00
#8 6.446 Collecting aiohttp-cors
#8 6.460   Downloading aiohttp_cors-0.7.0-py3-none-any.whl (27 kB)
#8 6.485 Collecting gpustat>=1.0.0
#8 6.498   Downloading gpustat-1.1.1.tar.gz (98 kB)
#8 6.502      ???????????????????????????????????????? 98.1/98.1 KB 38.5 MB/s eta 0:00:00
#8 6.598   Installing build dependencies: started
#8 9.548   Installing build dependencies: finished with status 'done'
#8 9.552   Getting requirements to build wheel: started
#8 9.699   Getting requirements to build wheel: finished with status 'error'
#8 9.704   error: subprocess-exited-with-error
#8 9.704   
#8 9.704   × Getting requirements to build wheel did not run successfully.
#8 9.704   ? exit code: 1
#8 9.704   ??> [48 lines of output]
#8 9.704       /tmp/pip-build-env-ga3bcuqj/overlay/local/lib/python3.10/dist-packages/setuptools_scm/_integration/setuptools.py:31: RuntimeWarning:
#8 9.704       ERROR: setuptools==59.6.0 is used in combination with setuptools-scm>=8.x
#8 9.704       
#8 9.704       Your build configuration is incomplete and previously worked by accident!
#8 9.704       setuptools-scm requires setuptools>=61
#8 9.704       
#8 9.704       Suggested workaround if applicable:
#8 9.704        - migrating from the deprecated setup_requires mechanism to pep517/518
#8 9.704          and using a pyproject.toml to declare build dependencies
#8 9.704          which are reliably pre-installed before running the build tools
#8 9.704       
#8 9.704         warnings.warn(
#8 9.704       Traceback (most recent call last):
#8 9.704         File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
#8 9.704           main()
#8 9.704         File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
#8 9.704           json_out['return_val'] = hook(**hook_input['kwargs'])
#8 9.704         File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
#8 9.704           return hook(config_settings)
#8 9.704         File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 162, in get_requires_for_build_wheel
#8 9.704           return self._get_build_requires(
#8 9.704         File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 143, in _get_build_requires
#8 9.704           self.run_setup()
#8 9.704         File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 267, in run_setup
#8 9.704           super(_BuildMetaLegacyBackend,
#8 9.704         File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 158, in run_setup
#8 9.704           exec(compile(code, __file__, 'exec'), locals())
#8 9.704         File "setup.py", line 101, in <module>
#8 9.704           setup(
#8 9.704         File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 153, in setup
#8 9.704           return distutils.core.setup(**attrs)
#8 9.704         File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 109, in setup
#8 9.704           _setup_distribution = dist = klass(attrs)
#8 9.704         File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 459, in __init__
#8 9.704           _Distribution.__init__(
#8 9.704         File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 293, in __init__
#8 9.704           self.finalize_options()
#8 9.704         File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 836, in finalize_options
#8 9.704           for ep in sorted(loaded, key=by_order):
#8 9.704         File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 835, in <lambda>
#8 9.704           loaded = map(lambda e: e.load(), filtered)
#8 9.704         File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2464, in load
#8 9.704           self.require(*args, **kwargs)
#8 9.704         File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2487, in require
#8 9.704           items = working_set.resolve(reqs, env, installer, extras=self.extras)
#8 9.704         File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 782, in resolve
#8 9.704           raise VersionConflict(dist, req).with_context(dependent_req)
#8 9.704       pkg_resources.VersionConflict: (setuptools 59.6.0 (/usr/lib/python3/dist-packages), Requirement.parse('setuptools>=61'))
#8 9.704       [end of output]
#8 9.704   
#8 9.704   note: This error originates from a subprocess, and is likely not a problem with pip.
#8 9.706 error: subprocess-exited-with-error
#8 9.706 
#8 9.706 × Getting requirements to build wheel did not run successfully.
#8 9.706 ? exit code: 1
#8 9.706 ??> See above for output.
#8 9.706 
#8 9.706 note: This error originates from a subprocess, and is likely not a problem with pip.
#8 ERROR: process "/bin/sh -c pip3 install -U -r /OpenROAD-flow-scripts/tools/AutoTuner/requirements.txt" did not complete successfully: exit code: 1
------
 > [2/3] RUN pip3 install -U -r /OpenROAD-flow-scripts/tools/AutoTuner/requirements.txt:
9.704       [end of output]
9.704   
9.704   note: This error originates from a subprocess, and is likely not a problem with pip.
9.706 error: subprocess-exited-with-error
9.706 
9.706 × Getting requirements to build wheel did not run successfully.
9.706 ? exit code: 1
9.706 ??> See above for output.
9.706 
9.706 note: This error originates from a subprocess, and is likely not a problem with pip.
------
Dockerfile.at:9
--------------------
   7 |     
   8 |     # install AT required packages
   9 | >>> RUN pip3 install -U -r /OpenROAD-flow-scripts/tools/AutoTuner/requirements.txt
  10 |     
  11 |     # ORFS installation dir
--------------------
ERROR: failed to solve: process "/bin/sh -c pip3 install -U -r /OpenROAD-flow-scripts/tools/AutoTuner/requirements.txt" did not complete successfully: exit code: 1

Screenshots

No response

Additional Context

No response
@jeffng-or jeffng-or added the autotuner Flow autotuner label Feb 28, 2025
@vvbandeira
Copy link
Member

Upgrading pip before installing the requirements allows the image building to finish.
You can use:

RUN pip3 install --upgrade pip
RUN pip3 install -U -r /OpenROAD-flow-scripts/tools/AutoTuner/requirements.txt

OR

RUN pip3 install --upgrade pip \
      && pip3 install -U -r /OpenROAD-flow-scripts/tools/AutoTuner/requirements.txt

However I see some error messages that are worth looking into:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
grpcio-status 1.70.0 requires protobuf<6.0dev,>=5.26.1, but you have protobuf 3.20.3 which is incompatible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vvbandeira vvbandeira

@luarss luarss

Labels
autotuner Flow autotuner
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vvbandeira @luarss @jeffng-or