Move setup configuration to pyproject.toml

[serge-sans-paille]

Related to scipy/scipy#19670

[rgommers]

setup.py bdist_wheel is deprecated, you probably want to remove wheel here and change the one test that it's needed for (test_setup_wheel_install) to python -m build which produces an sdist and then a wheel from the sdist.

[serge-sans-paille]

ack, I'll do that in another PR though.

[rgommers]

I don't think numpy is needed? I can build a wheel just fine with python -m build or pip install ., and I am not aware of the content of that wheel then depending on whether numpy was available in the build env.

[rgommers]

This comment still applies I think. For the rest, this PR now LGTM.

[serge-sans-paille]

Numpy is a required runtime dependency of Pythran. Isn't that what's captured by that field?

[rgommers]

No, these are the requirements to build a wheel only. That is (by default) done in an isolated venv, these packages won't be installed in the user's environment. So actually more is wrong. This list needs to be dependencies = under [project].

The requires = is only setuptools, nothing else I think.

[tylerjereddy]

When I point the cross-ref'd SciPy branch to a local checkout of this branch with

--- a/pyproject.toml
+++ b/pyproject.toml
@@ -13,7 +13,7 @@ requires = [
     "meson-python>=0.15.0,<0.16.0", # working with 0.15.x series at branch time
     "Cython>=0.29.35,!=3.0.3,<3.1.0",  # when updating version, also update check in meson.build
     "pybind11>=2.10.4,<2.12.0", # working with 2.11.x series at branch time
-    "pythran>=0.14.0,<0.15.0", # working with 0.14.x series at branch time
+    "pythran @ file:///home/treddy/github_projects/pythran", # local checkout of gh-2162

The distutils error is resolved when I try python -m pip wheel -v ., but I see another issue crop up:

  [791/1606] Compiling C++ object scipy/stats/_stats_pythran.cpython-312-x86_64-linux-gnu.so.p/meson-generated_..__stats_pythran.cpp.o
  FAILED: scipy/stats/_stats_pythran.cpython-312-x86_64-linux-gnu.so.p/meson-generated_..__stats_pythran.cpp.o
  c++ -Iscipy/stats/_stats_pythran.cpython-312-x86_64-linux-gnu.so.p -Iscipy/stats -I../scipy/stats -I../../../../../tmp/pip-build-env-108828f2/overlay/lib/python3.12/site-packages/pythran -I../../../../../tmp/pip-build-env-108828f2/overlay/lib/python3.12/site-packages/numpy/core/include -I/home/treddy/github_projects/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/python-3.12.0-xqzvew4twenwzsdf2usuartrzdghpwup/include/python3.12 -fvisibility=hidden -fvisibility-inlines-hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c++14 -O3 -fPIC -DENABLE_PYTHON_MODULE -D__PYTHRAN__=3 -DPYTHRAN_BLAS_NONE -Wno-cpp -Wno-deprecated-declarations -Wno-unused-but-set-variable -Wno-unused-function -Wno-unused-variable -Wno-int-in-bool-context -MD -MQ scipy/stats/_stats_pythran.cpython-312-x86_64-linux-gnu.so.p/meson-generated_..__stats_pythran.cpp.o -MF scipy/stats/_stats_pythran.cpython-312-x86_64-linux-gnu.so.p/meson-generated_..__stats_pythran.cpp.o.d -o scipy/stats/_stats_pythran.cpython-312-x86_64-linux-gnu.so.p/meson-generated_..__stats_pythran.cpp.o -c scipy/stats/_stats_pythran.cpp
  In file included from scipy/stats/_stats_pythran.cpp:34:
  ../../../../../tmp/pip-build-env-108828f2/overlay/lib/python3.12/site-packages/pythran/pythonic/include/numpy/ceil.hpp:8:10: fatal error: xsimd/xsimd.hpp: No such file or directory
      8 | #include <xsimd/xsimd.hpp>
        |          ^~~~~~~~~~~~~~~~~
  compilation terminated.

I believe we've been allowing xsimd to be an optional dependency on the SciPy side, could we guard the include or?

[rgommers]

I believe we've been allowing xsimd to be an optional dependency on the SciPy side, could we guard the include or?

Not quite optional - but it's either vendored within Pythran, or a transitive dependency of it. So this error should never happen, and xsimd is something we don't have to care about in SciPy (the optional thing you are thinking about was a now-resolved packaging bug in conda-forge).

[serge-sans-paille]

@rgommers : to fit in the pyproject.toml paradigm without writing extensions, the patch also changes the installation dir of the bundled xsimd and boost headers (see https://github.com/serge-sans-paille/pythran/pull/2162/files#diff-76bdcb439914b0c3b773139a2b33e7dfab727904613033afb2678af0afeae691). Maybe you should reflect that in scipy's buildsystem if you're not using pythran primitives to compile?

[rgommers]

That change will break not only SciPy, but also scikit-image and any other users of pythran.get_include as far as I can tell, since the result of that is no longer enough - it only contains one of two include dirs. If you do want to get rid of a separate third_party/ top-level directory to avoid the copy action in setup.py, it seems better to simply move them to the location they currently get copied to (so pythran/xsimd and pythran/boost)?

[tylerjereddy]

I noticed that a recent push was made to this branch, and confirmed locally that the wheel build scenario described above is now successful, at least on x86_64 Linux with Python 3.12. Let me know if anything else needs testing or what the plan will be, obviously I'm happy to test things to prevent slippage in the SciPy release cycle (though this is better than hacking our wheel builds, probably).

[serge-sans-paille]

@rgommers does that look good to you know? If so @tylerjereddy could you do a final sanity check?

[rgommers]

LGTM now!

[tylerjereddy]

The wheel build for SciPy works for me alongside this branch.

Actually, I spent a bit of time working with pythran master branch, and it seems that is "ok" as well--my original test for master was too strict I think (it was to build SciPy from source, with Pythran installed, but setuptools removed; but pip wheel . I think seems to be "ok" even with master of Pythran because I believe setup.py will still do the trick to pull setuptools in during the isolated build).

I think I got confused by one of the responses suggesting that pyproject.toml was mandatory for the transitive dependency. I'm not entirely sure that's true, but pyproject.toml is the modern way anyway I guess.

So this seems "good," I wonder about the non-wheel scenario a bit, where SciPy now only needs setuptools to be present at build time because of Pythran with 3.12, but that's maybe less important for the immediate issue.

[serge-sans-paille]

Thanks @rgommers & @tylerjereddy for the review and testing! 🙇