Migrating to pyproject.toml + cibuildwheel (#256)

* introduce pyproject.toml
* introduce cibuildwheel
* require cython during build (no silent fails anymore)
* introduce PyPI test

.github/workflows/python-build-wheels.yml

@@ -0,0 +1,88 @@
+name: Build wheels
+
+on: [workflow_dispatch]
+
+jobs:
+  make_sdist:
+    name: Make SDist
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        fetch-depth: 0  # Optional, use if you use setuptools_scm
+        submodules: true  # Optional, use if you have submodules
+
+    - name: Install dependencies
+      run: python -m pip install cython numpy pysam
+
+    - name: Build SDist
+      run: pipx run build --sdist
+
+    - uses: actions/upload-artifact@v4
+      with:
+        name: cibw-sdist
+        path: dist/*.tar.gz
+
+  build_wheels:
+    name: Build wheels on ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        # macos-13 is an intel runner, macos-14 is apple silicon
+        os: [ubuntu-latest]
+        #, windows-latest, macos-13, macos-14]
+        python-version: [ "3.11" ] # "3.7", "3.8", "3.9", "3.10",
+
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      # - name: Build wheels
+      #   uses: pypa/cibuildwheel@v2.21.0
+      #   # uses: pypa/cibuildwheel@v2.17.0
+      #   # env:
+      #   #   CIBW_SOME_OPTION: value
+      #   #    ...
+      #   # with:
+      #   #   package-dir: .
+      #   #   output-dir: wheelhouse
+      #   #   config-file: "{package}/pyproject.toml"
+
+      - name: Install cibuildwheel
+        run: python -m pip install cibuildwheel==2.22.0
+
+      - name: Build wheels
+        run: python -m cibuildwheel --output-dir dist
+        # to supply options, put them in 'env', like:
+        env:
+          #CIBW_BUILD_FRONTEND: "pip; args: --no-build-isolation"
+          CIBW_BUILD_FRONTEND: "build; args: --no-isolation"
+          CIBW_BEFORE_ALL: "yum install bzip2-devel xz-devel -y;"
+
+          # we have to recompile pysam so that repairwheel can later find various libraries (libssl, libnghttp2, etc)
+          #CIBW_BEFORE_ALL: "yum install bzip2-devel xz-devel openssl-devel openldap-devel krb5-devel libssh-devel libnghttp2-devel -y;"
+          CIBW_BEFORE_BUILD: "python -m pip install setuptools cython numpy pysam --no-binary pysam" 
+
+          # skip building 32-bit wheels (i686)
+          CIBW_ARCHS_LINUX: "auto64"
+
+          # we could use 2_28 to download pysam's wheel instead of compiling it ;
+          # HOWEVER THAT DIDN'T WORK BECAUSE PYSAM DEPENDS ON LIBSSL, LIBNGHTTP2, ETC, WHICH CANNOT BE FOUND
+          # SO WE ARE BACK TO COMPILING PYSAM'S WHEEL (no-binary pysam)
+          # CIBW_MANYLINUX_X86_64_IMAGE: "manylinux_2_28"
+
+          ## skip building pypy and musllinux
+          CIBW_SKIP: pp* *musllinux*
+
+          #CIBW_REPAIR_WHEEL_COMMAND: 'auditwheel -v repair -w {dest_dir} {wheel}'
+
+          #PIP_NO_CACHE_DIR: "false"
+          #PIP_NO_BUILD_ISOLATION: "false"
+          #PIP_NO_BINARY: "pysam"
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: cibw-wheels-${{ matrix.os }}-${{ strategy.job-index }}
+          path: ./dist/*.whl

.github/workflows/python-publish-test.yml

@@ -1,32 +1,51 @@
-
-# This workflows will upload a Python Package using Twine when a release is created
-# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries
-
 name: Publish Python Package to Test PyPI
 
 on:
-  release:
-    types: [prereleased]
+  # release:
+  #   types: [published]
+  workflow_dispatch:
 
 jobs:
-  deploy:
-
+  publish_all:
+    name: Publish wheels and sdist to Test PyPI
+
+    # if: github.event_name == 'release' && github.event.action == 'published'
+
+    environment: testpypi
+    permissions:
+      id-token: write
     runs-on: ubuntu-latest
-
     steps:
-    - uses: actions/checkout@v2
-    - name: Set up Python
-      uses: actions/setup-python@v2
-      with:
-        python-version: '3.10'
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip
-        pip install setuptools wheel twine cython numpy pysam
-    - name: Build and publish
-      env:
-        TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
-        TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
-      run: |
-        python setup.py sdist
-        twine upload --repository-url https://test.pypi.org/legacy/ dist/*
+      - uses: dawidd6/action-download-artifact@v7
+        with:
+          # Required, if the repo is private a Personal Access Token with `repo` scope is needed or GitHub token in a job where the permissions `action` scope set to `read`
+          #github_token: ${{secrets.GITHUB_TOKEN}}
+          # Optional, workflow file name or ID
+          # If not specified, will be inferred from run_id (if run_id is specified), or will be the current workflow
+          workflow: python-build-wheels.yml
+          # Optional, the status or conclusion of a completed workflow to search for
+          # Can be one of a workflow conclusion:
+          #   "failure", "success", "neutral", "cancelled", "skipped", "timed_out", "action_required"
+          # Or a workflow status:
+          #   "completed", "in_progress", "queued"
+          # Use the empty string ("") to ignore status or conclusion in the search
+          workflow_conclusion: success
+
+      - name: Publish distribution 📦 to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          packages-dir: cibw-sdist
+          repository-url: https://test.pypi.org/legacy/
+
+      - name: Publish distribution 📦 to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          packages-dir: cibw-wheels-ubuntu-latest-0
+          repository-url: https://test.pypi.org/legacy/
+
+
+
+
+
+
+

.github/workflows/python-publish.yml

@@ -1,68 +1,48 @@
-# # This workflow will upload a Python Package using Twine when a release is created
-# # For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries
-
-# name: Upload Python Package
-
-# on:
-#   release:
-#     types: [created]
-
-# jobs:
-#   deploy:
-
-#     runs-on: ubuntu-latest
-
-#     steps:
-#     - uses: actions/checkout@v2
-#     - name: Set up Python
-#       uses: actions/setup-python@v2
-#       with:
-#         python-version: '3.10'
-#     - name: Install dependencies
-#       run: |
-#         python -m pip install --upgrade pip
-#         pip install setuptools wheel twine cython pysam numpy
-#     - name: Build and publish
-#       env:
-#         TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
-#         TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
-#       run: |
-#         python setup.py sdist
-#         twine upload dist/*
-
-
 name: Publish Python Package to PyPI
 
 on:
-  release:
-    types: [published]
+  # release:
+  #   types: [published]
   workflow_dispatch:
 
 jobs:
-  Publish:
-    # prevents this action from running on forks
-    if: github.repository == 'open2c/pairtools'
-
-    runs-on: ubuntu-latest
+  publish_all:
+    name: Publish wheels and sdist to  PyPI
+
+    # if: github.event_name == 'release' && github.event.action == 'published'
+
+    environment: pypi
     permissions:
       id-token: write
-
+    runs-on: ubuntu-latest
     steps:
-      - name: Checkout
-        uses: actions/checkout@v4
+      - uses: dawidd6/action-download-artifact@v7
+        with:
+          # Required, if the repo is private a Personal Access Token with `repo` scope is needed or GitHub token in a job where the permissions `action` scope set to `read`
+          #github_token: ${{secrets.GITHUB_TOKEN}}
+          # Optional, workflow file name or ID
+          # If not specified, will be inferred from run_id (if run_id is specified), or will be the current workflow
+          workflow: python-build-wheels.yml
+          # Optional, the status or conclusion of a completed workflow to search for
+          # Can be one of a workflow conclusion:
+          #   "failure", "success", "neutral", "cancelled", "skipped", "timed_out", "action_required"
+          # Or a workflow status:
+          #   "completed", "in_progress", "queued"
+          # Use the empty string ("") to ignore status or conclusion in the search
+          workflow_conclusion: success
 
-      - name: Setup
-        uses: actions/setup-python@v5
+      - name: Publish distribution 📦 to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
         with:
-          python-version: "3.x"
-          cache: 'pip' # caching pip dependencies
-
-      - name: Install, build
-        run: |
-          python -m pip install --upgrade pip wheel setuptools build
-          python -m pip install --upgrade cython pysam
-          python -m pip install -r requirements-dev.txt
-          python -m build
+          packages-dir: cibw-sdist
 
       - name: Publish distribution 📦 to PyPI
         uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          packages-dir: cibw-wheels-ubuntu-latest-0
+
+
+
+
+
+

.github/workflows/python-package.yml → .github/workflows/python-test.yml

@@ -1,8 +1,7 @@
 # This workflow will install Python dependencies, run tests and lint with a variety of Python versions
 # For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
 
-name: Python package
-
+name: Test build, lint and test
 on:
   push:
     branches: [ master ]
@@ -16,7 +15,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
+        python-version: ["3.9", "3.10", "3.11", "3.12"]
 
     steps:
     - uses: actions/checkout@v2
@@ -26,10 +25,9 @@ jobs:
         python-version: ${{ matrix.python-version }}
     - name: Install dependencies
       run: |
-        python -m pip install --upgrade pip wheel setuptools
-        pip install numpy cython pysam
-        pip install -r requirements-dev.txt
-        pip install -e .
+        python -m pip install --upgrade pip wheel setuptools build 
+        pip install cython pysam numpy
+        pip install -e .[test] --no-build-isolation -v -v 
     - name: Lint with flake8
       run: |
         # stop the build if there are Python syntax errors or undefined names
@@ -40,3 +38,6 @@ jobs:
       run: |
         pip install pytest
         pytest
+
+
+        

README.md

@@ -48,23 +48,33 @@ Requirements:
 
 - Python 3.x
 - Python packages `cython`, `pysam`, `bioframe`, `pyyaml`, `numpy`, `scipy`, `pandas` and `click`.
-- Command-line utilities `sort` (the Unix version), `bgzip` (shipped with `samtools`)  and `samtools`. If available, `pairtools` can compress outputs with `pbgzip` and `lz4`.
+- Command-line utilities `sort` (the Unix version), `samtools` and `bgzip` (shipped with `samtools`). If available, `pairtools` can compress outputs with `pbgzip` and `lz4`.
 
-For the full list of recommended versions, see [requirements in the the GitHub repo](https://github.com/open2c/pairtools/blob/detect_mutations/requirements.txt). 
+For the full list of recommended versions, see [the requirements section in the pyproject.toml](https://github.com/open2c/pairtools/blob/main/pyproject.toml). 
 
-We highly recommend using the `conda` package manager to install `pairtools` together with all its dependencies. To get it, you can either install the full [Anaconda](https://www.continuum.io/downloads) Python distribution or just the standalone [conda](http://conda.pydata.org/miniconda.html) package manager.
+There are three options for installing pairtools:
 
-With `conda`, you can install `pairtools` and all of its dependencies from the [bioconda](https://bioconda.github.io/index.html) channel.
+1. We highly recommend using the `conda` package manager to install `pairtools` together with all its dependencies. To get it, you can either install the full [Anaconda](https://www.continuum.io/downloads) Python distribution or just the standalone [conda](http://conda.pydata.org/miniconda.html) package manager.
+
+With `conda`, you can install `pairtools` and all of its dependencies from the [bioconda](https://bioconda.github.io/index.html) channel:
 ```sh
 $ conda install -c conda-forge -c bioconda pairtools
 ```
 
-Alternatively, install non-Python dependencies and `pairtools` with Python-only dependencies from PyPI using pip:
+2. Alternatively, install non-Python dependencies (`sort`, `samtools`, `bgzip`, `pbgzip` and `lz4`) separately and download `pairtools` with Python dependencies from PyPI using pip:
 ```sh
-$ pip install numpy pysam cython
 $ pip install pairtools
 ```
 
+3. Finally, when the two options above don't work or when you want to modify `pairtools`, build `pairtools` from source via pip's "editable" mode:
+```sh
+$ pip install numpy cython pysam 
+$ git clone https://github.com/open2c/pairtools
+$ cd pairtools
+$ pip install -e ./ --no-build-isolation
+```
+
+
 ## Quick example
 
 Setup a new test folder and download a small Hi-C dataset mapped to sacCer3 genome:

doc/installation.rst

@@ -50,11 +50,17 @@ Then, you can compile and install `pairtools` in
 `the development mode <https://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode>`_, 
 which installs the package without moving it to a system folder and thus allows
 immediate live-testing any changes in the python code. Please, make sure that you 
-have `cython` installed!
+have `cython` and `pysam` installed!
 
 .. code-block:: bash
 
+    $ pip install cython pysam numpy
     $ cd pairtools 
-    $ pip install -e ./
-
-
+    $ pip install -e ./ --no-build-isolation
+
+A few notes on the installation:
+  - `pairtools` have to use `--no-build-isolation`, because it extends `pysam` via Cython and 
+    re-compiles it during the build process. When build isolation is enabled, these `pysam` objects 
+    get lost after the build.
+  - Because of the `--no-build-isolation` flag, build does not install build-requires, so you have to 
+    install `cython`, `pysam` and `numpy` manually before the build.

pairtools/lib/scaling.py

@@ -48,12 +48,13 @@ def _to_float(arr_or_scalar):
 
 
 def assign_regs(chroms, pos, regs):
-    gb_regs = regs.sort_values(["chrom", "start", "end"]).groupby(["chrom"])
+    gb_regs = regs.sort_values(["chrom", "start", "end"]).groupby("chrom")
 
     regs_dict = {
         chrom.encode(): regs_per_chrom[["start", "end"]]
-        .values.flatten()
-        .astype(np.int64)
+            .values
+            .flatten()
+            .astype(np.int64)
         for chrom, regs_per_chrom in gb_regs
     }