PUBLISH.md

Publish

Publish a new official release

Prerequisite: Have a dataClay's pypi and testpypi account with owner access.

1. Run nox to check lint and tests.

2. Push a new commit release version <VERSION> where you:

   • Update dataclay.__version__ from dataclay.__init__ to <VERSION>

3. Create and push a new tag:

   git tag -a <VERSION> -m "Release <VERSION>"
   git push origin <VERSION>

4. Follow the instructions to create a new release in GitHub.

5. Publish the dataclay:latest docker image:

   1. First, generate a Personal Access Token (PAT) on GitHub (if you don't have):

      • Go to GitHub Settings → Developer settings → Personal access tokens.
      • Select Tokens (classic) or fine-grained tokens (depending on GitHub updates).
      • Click Generate new token and configure:
        • Scopes: At least select the read:packages and write:packages.
      • Generate the token and copy it (you’ll need it for the login).

   2. Then, log in to GitHub Packages (ghcr.io) with your account name and token:

      docker login ghcr.io

   3. Build and publish the docker image. To use buildx you may need to sudo apt install qemu-user-static:

      # Set the version variable
      VERSION=<VERSION>
      
      # Create a new builder instance
      export DOCKER_BUILDKIT=1
      docker buildx create --use
      
      # Build and push Python 3.9 bookworm
      docker buildx build --platform linux/amd64,linux/arm64 \
      -t ghcr.io/bsc-dom/dataclay:$VERSION-py3.9-bookworm \
      --build-arg PYTHON_VERSION=3.9-bookworm --push .
      
      # Build and push Python 3.10 bookworm
      docker buildx build --platform linux/amd64,linux/arm64 \
      -t ghcr.io/bsc-dom/dataclay:$VERSION-py3.10-bookworm \
      -t ghcr.io/bsc-dom/dataclay:$VERSION \
      -t ghcr.io/bsc-dom/dataclay:latest \
      --build-arg PYTHON_VERSION=3.10-bookworm --push .
      
      # Build and push Python 3.11 bookworm
      docker buildx build --platform linux/amd64,linux/arm64 \
      -t ghcr.io/bsc-dom/dataclay:$VERSION-py3.11-bookworm \
      --build-arg PYTHON_VERSION=3.11-bookworm --push .
      
      # Build and push Python 3.12 bookworm
      docker buildx build --platform linux/amd64,linux/arm64 \
      -t ghcr.io/bsc-dom/dataclay:$VERSION-py3.12-bookworm \
      --build-arg PYTHON_VERSION=3.12-bookworm --push .
      
      # Build and push Python 3.13 bookworm
      docker buildx build --platform linux/amd64,linux/arm64 \
      -t ghcr.io/bsc-dom/dataclay:$VERSION-py3.13-bookworm \
      --build-arg PYTHON_VERSION=3.13-bookworm --push .

6. Publish the release distribution to PyPI:

   # Create and source new virtual environment
   python3 -m venv venv.deploy && source venv.deploy/bin/activate
   
   # Install the latest `build` and `twine`
   python3 -m pip install --upgrade build twine
   
   # Remove `dist` folder if exists
   rm -rf dist/
   
   # Build the release distribution
   python3 -m build
   
   # Publish the package to TestPyPI
   # and check that everything works fine
   python3 -m twine upload --repository testpypi dist/*
   
   # Publish the package to PyPI
   python3 -m twine upload dist/*

7. Push a new commit start version <NEW_VERSION> where you:

   • Update dataclay.__version__ from dataclay.__init__ to <NEW_VERSION>.dev
   • Update orchestration/spack to point the new version in PyPI

8. Update the active versions on ReadTheDocs, i.e. go to the versions page and activate/deactivate versions accordingly. You probably must add the newly added release, and maybe you will need to deactivate patch versions that are irrelevant.

Publish a development distribution to TestPyPI

To build and publish a development distribution:

# Remove `dist` folder if exists
rm -rf dist/

# Set NUM_TAG to 0 and increment its value 
# to publish multiple builds on the same day
NUM_TAG=0

# Build the development distribution with date tag
python3 -m build -C--global-option=egg_info -C--global-option=--tag-date -C--global-option=--tag-build=$NUM_TAG

# Publish the package to TestPyPI
python3 -m twine upload --repository testpypi dist/*

To install a development package from TestPyPI use:

python3 -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ dataclay

• --index-url tells pip to download the package form TestPyPI instead of PyPI
• --extra-index-url is used to install the package dependencies from PyPI

Build the dataclay dev image for testing

docker build -t ghcr.io/bsc-dom/dataclay:dev -f Dockerfile.dev \
--build-arg PYTHON_VERSION=3.10-bullseye .