Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update docs on using CI artifacts across PRs #577

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
+121 −28
Open

update docs on using CI artifacts across PRs #577

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2f02331
update docs on using CI artifacts across PRs
jameslamb Feb 14, 2025
3e682da
clarify comment
jameslamb Feb 14, 2025
44c1455
Apply suggestions from code review
jameslamb Feb 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
149 changes: 121 additions & 28 deletions resources/github-actions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,51 +176,144 @@ The example code below demonstrates building and testing with conda packages fro
Replace the pull request numbers and library names as needed.
Remember that changes to use CI artifacts should be _temporary_ and should be reverted prior to merging any required changes in that PR.

**Example 1:** Building `libcuml` (C++) using `librmm`, `libraft`, `libcumlprims_mg` PR artifacts.
**Example 1:** Building `libcuml` (C++) using `librmm` and `libraft` PR artifacts.

```sh
# ci/build_cpp.sh
Add a new file called `ci/use_conda_packages_from_prs.sh`.

LIBRMM_CHANNEL=$(rapids-get-pr-conda-artifact rmm 1095 cpp)
```shell
# ci/use_conda_packages_from_prs.sh

# download CI artifacts
LIBRAFT_CHANNEL=$(rapids-get-pr-conda-artifact raft 1388 cpp)
LIBCUMLPRIMS_CHANNEL=$(rapids-get-pr-conda-artifact cumlprims_mg 129 cpp)
LIBRMM_CHANNEL=$(rapids-get-pr-conda-artifact rmm 1095 cpp)

# make sure they can be found locally
conda config --system --add channels "${LIBRAFT_CHANNEL}"
conda config --system --add channels "${LIBRMM_CHANNEL}"
Comment on lines +190 to +192
Copy link
Contributor

@bdice bdice Feb 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: this will change with rattler-build, which doesn't use the system conda config afaik. Should we preface this with that disclaimer?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd written this before I knew that: rapidsai/gha-tools#146 (comment)

I don't think we should add a disclaimer like that. A note of the form "this won't work for builds using rattler-build" isn't something that folks reading this could do much with, by themselves.

It'd be better to add "update the docs on using CI artifacts" as part of the work for rapidsai/build-planning#47

```

# Build library packages with the CI artifact channels providing the updated dependencies
Then copy the following into every script in the `ci/` directory that is doing `conda` installs.

rapids-mamba-retry mambabuild \
--channel "${LIBRMM_CHANNEL}" \
--channel "${LIBRAFT_CHANNEL}" \
--channel "${LIBCUMLPRIMS_CHANNEL}" \
conda/recipes/libcuml
```shell
source ./ci/use_conda_packages_from_prs.sh
```

**Example 2:** Testing `cudf` (Python) using `librmm`, `rmm`, and `libkvikio` PR artifacts.

```sh
# ci/test_python_common.sh
It's important to include all of the recursive dependencies.
So, for example, Python testing jobs that use the `rmm` Python package also need the `librmm` C++ package.

```shell
# ci/use_conda_packages_from_prs.sh

# download CI artifacts
LIBKVIKIO_CHANNEL=$(rapids-get-pr-conda-artifact kvikio 224 cpp)
LIBRMM_CHANNEL=$(rapids-get-pr-conda-artifact rmm 1223 cpp)
RMM_CHANNEL=$(rapids-get-pr-conda-artifact rmm 1223 python)
LIBKVIKIO_CHANNEL=$(rapids-get-pr-conda-artifact kvikio 224 cpp)

# Install library packages with the CI artifact channels providing the updated dependencies for testing
# make sure they can be found locally
conda config --system --add channels "${LIBKVIKIO_CHANNEL}"
conda config --system --add channels "${LIBRMM_CHANNEL}"
conda config --system --add channels "${RMM_CHANNEL}"
```

Then copy the following into every script in the `ci/` directory that is doing `conda` installs.

rapids-mamba-retry install \
--channel "${CPP_CHANNEL}" \
--channel "${PYTHON_CHANNEL}" \
--channel "${LIBRMM_CHANNEL}" \
--channel "${LIBKVIKIO_CHANNEL}" \
--channel "${RMM_CHANNEL}" \
cudf libcudf
```shell
source ./ci/use_conda_packages_from_prs.sh
```

Note that the custom channel for PR artifacts is needed in the build scripts _and_ the test scripts, for C++ _and_ Python.
If building/testing a Python package that depends on a C++ library, it is necessary to use PR artifacts from that C++ library and not just Python (e.g. if testing `rmm` artifacts, you must use the corresponding `librmm` CI artifacts as well as `rmm`).
In some repos, the `test_python.sh` is quite complicated with multiple calls to conda/mamba.
We recommend that the Python and C++ artifact channels should be added to every call of `rapids-mamba-retry` / `rapids-conda-retry` "just in case."
**Note:** By default `rapids-get-pr-conda-artifact` uses the most recent commit from the specified PR.
A commit hash from the dependent PR can be added as an optional 4th argument to pin testing to a specific commit.

## Using Wheel CI Artifacts in Other PRs

To use wheels produced by other PRs' CI:

* download the wheels at the beginning of CI jobs
* constrain `pip` to always use them

Consider the following examples.

**Example:** Building `libcuml` (C++) using `librmm` and `libraft` PR artifacts.

Add a new file called `ci/use_wheels_from_prs.sh`.

```shell
# ci/use_wheels_from_prs.sh

RAPIDS_PY_CUDA_SUFFIX=$(rapids-wheel-ctk-name-gen "${RAPIDS_CUDA_VERSION}")

# download wheels, store the directories holding them in variables
LIBRMM_WHEELHOUSE=$(
RAPIDS_PY_WHEEL_NAME="librmm_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact rmm 1678 cpp
)
LIBRAFT_WHEELHOUSE=$(
RAPIDS_PY_WHEEL_NAME="libraft_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact raft 2433 cpp
)

# write a pip constraints file saying e.g. "whenever you encounter a requirement for 'librmm-cu12', use this wheel"
cat > /tmp/constraints.txt <<EOF
librmm-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${LIBRMM_WHEELHOUSE}/librmm_*.whl)
libraft-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${LIBRAFT_WHEELHOUSE}/libraft_*.whl)
EOF

export PIP_CONSTRAINT=/tmp/constraints.txt
```

Then copy the following into every script in the `ci/` directory that is doing `pip` installs or wheels builds with e.g. `pip wheel`.

```shell
source ./ci/use_wheels_from_prs.sh
```

This should generally be enough.
If any of the other CI scripts are already setting environment variable `PIP_CONSTRAINT`, you may need to
modify them slightly to ensure they **append to**, instead of **overwriting**, the constraints set up by `use_wheels_from_prs.sh`.

**Example 2:** Testing `cudf` (Python) using `librmm`, `rmm`, and `libkvikio` PR artifacts.

It's important to include all of the recursive dependencies.
So, for example, Python testing jobs that use the `rmm` Python package also need the `librmm` C++ package.

```shell
# ci/use_wheels_from_prs.sh

RAPIDS_PY_CUDA_SUFFIX=$(rapids-wheel-ctk-name-gen "${RAPIDS_CUDA_VERSION}")

# download wheels, store the directories holding them in variables
LIBKVIKIO_WHEELHOUSE=$(
RAPIDS_PY_WHEEL_NAME="libkvikio_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact kvikio 510 cpp
)
LIBRMM_WHEELHOUSE=$(
RAPIDS_PY_WHEEL_NAME="librmm_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact rmm 1678 cpp
)
RMM_WHEELHOUSE=$(
RAPIDS_PY_WHEEL_NAME="rmm_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact rmm 1678 python
)

# write a pip constraints file saying e.g. "whenever you encounter a requirement for 'librmm-cu12', use this wheel"
cat > /tmp/constraints.txt <<EOF
libkvikio-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${LIBKVIKIO_WHEELHOUSE}/libkvikio_*.whl)
librmm-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${LIBRMM_WHEELHOUSE}/librmm_*.whl)
rmm-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${RMM_WHEELHOUSE}/librmm_*.whl)
EOF

export PIP_CONSTRAINT=/tmp/constraints.txt
```

Then copy the following into every script in the `ci/` directory that is doing `pip` installs or wheels builds with e.g. `pip wheel`.

```shell
source ./ci/use_wheels_from_prs.sh
```

As above, if any of the other CI scripts are already setting environment variable `PIP_CONSTRAINT`, you may need to
modify them slightly to ensure they **append to**, instead of **overwriting**, the constraints set up by `use_wheels_from_prs.sh`.

**Note:** By default `rapids-get-pr-wheel-artifact` uses the most recent commit from the specified PR.
A commit hash from the dependent PR can be added as an optional 4th argument to pin testing to a specific commit.

Note: By default `rapids-get-pr-conda-artifact` uses the most recent commit from the specified PR.
A commit hash from the dependent PR can be added as an optional 4th argument to test with an earlier commit or to pin testing to a commit even if the dependent PR is updated.
## Skipping CI for Commits

See the GitHub Actions documentation page below on how to prevent GitHub Actions from running on certain commits. This is useful for preventing GitHub Actions from running on pull requests that are not fully complete. This also helps preserve the finite GPU resources provided by the RAPIDS Ops team.
Expand Down