diff --git a/resources/github-actions.md b/resources/github-actions.md index 4557a8c963b..dfa4e6157f8 100644 --- a/resources/github-actions.md +++ b/resources/github-actions.md @@ -176,51 +176,144 @@ The example code below demonstrates building and testing with conda packages fro Replace the pull request numbers and library names as needed. Remember that changes to use CI artifacts should be _temporary_ and should be reverted prior to merging any required changes in that PR. -**Example 1:** Building `libcuml` (C++) using `librmm`, `libraft`, `libcumlprims_mg` PR artifacts. +**Example 1:** Building `libcuml` (C++) using `librmm` and `libraft` PR artifacts. -```sh -# ci/build_cpp.sh +Add a new file called `ci/use_conda_packages_from_prs.sh`. -LIBRMM_CHANNEL=$(rapids-get-pr-conda-artifact rmm 1095 cpp) +```shell +# ci/use_conda_packages_from_prs.sh + +# download CI artifacts LIBRAFT_CHANNEL=$(rapids-get-pr-conda-artifact raft 1388 cpp) -LIBCUMLPRIMS_CHANNEL=$(rapids-get-pr-conda-artifact cumlprims_mg 129 cpp) +LIBRMM_CHANNEL=$(rapids-get-pr-conda-artifact rmm 1095 cpp) + +# make sure they can be found locally +conda config --system --add channels "${LIBRAFT_CHANNEL}" +conda config --system --add channels "${LIBRMM_CHANNEL}" +``` -# Build library packages with the CI artifact channels providing the updated dependencies +Then copy the following into every script in the `ci/` directory that is doing `conda` installs. -rapids-mamba-retry mambabuild \ - --channel "${LIBRMM_CHANNEL}" \ - --channel "${LIBRAFT_CHANNEL}" \ - --channel "${LIBCUMLPRIMS_CHANNEL}" \ - conda/recipes/libcuml +```shell +source ./ci/use_conda_packages_from_prs.sh ``` **Example 2:** Testing `cudf` (Python) using `librmm`, `rmm`, and `libkvikio` PR artifacts. -```sh -# ci/test_python_common.sh +It's important to include all of the recursive dependencies. +So, for example, Python testing jobs that use the `rmm` Python package also need the `librmm` C++ package. + +```shell +# ci/use_conda_packages_from_prs.sh +# download CI artifacts +LIBKVIKIO_CHANNEL=$(rapids-get-pr-conda-artifact kvikio 224 cpp) LIBRMM_CHANNEL=$(rapids-get-pr-conda-artifact rmm 1223 cpp) RMM_CHANNEL=$(rapids-get-pr-conda-artifact rmm 1223 python) -LIBKVIKIO_CHANNEL=$(rapids-get-pr-conda-artifact kvikio 224 cpp) -# Install library packages with the CI artifact channels providing the updated dependencies for testing +# make sure they can be found locally +conda config --system --add channels "${LIBKVIKIO_CHANNEL}" +conda config --system --add channels "${LIBRMM_CHANNEL}" +conda config --system --add channels "${RMM_CHANNEL}" +``` + +Then copy the following into every script in the `ci/` directory that is doing `conda` installs. -rapids-mamba-retry install \ - --channel "${CPP_CHANNEL}" \ - --channel "${PYTHON_CHANNEL}" \ - --channel "${LIBRMM_CHANNEL}" \ - --channel "${LIBKVIKIO_CHANNEL}" \ - --channel "${RMM_CHANNEL}" \ - cudf libcudf +```shell +source ./ci/use_conda_packages_from_prs.sh ``` -Note that the custom channel for PR artifacts is needed in the build scripts _and_ the test scripts, for C++ _and_ Python. -If building/testing a Python package that depends on a C++ library, it is necessary to use PR artifacts from that C++ library and not just Python (e.g. if testing `rmm` artifacts, you must use the corresponding `librmm` CI artifacts as well as `rmm`). -In some repos, the `test_python.sh` is quite complicated with multiple calls to conda/mamba. -We recommend that the Python and C++ artifact channels should be added to every call of `rapids-mamba-retry` / `rapids-conda-retry` "just in case." +**Note:** By default `rapids-get-pr-conda-artifact` uses the most recent commit from the specified PR. +A commit hash from the dependent PR can be added as an optional 4th argument to pin testing to a specific commit. + +## Using Wheel CI Artifacts in Other PRs + +To use wheels produced by other PRs' CI: + +* download the wheels at the beginning of CI jobs +* constrain `pip` to always use them + +Consider the following examples. + +**Example:** Building `libcuml` (C++) using `librmm` and `libraft` PR artifacts. + +Add a new file called `ci/use_wheels_from_prs.sh`. + +```shell +# ci/use_wheels_from_prs.sh + +RAPIDS_PY_CUDA_SUFFIX=$(rapids-wheel-ctk-name-gen "${RAPIDS_CUDA_VERSION}") + +# download wheels, store the directories holding them in variables +LIBRMM_WHEELHOUSE=$( + RAPIDS_PY_WHEEL_NAME="librmm_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact rmm 1678 cpp +) +LIBRAFT_WHEELHOUSE=$( + RAPIDS_PY_WHEEL_NAME="libraft_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact raft 2433 cpp +) + +# write a pip constraints file saying e.g. "whenever you encounter a requirement for 'librmm-cu12', use this wheel" +cat > /tmp/constraints.txt < /tmp/constraints.txt <