Fix CUDA plugin CI.

[ysiraichi]

Fix: #8577

This PR reverts #8286, and bumps CUDA version to 12.3. The latter is needed for successfully compiling GPU dependent source code that makes use of CUgraphConditionalHandle (not available in 12.1) driver API typedef.

[amjames]

Looks like the failing jobs are due to a failed clone from kleidiai's gitlab. Is that a widespread issue or spurious failure?

[ysiraichi]

It doesn't look widespread (haven't seen in other PRs). I will try rebasing this PR.

[tengyifei]

@ysiraichi from pytorch/pytorch#138609 (comment), it looks like PyTorch upstream decided to release with some specific set of CUDA versions (see issue). Can we use one of their chosen versions, for example CUDA 12.4 instead of CUDA 12.3?

[ysiraichi]

Problem is: I didn't find a docker image with CUDA 12.4. Also, I'm not sure how to create one, since it seems something internal.

[tengyifei]

Problem is: I didn't find a docker image with CUDA 12.4. Also, I'm not sure how to create one, since it seems something internal.

Could you clarify this challenge? Do you mean that you were hoping to find a torch_xla CUDA 12.4 docker build?

[ysiraichi]

As far as I understand, PyTorch/XLA CI relies on docker images (see dev-image). My point is that there is no docker image with CUDA 12.4 in that registry.

xla/.github/workflows/build_and_test.yml

Lines 44 to 52 in fbbdfca

	# build-torch-with-cuda:
	# name: "Build PyTorch with CUDA"
	# uses: ./.github/workflows/_build_torch_with_cuda.yml
	# needs: get-torch-commit
	# with:
	# # note that to build a torch wheel with CUDA enabled, we do not need a GPU runner.
	# dev-image: us-central1-docker.pkg.dev/tpu-pytorch-releases/docker/development:3.10_cuda_12.1
	# torch-commit: ${{needs.get-torch-commit.outputs.torch_commit}}
	# runner: linux.24xlarge

[tengyifei]

As far as I understand, PyTorch/XLA CI relies on docker images (see dev-image). My point is that there is no docker image with CUDA 12.4 in that registry.

@ysiraichi got it. thanks for the explanation. I think using CUDA 12.3 for now is a-okay. IIUC, most of the time we're only using torch CPU + torch_xla GPU in any case.

[tengyifei]

LMK when I should review. It looks like there are still some failed tests.

[ysiraichi]

After discussing this issue with @lsy323, I think the issue could be due to the older GPU in these instances. Is it okay if I try them out with a linux.g4dn.12xlarge.nvidia.gpu?

Current: g3.8xlarge 2x Tesla M60 (CC 5.2)
New: g4dn.12xlarge 4x Tesla T4 (CC 7.5)

@miladm @tengyifei
Let me know what you think.

[tengyifei]

Seems reasonable to me -- for comparison, we have created similar or more advanced dev VMs before

[ysiraichi]

There are some GPU tests and Triton tests that are still failing, right now. I think we should merge this PR, skipping them for now. I will make sure to open issues for each of them.

@miladm @tengyifei What do you think?

[tengyifei]

I think we should merge this PR, skipping them for now. I will make sure to open issues for each of them.

That sounds good.

[tengyifei]

I didn't know that we also need to build PyTorch with CUDA. Does building PyTorch with CPU not work for some reason? If we need to build PyTorch with CUDA, do you know if 12.3 is a supported CUDA version to build PyTorch with? From pytorch/pytorch#138609 it looks like upstream picked 12.4 or 12.6