Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom backend support #1466

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Conversation

agerasev
Copy link
Contributor

For now, Candle only has a fixed set of backends (CPU, CUDA and Metal). Some other frameworks like PyTorch or Burn supports custom backends.
This PR contains changes enabling to create and use custom backend.
This can be useful when user wants to use a different GPGPU framework (like OpenCL) or some specific hardware.

The PR consists of two parts:

  • Refactoring of BackendDevice and BackendStorage traits (removed new method, made location an associated type, renamed some methods to avoid ambiguity and other minor changes) and implementing them also for Device and Storage.
  • Creating CustomDevice and CustomStorage that encapsulate a custom BackendDevice and BackendStorage implementation (which is relatively tricky because these traits contain associated types and generic methods, but Tesnor doesn't have generic parameters, so implementation should go through dynamic dispatch, but this isn't exposed to user) and making types used by these traits public.

Has it a chance to be merged?
If it is undesirable to stabilize backend API, then custom backends may be enabled by feature with a note that it is unstable.

@xnorpx
Copy link
Contributor

xnorpx commented Dec 26, 2023

+1 to be able to implement and running another backend without needing to fork would be very useful.

@LaurentMazare
Copy link
Collaborator

I don't think this would be in the immediate priorities of the project. The diff is pretty large and it would be quite tricky to review it and assess that it doesn't break anything, we would have to brainstorm about how to make it in a way that the changes are less invasive than this. Before making such changes, I would suggest asking in the discord server if there are some thoughts about it just to avoid putting a lot of effort in such PRs.

@juntyr
Copy link

juntyr commented Dec 4, 2024

+1 on supporting custom backends in some way. This also allows wrapping an existing backend to modify some of its operations without needing to fork.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants