Replies: 1 comment 1 reply
-
Hi @anwai98 1- First, quantize your Vision Transformers with accelerate & bitsandbytes: https://huggingface.co/docs/accelerate/v0.29.3/en/usage_guides/quantization#bitsandbytes-integration to quantize it in 4-bit precision. You might leave the classification head un-quantized (that's what we do for LLMs) Hope this helps ! |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi community,
Thanks to the huggingface team for this awesome repo 😉
I have a question: I would like to check out QLoRa on vision transformers. I found this gem which helps me to get a rough idea of how to integrate LoRa in computer vision models (haven't tested yet), and I see some hints here on quantization. Could someone guide me on how to integrate the two? (basically to achieve QLoRa-based finetuning for pretrained vision transformers).
(My best guess is both the config files (one for LoRa and/or one from the quantization library) come together in one place, I am a bit unsure of the right way to go ahead with this)
Thanks!
Beta Was this translation helpful? Give feedback.
All reactions