Add DeepSparse backend for CLIP inference

[mgoin]

DeepSparse is an inference runtime for fast sparse model inference on CPUs.
There is a backend available within clip-retrieval by installing it with pip install deepsparse-nightly[clip], and specifying a clip_model with a prepended "nm:", such as "nm:neuralmagic/CLIP-ViT-B-32-256x256-DataComp-s34B-b86K-quant-ds" or "nm:mgoin/CLIP-ViT-B-32-laion2b_s34b_b79k-ds".

Install

git clone https://github.com/mgoin/clip-retrieval
cd clip-retrieval 
git checkout deepsparse-clip-backend 
pip install -e . 
pip install "deepsparse-nightly[clip]"

Setup some example images

pip install img2dataset
echo 'https://placekitten.com/200/305' >> myimglist.txt
echo 'https://placekitten.com/200/304' >> myimglist.txt
echo 'https://placekitten.com/200/303' >> myimglist.txt
img2dataset --url_list=myimglist.txt --output_folder=image_folder --thread_count=64 --image_size=256

Inference!

clip-retrieval inference --batch_size 4 --clip_model=nm:neuralmagic/CLIP-ViT-B-32-256x256-DataComp-s34B-b86K-quant-ds --input_dataset image_folder --output_folder embeddings_folder
The number of samples has been estimated to be 3
Starting the worker
dataset is 12
Starting work on task 0
Fetching 7 files: 100%|███████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 61422.86it/s]
DeepSparse, Copyright 2021-present / Neuralmagic, Inc. version: 1.7.0.20231210 COMMUNITY | (99472380) (release) (optimized) (system=avx512_vnni, binary=avx512)
warming up with batch size 4 on cpu
done warming up in 0.5746381282806396s
 sample_per_sec 0 ; sample_count 3 

[rom1504]

would be great to do more benchmarks with this and add them to the readme

[rom1504]

thanks for the pr!