From daa8fd87257ce9d3b11656c38e20829a60b9c150 Mon Sep 17 00:00:00 2001 From: Andy Linfoot <78757007+andy-neuma@users.noreply.github.com> Date: Wed, 24 Jul 2024 13:42:31 -0400 Subject: [PATCH] add Benchmark link (#5) * add Benchmarking session * adjust link scope --------- Co-authored-by: andy-neuma --- README.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 186bc27..d28141f 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,17 @@ # nm-vllm-certs -## Overview +## Overview The `nm-vllm` packages published in this repository are Neuralmagic Enterprise Edition of [vLLM](https://github.com/vllm-project/vllm). Packages are versioned Python wheels and docker images. These are released as "production level" official releases and "beta level" Nightly's. Official releases are made at the discretion of Neuralmagic, but typically track with `vllm` releases. These wheels are available via "public pypi" as well as ["nm-pypi"](https://pypi.neuralmagic.com). Nightly's are released every night given green runs in automation. The wheels are available at ["nm-pypi"](https://pypi.neuralmagic.com). + ## Installation + ### PyPI The [nm-vllm PyPi package](https://pypi.neuralmagic.com/simple/nm-vllm/index.html) includes pre-compiled binaries for CUDA (version 12.1) kernels. For other PyTorch or CUDA versions, please compile the package from source. @@ -23,9 +25,10 @@ To utilize the weight sparsity features, include the optional `sparse` dependenc pip install nm-vllm[sparse] --extra-index-url https://pypi.neuralmagic.com/simple ``` + ### Docker -The [`nm-vllm-ent` container registry](https://github.com/neuralmagic/nm-vllm-certs/pkgs/container/nm-vllm-ent) includes premade docker images. +The `nm-vllm-ent` [container registry](https://github.com/neuralmagic/nm-vllm-certs/pkgs/container/nm-vllm-ent) includes premade docker images. Launch the OpenAI-compatible server with: @@ -34,6 +37,12 @@ MODEL_ID=Qwen/Qwen2-0.5B-Instruct docker run --gpus all --shm-size 2g ghcr.io/neuralmagic/nm-vllm-ent:latest --model $MODEL_ID ``` + +## Benchmarks + +Please see our benchmarking results [here]( https://neuralmagic.github.io/nm-vllm-certs/dev/bench/). + + ## Models Neural Magic maintains a variety of optimized models on our Hugging Face organization profiles: