Code

GPTQ Fused MoE class#8

Code

Closed

ElizaWszola wants to merge 82 commits intomarlin-moe-8-bitneuralmagic/vllm:marlin-moe-8-bitfrom gptq_fused_moeneuralmagic/vllm:gptq_fused_moeCopy head branch name to clipboard

+6,722-1,916

Commits

Commits on Sep 3, 2024

• GPTQ Fused MoE class

  ElizaWszola

  committed
  db1f07e

  Copy full SHA for db1f07e

• Add GPTQMarlinMoEMethod to gptq_marlin.py

  ElizaWszola

  committed
  6753789

  Copy full SHA for 6753789

Commits on Sep 4, 2024

• Use FusedMoE layer for all loads

  ElizaWszola

  committed
  7df4014

  Copy full SHA for 7df4014

• Merge branch 'marlin-moe-8-bit' into gptq_fused_moe

  ElizaWszola

  committed
  c3dc249

  Copy full SHA for c3dc249

• Make sure that GPTQ runs through mixtral.py

  ElizaWszola

  committed
  2fa03e5

  Copy full SHA for 2fa03e5

• enforce float16A/scales for marlin moe

  ElizaWszola

  committed
  8a504d9

  Copy full SHA for 8a504d9

• Merge branch 'marlin-moe-8-bit' into gptq_fused_moe

  ElizaWszola

  committed
  689ea0a

  Copy full SHA for 689ea0a

• cleanup

  ElizaWszola

  committed
  ec47561

  Copy full SHA for ec47561

• [MISC] Consolidate FP8 kv-cache tests (vllm-project#8131)

  comaniac

  authored
  2ad2e56

  Copy full SHA for 2ad2e56

• [CI/Build][ROCm] Enabling LoRA tests on ROCm (vllm-project#7369)

  Show description for d1dec64
  

  alexeykondrat

  and

  simon-mo

  authored
  d1dec64

  Copy full SHA for d1dec64

• [CI] Change test input in Gemma LoRA test (vllm-project#8163)

  WoosukKwon

  authored
  561d6f8

  Copy full SHA for 561d6f8

• [Feature] OpenAI-Compatible Tools API + Streaming for Hermes & Mistral models (vllm-project#5649)

  Show description for e02ce49
  

  K-Mistele

  and

  constellate

  authored
  e02ce49

  Copy full SHA for e02ce49

• [MISC] Replace input token throughput with total token throughput (vllm-project#8164)

  Show description for 77d9e51
  

  comaniac

  and

  mgoin

  authored
  77d9e51

  Copy full SHA for 77d9e51

• [Neuron] Adding support for adding/ overriding neuron configuration a… (vllm-project#8062)

  Show description for 008cf88
  

  hbikki

  and

  Harsha Bikki

  authored
  008cf88

  Copy full SHA for 008cf88

• Bump version to v0.6.0 (vllm-project#8166)

  simon-mo

  authored
  32e7db2

  Copy full SHA for 32e7db2

• [Doc] [Misc] Create CODE_OF_CONDUCT.md (vllm-project#8161)

  mmcelaney

  authored
  e01c2be

  Copy full SHA for e01c2be

Commits on Sep 5, 2024

• [bugfix] >1.43 constraint for openai (vllm-project#8169)

  Show description for 1afc931
  

  SolitaryThinker

  and

  mgoin

  authored
  1afc931

  Copy full SHA for 1afc931

• [Misc] Clean up RoPE forward_native (vllm-project#8076)

  WoosukKwon

  authored
  4624d98

  Copy full SHA for 4624d98

• [ci] Mark LoRA test as soft-fail (vllm-project#8160)

  Show description for ba262c4
  khluu

  authored
  ba262c4

  Copy full SHA for ba262c4

• [Core/Bugfix] Add query dtype as per FlashInfer API requirements. (vllm-project#8173)

  elfiegg

  authored
  e39ebf5

  Copy full SHA for e39ebf5

• [Doc] Indicate more information about supported modalities (vllm-project#8181)

  DarkLight1337

  authored
  288a938

  Copy full SHA for 288a938

• Inclusion of InternVLChatModel In PP_SUPPORTED_MODELS(Pipeline Parallelism) (vllm-project#7860)

  Manikandan-Thangaraj-ZS0321

  authored
  8685ba1

  Copy full SHA for 8685ba1

• [MODEL] Qwen Multimodal Support (Qwen-VL / Qwen-VL-Chat) (vllm-project#8029)

  Show description for 9da25a8
  

  alex-jw-brooks

  and

  DarkLight1337

  authored
  9da25a8

  Copy full SHA for 9da25a8

• Move verify_marlin_supported to GPTQMarlinLinearMethod (vllm-project#8165)

  mgoin

  authored
  2ee4528

  Copy full SHA for 2ee4528

• [Documentation][Spec Decode] Add documentation about lossless guarantees in Speculative Decoding in vLLM (vllm-project#7962)

  sroy745

  authored
  2febcf2

  Copy full SHA for 2febcf2

• update/fix weight loading to support tp

  dsikka

  committed
  9f97b3b

  Copy full SHA for 9f97b3b

Commits on Sep 6, 2024

• [Core] Support load and unload LoRA in api server (vllm-project#6566)

  Show description for db3bf7c
  

  Jeffwan

  and

  jeejeelee

  authored
  db3bf7c

  Copy full SHA for db3bf7c

• [BugFix] Fix Granite model configuration (vllm-project#8216)

  njhill

  authored
  baa5467

  Copy full SHA for baa5467

• remove 8-bit stuff for now

  ElizaWszola

  committed
  b841ac4

  Copy full SHA for b841ac4

• Merge branch 'gptq_fused_moe' of https://github.com/neuralmagic/vllm into gptq_fused_moe

  ElizaWszola

  committed
  a245032

  Copy full SHA for a245032

• fix; update large model testing cases

  dsikka

  committed
  9d8a80c

  Copy full SHA for 9d8a80c

• [Frontend] Add --logprobs argument to benchmark_serving.py (vllm-project#8191)

  afeldman-nm

  authored
  e5cab71

  Copy full SHA for e5cab71

• add hack to support unfused mixtral pathway for int8

  dsikka

  committed
  315e22f

  Copy full SHA for 315e22f

• [Misc] Use ray[adag] dependency instead of cuda (vllm-project#7938)

  ruisearch42

  authored
  de80783

  Copy full SHA for de80783

• fix install for tpu test

  dsikka

  committed
  565cc43

  Copy full SHA for 565cc43

• [CI/Build] Increasing timeout for multiproc worker tests (vllm-project#8203)

  alexeykondrat

  authored
  1447c97

  Copy full SHA for 1447c97

• [Kernel] [Triton] Memory optimization for awq_gemm and awq_dequantize, 2x throughput (vllm-project#8248)

  rasmith

  authored
  9db52ea

  Copy full SHA for 9db52ea

• [Misc] Remove SqueezeLLM (vllm-project#8220)

  dsikka

  authored
  23f3222

  Copy full SHA for 23f3222

• [Model] Allow loading from original Mistral format (vllm-project#8168)

  Show description for 29f49cd
  

  patrickvonplaten

  and

  mgoin

  authored
  29f49cd

  Copy full SHA for 29f49cd

Commits on Sep 7, 2024

• [misc] [doc] [frontend] LLM torch profiler support (vllm-project#7943)

  SolitaryThinker

  authored
  12dd715

  Copy full SHA for 12dd715

• [Bugfix] Fix Hermes tool call chat template bug (vllm-project#8256)

  Show description for 41e95c5
  K-Mistele

  authored
  41e95c5

  Copy full SHA for 41e95c5

• [Model] Multi-input support for LLaVA (vllm-project#8238)

  DarkLight1337

  authored
  2f707fc

  Copy full SHA for 2f707fc

• Enable Random Prefix Caching in Serving Profiling Tool (benchmark_serving.py) (vllm-project#8241)

  wschin

  authored
  795b662

  Copy full SHA for 795b662

• [tpu][misc] fix typo (vllm-project#8260)

  youkaichao

  authored
  ce2702a

  Copy full SHA for ce2702a

• [Bugfix] Fix broken OpenAI tensorizer test (vllm-project#8258)

  DarkLight1337

  authored
  9f68e00

  Copy full SHA for 9f68e00

• [Model][VLM] Support multi-images inputs for InternVL2 models (vllm-project#8201)

  Isotr0py

  authored
  e807125

  Copy full SHA for e807125

• Move float16 typecast hack to gptq marlin moe method

  ElizaWszola

  committed
  8886423

  Copy full SHA for 8886423

• Move output type conversion to gptq method as well

  ElizaWszola

  committed
  ab27497

  Copy full SHA for ab27497

• [Model][VLM] Decouple weight loading logic for Paligemma (vllm-project#8269)

  Isotr0py

  authored
  36bf815

  Copy full SHA for 36bf815

• ppc64le: Dockerfile fixed, and a script for buildkite (vllm-project#8026)

  sumitd2

  authored
  b962ee1

  Copy full SHA for b962ee1

• [CI/Build] Use python 3.12 in cuda image (vllm-project#8133)

  Show description for cfe712b
  joerunde

  authored
  cfe712b

  Copy full SHA for cfe712b

Commits on Sep 8, 2024

• [Bugfix] Fix async postprocessor in case of preemption (vllm-project#8267)

  alexm-redhat

  authored
  4ef41b8

  Copy full SHA for 4ef41b8

Commits on Sep 9, 2024

• Enable 8-bit weights in Fused Marlin MoE

  

  ElizaWszola

  authored and

  dsikka

  committed
  847e860

  Copy full SHA for 847e860

• fix rocm

  

  ElizaWszola

  authored and

  dsikka

  committed
  430a9cb

  Copy full SHA for 430a9cb

• bad paste

  

  ElizaWszola

  authored and

  dsikka

  committed
  48047aa

  Copy full SHA for 48047aa

• add test case; fix imports for tests

  dsikka

  committed
  bfc4fae

  Copy full SHA for bfc4fae

• fix to adapt custom_routin_function

  dsikka

  committed
  c5a2f62

  Copy full SHA for c5a2f62

• Use select_experts to compute top_k tensors in fused moe

  

  ElizaWszola

  authored and

  dsikka

  committed
  2b308c4

  Copy full SHA for 2b308c4

• bring back fused_moe_marlin -> fused_marlin_moe

  

  ElizaWszola

  authored and

  dsikka

  committed
  71256d4

  Copy full SHA for 71256d4

• GPTQ Fused MoE class

  

  ElizaWszola

  authored and

  dsikka

  committed
  7aa844c

  Copy full SHA for 7aa844c

• Add GPTQMarlinMoEMethod to gptq_marlin.py

  

  ElizaWszola

  authored and

  dsikka

  committed
  0f7bec3

  Copy full SHA for 0f7bec3

• Use FusedMoE layer for all loads

  

  ElizaWszola

  authored and

  dsikka

  committed
  cb0001e

  Copy full SHA for cb0001e

• Make sure that GPTQ runs through mixtral.py

  

  ElizaWszola

  authored and

  dsikka

  committed
  33090a3

  Copy full SHA for 33090a3

• enforce float16A/scales for marlin moe

  

  ElizaWszola

  authored and

  dsikka

  committed
  d479837

  Copy full SHA for d479837

• remove large model

  dsikka

  committed
  8baaec6

  Copy full SHA for 8baaec6

• Cleanup, comments

  

  ElizaWszola

  authored and

  dsikka

  committed
  8fbc181

  Copy full SHA for 8fbc181

• cleanup

  

  ElizaWszola

  authored and

  dsikka

  committed
  839915f

  Copy full SHA for 839915f

• remove 8-bit stuff for now

  

  ElizaWszola

  authored and

  dsikka

  committed
  a5bc626

  Copy full SHA for a5bc626

• update/fix weight loading to support tp

  dsikka

  committed
  c573fa1

  Copy full SHA for c573fa1

• fix; update large model testing cases

  dsikka

  committed
  a991d82

  Copy full SHA for a991d82

• add hack to support unfused mixtral pathway for int8

  dsikka

  committed
  d57804d

  Copy full SHA for d57804d

• fix install for tpu test

  dsikka

  committed
  96fa486

  Copy full SHA for 96fa486

• Move float16 typecast hack to gptq marlin moe method

  

  ElizaWszola

  authored and

  dsikka

  committed
  1faab90

  Copy full SHA for 1faab90

• Move output type conversion to gptq method as well

  

  ElizaWszola

  authored and

  dsikka

  committed
  970e06a

  Copy full SHA for 970e06a

• typo fix; fix comment

  dsikka

  committed
  fd0a4f2

  Copy full SHA for fd0a4f2

• Merge branch 'gptq_fused_moe' of https://github.com/neuralmagic/vllm into gptq_fused_moe

  ElizaWszola

  committed
  3ac9273

  Copy full SHA for 3ac9273

• Clarify comment, change how we process bias

  ElizaWszola

  committed
  d51a2f4

  Copy full SHA for d51a2f4

• [Bugfix] Streamed tool calls now more strictly follow OpenAI's format; ensures Vercel AI SDK compatibility (vllm-project#8272)

  K-Mistele

  authored
  08287ef

  Copy full SHA for 08287ef

• [Frontend] Add progress reporting to run_batch.py (vllm-project#8060)

  Show description for 58fcc85
  

  alugowski

  and

  Adam Lugowski

  authored
  58fcc85

  Copy full SHA for 58fcc85

• [Bugfix] Correct adapter usage for cohere and jamba (vllm-project#8292)

  vladislavkruglikov

  authored
  f9b4a2d

  Copy full SHA for f9b4a2d

• [Misc] GPTQ Activation Ordering (vllm-project#8135)

  kylesayrs

  authored
  c7cb5c3

  Copy full SHA for c7cb5c3

• Merge branch 'main' into gptq_fused_moe

  dsikka

  authored
  12f05c5

  Copy full SHA for 12f05c5