Skip to content

Commit

Permalink
Update example to not fail hessian inversion (#904)
Browse files Browse the repository at this point in the history
* update

Signed-off-by: Dipika <dipikasikka1@gmail.com>

* quality

---------

Signed-off-by: Dipika <dipikasikka1@gmail.com>
Co-authored-by: Rahul Tuli <rahul@neuralmagic.com>
  • Loading branch information
dsikka and rahul-tuli authored Nov 9, 2024
1 parent 644a500 commit a173a0c
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion examples/big_models_with_accelerate/multi_gpu_int8.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,9 @@ def tokenize(sample):
# * quantize the weights to int8 with GPTQ (static per channel)
# * quantize the activations to int8 (dynamic per token)
recipe = [
GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head"]),
GPTQModifier(
targets="Linear", scheme="W8A8", ignore=["lm_head"], dampening_frac=0.1
),
]

# 4) Apply algorithms and save in `compressed-tensors` format.
Expand Down

0 comments on commit a173a0c

Please sign in to comment.