From df14597f92f51a8d80f6e89504a2e2c3877de023 Mon Sep 17 00:00:00 2001 From: Benjamin Bossan Date: Fri, 7 Feb 2025 16:51:39 +0100 Subject: [PATCH 1/2] DOC Explain uninitialized weights warning Users sometimes get confused by the warning from transformers that some weights are uninitialized and need to be trained when they use models for classification. A recent example is #2367. Even though the warning does not come from PEFT, let's add a section to the docs to explain this warning, as the situation is a bit different here. --- .../developer_guides/troubleshooting.md | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/docs/source/developer_guides/troubleshooting.md b/docs/source/developer_guides/troubleshooting.md index a0d0c91875..4ab2bf8e22 100644 --- a/docs/source/developer_guides/troubleshooting.md +++ b/docs/source/developer_guides/troubleshooting.md @@ -148,6 +148,34 @@ For inference, load the base model first and resize it the same way you did befo For a complete example, please check out [this notebook](https://github.com/huggingface/peft/blob/main/examples/causal_language_modeling/peft_lora_clm_with_additional_tokens.ipynb). +### Getting a warning about "weights not being initialized from the model checkpoint" + +When you load your PEFT model which has been trained on a classification task, you may get a warning like: + +> Some weights of LlamaForSequenceClassification were not initialized from the model checkpoint at meta-llama/Llama-3.2-1B and are newly initialized: ['score.weight']. You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. + +Although this looks scary, it is most likely nothing to worry about. It is in fact not a PEFT specific warning, instead it comes from `transformers`. The reason why you get is probably because you used something like `AutoModelForSequenceClassification`. This will attach a randomly initialized classification head to the base model (called `"score"` in this case). This head must be trained to produce sensible predictions, which is what the warning is telling you. + +When you get this warning _before_ training the model, there is thus no cause for concern. PEFT will automatically take care of making the classification head trainable if you correctly passed the `task_type` argument to the PEFT config, e.g. like so: + +```python +from peft import LoraConfig, TaskType + +lora_config = LoraConfig(..., task_type=TaskType.SEQ_CLS) +``` + +If your classification head does not follow the usual naming conventions from `transformers` (which is rare), you have to explicitly tell PEFT how the head is called using the `modules_to_save` argument: + +```python +lora_config = LoraConfig(..., modules_to_save=["name-of-classification-head"]) +``` + +To check the name of the classification head, just print the model, it should be the last module. + +If you get this warning from you inference code, i.e. _after_ training model, there is also nothing to worry about. When you load the PEFT model, you always first have to load the `transformers` model. Since `transformers` does not know that you will load PEFT weights afterwards, it still gives the warning. + +As always, it is best practice to ensure that the model works correctly for inference by running some validation on it. But the fact that you see this warning is no cause for concern. + ### Check layer and model status Sometimes a PEFT model can end up in a bad state, especially when handling multiple adapters. There can be some confusion around what adapters exist, which one is active, which one is merged, etc. To help investigate this issue, call the [`~peft.PeftModel.get_layer_status`] and the [`~peft.PeftModel.get_model_status`] methods. From eac162698d7354d554f8749e984edb420e928ac0 Mon Sep 17 00:00:00 2001 From: Benjamin Bossan Date: Mon, 10 Feb 2025 11:52:18 +0100 Subject: [PATCH 2/2] Apply suggestions from code review Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> --- docs/source/developer_guides/troubleshooting.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/source/developer_guides/troubleshooting.md b/docs/source/developer_guides/troubleshooting.md index 4ab2bf8e22..13c7de2238 100644 --- a/docs/source/developer_guides/troubleshooting.md +++ b/docs/source/developer_guides/troubleshooting.md @@ -150,13 +150,13 @@ For a complete example, please check out [this notebook](https://github.com/hugg ### Getting a warning about "weights not being initialized from the model checkpoint" -When you load your PEFT model which has been trained on a classification task, you may get a warning like: +When you load your PEFT model which has been trained on a task (for example, classification), you may get a warning like: > Some weights of LlamaForSequenceClassification were not initialized from the model checkpoint at meta-llama/Llama-3.2-1B and are newly initialized: ['score.weight']. You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. -Although this looks scary, it is most likely nothing to worry about. It is in fact not a PEFT specific warning, instead it comes from `transformers`. The reason why you get is probably because you used something like `AutoModelForSequenceClassification`. This will attach a randomly initialized classification head to the base model (called `"score"` in this case). This head must be trained to produce sensible predictions, which is what the warning is telling you. +Although this looks scary, it is most likely nothing to worry about. This warning comes from Transformers, and it isn't a PEFT specific warning. It lets you know that a randomly initialized classification head (`score`) is attached to the base model, and the head must be trained to produce sensible predictions. -When you get this warning _before_ training the model, there is thus no cause for concern. PEFT will automatically take care of making the classification head trainable if you correctly passed the `task_type` argument to the PEFT config, e.g. like so: +When you get this warning _before_ training the model, PEFT automatically takes care of making the classification head trainable if you correctly passed the `task_type` argument to the PEFT config. ```python from peft import LoraConfig, TaskType @@ -164,17 +164,17 @@ from peft import LoraConfig, TaskType lora_config = LoraConfig(..., task_type=TaskType.SEQ_CLS) ``` -If your classification head does not follow the usual naming conventions from `transformers` (which is rare), you have to explicitly tell PEFT how the head is called using the `modules_to_save` argument: +If your classification head does not follow the usual naming conventions from Transformers (which is rare), you have to explicitly tell PEFT the name of the head in `modules_to_save`. ```python lora_config = LoraConfig(..., modules_to_save=["name-of-classification-head"]) ``` -To check the name of the classification head, just print the model, it should be the last module. +To check the name of the classification head, print the model and it should be the last module. -If you get this warning from you inference code, i.e. _after_ training model, there is also nothing to worry about. When you load the PEFT model, you always first have to load the `transformers` model. Since `transformers` does not know that you will load PEFT weights afterwards, it still gives the warning. +If you get this warning from your inference code, i.e. _after_ training the model, when you load the PEFT model, you always have to load the Transformers model first. Since Transformers does not know that you will load PEFT weights afterwards, it still gives the warning. -As always, it is best practice to ensure that the model works correctly for inference by running some validation on it. But the fact that you see this warning is no cause for concern. +As always, it is best practice to ensure the model works correctly for inference by running some validation on it. ### Check layer and model status