Merge branch 'develop' into master

.github/workflows/tests_torch.yml

@@ -17,6 +17,7 @@ on:
       - 'templates/**'
       - 'tests/**'
       - 'utils/**'
+  workflow_dispatch:
 
 jobs:
   check_code_quality:
@@ -60,4 +61,4 @@ jobs:
           pip install datasets
       - name: Test
         run: |
-          make test-reduced
+          make test-adapters

.gitignore

@@ -167,3 +167,6 @@ scripts/git-strip-merge
 
 # .lock
 *.lock
+
+# DS_Store (MacOS)
+.DS_Store

CONTRIBUTING.md

@@ -175,51 +175,82 @@ Follow these steps to start contributing:
 5. Develop the features on your branch.
 
    As you work on the features, you should make sure that the test suite
-   passes:
+   passes. You should run the tests impacted by your changes like this:
+
+   ```bash
+   $ pytest tests/<TEST_TO_RUN>.py
+   ```
+
+   You can also run the full suite with the following command, but it takes
+   a beefy machine to produce a result in a decent amount of time now that
+   Transformers has grown a lot. Here is the command for it:
 
    ```bash
    $ make test
    ```
 
-   Note, that this command uses `-n auto` pytest flag, therefore, it will start as many parallel `pytest` processes as the number of your computer's CPU-cores, and if you have lots of those and a few GPUs and not a great amount of RAM, it's likely to overload your computer. Therefore, to run the test suite, you may want to consider using this command instead:
+   For more information about tests, check out the
+   [dedicated documentation](https://huggingface.co/docs/transformers/testing)
+
+   🤗 Transformers relies on `black` and `isort` to format its source code
+   consistently. After you make changes, apply automatic style corrections and code verifications
+   that can't be automated in one go with:
 
    ```bash
-   $ python -m pytest -n 3 --dist=loadfile -s -v ./tests/
+   $ make fixup
    ```
 
-   Adjust the value of `-n` to fit the load your hardware can support.
+   This target is also optimized to only work with files modified by the PR you're working on.
 
-   `adapter-transformers` relies on `black` and `isort` to format its source code
-   consistently. After you make changes, format them with:
+   If you prefer to run the checks one after the other, the following command apply the
+   style corrections:
 
    ```bash
    $ make style
    ```
 
-   `adapter-transformers` also uses `flake8` to check for coding mistakes. Quality
+   `adapter-transformers` also uses `flake8` and a few custom scripts to check for coding mistakes. Quality
    control runs in CI, however you can also run the same checks with:
 
    ```bash
    $ make quality
    ```
-   You can do the automatic style corrections and code verifications that can't be automated in one go:
+
+   Finally we have a lot of scripts that check we didn't forget to update
+   some files when adding a new model, that you can run with
 
    ```bash
-   $ make fixup
+   $ make repo-consistency
    ```
 
-   This target is also optimized to only work with files modified by the PR you're working on.
+   To learn more about those checks and how to fix any issue with them, check out the
+   [documentation](https://huggingface.co/docs/transformers/pr_checks)
 
    If you're modifying documents under `docs/source`, make sure to validate that
    they can still be built. This check also runs in CI. To run a local check
-   make sure you have installed the documentation builder requirements, by
-   running `pip install .[tf,torch,docs]` once from the root of this repository
-   and then run:
+   make sure you have installed the documentation builder requirements. First you will need to clone the
+   repository containing our tools to build the documentation:
+
+   ```bash
+   $ pip install git+https://github.com/huggingface/doc-builder
+   ```
 
+   Then, make sure you have all the dependencies to be able to build the doc with:
+
    ```bash
-   $ make docs
+   $ pip install ".[docs]"
    ```
 
+   Finally run the following command from the root of the repository:
+
+   ```bash
+   $ doc-builder build transformers docs/source/ --build_dir ~/tmp/test-build
+   ```
+
+   This will build the documentation in the `~/tmp/test-build` folder where you can inspect the generated
+   Markdown files with your favorite editor. You won't be able to see the final rendering on the website
+   before your PR is merged, we are actively working on adding a tool for this.
+
    Once you're happy with your changes, add changed files using `git add` and
    make a commit with `git commit` to record your changes locally:
 
@@ -273,8 +304,15 @@ Follow these steps to start contributing:
    - If you are adding a new tokenizer, write tests, and make sure
      `RUN_SLOW=1 python -m pytest tests/test_tokenization_{your_model_name}.py` passes.
    CircleCI does not run the slow tests, but github actions does every night!
-6. All public methods must have informative docstrings that work nicely with sphinx. See `modeling_ctrl.py` for an
+6. All public methods must have informative docstrings that work nicely with sphinx. See `modeling_bert.py` for an
    example.
+7. Due to the rapidly growing repository, it is important to make sure that no files that would significantly weigh down the repository are added. This includes images, videos and other non-text files. We prefer to leverage a hf.co hosted `dataset` like
+   the ones hosted on [`hf-internal-testing`](https://huggingface.co/hf-internal-testing) in which to place these files and reference 
+   them by URL. We recommend putting them in the following dataset: [huggingface/documentation-images](https://huggingface.co/datasets/huggingface/documentation-images).
+   If an external contribution, feel free to add the images to your PR and ask a Hugging Face member to migrate your images
+   to this dataset.
+
+See more about the checks run on a pull request in our [PR guide](pr_checks)
 
 ### Tests
 
@@ -326,7 +364,7 @@ $ python -m unittest discover -s examples -t examples -v
 
 ### Style guide
 
-For documentation strings, `transformers` follows the [google style](https://google.github.io/styleguide/pyguide.html).
+For documentation strings, 🤗 Transformers follows the [google style](https://google.github.io/styleguide/pyguide.html).
 Check our [documentation writing guide](https://github.com/huggingface/transformers/tree/master/docs#writing-documentation---specification)
 for more information.
 
@@ -350,7 +388,7 @@ You can now use `make` from any terminal (Powershell, cmd.exe, etc) 🎉
 
 ### Syncing forked master with upstream (HuggingFace) master
 
-To avoid pinging the upstream repository which adds reference notes to each upstream PR and sends unnessary notifications to the developers involved in these PRs,
+To avoid pinging the upstream repository which adds reference notes to each upstream PR and sends unnecessary notifications to the developers involved in these PRs,
 when syncing the master branch of a forked repository, please, follow these steps:
 1. When possible, avoid syncing with the upstream using a branch and PR on the forked repository. Instead merge directly into the forked master.
 2. If a PR is absolutely necessary, use the following steps after checking out your branch:

Makefile

@@ -31,25 +31,25 @@ deps_table_check_updated:
 
 autogenerate_code: deps_table_update
 
-# Check that source code meets quality standards
+# Check that the repo is in a good state
 
 # NOTE FOR adapter-transformers: The following check is skipped as not all copies implement adapters yet
 # python utils/check_copies.py
 # python utils/check_table.py
 # python utils/check_dummies.py
-# python utils/tests_fetcher.py --sanity_check
-extra_quality_checks:
+repo-consistency:
 	python utils/check_repo.py
 	python utils/check_inits.py
 	python utils/check_adapters.py
 
 # this target runs checks on all files
+
 quality:
 	black --check $(check_dirs)
 	isort --check-only $(check_dirs)
 	python utils/custom_init_isort.py --check_only
 	flake8 $(check_dirs)
-	${MAKE} extra_quality_checks
+	python utils/style_doc.py src/transformers docs/source --max_len 119 --check_only
 
 # Format source code automatically and check is there are any problems left that need manual fixing
 
@@ -58,6 +58,7 @@ extra_style_checks:
 	python utils/style_doc.py src/transformers docs/source --max_len 119
 
 # this target runs checks on all files and potentially modifies some of them
+
 style:
 	black $(check_dirs)
 	isort $(check_dirs)
@@ -66,7 +67,7 @@ style:
 
 # Super fast fix and check target that only works on relevant modified files since the branch was made
 
-fixup: modified_only_fixup extra_style_checks autogenerate_code extra_quality_checks
+fixup: modified_only_fixup extra_style_checks autogenerate_code repo-consistency
 
 # Make marked copies of snippets of codes conform to the original
 
@@ -82,16 +83,8 @@ test:
 
 # Run the adapter tests
 
-test-adapter:
-	python -m pytest -n auto --dist=loadfile -s -v\
-		-k test_adapter\
-		--ignore-glob='tests/test_tokenization*'\
-		--ignore-glob='tests/test_processor*'\
-		./tests/
-
-# Run a reduced test suite in the CI pipeline of adapter-transformers
-test-reduced:
-	python utils/run_tests.py
+test-adapters:
+	python -m pytest -n auto --dist=loadfile -s -v ./tests_adapters/
 
 # Run tests for examples
 
@@ -104,11 +97,6 @@ test-sagemaker: # install sagemaker dependencies in advance with pip install .[s
 	TEST_SAGEMAKER=True python -m pytest -n auto  -s -v ./tests/sagemaker
 
 
-# Check that docs can build
-
-docs:
-	cd docs && make html SPHINXOPTS="-W -j 4"
-
 # Release stuff
 
 pre-release:

README.md

@@ -62,6 +62,23 @@ To get started with adapters, refer to these locations:
 - **https://adapterhub.ml** to explore available pre-trained adapter modules and share your own adapters
 - **[Examples folder](https://github.com/Adapter-Hub/adapter-transformers/tree/master/examples)** of this repository containing HuggingFace's example training scripts, many adapted for training adapters
 
+## Implemented Methods
+
+Currently, adapter-transformers integrates all architectures and methods listed below:
+
+| Method | Paper(s) | Quick Links |
+| --- | --- | --- |
+| Bottleneck adapters | [Houlsby et al. (2019)](https://arxiv.org/pdf/1902.00751.pdf)<br> [Bapna and Firat (2019)](https://arxiv.org/pdf/1909.08478.pdf) | [Quickstart](https://docs.adapterhub.ml/quickstart.html), [Notebook](https://colab.research.google.com/github/Adapter-Hub/adapter-transformers/blob/master/notebooks/01_Adapter_Training.ipynb) |
+| AdapterFusion | [Pfeiffer et al. (2021)](https://aclanthology.org/2021.eacl-main.39.pdf) | [Docs: Training](https://docs.adapterhub.ml/training.html#train-adapterfusion), [Notebook](https://colab.research.google.com/github/Adapter-Hub/adapter-transformers/blob/master/notebooks/03_Adapter_Fusion.ipynb) |
+| MAD-X,<br> Invertible adapters | [Pfeiffer et al. (2020)](https://aclanthology.org/2020.emnlp-main.617/) | [Notebook](https://colab.research.google.com/github/Adapter-Hub/adapter-transformers/blob/master/notebooks/04_Cross_Lingual_Transfer.ipynb) |
+| AdapterDrop | [Rücklé et al. (2021)](https://arxiv.org/pdf/2010.11918.pdf) | [Notebook](https://colab.research.google.com/github/Adapter-Hub/adapter-transformers/blob/master/notebooks/05_Adapter_Drop_Training.ipynb) |
+| MAD-X 2.0,<br> Embedding training | [Pfeiffer et al. (2021)](https://arxiv.org/pdf/2012.15562.pdf) | [Docs: Embeddings](https://docs.adapterhub.ml/embeddings.html), [Notebook](https://colab.research.google.com/github/Adapter-Hub/adapter-transformers/blob/master/notebooks/08_NER_Wikiann.ipynb) |
+| Prefix Tuning | [Li and Liang (2021)](https://arxiv.org/pdf/2101.00190.pdf) | [Docs](https://adapterhub.ml/overview#prefix-tuning) |
+| Parallel adapters,<br> Mix-and-Match adapters | [He et al. (2021)](https://arxiv.org/pdf/2110.04366.pdf) | [Docs](https://adapterhub.ml/overview#combinations-mix-and-match-adapters) |
+
+## Supported Models
+
+We currently support the PyTorch versions of all models listed on the **[Model Overview](https://docs.adapterhub.ml/model_overview.html) page** in our documentation.
 
 ## Citation
 

adapter_docs/_static/custom.css

@@ -22,6 +22,9 @@ a {
 }
 /* The literal code blocks */
 .rst-content tt.literal, .rst-content tt.literal, .rst-content code.literal {
+    color: #666;
+}
+.rst-content a code.literal {
     color: #39B3C6;
 }
 /* Sidebar scroll space for version switcher */

adapter_docs/adapter_composition.md

@@ -4,6 +4,8 @@ One of the great advantages of using adapters is the possibility to combine mult
 To enable such adapter compositions, `adapter-transformers` comes with a modular and flexible concept to define how the input to the model should flow through the available adapters.
 This not only allows stacking ([_MAD-X_](https://arxiv.org/pdf/2005.00052.pdf)) and fusing ([_AdapterFusion_](https://arxiv.org/pdf/2005.00247.pdf)) adapters, but also even more complex adapter setups.
 
+## Adapter activation
+
 The single location where all the adapter composition magic happens is the `active_adapters` property of the model class.
 In the simplest case, you can set the name of a single adapter here to activate it:
 ```python
@@ -20,6 +22,17 @@ Note that we also could have used `model.set_active_adapters("adapter_name")` wh
     - All adapters not mentioned anywhere in the ``active_adapters`` setup are ignored although they might be loaded into the model. Thus, after adding an adapter, make sure to activate it.
 ```
 
+Alternatively, the [`AdapterSetup`](transformers.AdapterSetup) context manager allows dynamic configuration of activated setups without changing the model state:
+
+```python
+model = ...
+model.add_adapter("adapter_name")
+
+with AdapterSetup("adapter_name"):
+    # will use the adapter named "adapter_name" in the forward pass
+    outputs = model(**inputs)
+```
+
 The basic building blocks of the more advanced setups are simple objects derived from `AdapterCompositionBlock`,
 each representing a different possibility to combine single adapters.
 They are presented in more detail in the following.
@@ -162,7 +175,7 @@ In the following example, we load two adapters for semantic textual similarity (
 We activate a parallel setup where the input is passed through both adapters and their respective prediction heads.
 
 ```python
-model = AutoModelWithHeads.from_pretrained("distilbert-base-uncased")
+model = AutoAdapterModel.from_pretrained("distilbert-base-uncased")
 tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
 
 adapter1 = model.load_adapter("sts/sts-b@ukp")