Merge pull request #1456 from PAIR-code/dev

Merge dev into main

.github/workflows/ci.yml

@@ -33,7 +33,7 @@ jobs:
     strategy:
       matrix:
         node-version: [18]
-        python-version: ["3.10"]
+        python-version: ["3.10", "3.11"]
     defaults:
       run:
         shell: bash -l {0}

docs/documentation/_sources/components.md.txt

@@ -460,20 +460,34 @@ The UI supports multiple options for analysis, including:
 
 For a walkthrough of how to use sequence salience to debug LLMs, check out the
 Responsible Generative AI Toolkit at
-https://ai.google.dev/responsible/model_behavior.
+https://ai.google.dev/responsible/model_behavior and for more on design of the
+system see our paper at https://arxiv.org/abs/2404.07498.
+
+If you find this useful in your work, please cite Sequence Salience as:
+
+```
+@article{tenney2024interactive,
+  title={Interactive Prompt Debugging with Sequence Salience},
+  author={Tenney, Ian and Mullins, Ryan and Du, Bin and Pandya, Shree and Kahng, Minsuk and Dixon, Lucas},
+  journal={arXiv preprint arXiv:2404.07498},
+  year={2024}
+}
+```
 
 **Code:**
 
-*   LIT-for-Gemma Colab: [`lit_gemma.ipynb`](https://colab.research.google.com/github/google/generative-ai-docs/blob/main/site/en/gemma/docs/lit_gemma.ipynb)
+Currently, this works out-of-the-box with Gemma, Llama 2, Mistral, and GPT-2,
+using either KerasNLP or Transformers.
+
+*   LIT-for-Gemma Colab:
+    [`lit_gemma.ipynb`](https://colab.research.google.com/github/google/generative-ai-docs/blob/main/site/en/gemma/docs/lit_gemma.ipynb)
 *   Demo binary:
     [`lm_salience_demo.py`](https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples/lm_salience_demo.py)
 *   KerasNLP model wrappers:
     [`instrumented_keras_lms.py`](https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples/models/instrumented_keras_lms.py)
 *   Transformers model wrappers:
     [`pretrained_lms.py`](https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples/models/pretrained_lms.py)
 
-Currently, this works out-of-the-box
-with Gemma models (using Keras) as well as with GPT-2.
 
 
 ## Salience Clustering

docs/documentation/components.html

@@ -637,19 +637,30 @@ <h2>Sequence Salience<a class="headerlink" href="#sequence-salience" title="Link
 </ul>
 <p>For a walkthrough of how to use sequence salience to debug LLMs, check out the
 Responsible Generative AI Toolkit at
-<a class="reference external" href="https://ai.google.dev/responsible/model_behavior">https://ai.google.dev/responsible/model_behavior</a>.</p>
+<a class="reference external" href="https://ai.google.dev/responsible/model_behavior">https://ai.google.dev/responsible/model_behavior</a> and for more on design of the
+system see our paper at <a class="reference external" href="https://arxiv.org/abs/2404.07498">https://arxiv.org/abs/2404.07498</a>.</p>
+<p>If you find this useful in your work, please cite Sequence Salience as:</p>
+<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="nd">@article</span><span class="p">{</span><span class="n">tenney2024interactive</span><span class="p">,</span>
+  <span class="n">title</span><span class="o">=</span><span class="p">{</span><span class="n">Interactive</span> <span class="n">Prompt</span> <span class="n">Debugging</span> <span class="k">with</span> <span class="n">Sequence</span> <span class="n">Salience</span><span class="p">},</span>
+  <span class="n">author</span><span class="o">=</span><span class="p">{</span><span class="n">Tenney</span><span class="p">,</span> <span class="n">Ian</span> <span class="ow">and</span> <span class="n">Mullins</span><span class="p">,</span> <span class="n">Ryan</span> <span class="ow">and</span> <span class="n">Du</span><span class="p">,</span> <span class="n">Bin</span> <span class="ow">and</span> <span class="n">Pandya</span><span class="p">,</span> <span class="n">Shree</span> <span class="ow">and</span> <span class="n">Kahng</span><span class="p">,</span> <span class="n">Minsuk</span> <span class="ow">and</span> <span class="n">Dixon</span><span class="p">,</span> <span class="n">Lucas</span><span class="p">},</span>
+  <span class="n">journal</span><span class="o">=</span><span class="p">{</span><span class="n">arXiv</span> <span class="n">preprint</span> <span class="n">arXiv</span><span class="p">:</span><span class="mf">2404.07498</span><span class="p">},</span>
+  <span class="n">year</span><span class="o">=</span><span class="p">{</span><span class="mi">2024</span><span class="p">}</span>
+<span class="p">}</span>
+</pre></div>
+</div>
 <p><strong>Code:</strong></p>
+<p>Currently, this works out-of-the-box with Gemma, Llama 2, Mistral, and GPT-2,
+using either KerasNLP or Transformers.</p>
 <ul class="simple">
-<li><p>LIT-for-Gemma Colab: <a class="reference external" href="https://colab.research.google.com/github/google/generative-ai-docs/blob/main/site/en/gemma/docs/lit_gemma.ipynb"><code class="docutils literal notranslate"><span class="pre">lit_gemma.ipynb</span></code></a></p></li>
+<li><p>LIT-for-Gemma Colab:
+<a class="reference external" href="https://colab.research.google.com/github/google/generative-ai-docs/blob/main/site/en/gemma/docs/lit_gemma.ipynb"><code class="docutils literal notranslate"><span class="pre">lit_gemma.ipynb</span></code></a></p></li>
 <li><p>Demo binary:
 <a class="reference external" href="https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples/lm_salience_demo.py"><code class="docutils literal notranslate"><span class="pre">lm_salience_demo.py</span></code></a></p></li>
 <li><p>KerasNLP model wrappers:
 <a class="reference external" href="https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples/models/instrumented_keras_lms.py"><code class="docutils literal notranslate"><span class="pre">instrumented_keras_lms.py</span></code></a></p></li>
 <li><p>Transformers model wrappers:
 <a class="reference external" href="https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples/models/pretrained_lms.py"><code class="docutils literal notranslate"><span class="pre">pretrained_lms.py</span></code></a></p></li>
 </ul>
-<p>Currently, this works out-of-the-box
-with Gemma models (using Keras) as well as with GPT-2.</p>
 </section>
 <section id="salience-clustering">
 <h2>Salience Clustering<a class="headerlink" href="#salience-clustering" title="Link to this heading">#</a></h2>

docs/tutorials/sequence-salience/index.html

@@ -112,10 +112,11 @@ <h2>Prompt Engineering with Sequence Salience</h2>
 LIT supports additional LLMs, including <a href="https://llama.meta.com/">Llama 2</a> and <a href="https://mistral.ai/news/announcing-mistral-7b/">Mistral</a>,
 via the HuggingFace Transformers and KerasNLP libraries.</p>
 <p>This tutorial was adapted from and expands upon LIT's contributions to the
-<a href="https://ai.google.dev/responsible">Responsible Generative AI Tookit</a> and the related paper and
-<a href="https://youtu.be/EZgUlnWdh0w">video</a> submitted to the ACL 2024 Systems Demonstration track.
-This is an active and ongoing research area for the LIT team, so expect changes
-and further expansions to this tutorial over time.</p>
+<a href="https://ai.google.dev/responsible">Responsible Generative AI Tookit</a> and the related
+<a href="https://arxiv.org/abs/2404.07498">paper</a> and <a href="https://youtu.be/EZgUlnWdh0w">video</a> submitted to the ACL 2024
+System Demonstrations track. This is an active and ongoing research area for
+the LIT team, so expect changes and further expansions to this tutorial over
+time.</p>
 <h2>Case Study 1: Debugging Few-Shot Prompts</h2>
 <p>Few-shot prompting was introduced with <a href="https://cdn.openai.com/better-language-models/language-models.pdf">GPT-2</a>: an ML developer provides
 examples of how to perform a task in a prompt, affixes user-provided content at

lit_nlp/examples/models/tfx_model_test.py

@@ -12,7 +12,8 @@ def setUp(self):
     super(TfxModelTest, self).setUp()
     self._path = tempfile.mkdtemp()
     input_layer = tf.keras.layers.Input(
-        shape=(1), dtype=tf.string, name='example')
+        shape=(1,), dtype=tf.string, name='example'
+    )
     parsed_input = tf.io.parse_example(
         tf.reshape(input_layer, [-1]),
         {'input_0': tf.io.FixedLenFeature([1], dtype=tf.float32)})

pyproject.toml

@@ -79,10 +79,11 @@ keywords = [
 [project.optional-dependencies]
 # LINT.IfChange
 examples = [
-  "gunicorn==20.1.0",
-  "tensorflow==2.10.0",
-  "tensorflow-datasets==4.8.0",
-  "tensorflow-text==2.10.0",
+  "gunicorn>=20.1.0",
+  "sentencepiece==0.1.99",
+  "tensorflow>=2.10.0,<2.16.0",
+  "tensorflow-datasets>=4.9.0",
+  "tensorflow-text>=2.10.0,<2.16.0",
   "torch>=2.0.0",
   "transformers>=4.27.1",
 ]

requirements_examples.txt

@@ -13,11 +13,11 @@
 # limitations under the License.
 # ==============================================================================
 # LINT.IfChange
-gunicorn==20.1.0
+gunicorn>=20.1.0
 sentencepiece==0.1.99
-tensorflow==2.10.0
-tensorflow-datasets==4.8.0
-tensorflow-text==2.10.0
+tensorflow>=2.10.0,<2.16.0
+tensorflow-datasets>=4.9.0
+tensorflow-text>=2.10.0,<2.16.0
 torch>=2.0.0
 transformers>=4.27.1
 # LINT.ThenChange(./pyproject.toml)

website/sphinx_src/components.md

@@ -460,20 +460,34 @@ The UI supports multiple options for analysis, including:
 
 For a walkthrough of how to use sequence salience to debug LLMs, check out the
 Responsible Generative AI Toolkit at
-https://ai.google.dev/responsible/model_behavior.
+https://ai.google.dev/responsible/model_behavior and for more on design of the
+system see our paper at https://arxiv.org/abs/2404.07498.
+
+If you find this useful in your work, please cite Sequence Salience as:
+
+```
+@article{tenney2024interactive,
+  title={Interactive Prompt Debugging with Sequence Salience},
+  author={Tenney, Ian and Mullins, Ryan and Du, Bin and Pandya, Shree and Kahng, Minsuk and Dixon, Lucas},
+  journal={arXiv preprint arXiv:2404.07498},
+  year={2024}
+}
+```
 
 **Code:**
 
-*   LIT-for-Gemma Colab: [`lit_gemma.ipynb`](https://colab.research.google.com/github/google/generative-ai-docs/blob/main/site/en/gemma/docs/lit_gemma.ipynb)
+Currently, this works out-of-the-box with Gemma, Llama 2, Mistral, and GPT-2,
+using either KerasNLP or Transformers.
+
+*   LIT-for-Gemma Colab:
+    [`lit_gemma.ipynb`](https://colab.research.google.com/github/google/generative-ai-docs/blob/main/site/en/gemma/docs/lit_gemma.ipynb)
 *   Demo binary:
     [`lm_salience_demo.py`](https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples/lm_salience_demo.py)
 *   KerasNLP model wrappers:
     [`instrumented_keras_lms.py`](https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples/models/instrumented_keras_lms.py)
 *   Transformers model wrappers:
     [`pretrained_lms.py`](https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples/models/pretrained_lms.py)
 
-Currently, this works out-of-the-box
-with Gemma models (using Keras) as well as with GPT-2.
 
 
 ## Salience Clustering

website/src/tutorials/sequence-salience.md

@@ -55,10 +55,11 @@ LIT supports additional LLMs, including [Llama 2][llama] and [Mistral][mistral],
 via the HuggingFace Transformers and KerasNLP libraries.
 
 This tutorial was adapted from and expands upon LIT's contributions to the
-[Responsible Generative AI Tookit][rai_toolkit] and the related paper and
-[video][seqsal_video] submitted to the ACL 2024 Systems Demonstration track.
-This is an active and ongoing research area for the LIT team, so expect changes
-and further expansions to this tutorial over time.
+[Responsible Generative AI Tookit][rai_toolkit] and the related
+[paper][seqsal_paper] and [video][seqsal_video] submitted to the ACL 2024
+System Demonstrations track. This is an active and ongoing research area for
+the LIT team, so expect changes and further expansions to this tutorial over
+time.
 
 ## Case Study 1: Debugging Few-Shot Prompts
 
@@ -486,6 +487,7 @@ helpful guides that can help you develop better prompts, including:
 [salience_research_1]: https://dl.acm.org/doi/full/10.1145/3639372
 [salience_research_2]: https://arxiv.org/abs/2402.01761
 [seqsal_docs]: ../../documentation/components.html#sequence-salience
+[seqsal_paper]: https://arxiv.org/abs/2404.07498
 [seqsal_video]: https://youtu.be/EZgUlnWdh0w
 [synapis]: https://scholarspace.manoa.hawaii.edu/items/65312e48-5954-4a5f-a1e8-e5119e6abc0a
 [toolformer]: https://arxiv.org/abs/2302.04761