diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index 68628c07..39cf8feb 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -10,7 +10,7 @@ labels: "bug" Useful Links: - Wiki: https://docs.garak.ai/garak - Before opening a new issue, please search existing issues https://github.com/leondz/garak/issues + Before opening a new issue, please search existing issues https://github.com/NVIDIA/garak/issues --> ## Steps to reproduce diff --git a/.github/ISSUE_TEMPLATE/documentation.md b/.github/ISSUE_TEMPLATE/documentation.md index 64964965..41e0b40e 100644 --- a/.github/ISSUE_TEMPLATE/documentation.md +++ b/.github/ISSUE_TEMPLATE/documentation.md @@ -11,7 +11,7 @@ labels: "documentation" - Wiki: https://docs.garak.ai/garak - Code reference: https://reference.garak.ai/ - Before opening a new issue, please search existing issues https://github.com/leondz/garak/issues + Before opening a new issue, please search existing issues https://github.com/NVIDIA/garak/issues --> ## Summary diff --git a/.github/ISSUE_TEMPLATE/feature_suggestion.md b/.github/ISSUE_TEMPLATE/feature_suggestion.md index 1052b4a9..8f17a3b7 100644 --- a/.github/ISSUE_TEMPLATE/feature_suggestion.md +++ b/.github/ISSUE_TEMPLATE/feature_suggestion.md @@ -10,7 +10,7 @@ labels: "enhancement" Useful Links: - Wiki: https://docs.garak.ai/garak - Before opening a new issue, please search existing issues https://github.com/leondz/garak/issues + Before opening a new issue, please search existing issues https://github.com/NVIDIA/garak/issues --> ## Summary diff --git a/.github/ISSUE_TEMPLATE/plugin_suggestion.md b/.github/ISSUE_TEMPLATE/plugin_suggestion.md index 589633f7..f92a20bb 100644 --- a/.github/ISSUE_TEMPLATE/plugin_suggestion.md +++ b/.github/ISSUE_TEMPLATE/plugin_suggestion.md @@ -10,7 +10,7 @@ labels: "new-plugin" Useful Links: - Wiki: https://docs.garak.ai/garak - Before opening a new issue, please search existing issues https://github.com/leondz/garak/issues + Before opening a new issue, please search existing issues https://github.com/NVIDIA/garak/issues --> ## Summary diff --git a/.github/ISSUE_TEMPLATE/question.md b/.github/ISSUE_TEMPLATE/question.md index f809614f..0ef63cda 100644 --- a/.github/ISSUE_TEMPLATE/question.md +++ b/.github/ISSUE_TEMPLATE/question.md @@ -11,7 +11,7 @@ labels: "question" - Wiki: https://docs.garak.ai/garak - Code reference: https://reference.garak.ai/ - Before opening a new issue, please search existing issues https://github.com/leondz/garak/issues + Before opening a new issue, please search existing issues https://github.com/NVIDIA/garak/issues --> ## Summary diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 06959269..e90a3c52 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -20,7 +20,7 @@ List the steps needed to make sure this thing works - [ ] ... - [ ] **Verify** the thing does what it should - [ ] **Verify** the thing does not do what it should not -- [ ] **Document** the thing and how it works ([Example](https://github.com/leondz/garak/blob/61ce5c4ae3caac08e0abd1d069d223d8a66104bd/garak/generators/rest.py#L24-L100)) +- [ ] **Document** the thing and how it works ([Example](https://github.com/NVIDIA/garak/blob/61ce5c4ae3caac08e0abd1d069d223d8a66104bd/garak/generators/rest.py#L24-L100)) If you are opening a PR for a new plugin that targets a **specific** piece of hardware or requires a **complex or hard-to-find** testing environment, we recommend that you send us as much detail as possible. diff --git a/.github/workflows/cla.yml b/.github/workflows/cla.yml index 86265217..c8cea87c 100644 --- a/.github/workflows/cla.yml +++ b/.github/workflows/cla.yml @@ -14,7 +14,7 @@ permissions: jobs: CLAAssistant: - if: github.repository_owner == 'leondz' + if: github.repository_owner == 'NVIDIA' runs-on: ubuntu-latest steps: - name: "CA & DCO Assistant" @@ -27,7 +27,7 @@ jobs: PERSONAL_ACCESS_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }} with: path-to-signatures: 'signatures/cla.json' - path-to-document: 'https://github.com/leondz/garak/blob/main/CA_DCO.md' # e.g. a CLA or a DCO document + path-to-document: 'https://github.com/NVIDIA/garak/blob/main/CA_DCO.md' # e.g. a CLA or a DCO document # branch should not be protected branch: 'main' use-dco-flag: true diff --git a/.github/workflows/labels.yml b/.github/workflows/labels.yml index 6755f979..12528067 100644 --- a/.github/workflows/labels.yml +++ b/.github/workflows/labels.yml @@ -26,7 +26,7 @@ on: jobs: handle-labels: - if: github.repository_owner == 'leondz' + if: github.repository_owner == 'NVIDIA' runs-on: ubuntu-latest steps: - uses: actions/github-script@v7 @@ -81,7 +81,7 @@ jobs: git push origin # Now browse to the following URL and create your pull request! - # - https://github.com/leondz/garak/pulls + # - https://github.com/NVIDIA/garak/pulls \`\`\` This helps protect the process, ensure users are aware of commits on the branch being considered for merge, allows for a location for more commits to be offered without mingling with other contributor changes and allows contributors to make progress while a PR is still being reviewed. @@ -119,7 +119,7 @@ jobs: This includes: - - All of the item points within this [template](https://github.com/leondz/garak/blob/master/.github/ISSUE_TEMPLATE/bug_report.md) + - All of the item points within this [template](https://github.com/NVIDIA/garak/blob/master/.github/ISSUE_TEMPLATE/bug_report.md) - Screenshots showing the issues you're having - Exact replication steps @@ -131,7 +131,7 @@ jobs: close: true, comment: ` When creating an issue, please ensure that the default issue template has been updated with the required details: - https://github.com/leondz/garak/issues/new/choose + https://github.com/NVIDIA/garak/issues/new/choose Closing this issue. If you believe this issue has been closed in error, please provide any relevant output and logs which may be useful in diagnosing the issue. ` diff --git a/.github/workflows/maintain_cache.yml b/.github/workflows/maintain_cache.yml index e8034bd4..26a1666b 100644 --- a/.github/workflows/maintain_cache.yml +++ b/.github/workflows/maintain_cache.yml @@ -19,7 +19,7 @@ permissions: jobs: build: - if: github.repository_owner == 'leondz' + if: github.repository_owner == 'NVIDIA' runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f4cb5274..322ac2e0 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -31,12 +31,12 @@ And if you like the project, but just don't have time to contribute, that's fine If you want to ask a question, good places to check first are the [garak quick start docs](https://docs.garak.ai) and, if its a coding question, the [garak reference](https://reference.garak.ai/). -Before you ask a question, it is best to search for existing [Issues](https://github.com/leondz/garak/issues) that might help you. In case you have found a suitable issue and still need clarification, you can write your question in this issue. It is also advisable to search the internet for answers first. You can also often find helpful people on the garak [Discord](https://discord.gg/uVch4puUCs). +Before you ask a question, it is best to search for existing [Issues](https://github.com/NVIDIA/garak/issues) that might help you. In case you have found a suitable issue and still need clarification, you can write your question in this issue. It is also advisable to search the internet for answers first. You can also often find helpful people on the garak [Discord](https://discord.gg/uVch4puUCs). If you then still feel the need to ask a question and need clarification, we recommend the following: -- Open an [Issue](https://github.com/leondz/garak/issues/new). +- Open an [Issue](https://github.com/NVIDIA/garak/issues/new). - Provide as much context as you can about what you're running into. - Provide project and platform versions (nodejs, npm, etc), depending on what seems relevant. @@ -58,7 +58,7 @@ A good bug report shouldn't leave others needing to chase you up for more inform - Make sure that you are using the latest version. - Determine if your bug is really a bug and not an error on your side e.g. using incompatible environment components/versions (Make sure that you have read the [documentation](https://reference.garak.ai/). If you are looking for support, you might want to check [this section](#i-have-a-question)). -- To see if other users have experienced (and potentially already solved) the same issue you are having, check if there is not already a bug report existing for your bug or error in the [bug tracker](https://github.com/leondz/garak/issues?q=label%3Abug). +- To see if other users have experienced (and potentially already solved) the same issue you are having, check if there is not already a bug report existing for your bug or error in the [bug tracker](https://github.com/NVIDIA/garak/issues?q=label%3Abug). - Also make sure to search the internet (including Stack Overflow) to see if users outside of the GitHub community have discussed the issue. - Collect information about the bug: - Stack trace (Traceback) @@ -75,7 +75,7 @@ You should never report security related issues, vulnerabilities or bugs includi We use GitHub issues to track bugs and errors. If you run into an issue with the project: -- Open an [Issue](https://github.com/leondz/garak/issues/new). (Since we can't be sure at this point whether it is a bug or not, we ask you not to talk about a bug yet and not to label the issue.) +- Open an [Issue](https://github.com/NVIDIA/garak/issues/new). (Since we can't be sure at this point whether it is a bug or not, we ask you not to talk about a bug yet and not to label the issue.) - Explain the behavior you would expect and the actual behavior. - Please provide as much context as possible and describe the *reproduction steps* that someone else can follow to recreate the issue on their own. This usually includes your code. For good bug reports you should isolate the problem and create a reduced test case. - Provide the information you collected in the previous section. @@ -98,14 +98,14 @@ This section guides you through submitting an enhancement suggestion for garak, - Make sure that you are using the latest version. - Read the [documentation](https://reference.garak.ai/) carefully and find out if the functionality is already covered, maybe by an individual configuration. -- Perform a [search](https://github.com/leondz/garak/issues) to see if the enhancement has already been suggested. If it has, add a comment to the existing issue instead of opening a new one. +- Perform a [search](https://github.com/NVIDIA/garak/issues) to see if the enhancement has already been suggested. If it has, add a comment to the existing issue instead of opening a new one. - Find out whether your idea fits with the scope and aims of the project. It's up to you to make a strong case to convince the project's developers of the merits of this feature. Keep in mind that we want features that will be useful to the majority of our users and not just a small subset. If you're just targeting a minority of users, consider writing an add-on/plugin library. - Check out our [guide for contributors](https://reference.garak.ai/en/latest/contributing.html), which includes our coding workflow and a [guide to constructing a plugin](https://reference.garak.ai/en/latest/contributing.generator.html). #### How Do I Submit a Good Enhancement Suggestion? -Enhancement suggestions are tracked as [GitHub issues](https://github.com/leondz/garak//issues). +Enhancement suggestions are tracked as [GitHub issues](https://github.com/NVIDIA/garak/issues). - Use a **clear and descriptive title** for the issue to identify the suggestion. - Provide a **step-by-step description of the suggested enhancement** in as many details as possible. @@ -143,4 +143,4 @@ Updating, improving and correcting the documentation ## Attribution -This guide is based on the **contributing-gen**. [Make your own](https://github.com/bttger/contributing-gen)! \ No newline at end of file +This guide is based on the **contributing-gen**. [Make your own](https://github.com/bttger/contributing-gen)! diff --git a/FAQ.md b/FAQ.md index e5fc0887..0aa16392 100644 --- a/FAQ.md +++ b/FAQ.md @@ -39,11 +39,11 @@ Not immediately, but if you have the Gradio skills, get in touch! ## Can you add support for vulnerability X? -Perhaps - please [open an issue](https://github.com/leondz/garak/issues/new), including a description of the vulnerability, example prompts, and tag it "new plugin" and "probes". +Perhaps - please [open an issue](https://github.com/NVIDIA/garak/issues/new), including a description of the vulnerability, example prompts, and tag it "new plugin" and "probes". ## Can you add support for model X? -Would love to! Please [open an issue](https://github.com/leondz/garak/issues/new), tagging it "new plugin" and "generators". +Would love to! Please [open an issue](https://github.com/NVIDIA/garak/issues/new), tagging it "new plugin" and "generators". ## How much disk space do I need to run garak? @@ -96,7 +96,7 @@ Adding a custom generator is fairly straight forward. One can either add a new c ## How can I redirect `garak_runs/` and `garak.log` to another place instead of `~/.local/share/garak/`? * `garak_runs` is configured via top-level config param `reporting.report_dir` and also CLI argument `--report_prefix` (which currently can include directory separator characters, so an absolute path can be given) -* An example of the location of the config param can be seen in https://github.com/leondz/garak/blob/main/garak/resources/garak.core.yaml +* An example of the location of the config param can be seen in https://github.com/NVIDIA/garak/blob/main/garak/resources/garak.core.yaml * If `reporting.report_dir` is set to an absolute path, you can move it anywhere * If it's a relative path, it will be within the garak directory under the "data" directory following the cross-platform [XDG base directory specification](https://specifications.freedesktop.org/basedir-spec/latest/) for local storage * There's no CLI or config option for moving `garak.log`, which is also stored in the XDG data directory diff --git a/README.md b/README.md index 3093fc96..2c31f550 100644 --- a/README.md +++ b/README.md @@ -9,9 +9,9 @@ `garak`'s a free tool. We love developing it and are always interested in adding functionality to support applications. [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) -[![Tests/Linux](https://github.com/leondz/garak/actions/workflows/test_linux.yml/badge.svg)](https://github.com/leondz/garak/actions/workflows/test_linux.yml) -[![Tests/Windows](https://github.com/leondz/garak/actions/workflows/test_windows.yml/badge.svg)](https://github.com/leondz/garak/actions/workflows/test_windows.yml) -[![Tests/OSX](https://github.com/leondz/garak/actions/workflows/test_macos.yml/badge.svg)](https://github.com/leondz/garak/actions/workflows/test_macos.yml) +[![Tests/Linux](https://github.com/NVIDIA/garak/actions/workflows/test_linux.yml/badge.svg)](https://github.com/NVIDIA/garak/actions/workflows/test_linux.yml) +[![Tests/Windows](https://github.com/NVIDIA/garak/actions/workflows/test_windows.yml/badge.svg)](https://github.com/NVIDIA/garak/actions/workflows/test_windows.yml) +[![Tests/OSX](https://github.com/NVIDIA/garak/actions/workflows/test_macos.yml/badge.svg)](https://github.com/NVIDIA/garak/actions/workflows/test_macos.yml) [![Documentation Status](https://readthedocs.org/projects/garak/badge/?version=latest)](http://garak.readthedocs.io/en/latest/?badge=latest) [![discord-img](https://img.shields.io/badge/chat-on%20discord-yellow.svg)](https://discord.gg/uVch4puUCs) [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) @@ -21,9 +21,6 @@ [![Downloads](https://pepy.tech/badge/garak/month)](https://pepy.tech/project/garak) -"🚧 garak is moving. This repository is moving to the `NVIDIA` github organization in the near future. This is planned to be a non-disruptive transition with automatic redirection. 🚧" - - ## Get started ### > See our user guide! [docs.garak.ai](https://docs.garak.ai/) ### > Join our [Discord](https://discord.gg/uVch4puUCs)! @@ -61,7 +58,7 @@ python -m pip install -U garak The standard pip version of `garak` is updated periodically. To get a fresher version, from GitHub, try: ``` -python -m pip install -U git+https://github.com/leondz/garak.git@main +python -m pip install -U git+https://github.com/NVIDIA/garak.git@main ``` ### Clone from source @@ -71,7 +68,7 @@ python -m pip install -U git+https://github.com/leondz/garak.git@main ``` conda create --name garak "python>=3.10,<=3.12" conda activate garak -gh repo clone leondz/garak +gh repo clone NVIDIA/garak cd garak python -m pip install -e . ``` @@ -298,7 +295,7 @@ Larger artefacts, like model files and bigger corpora, are kept out of the repos ## FAQ -We have an FAQ [here](https://github.com/leondz/garak/blob/main/FAQ.md). Reach out if you have any more questions! [leon@garak.ai](mailto:leon@garak.ai) +We have an FAQ [here](https://github.com/NVIDIA/garak/blob/main/FAQ.md). Reach out if you have any more questions! [leon@garak.ai](mailto:leon@garak.ai) Code reference documentation is at [garak.readthedocs.io](https://garak.readthedocs.io/en/latest/). diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 00000000..3a818bdf --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,24 @@ + ## Security + +NVIDIA is dedicated to the security and trust of our software products and services, including all source code repositories managed through our organization. + +If you need to report a security issue, please use the appropriate contact points outlined below. **Please do not report security vulnerabilities through GitHub.** + +## Reporting Potential Security Vulnerability in an NVIDIA Product + +To report a potential security vulnerability in any NVIDIA product: +- Web: [Security Vulnerability Submission Form](https://www.nvidia.com/object/submit-security-vulnerability.html) +- E-Mail: psirt@nvidia.com + - We encourage you to use the following PGP key for secure email communication: [NVIDIA public PGP Key for communication](https://www.nvidia.com/en-us/security/pgp-key) + - Please include the following information: + - Product/Driver name and version/branch that contains the vulnerability + - Type of vulnerability (code execution, denial of service, buffer overflow, etc.) + - Instructions to reproduce the vulnerability + - Proof-of-concept or exploit code + - Potential impact of the vulnerability, including how an attacker could exploit the vulnerability + +While NVIDIA currently does not have a bug bounty program, we do offer acknowledgement when an externally reported security issue is addressed under our coordinated vulnerability disclosure policy. Please visit our [Product Security Incident Response Team (PSIRT)](https://www.nvidia.com/en-us/security/psirt-policies/) policies page for more information. + +## NVIDIA Product Security + +For all security-related concerns, please visit NVIDIA's Product Security portal at https://www.nvidia.com/en-us/security diff --git a/docs/source/cliref.rst b/docs/source/cliref.rst index 4ef57234..23023180 100644 --- a/docs/source/cliref.rst +++ b/docs/source/cliref.rst @@ -3,7 +3,7 @@ CLI reference for garak :: - garak LLM vulnerability scanner v0.10.0 ( https://github.com/leondz/garak ) at 2024-10-31T09:41:09.763226 + garak LLM vulnerability scanner v0.10.0 ( https://github.com/NVIDIA/garak ) at 2024-10-31T09:41:09.763226 usage: python -m garak [-h] [--verbose] [--report_prefix REPORT_PREFIX] [--narrow_output] [--parallel_requests PARALLEL_REQUESTS] @@ -93,4 +93,4 @@ CLI reference for garak with JSON containing a prompt and target --interactive.py Launch garak in interactive.py mode - See https://github.com/leondz/garak + See https://github.com/NVIDIA/garak diff --git a/docs/source/configurable.rst b/docs/source/configurable.rst index 72a89063..86abb8ed 100644 --- a/docs/source/configurable.rst +++ b/docs/source/configurable.rst @@ -195,7 +195,7 @@ You can see the parameters for any given plugin using garak ``--plugin_info``. .. code-block:: text $ python -m garak --plugin_info probes.grandma.Win10 - garak LLM vulnerability scanner v0.10.0 ( https://github.com/leondz/garak ) at 2024-09-11T11:54:07.215242 + garak LLM vulnerability scanner v0.10.0 ( https://github.com/NVIDIA/garak ) at 2024-09-11T11:54:07.215242 Configured info on probes.grandma.Win10: description: Will the model yield a Windows 10 product key if we beg it to? DEFAULT_PARAMS: {'generations': 1} diff --git a/docs/source/contributing.generator.rst b/docs/source/contributing.generator.rst index 672d94db..f4f47383 100644 --- a/docs/source/contributing.generator.rst +++ b/docs/source/contributing.generator.rst @@ -230,7 +230,7 @@ Testing Now that the pieces for our generator are in place - a subclass of ``garak.generators.base.Generator``, with some customisation in the constructor, and an overridden ``_call_model()`` method, plus a ``DEFAULT_CLASS`` given at module level - we can start to test. -A good first step is to fire up the Python interpreter and try to import the module. Garak supports a specific range of tested Python versions (listed in `pyproject.toml `_, under the ``classifiers`` descriptor), so remember to use the right Python version for testing. +A good first step is to fire up the Python interpreter and try to import the module. Garak supports a specific range of tested Python versions (listed in `pyproject.toml `_, under the ``classifiers`` descriptor), so remember to use the right Python version for testing. .. code-block:: bash @@ -299,7 +299,7 @@ The next step is to try some integration tests - executing garak from the comman Add some of your own tests if there are edge-case behaviours, general validation, or other things in ``__init__()``, ``_call_model()``, and other new methods that can be checked. Plugin-specific tests should go into a new file, ``tests/generators/test_[modulename].py``. -If you want to see the full, live code for the Replicate garak generator, it's here: `garak/generators/replicate.py `_ . +If you want to see the full, live code for the Replicate garak generator, it's here: `garak/generators/replicate.py `_ . Done! ===== @@ -318,4 +318,4 @@ This tutorial covered a tool that takes text as input and produces text as outpu modality: dict = {"in": {"text"}, "out": {"text"}} -For an example of a multimodal model, check out LLaVa in `garak.generators.huggingface `_ . \ No newline at end of file +For an example of a multimodal model, check out LLaVa in `garak.generators.huggingface `_ . \ No newline at end of file diff --git a/docs/source/contributing.rst b/docs/source/contributing.rst index 32c0f61a..864bfdb8 100644 --- a/docs/source/contributing.rst +++ b/docs/source/contributing.rst @@ -15,7 +15,7 @@ Checking your contribution is within scope ``garak`` is a security toolkit rather than a content safety or bias toolkit. The project scope relates primarily to LLM & dialog system security. -This is a huge area, and you can get an idea of the kind of contributions that are in scope from our `FAQ _` and our `Github issues `_ page. +This is a huge area, and you can get an idea of the kind of contributions that are in scope from our `FAQ _` and our `Github issues `_ page. Connecting with the ``garak`` team & community @@ -24,7 +24,7 @@ Connecting with the ``garak`` team & community If you're going to contribute, it's a really good idea to reach out, so you have a source of help nearby, and so that we can make sure your valuable coding time is spent efficiently as a contributor. There are a number of ways you can reach out to us: -* GitHub discussions: ``_ +* GitHub discussions: ``_ * Twitter: ``_ * Discord: ``_ @@ -35,8 +35,8 @@ Checklist for contributing -------------------------- 1. Set up a `Github `_ account, if you don't have one already. We develop in the open and the public repository is the authoritative one. -1. Fork the ``garak`` repository - ``_ -1. Work out what you're doing. If it's from a good first issue (`see the list `_), drop a note on that issue so that we know you're working on it, and so that nobody else also starts working on it. +1. Fork the ``garak`` repository - ``_ +1. Work out what you're doing. If it's from a good first issue (`see the list `_), drop a note on that issue so that we know you're working on it, and so that nobody else also starts working on it. 1. Before you code anything: create a new branch for your work, e.g. ``git checkout -b feature/spicy_probe`` 1. Check out the rest of this page which includes links to detailed step-by-step guides to developing garak plugins 1. Code! diff --git a/docs/source/index.rst b/docs/source/index.rst index c420df1f..0c5a3357 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -38,7 +38,7 @@ Using garak how usage - FAQ + FAQ Advanced usage ^^^^^^^^^^^^^^ diff --git a/docs/source/reporting.calibration.rst b/docs/source/reporting.calibration.rst index c89b6af8..974a134b 100644 --- a/docs/source/reporting.calibration.rst +++ b/docs/source/reporting.calibration.rst @@ -21,7 +21,7 @@ We look for the following things when composing the model bag for calibrating ga * **Provider** - No more than two models in the bag from the same provider * **Openness** - Open weights models are easiest for us to survey, so we prefer to use those -One can read about which models are in the current calibration, and what configuration was used, from the source in `bag.md `_. +One can read about which models are in the current calibration, and what configuration was used, from the source in `bag.md `_. Z-scores ^^^^^^^^ diff --git a/docs/source/usage.rst b/docs/source/usage.rst index 8c9bf4aa..c0eee1cf 100644 --- a/docs/source/usage.rst +++ b/docs/source/usage.rst @@ -28,7 +28,7 @@ The standard pip version of ``garak`` is updated periodically. To get a fresher .. code-block:: console - python3 -m pip install -U git+https://github.com/leondz/garak.git@main + python3 -m pip install -U git+https://github.com/NVIDIA/garak.git@main For development: clone from `git` @@ -42,7 +42,7 @@ You can also clone the source and run ``garak`` directly. This works fine and is conda create --name garak "python>=3.10,<=3.12" conda activate garak - gh repo clone leondz/garak + gh repo clone NVIDIA/garak cd garak python3 -m pip install -r requirements.txt diff --git a/garak/analyze/report_avid.py b/garak/analyze/report_avid.py index 7b085807..77455e09 100644 --- a/garak/analyze/report_avid.py +++ b/garak/analyze/report_avid.py @@ -68,7 +68,7 @@ Reference( type="source", label="garak, an LLM vulnerability scanner", - url="https://github.com/leondz/garak", + url="https://github.com/NVIDIA/garak", ) ] report_template.reported_date = date.today() diff --git a/garak/analyze/templates/digest_about_z.jinja b/garak/analyze/templates/digest_about_z.jinja index 2c989a55..81ab13b5 100644 --- a/garak/analyze/templates/digest_about_z.jinja +++ b/garak/analyze/templates/digest_about_z.jinja @@ -6,7 +6,7 @@

About Z-scores in this analysis:

  • Positive Z-scores mean better than average, negative Z-scores mean worse than average.
    • -
  • "Average" is determined over a bag of models of varying sizes, updated periodically. Details
    • +
  • "Average" is determined over a bag of models of varying sizes, updated periodically. Details
  • For any probe, roughly two-thirds of models get a Z-score between -1.0 and +1.0.
  • The middle 10% of models score -0.125 to +0.125. This is labelled "competitive".
  • A Z-score of +1.0 means the score was one standard deviation better than the mean score other models achieved for this probe & metric
    • diff --git a/garak/buffs/base.py b/garak/buffs/base.py index d0b51b3b..bc9259e3 100644 --- a/garak/buffs/base.py +++ b/garak/buffs/base.py @@ -83,7 +83,7 @@ def buff( leave=False, ): # create one or more untransformed new attempts - # don't include the original attempt/prompt in the buffs: https://github.com/leondz/garak/issues/373 + # don't include the original attempt/prompt in the buffs: https://github.com/NVIDIA/garak/issues/373 new_attempts = [] new_attempts.append( self._derive_new_attempt(source_attempt, source_attempt.seq) diff --git a/garak/cli.py b/garak/cli.py index 602d9560..d0c2dafa 100644 --- a/garak/cli.py +++ b/garak/cli.py @@ -29,7 +29,7 @@ def main(arguments=None) -> None: _config.load_base_config() print( - f"garak {__description__} v{_config.version} ( https://github.com/leondz/garak ) at {_config.transient.starttime_iso}" + f"garak {__description__} v{_config.version} ( https://github.com/NVIDIA/garak ) at {_config.transient.starttime_iso}" ) import argparse @@ -37,7 +37,7 @@ def main(arguments=None) -> None: parser = argparse.ArgumentParser( prog="python -m garak", description="LLM safety & security scanning tool", - epilog="See https://github.com/leondz/garak", + epilog="See https://github.com/NVIDIA/garak", ) ## SYSTEM diff --git a/garak/generators/function.py b/garak/generators/function.py index e745d6c6..3a439c8a 100644 --- a/garak/generators/function.py +++ b/garak/generators/function.py @@ -56,7 +56,7 @@ class Single(Generator): DEFAULT_PARAMS = { "kwargs": {}, } - doc_uri = "https://github.com/leondz/garak/issues/137" + doc_uri = "https://github.com/NVIDIA/garak/issues/137" generator_family_name = "function" supports_multiple_generations = False diff --git a/garak/generators/ggml.py b/garak/generators/ggml.py index e47bcb70..c75a4d0e 100644 --- a/garak/generators/ggml.py +++ b/garak/generators/ggml.py @@ -8,7 +8,7 @@ or as the constructor parameter when instantiating LLaMaGgmlGenerator. Compatibility or other problems? Please let us know! - https://github.com/leondz/garak/issues + https://github.com/NVIDIA/garak/issues """ import logging diff --git a/garak/generators/huggingface.py b/garak/generators/huggingface.py index 1b72b86b..86a94bee 100644 --- a/garak/generators/huggingface.py +++ b/garak/generators/huggingface.py @@ -5,7 +5,7 @@ Not all models on HF Hub work well with pipelines; try a Model generator if there are problems. Otherwise, please let us know if it's still not working! - https://github.com/leondz/garak/issues + https://github.com/NVIDIA/garak/issues If you use the inference API, it's recommended to put your Hugging Face API key in an environment variable called HF_INFERENCE_TOKEN , else the rate limiting can @@ -350,13 +350,13 @@ def _call_model( ) else: raise TypeError( - f"Unsure how to parse 🤗 API response dict: {response}, please open an issue at https://github.com/leondz/garak/issues including this message" + f"Unsure how to parse 🤗 API response dict: {response}, please open an issue at https://github.com/NVIDIA/garak/issues including this message" ) elif isinstance(response, list): return [g["generated_text"] for g in response] else: raise TypeError( - f"Unsure how to parse 🤗 API response type: {response}, please open an issue at https://github.com/leondz/garak/issues including this message" + f"Unsure how to parse 🤗 API response type: {response}, please open an issue at https://github.com/NVIDIA/garak/issues including this message" ) def _pre_generate_hook(self): diff --git a/garak/report.py b/garak/report.py index c74d8c65..acd70396 100644 --- a/garak/report.py +++ b/garak/report.py @@ -103,7 +103,7 @@ def export(self): # TODO: add html format ac.Reference( type="source", label="garak, an LLM vulnerability scanner", - url="https://github.com/leondz/garak", + url="https://github.com/NVIDIA/garak", ) ] report_template.reported_date = date.today() diff --git a/pyproject.toml b/pyproject.toml index be8a4290..43bc1000 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -102,8 +102,8 @@ calibration = [ ] [project.urls] -"Homepage" = "https://github.com/leondz/garak" -"Bug Tracker" = "https://github.com/leondz/garak/issues" +"Homepage" = "https://github.com/NVIDIA/garak" +"Bug Tracker" = "https://github.com/NVIDIA/garak/issues" [project.scripts] garak = "garak.__main__:main"