Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Give error when initializing tokenizer with too high stride #1306

Merged
merged 5 commits into from
Jul 28, 2023
Merged

Give error when initializing tokenizer with too high stride #1306

merged 5 commits into from
Jul 28, 2023

Commits on Jul 27, 2023

  1. Split get_n_added_tokens into separate method

    @boyleconnor
    boyleconnor committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    ec31bd2 View commit details
    Browse the repository at this point in the history

  2. Modify TokenizerImpl.with_truncation() to raise an error if given b… 

    …ad parameters
    @boyleconnor
    boyleconnor committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    ffce311 View commit details
    Browse the repository at this point in the history

  3. Return Python error if tokenizer.with_truncation() fails

    @boyleconnor
    boyleconnor committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    e812b42 View commit details
    Browse the repository at this point in the history

  4. Add dummy variable assignment for no_truncation() case

    @boyleconnor
    boyleconnor committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    06261db View commit details
    Browse the repository at this point in the history

  5. Unrelated fmt fix.

    @Narsil
    Narsil committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    16af73a View commit details
    Browse the repository at this point in the history