Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

resolve GPU tokenize padding issue #13

Open
WeihaoGe1009 opened this issue Aug 31, 2024 · 1 comment
Open

resolve GPU tokenize padding issue #13

WeihaoGe1009 opened this issue Aug 31, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@WeihaoGe1009
Copy link
Collaborator

Currently, when running the code on a GPU cluster, it doesn't recognize padding.

I got an error of:

Asking to pad, but the tokenizer does not have a padding token. 
Please select a token to use as 'pad_token'
'(tokenizer.pad_token = tokenizer.eos_token e.g.)' 
or add a new pad token via 'tokenizer.add_special_tokens({'pad_token': '[PAD]'})'.

This issue doesn't occur when I was running on the CPU-only cluster.

On GPU the following codes got ignored:

tokenizer.pad_token = tokenizer.eos_token

or

if tokenizer.pad_token is None:
    tokenizer.add_special_tokens({'pad_token': '[PAD]'})
    model.resize_token_embeddings(len(tokenizer))
@WeihaoGe1009 WeihaoGe1009 added the bug Something isn't working label Aug 31, 2024
@WeihaoGe1009
Copy link
Collaborator Author

WeihaoGe1009 commented Sep 3, 2024

try without padding

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant