Why is it necessary to include b" " before the prompt for tokenization? #545
Unanswered
viniciusarruda
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
How you figured out that it was needed to include a
b" "
in front of the prompt for tokenization to be similar to Meta tokenization?I think this might solve an issue in the original
llama.cpp
repo.Beta Was this translation helpful? Give feedback.
All reactions