Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tokenizer support #148

Closed
Dreaming-Codes opened this issue Nov 17, 2023 · 3 comments
Closed

Tokenizer support #148

Dreaming-Codes opened this issue Nov 17, 2023 · 3 comments
Labels
out of scope Requests which are not related to OpenAI API

Comments

@Dreaming-Codes
Copy link

Whenever there are updates to this library, it causes compatibility issues with tiktoken-rs. It could be better to flip the script and have this library depend on tiktoken-rs instead, especially since this library often needs changes to keep up with OpenAI API updates.

zurawiki/tiktoken-rs#50
@64bit
Copy link
Owner

64bit commented Nov 25, 2023

Hey thanks for the issue and description. I dont think this fits into scope of async-openai.

@Dreaming-Codes
Copy link
Author

I see your point, but carrying this out seems impractical, considering this library is updated with every OpenAI release. Unless I'm missing something, I think isn't possible to depend on a specific range of library versions.

@64bit 64bit added the out of scope Requests which are not related to OpenAI API label Jan 9, 2024
@64bit
Copy link
Owner

64bit commented Mar 2, 2024

Sorry I don't have further input on this - you and zurawiki are the right subject matter experts on this.

@64bit 64bit closed this as completed May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
out of scope Requests which are not related to OpenAI API
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@64bit @Dreaming-Codes