Skip to content

bpe Python Bindings? #61

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
coopslarhette opened this issue Apr 28, 2025 · 1 comment
Open

bpe Python Bindings? #61

coopslarhette opened this issue Apr 28, 2025 · 1 comment

Comments

@coopslarhette
Copy link

Amazing work on bpe!

Wanted to see if there were any known Python bindings a la the snippets from tiktoken below?

import tiktoken
enc = tiktoken.get_encoding("o200k_base")
assert enc.decode(enc.encode("hello world")) == "hello world"

# To get the tokeniser corresponding to a specific model in the OpenAI API:
enc = tiktoken.encoding_for_model("gpt-4o")
@micycle1
Copy link

micycle1 commented May 8, 2025

Bindings here: https://github.com/micycle1/pybpe_openai.
I'll create a 0.3.0 release on (non-test) PyPi when it gets finalised in this repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants