Metadata-Version: |
2.1 |
Name: |
bpeasy |
Version: |
0.1.2 |
Summary: |
Fast bare-bones BPE for modern tokenizer training |
Author-Email: |
<gautier.dagan[at]ed.ac.uk> |
License: |
MIT |
Keywords: |
tokenizer,tokenization,bpe |
Classifier: |
Programming Language :: Rust |
Classifier: |
Programming Language :: Python :: Implementation :: CPython |
Classifier: |
Programming Language :: Python :: Implementation :: PyPy |
Classifier: |
Programming Language :: Python :: 3.8 |
Classifier: |
Programming Language :: Python :: 3.9 |
Classifier: |
Programming Language :: Python :: 3.10 |
Classifier: |
Programming Language :: Python :: 3.11 |
Classifier: |
Programming Language :: Python :: 3.12 |
Classifier: |
License :: OSI Approved :: MIT License |
Requires-Python: |
>=3.8 |
Requires-Dist: |
tiktoken (>=0.4.0) |
Requires-Dist: |
pytest; extra == "dev" |
Requires-Dist: |
pytest-cov; extra == "dev" |
Requires-Dist: |
black; extra == "dev" |
Requires-Dist: |
tokenizers; extra == "dev" |
Requires-Dist: |
tqdm; extra == "dev" |
Provides-Extra: |
dev |
Description-Content-Type: |
text/markdown; charset=UTF-8; variant=GFM |
License-File: |
LICENSE |