| Metadata-Version: |
2.1 |
| Name: |
bpeasy |
| Version: |
0.1.2 |
| Summary: |
Fast bare-bones BPE for modern tokenizer training |
| Author-Email: |
<gautier.dagan[at]ed.ac.uk> |
| License: |
MIT |
| Keywords: |
tokenizer,tokenization,bpe |
| Classifier: |
Programming Language :: Rust |
| Classifier: |
Programming Language :: Python :: Implementation :: CPython |
| Classifier: |
Programming Language :: Python :: Implementation :: PyPy |
| Classifier: |
Programming Language :: Python :: 3.8 |
| Classifier: |
Programming Language :: Python :: 3.9 |
| Classifier: |
Programming Language :: Python :: 3.10 |
| Classifier: |
Programming Language :: Python :: 3.11 |
| Classifier: |
Programming Language :: Python :: 3.12 |
| Classifier: |
License :: OSI Approved :: MIT License |
| Requires-Python: |
>=3.8 |
| Requires-Dist: |
tiktoken (>=0.4.0) |
| Requires-Dist: |
pytest; extra == "dev" |
| Requires-Dist: |
pytest-cov; extra == "dev" |
| Requires-Dist: |
black; extra == "dev" |
| Requires-Dist: |
tokenizers; extra == "dev" |
| Requires-Dist: |
tqdm; extra == "dev" |
| Provides-Extra: |
dev |
| Description-Content-Type: |
text/markdown; charset=UTF-8; variant=GFM |
| License-File: |
LICENSE |