Wheelodex » Projects » Japanese-BPE-Tokenizer

Japanese-BPE-Tokenizer

View on PyPI — Reverse Dependencies (0)

0.1.2

Japanese_BPE_Tokenizer-0.1.2-py3-none-any.whl

Wheel Details

Project:	Japanese-BPE-Tokenizer
Version:	0.1.2
Filename:	Japanese_BPE_Tokenizer-0.1.2-py3-none-any.whl
Download:	[link]
Size:	5982
MD5:	`6077d47b40cbcccd00dbed5e4adcb514`
SHA256:	`338c080206eb94b52e1eba9251d4617b6dd9e5e5a1a43441f94bb694feb60b03`
Uploaded:	2021-06-05 13:48:44 +0000

dist-info

METADATA · WHEEL · RECORD · top_level.txt

METADATA

Metadata-Version:	2.1
Name:	Japanese-BPE-Tokenizer
Version:	0.1.2
Summary:	Mecab-based BPE toknizer for Japanese text
Author:	Shiro T.
Home-Page:	http://github.com/stsuchi/Japanese-BPE-Tokenizer
License:	MIT
Requires-Dist:	transformers
Requires-Dist:	fugashi
Requires-Dist:	unidic-lite
Description-Content-Type:	text/markdown

[Description omitted; length: 837 characters]

WHEEL

Wheel-Version:	1.0
Generator:	bdist_wheel (0.36.2)
Root-Is-Purelib:	true
Tag:	py3-none-any

RECORD

Path	Digest	Size
jpn_bpe_tokenizer/__init__.py	sha256=CQbdmrcd0SK6lXT5n3vUEZxzCQZMmnWDCRRsUO7my30	95
jpn_bpe_tokenizer/mecab_bpe_tokenizer.py	sha256=o5wd95QLGQmCQwMdYhm68v03TXkBPqKI3bKfMv0aYw0	6621
jpn_bpe_tokenizer/pre_tokenizers.py	sha256=iUIMH5iGRDCWYPEaBFMRZ5n9w0W9tuuPqKr6DkdqVNI	1626
jpn_bpe_tokenizer/trainer.py	sha256=vjSFdJJXl7ERy3TCadZr-8AVVbVGdpd4j1nPMzQvOBk	792
Japanese_BPE_Tokenizer-0.1.2.dist-info/LICENSE	sha256=WfetpJ1591TGgEgmWIPUvrIDVsKxEVg2R-Lzn9rqR3s	1065
Japanese_BPE_Tokenizer-0.1.2.dist-info/METADATA	sha256=YIMDQ5_PJJEdNHWEMzbr4PuQ6v7UGFhL-lL3We_NtpE	1182
Japanese_BPE_Tokenizer-0.1.2.dist-info/WHEEL	sha256=OqRkF0eY5GHssMorFjlbTIq072vpHpF60fIQA6lS9xA	92
Japanese_BPE_Tokenizer-0.1.2.dist-info/top_level.txt	sha256=19Bn_IURfTB-v-hB4UqCl6-v_toG3g9vhz9KnC1vL-U	18
Japanese_BPE_Tokenizer-0.1.2.dist-info/RECORD	—	—

top_level.txt

jpn_bpe_tokenizer