Wheelodex » Projects » code-tokenizers

code-tokenizers

View on PyPI — Reverse Dependencies (1)

0.0.5 code_tokenizers-0.0.5-py3-none-any.whl

Wheel Details

Project: code-tokenizers
Version: 0.0.5
Filename: code_tokenizers-0.0.5-py3-none-any.whl
Download: [link]
Size: 112642
MD5: b5be15d77a1bdf4376309d092e4f4d78
SHA256: b4ce9108c840370cc8dd2582dc9a45451722806c9c8a3226f783870bbd4a9074
Uploaded: 2023-02-06 03:46:50 +0000

dist-info

METADATA · WHEEL · RECORD · top_level.txt · entry_points.txt

METADATA

Metadata-Version: 2.1
Name: code-tokenizers
Version: 0.0.5
Summary: Aligning BPE and AST
Author: ncoop57
Author-Email: nacooper01[at]wm.edu
Home-Page: https://github.com/ncoop57/code_tokenizers
License: Apache Software License 2.0
Keywords: nbdev jupyter notebook python tokenizer bpe ast
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: License :: OSI Approved :: Apache Software License
Requires-Python: >=3.7
Requires-Dist: fastcore
Requires-Dist: pandas
Requires-Dist: transformers (<5)
Requires-Dist: tree-sitter (==0.20.1)
Requires-Dist: black[jupyter]; extra == "dev"
Requires-Dist: datasets (<3); extra == "dev"
Requires-Dist: nbdev (<3); extra == "dev"
Requires-Dist: twine; extra == "dev"
Provides-Extra: dev
Description-Content-Type: text/markdown
License-File: LICENSE
[Description omitted; length: 2627 characters]

WHEEL

Wheel-Version: 1.0
Generator: bdist_wheel (0.37.1)
Root-Is-Purelib: true
Tag: py3-none-any

RECORD

Path Digest Size
code_tokenizers/__init__.py sha256=S7u1lbuWmM3A3ajykBialmPoJUK6Jg-WmNqM-9OZFdk 22
code_tokenizers/_modidx.py sha256=Lsure95gz56AmleV9E3KAGYYiSxhWreWcMSskDblon4 3807
code_tokenizers/cli.py sha256=HFydce2N0lmrhKr320I6gfHLcQXfBRHxg69lM8SUiPs 277
code_tokenizers/core.py sha256=ppBTysvYw71sub27FPednufQPncuPHAfGucEeXErqV0 10103
code_tokenizers/helpers.py sha256=IB7NROXL2pVm8-_qNLipNi8bSGdHuNkE6HiY8m1b8WA 4259
code_tokenizers/grammars/__init__.py sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU 0
code_tokenizers/grammars/python/__init__.py sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU 0
code_tokenizers/grammars/python/examples/__init__.py sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU 0
code_tokenizers/grammars/python/examples/compound-statement-without-trailing-newline.py sha256=FcTiU7yDQreycfwNAQZndvLbXhdY_Z-W6pLvhERdV_8 38
code_tokenizers/grammars/python/examples/crlf-line-endings.py sha256=zOWqR-touD9wmoZ2bd_XWnFB09lwi-6twaRlrNhEF5Y 51
code_tokenizers/grammars/python/examples/mixed-spaces-tabs.py sha256=L-d7AebpibwSDdSYW74GvrFauE6PuBNQ_FO0gPXxzZs 74
code_tokenizers/grammars/python/examples/multiple-newlines.py sha256=Q7nqUPWQE7AgTqyRwU68zW6FB3gJI1Fvfw1AFGrcu2g 73
code_tokenizers/grammars/python/examples/python2-grammar-crlf.py sha256=QukBTt-uYDJsniZWjhzgTKJmYWQQuBrFvPCasUJGSb0 31929
code_tokenizers/grammars/python/examples/python2-grammar.py sha256=UAjQNXhR90QyHw7D6MZWghTV3YjiUcmoYa1TOxVX2So 30981
code_tokenizers/grammars/python/examples/python3-grammar-crlf.py sha256=PKWswVroLU9fOA5NTqgwhz0iWC6FMBScFM2iskAFN00 31722
code_tokenizers/grammars/python/examples/python3-grammar.py sha256=7dw4ivbXc5QZfVEnYmO5Co9yK8EsWDMIKaIrHOS6rjg 30784
code_tokenizers/grammars/python/examples/python3.8_grammar.py sha256=EljPKYctItnGGHBJTyBpwuvW2Jj2IJSpAMgZAcuiQw4 50403
code_tokenizers/grammars/python/examples/simple-statements-without-trailing-newline.py sha256=NbHO_av03f4XEHOu0D2QOvQnE6waq4VOlOTDrAnuQtw 16
code_tokenizers/grammars/python/examples/tabs.py sha256=bzxDW-wlS7r_afCDVoXwGNsj0h5sroy3eqqv8DCggHU 1236
code_tokenizers/grammars/python/examples/trailing-whitespace.py sha256=fWOvOnvwrV-FEAWlQNnA7mu5LAf5CWk2EEerBjKgnrM 58
code_tokenizers/grammars/python/test/__init__.py sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU 0
code_tokenizers/grammars/python/test/highlight/__init__.py sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU 0
code_tokenizers/grammars/python/test/highlight/keywords.py sha256=UgnP5iuu-kifIfAMpFeICgkIxU42E6tfc_sNVo49nWk 352
code_tokenizers/grammars/python/test/highlight/parameters.py sha256=ca65og9nUjEPklbLDFp6vymo4BrrSfSFfGrwNi7gwHI 99
code_tokenizers/grammars/python/test/highlight/pattern_matching.py sha256=n1wQ7WCIk__LwNheqUq2flD5A8Ja7imCrSn4BvgpiZg 1297
code_tokenizers/grammars/tree-sitter-python/__init__.py sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU 0
code_tokenizers/grammars/tree-sitter-python/examples/__init__.py sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU 0
code_tokenizers/grammars/tree-sitter-python/examples/compound-statement-without-trailing-newline.py sha256=FcTiU7yDQreycfwNAQZndvLbXhdY_Z-W6pLvhERdV_8 38
code_tokenizers/grammars/tree-sitter-python/examples/crlf-line-endings.py sha256=zOWqR-touD9wmoZ2bd_XWnFB09lwi-6twaRlrNhEF5Y 51
code_tokenizers/grammars/tree-sitter-python/examples/mixed-spaces-tabs.py sha256=L-d7AebpibwSDdSYW74GvrFauE6PuBNQ_FO0gPXxzZs 74
code_tokenizers/grammars/tree-sitter-python/examples/multiple-newlines.py sha256=Q7nqUPWQE7AgTqyRwU68zW6FB3gJI1Fvfw1AFGrcu2g 73
code_tokenizers/grammars/tree-sitter-python/examples/python2-grammar-crlf.py sha256=QukBTt-uYDJsniZWjhzgTKJmYWQQuBrFvPCasUJGSb0 31929
code_tokenizers/grammars/tree-sitter-python/examples/python2-grammar.py sha256=UAjQNXhR90QyHw7D6MZWghTV3YjiUcmoYa1TOxVX2So 30981
code_tokenizers/grammars/tree-sitter-python/examples/python3-grammar-crlf.py sha256=PKWswVroLU9fOA5NTqgwhz0iWC6FMBScFM2iskAFN00 31722
code_tokenizers/grammars/tree-sitter-python/examples/python3-grammar.py sha256=7dw4ivbXc5QZfVEnYmO5Co9yK8EsWDMIKaIrHOS6rjg 30784
code_tokenizers/grammars/tree-sitter-python/examples/python3.8_grammar.py sha256=EljPKYctItnGGHBJTyBpwuvW2Jj2IJSpAMgZAcuiQw4 50403
code_tokenizers/grammars/tree-sitter-python/examples/simple-statements-without-trailing-newline.py sha256=NbHO_av03f4XEHOu0D2QOvQnE6waq4VOlOTDrAnuQtw 16
code_tokenizers/grammars/tree-sitter-python/examples/tabs.py sha256=bzxDW-wlS7r_afCDVoXwGNsj0h5sroy3eqqv8DCggHU 1236
code_tokenizers/grammars/tree-sitter-python/examples/trailing-whitespace.py sha256=fWOvOnvwrV-FEAWlQNnA7mu5LAf5CWk2EEerBjKgnrM 58
code_tokenizers/grammars/tree-sitter-python/test/__init__.py sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU 0
code_tokenizers/grammars/tree-sitter-python/test/highlight/__init__.py sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU 0
code_tokenizers/grammars/tree-sitter-python/test/highlight/keywords.py sha256=UgnP5iuu-kifIfAMpFeICgkIxU42E6tfc_sNVo49nWk 352
code_tokenizers/grammars/tree-sitter-python/test/highlight/parameters.py sha256=ca65og9nUjEPklbLDFp6vymo4BrrSfSFfGrwNi7gwHI 99
code_tokenizers/grammars/tree-sitter-python/test/highlight/pattern_matching.py sha256=n1wQ7WCIk__LwNheqUq2flD5A8Ja7imCrSn4BvgpiZg 1297
code_tokenizers-0.0.5.dist-info/LICENSE sha256=xV8xoN4VOL0uw9X8RSs2IMuD_Ss_a9yAbtGNeBWZwnw 11337
code_tokenizers-0.0.5.dist-info/METADATA sha256=D_ehkxZInJi3XqdN44ZIwam7aYVRsyoCcAIRRdRrU8Q 3701
code_tokenizers-0.0.5.dist-info/WHEEL sha256=G16H4A3IeoQmnOrYV4ueZGKSjhipXx8zc8nu9FGlvMA 92
code_tokenizers-0.0.5.dist-info/entry_points.txt sha256=w9wZJ_AxThraECWww1QwbG36u80_NLIAoQvhcbsN9JE 52
code_tokenizers-0.0.5.dist-info/top_level.txt sha256=M8E0z-EttES-8PPbYv81DF68FrHWDZT0rMEEQRDvYJo 16
code_tokenizers-0.0.5.dist-info/RECORD — —

top_level.txt

code_tokenizers

entry_points.txt

[nbdev]
code_tokenizers = code_tokenizers._modidx:d