Simple Tokenizers

This is a collection of tokenization implementations focused on transparency and readability.

Installation

You can install Tokenizers directly from GitHub.

pip install git+https://github.com/dakofler/simple_tokenizers.git

Usage

There are example-notebooks included that show how to use the package, see Examples.

Author

Daniel Kofler - AI Research Associate (dkofler@outlook.com)

License

MIT

Cheers,
Daniel