Tokenizers

HuggingFace Tokenizers in UE

                                                                                                             

Table of Contents

1. About the Plugin
2. Getting Started
   • Prerequisites
   • Setup
3. Guides and Tutorials
   • YouTube Tutorials
   • Wiki
4. Contributing
5. Credits and Licenses

About the Plugin

Tokenizers is an experimental plugin which enables developers to use tokenizers inside Unreal Engine's environment.

With this plugin you can:

• Initialize tokenizers from JSON blob or file configuration
• Encode and Decode text
• Use every feature in both C++ and Blueprints

(back to top)

Getting Started

To use this plugin, you'll need the C static library from Tokenizers-cpp. You can either download it directly from the Releases page of this repository or compile it yourself from the Tokenizers-cpp source.

Prerequisites

• OS: Windows - 64 bit
• UE: version 5.0 - 5.3

Setup

1. In your Unreal Engine project, create a Plugins folder if it doesn't already exist.
2. Navigate to the Releases page.
3. Download the source code for the release you want to use.
4. Extract the downloaded source code into the Plugins directory.
5. Navigate to Plugins/Tokenizers-UE5/Source/ThirdParty/TokenizersLibrary/Win64.
6. From the same release page, download tokenizers_c.lib and place it inside the Win64 folder.
7. Delete the placeholder file named PLACE STATIC LIB HERE from the Win64 folder.

(back to top)

Guides and Tutorials

YouTube Tutorials

• Setup and Basic Usage
• Working with Meta's Llama

Wiki

• https://github.com/gpt-3d/pytuech/wiki

(back to top)

Contributing

Want to contribute to? Awesome! Check out the contributing guidelines to get involved. Contributors are encouraged join to the community Discord server.

(back to top)

Credits and Licenses

This project is licensed under the MIT License, except for specific files noted below. See the LICENSE file for more information.

• Tokenizers-cpp:
  • Source: Tokenizers-cpp GitHub
  • License for tokenizers_c.h: Apache License 2.0

This project is based on MLC-AI's C/C++ implementation of HuggingFace's Tokenizers library.

(back to top)