Question about tokeniser codebook lengths #60

Open

opened

on Nov 13, 2025

Hi! Is there a table of tokeniser codebook lengths per modality? A clean table would help others who want to build off of the pre-trained AION tokenisers

Metadata

Assignees

No one assigned

Labels

No labels

No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests