text dataset is not shuffled before tokenization #152

Open

opened

on May 28, 2024

not an issue for our stories-* suite as stories dataset is shuffled, but could be an issue for other datasets, because of concatenation during tokenization

Metadata

Assignees

No one assigned

Labels

No labels

No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests