feat: use ListLike instead of list in path type hints#8322
Open
N-S8990 wants to merge 2 commits into
Open
Conversation
Change type annotations from list[PathLike] to Sequence_[PathLike] in from_csv, from_json, from_parquet, and from_text methods. This allows callers to pass tuples and other sequence types without type checker errors. Closes huggingface#5354
Change path_or_paths type annotation from list[PathLike] to ListLike[PathLike] in from_csv, from_json, from_parquet, and from_text. list is invariant, so passing a tuple of paths fails mypy despite working at runtime. ListLike (defined in utils/typing.py) accepts both list and tuple and is already used elsewhere in this file. Type-annotation-only change, no runtime behavior difference. Verified with full test suite (151 passed) and before/after mypy diff. Closes huggingface#5354
N-S8990
changed the title
ListLike instead of list in path type hints
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Changed type annotations from
list[PathLike]toListLike[PathLike]infrom_csv,from_json,from_parquet, andfrom_text.listtype hints are invariant — passing atupleof paths fails mypy eventhough it works fine at runtime.
ListLike(defined insrc/datasets/utils/typing.py) accepts bothlistandtuple, and is alreadyused elsewhere in this file (see
_getitem), so this follows an existingproject convention rather than introducing a new type alias.
Changes
src/datasets/arrow_dataset.py: changed 4 parameter type hints(
from_csv,from_json,from_parquet,from_text) and updated theirdocstrings to match (
path-like or list of path-like→path-like or Sequence of path-like).Verification
path_or_pathsis passed through unchanged to the underlying*DatasetReaderclasses.(5 unrelated failures are pre-existing, caused by a missing
pytest-datadirfixture in compression-writer tests).reports a type mismatch between these methods and the internal
*DatasetReaderconstructors (CsvDatasetReader,JsonDatasetReader,ParquetDatasetReader,TextDatasetReader), which are typed to acceptlist | dictonly. Confirmed via diff that this mismatch is pre-existingon
main(it appeared even when usingSequence_[PathLike]beforeswitching to
ListLike) — it is out of scope for this PR and would need aseparate fix to the
io/*.pyreader constructor signatures.Related Issue
Closes #5354
AI Disclosure
AI-assisted (Hermes Agent) — reviewed and tested manually by the author.