Skip to content

Add skip_origin_metadata flag to optimize streaming startup time#8291

Open
yuxin00j wants to merge 6 commits into
huggingface:mainfrom
yuxin00j:fix-8197
Open

Add skip_origin_metadata flag to optimize streaming startup time#8291
yuxin00j wants to merge 6 commits into
huggingface:mainfrom
yuxin00j:fix-8197

Conversation

@yuxin00j

@yuxin00j yuxin00j commented Jun 30, 2026

Copy link
Copy Markdown

Fix #8197
This PR introduces the skip_origin_metadata kwarg to skip costly origin metadata fetching (ETags/mtime) when it's not needed, improving streaming initialization latency.

Changes

  • Kwarg Plumbing: Routes the skip_origin_metadata kwarg from load_dataset(...) safely down to DatasetBuilder while stripping it securely from kwargs (via .pop()) so it's not leaked into underlying configuration classes that don't expect it (such as CsvConfig()).
  • Streaming Validations:
    • Emits a warning when skip_origin_metadata=True is passed to load_dataset without streaming=True.
    • Disables download_and_prepare() entirely when builder._skip_origin_metadata is true by raising a ValueError("This function is not intended for streaming.").
  • Test Coverage: Added robust test coverage spanning tests/test_data_files.py and tests/test_load.py:
    • test_DataFilesList_from_patterns_skip_origin_metadata
    • test_DataFilesList_from_patterns_skip_origin_metadata_raises_FileNotFoundError
    • test_load_dataset_builder_skip_origin_metadata (covers kwarg mapping & ValueError)
    • test_load_dataset_skip_origin_metadata_warning

@yuxin00j yuxin00j marked this pull request as ready for review June 30, 2026 02:01
Comment thread src/datasets/load.py
Comment thread src/datasets/load.py
Comment thread src/datasets/load.py
Comment thread src/datasets/load.py
Comment thread src/datasets/load.py
Comment thread src/datasets/load.py
Comment thread src/datasets/load.py
Comment thread src/datasets/load.py
Comment thread src/datasets/builder.py
Comment thread src/datasets/builder.py
Comment thread src/datasets/load.py Outdated
data_dir=data_dir,
data_files=data_files,
cache_dir=cache_dir,
skip_origin_metadata=skip_origin_metadata,

@zhixiangli zhixiangli Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: why not skip_origin_metadata=config_kwargs.get("skip_origin_metadata", False)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in the latest commit! I've inlined the .get() call here and then properly extracted it with .pop() when instantiating the builder later to prevent kwargs leakage.

Comment thread src/datasets/builder.py Outdated
Comment thread tests/test_load.py
yuxin00j

This comment was marked as outdated.

@yuxin00j

Copy link
Copy Markdown
Author

Hi @lhoestq, could you take a look at this PR? Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request / Discussion: Skip origin_metadata fetch when streaming=True?

2 participants