Add timeout and HEAD request validation to get_dataframe

[Copilot]

get_dataframe can hang indefinitely when downloads take too long, and bad URLs aren't detected until after a full download attempt.

Changes

• Timeout on download: Adds a configurable timeout parameter (default 30s) to get_dataframe, passed through to get_gzipped_url() and its underlying requests.get() call, so downloads respect the timeout.
• HEAD request pre-check: Sends requests.head() before downloading to fail fast on bad URLs (404s, etc.) via raise_for_status().
• Tests: Added test_get_dataframe_bad_url using mocks. Updated VCR cassette to include the HEAD interaction.

Usage

from atcf_data_parser import get_dataframe

# Default 30s timeout
df = get_dataframe("https://ftp.nhc.noaa.gov/atcf/aid_public/aep182023.dat.gz")

# Custom timeout
df = get_dataframe(url, timeout=10)

# Bad URL raises requests.HTTPError immediately
df = get_dataframe("https://ftp.nhc.noaa.gov/atcf/aid_public/nonexistent.dat.gz")

[Copilot]

Pull request overview

This PR updates get_dataframe to fail faster and avoid indefinite hangs by adding a pre-download URL validation step and a configurable parsing timeout.

Changes:

• Add a timeout parameter to get_dataframe and run pd.read_fwf in an executor with a timeout.
• Perform a HEAD request pre-check (raise_for_status) before downloading.
• Add tests for timeout and bad-URL behavior, and update the VCR cassette to include the new HEAD interaction.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 4 comments.

File	Description
atcf_data_parser/init.py	Adds timeout to get_dataframe, performs HEAD validation, and wraps parsing with an executor timeout.
tests/test_get.py	Adds mocked unit tests for timeout behavior and bad URL handling.
tests/cassettes/test_get_dataframe.yaml	Records the additional HEAD request interaction for VCR.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[palewire]

@copy, does the read_fwf need the timeout and future if the actual failure is happening when the URL's data in fetched? Could we simply do the probe and the timeout there, since the data will be in memory by the time it gets to pandas? Also: Make sure all linting tests pass before committing.

[palewire]

@copilot, does the read_fwf need the timeout and future if the actual failure is happening when the URL's data in fetched? Could we simply do the probe and the timeout there, since the data will be in memory by the time it gets to pandas? Also: Make sure all linting tests pass before committing.

[Copilot]

@copilot, does the read_fwf need the timeout and future if the actual failure is happening when the URL's data in fetched? Could we simply do the probe and the timeout there, since the data will be in...

You're right — since the data is fully in memory before read_fwf runs, the executor/future pattern is unnecessary. Removed concurrent.futures, ThreadPoolExecutor, and the timeout test in cf51843. The function now calls pd.read_fwf directly. All linting (flake8, black, isort) and tests pass.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.