Skip to content

Add bundle support#9

Closed
kvkenyon wants to merge 1 commit into
mainfrom
rewrite_archive_and_fix_60d_reports
Closed

Add bundle support#9
kvkenyon wants to merge 1 commit into
mainfrom
rewrite_archive_and_fix_60d_reports

Conversation

@kvkenyon
Copy link
Copy Markdown
Owner

@kvkenyon kvkenyon commented Dec 30, 2025

Note

Introduces bundle downloads and refactors historical retrieval to support multi-file archives and MIS documents.

  • New: ERCOTArchiveBundle with bundles(), one_bundle(), and all() to fetch and extract large archive bundles
  • Archive refactor: ERCOTArchive now uses ERCOT.make_request; fetch_historical returns a dict of DataFrames and parses nested zip-of-zips; ArchiveLink converted to @define
  • Client: adds ERCOTBase.make_request for authenticated httpx calls reused by archive/bundle helpers
  • Documents/MIS: new report IDs and helpers (get_60d_sced_disclosure, get_system_wide_actuals_docs, get_state_estimator_load_report, get_state_estimator_dc_ties_flows_report)
  • API changes: rename get_dc_tie_flowsget_se_dc_tie_flows; replace get_total_generation with get_se_load; get_system_wide_actuals and 60‑day DAM/SCED disclosures now aggregate/return parsed MIS/archive data (DAM returns mapped dict of files)
  • Tests/Examples: update tests for new signatures/behaviors; add examples/demo.py and notebooks
  • Build: add returns dep and ty in dev; minor README/typos and small transform cleanup

Written by Cursor Bugbot for commit 937a0f9. This will update automatically on new commits. Configure here.

Comment thread tinygrid/ercot/archive.py
"""
all_dataframes: list[pd.DataFrame] = []
for bundle_link in bundles.links:
dfs = self.one(bundle_link)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Method self.one does not exist, should be self.one_bundle

The all method calls self.one(bundle_link) but no method named one exists in ERCOTArchiveBundle. The correct method name is one_bundle (defined at line 352). This will raise an AttributeError at runtime when all is called.

Fix in Cursor Fix in Web

Comment thread tinygrid/ercot/archive.py
all_dataframes: list[pd.DataFrame] = []
for bundle_link in bundles.links:
dfs = self.one(bundle_link)
all_dataframes.extend(dfs)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using extend on DataFrame instead of append

The all method calls all_dataframes.extend(dfs) where dfs is the return value of one_bundle, which returns a single pd.DataFrame. Calling extend on a DataFrame iterates over its column names (strings), not the DataFrame itself. This would incorrectly populate all_dataframes with column name strings instead of DataFrames. The code likely intended to use append instead of extend.

Fix in Cursor Fix in Web

Comment thread tinygrid/ercot/archive.py
io.BytesIO(inner_file.read()),
inner_name,
)
)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Double context manager closes ZipFile prematurely

The _extract_multiple_files_from_zip method wraps the passed-in zfile parameter in a with zfile as zf: block, which closes the ZipFile when exiting. However, the caller at line 226-228 already has its own with ZipFile(bytes_io) as zfile: context manager. This results in the ZipFile being closed twice. While Python's ZipFile handles double-close gracefully, this pattern is error-prone and the inner with statement is unnecessary.

Fix in Cursor Fix in Web

df = pd.read_csv(io.BytesIO(content), compression="zip")
results.append(df)

df = pd.concat(results)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pd.concat on empty list raises ValueError

When keep_docs is empty after date filtering (no documents match the date range criteria), the results list remains empty. Calling pd.concat(results) on an empty list raises a ValueError: No objects to concatenate. This affects get_system_wide_actuals_docs and _get_reports. Unlike similar patterns earlier in the file that return an empty DataFrame when no docs are found, these methods proceed to concatenate without checking if results is empty.

Additional Locations (1)

Fix in Cursor Fix in Web

@kvkenyon kvkenyon closed this Jan 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant