Feature/ercot data access gaps

CLAUDE.md

@@ -6,6 +6,35 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 Tiny Grid is a Python SDK for accessing electricity grid data from US Independent System Operators (ISOs). Currently supports 100+ ERCOT endpoints with plans to support other ISOs (CAISO, PJM, NYISO, ISO-NE, MISO, SPP).
 
+## ERCOT Developer Resources
+
+- **Developer Portal**: https://developer.ercot.com
+- **API Explorer**: https://apiexplorer.ercot.com (register for subscription key)
+- **OpenAPI Specs**: https://github.com/ercot/api-specs
+- **API Base URL**: https://api.ercot.com/api/public-reports
+
+## ERCOT API Data Availability (Important Limitations)
+
+1. **API Data Start Date**: December 11, 2023
+   - The REST API only contains data from this date forward
+   - For earlier data, use MIS document downloads or historical archives
+   - Historical archives extend 7+ years back
+
+2. **Data Delay**: Approximately 1 hour
+   - Real-time data is NOT truly real-time
+   - There is ~1 hour delay from actual grid operations to API availability
+
+3. **Geographic Restriction**: US IP addresses only
+   - API blocks requests from non-US IP addresses
+   - Users outside the US need a VPN or US-based proxy
+
+4. **Rate Limit**: 30 requests per minute
+   - Exceeding this limit results in HTTP 429 errors
+   - SDK includes built-in rate limiter (enabled by default)
+
+5. **Bulk Download Limit**: 1,000 documents per request
+   - Archive bulk downloads limited to 1,000 files per POST request
+
 ## Architecture
 
 The project follows a three-layer architecture:
@@ -95,6 +124,7 @@ uv run pyright
 The `ERCOT` class is the main entry point. It wraps pyercot's generated client with:
 - Automatic token management via `ERCOTAuth`
 - Retry logic with exponential backoff (configurable via `max_retries`)
+- Rate limiting (30 requests/minute, configurable via `rate_limit_enabled` and `requests_per_minute`)
 - Context manager support for resource cleanup
 - Pagination handling (page_size defaults to 10000)
 
@@ -108,6 +138,9 @@ auth = ERCOTAuth(ERCOTAuthConfig(
     subscription_key="key"
 ))
 ercot = ERCOT(auth=auth, max_retries=3)
+
+# Disable rate limiting for testing (not recommended for production)
+ercot = ERCOT(auth=auth, rate_limit_enabled=False)
 ```
 
 ### Data Fetching Patterns

README.md

@@ -66,11 +66,13 @@ ercot = ERCOT()
 # Get actual system load by weather zone
 load = ercot.get_load(start="today", by="weather_zone")
 
-# Get wind generation forecast
+# Get wind generation forecast (hourly or 5-minute resolution)
 wind = ercot.get_wind_forecast(start="today")
+wind_5min = ercot.get_wind_forecast(start="today", resolution="5min")
 
 # Get solar generation forecast  
 solar = ercot.get_solar_forecast(start="today")
+solar_5min = ercot.get_solar_forecast(start="today", resolution="5min", by_region=True)
 
 # Get load forecast
 load_forecast = ercot.get_load_forecast_by_weather_zone(
@@ -79,6 +81,33 @@ load_forecast = ercot.get_load_forecast_by_weather_zone(
 )
 ```
 
+### Dashboard Data (No Authentication Required)
+
+Access real-time grid status from ERCOT's public dashboard:
+
+```python
+from tinygrid import ERCOT
+
+ercot = ERCOT()
+
+# Get current grid status
+status = ercot.get_status()
+print(f"Condition: {status.condition}")
+print(f"Load: {status.current_load:,.0f} MW")
+print(f"Reserves: {status.reserves:,.0f} MW")
+
+# Get current fuel mix
+fuel_mix = ercot.get_fuel_mix()
+
+# Get renewable generation status
+renewables = ercot.get_renewable_generation()
+print(f"Wind: {renewables.wind_mw:,.0f} MW")
+print(f"Solar: {renewables.solar_mw:,.0f} MW")
+
+# Get supply/demand data
+supply_demand = ercot.get_supply_demand()
+```
+
 ### Historical Yearly Data
 
 Access complete yearly historical data from ERCOT's MIS document system:
@@ -128,6 +157,50 @@ forecast = ercot.get_load_forecast_by_weather_zone(
 )
 ```
 
+### Polling for Real-Time Updates
+
+For continuous data monitoring, use the polling utilities:
+
+```python
+from tinygrid import ERCOT
+from tinygrid.ercot import ERCOTPoller, poll_latest
+
+ercot = ERCOT(auth=auth)
+
+# Simple generator pattern
+for df in poll_latest(ercot, ercot.get_spp, interval=60, max_iterations=10):
+    print(f"Latest prices: {len(df)} rows")
+
+# Using ERCOTPoller with callback
+poller = ERCOTPoller(client=ercot, interval=60)
+
+def handle_data(result):
+    if result.success:
+        print(f"Got {len(result.data)} rows at {result.timestamp}")
+
+poller.poll(method=ercot.get_spp, callback=handle_data, max_iterations=10)
+```
+
+### EIA Integration (Supplementary Data)
+
+Access ERCOT data via the EIA API for historical data before December 2023:
+
+```python
+from tinygrid.ercot import EIAClient
+
+# Requires free API key from https://www.eia.gov/opendata/register.php
+eia = EIAClient(api_key="your-eia-key")
+
+# Get hourly demand
+demand = eia.get_demand(start="2022-01-01", end="2022-01-07")
+
+# Get generation by fuel type
+gen_by_fuel = eia.get_generation_by_fuel(start="2022-01-01")
+
+# Get net interchange
+interchange = eia.get_interchange(start="2022-01-01")
+```
+
 See [`examples/ercot_demo.ipynb`](examples/ercot_demo.ipynb) for complete examples.
 
 ## Unified API Methods
@@ -148,10 +221,28 @@ These methods provide a simpler interface with automatic routing, date parsing,
 
 | Method | Description |
 |--------|-------------|
-| `get_wind_forecast()` | Wind power forecast (system-wide or by region) |
-| `get_solar_forecast()` | Solar power forecast (system-wide or by region) |
+| `get_wind_forecast()` | Wind power forecast (hourly or 5-minute, system-wide or by region) |
+| `get_solar_forecast()` | Solar power forecast (hourly or 5-minute, system-wide or by region) |
 | `get_load()` | Actual system load by weather or forecast zone |
 
+### System-Wide Data Methods
+
+| Method | Description |
+|--------|-------------|
+| `get_dc_tie_flows()` | DC tie flow data (connections to Eastern Interconnection/Mexico) |
+| `get_total_generation()` | Total ERCOT system generation |
+| `get_system_wide_actuals()` | System-wide actual values per SCED interval |
+
+### Dashboard Methods (No Auth Required)
+
+| Method | Description |
+|--------|-------------|
+| `get_status()` | Grid operating condition, load, capacity, reserves |
+| `get_fuel_mix()` | Current generation by fuel type |
+| `get_renewable_generation()` | Wind and solar output with forecasts |
+| `get_supply_demand()` | Hourly supply/demand data |
+| `get_daily_prices()` | Daily price summary |
+
 ### Direct Endpoint Methods
 
 For full control, 100+ low-level endpoint methods are available:
@@ -177,6 +268,9 @@ For full control, 100+ low-level endpoint methods are available:
 - **Location filtering**: Filter by load zones, trading hubs, or specific settlement points
 - **Market selection**: Choose between real-time and day-ahead markets
 - **Standardized columns**: Consistent column names across all endpoints
+- **Rate limiting**: Built-in rate limiter (30 req/min) to prevent API throttling
+- **5-minute resolution**: Wind and solar forecasts available in 5-minute granularity
+- **Retry with backoff**: Automatic retry for transient failures
 
 ## ERCOT API Credentials
 
@@ -188,6 +282,20 @@ Authentication is required for some endpoints. To get credentials:
 
 **Note:** Dashboard methods (`get_status()`, `get_fuel_mix()`, etc.) do not require authentication.
 
+## API Data Availability
+
+Important limitations to be aware of:
+
+| Limitation | Details |
+|------------|---------|
+| **API Data Start Date** | December 11, 2023 - use archive API or EIA for earlier data |
+| **Data Delay** | ~1 hour from real-time to API availability |
+| **Geographic Restriction** | US IP addresses only (VPN required for international) |
+| **Rate Limit** | 30 requests per minute (built-in rate limiter enforces this) |
+| **Bulk Download Limit** | 1,000 documents per archive request |
+
+For data before December 2023, use `get_rtm_spp_historical()`, `get_dam_spp_historical()`, or the EIA integration.
+
 ## Available ERCOT Endpoints
 
 Direct access to 100+ ERCOT endpoints organized by category:
@@ -247,20 +355,22 @@ tinygrid/
 ├── tinygrid/              # SDK layer
 │   ├── ercot/             # ERCOT client package
 │   │   ├── __init__.py    # Main ERCOT class (combining mixins)
-│   │   ├── client.py      # ERCOTBase with auth, retry, pagination
+│   │   ├── client.py      # ERCOTBase with auth, retry, pagination, rate limiting
 │   │   ├── endpoints.py   # Low-level pyercot wrappers (~100 methods)
 │   │   ├── api.py         # High-level unified API methods
 │   │   ├── archive.py     # Historical archive access
 │   │   ├── dashboard.py   # Public dashboard methods (no auth)
 │   │   ├── documents.py   # MIS document fetching
+│   │   ├── eia.py         # EIA API integration for supplementary data
+│   │   ├── polling.py     # Real-time polling utilities
 │   │   └── transforms.py  # Data filtering/transformation utilities
 │   ├── auth/              # Authentication handling
 │   ├── constants/         # Market types, location enums, endpoint mappings
-│   ├── utils/             # Date parsing, timezone handling, decorators
+│   ├── utils/             # Date parsing, timezone, decorators, rate limiting
 │   └── errors.py          # Error types
 ├── pyercot/               # Auto-generated ERCOT API client (from OpenAPI spec)
 ├── examples/              # Usage examples
-└── tests/                 # Test suite (505 tests)
+└── tests/                 # Test suite (746 tests, 95% coverage)
 ```
 
 ## Development