Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 32 additions & 2 deletions docs/forecasts.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,14 +64,14 @@ rows roll into the correct calendar settlement.
| `hrrr` | ✓ wired | CONUS 3km | hourly | 2014-07-30 | High-resolution rapid refresh |
| `hrrrak` | ✓ wired | Alaska 3km | 3-hourly | 2018-01-01 | HRRR for Alaska |
| `gfs` | ✓ wired | Global 0.25° | 6-hourly | 2021-01-01 | Standard global model |
| `gefs` | ✓ wired | Global 0.5° ensemble (32 members) | 6-hourly | 2017-01-01 | Default member `c00`; opt in via `member=` |
| `gefs` | ✓ wired | Global 0.5° 31-member ensemble (`c00` + `p01`..`p30`) plus `avg`/`spr` statistical products | 6-hourly | 2017-01-01 | Default member `c00`; opt in via `member=` (e.g. `member="p05"`) |
| `gdas` | ✓ wired | Global 0.25° (short-range) | 6-hourly | 2021-01-01 | GFS analysis system |
| `nbm` | ✓ wired | Regional blend | hourly | 2020-01-01 | National Blend; `fxx=0` auto-bumps to `1` |
| `rap` | ✓ wired | CONUS 13km | hourly | 2020-01-01 | Rapid refresh |
| `rrfs` | ✓ wired | CONUS 3km | hourly | 2024-01-01 | HRRR successor (pre-operational) |
| `rtma` | ✓ wired | CONUS 2.5km analysis | hourly | 2024-01-01 | Real-time mesoscale analysis (`fxx=0` only) |
| `urma` | ✓ wired | CONUS 2.5km analysis | hourly | 2024-01-01 | Un-Restricted MA (`fxx=0` only) |
| `cfs` | ✓ wired | Global 1° (4-member) | 6-hourly | 2011-01-01 | Climate Forecast System |
| `cfs` | ✓ wired | Global 1° 4-member ensemble (`01`..`04`) | 6-hourly | 2011-01-01 | Climate Forecast System; default member `01`, opt in via `member=` (e.g. `member="03"`) |

All 11 NCEP-family models are end-to-end wired in v1.0.

Expand Down Expand Up @@ -207,6 +207,36 @@ df = pd.concat(frames, ignore_index=True)
BDP depths are documented above; older cycles raise
`HistoricalDepthError`.

### Ensemble members (`member=`)

The ensemble models **GEFS** and **CFS** accept a `member=` selector
(issue #74). It threads to the path builder so you fetch a specific
ensemble member instead of the default control run:

```python
from mostlyright.forecasts import forecast_nwp

# GEFS perturbation member p05 (default is the c00 control run)
df = forecast_nwp(station="KNYC", model="gefs", member="p05")

# CFS member 03 (default is 01)
df = forecast_nwp(station="KNYC", model="cfs", member="03")
```

- **GEFS** members: `c00` (control, default), `p01`..`p30`
(perturbations), plus `avg` / `spr` statistical products — 33 values.
- **CFS** members: `01`..`04` (default `01`).
- `member=` is **only** valid for `gefs` / `cfs`. Passing it for any
other model raises `ValueError`. An out-of-enum member value also
raises `ValueError` listing the valid members. Both errors fire before
the `[nwp]` extra is imported.
- `member=None` (the default) is byte-identical to pre-#74 behavior — no
`member` is threaded and the path-builder default is used.

> **Note:** `member=` does not (yet) add a `member` column to the output
> DataFrame — it selects which member's grid is fetched. A per-row
> `member` column is tracked as future work.

### Settlement-day envelope (Mode 2)

`research(include_forecast=True, forecast_models=[...])` fetches a
Expand Down
6 changes: 6 additions & 0 deletions packages-ts/weather/src/forecasts/nwp-stub.ts
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,12 @@ export interface ForecastNwpOptions {
readonly fxx?: number;
/** Force a mirror (e.g. `"aws_bdp"`). */
readonly mirror?: string;
/**
* Ensemble member id (e.g. GEFS `"p05"`, CFS `"03"`). Mirrors the
* Python `member=` kwarg (issue #74); only meaningful for GEFS / CFS.
* Signature-forward only — TS NWP execution lands in v2.0+.
*/
readonly member?: string;
}

/**
Expand Down
13 changes: 12 additions & 1 deletion packages-ts/weather/tests/forecasts/nwp-stub.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import { describe, expect, it } from "vitest";

import { DataAvailabilityError, NwpNotAvailableError } from "@mostlyrightmd/core";

import { forecastNwp } from "../../src/forecasts/index.js";
import { type ForecastNwpOptions, forecastNwp } from "../../src/forecasts/index.js";

describe("forecastNwp (Phase 21 21-07 messaging)", () => {
it("raises NwpNotAvailableError (subclass of DataAvailabilityError)", async () => {
Expand Down Expand Up @@ -115,4 +115,15 @@ describe("forecastNwp (Phase 21 21-07 messaging)", () => {
// Exercise one call to lock the runtime behavior.
await expect(forecastNwp("KNYC", models[0])).rejects.toThrow();
});

it("ForecastNwpOptions accepts an optional member (issue #74 parity)", async () => {
// Compile-level check: `member?` must exist on ForecastNwpOptions, or
// tsc fails. Runtime still throws the v1.x stub error.
const opts: ForecastNwpOptions = { member: "p05" };
await expect(forecastNwp("KNYC", "gefs", opts)).rejects.toThrow(NwpNotAvailableError);
// Inline-literal form mirrors the Python call shape.
await expect(forecastNwp("KNYC", "cfs", { member: "03" })).rejects.toThrow(
NwpNotAvailableError,
);
});
});
10 changes: 6 additions & 4 deletions packages/core/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -72,10 +72,12 @@ parquet = [
# pandas upper bound aligned with `parquet` extra at <4.0; both backends
# are exercised by the dual-pandas CI matrix.
research = [
# >=1.6.0: research() now calls fetch_open_meteo(variables=...), a kwarg
# introduced in mostlyrightmd-weather 1.6.0 (#64). An older weather pin would
# raise TypeError on research(..., forecast_source="open_meteo") (codex P1).
"mostlyrightmd-weather>=1.6.0,<2.0",
# >=1.7.0: forecast_nwp() threads member= to the weather impl, a kwarg
# introduced in mostlyrightmd-weather 1.7.0 (#74) — an older weather would
# TypeError on forecast_nwp(..., member=...). (Supersedes the >=1.6.0 floor
# for fetch_open_meteo(variables=...), #64 codex P1; default calls stay
# skew-tolerant via the core wrapper's conditional member threading.)
"mostlyrightmd-weather>=1.7.0,<2.0",
"pyarrow>=17.0,<24.0",
"pandas>=2.2,<4.0",
]
Expand Down
20 changes: 19 additions & 1 deletion packages/core/src/mostlyright/forecasts.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@ def forecast_nwp(
cycle_range_end: datetime | None = None,
fxx: int | None = None,
mirror: str | None = None,
member: str | None = None,
client: httpx.Client | None = None,
) -> pd.DataFrame:
"""Fetch an NWP forecast from NOAA Big Data Program direct-fetch.
Expand All @@ -149,6 +150,12 @@ def forecast_nwp(
fxx: Forecast hour ahead of ``cycle``. Default ``1``.
mirror: Force a specific NOAA BDP mirror (``"aws_bdp"`` or
``"nomads"``). Default: try AWS then NOMADS.
member: Ensemble member id — only valid for the member-capable
models GEFS (``"c00"``/``"p01"``..``"p30"``/``"avg"``/``"spr"``,
default ``"c00"``) and CFS (``"01"``..``"04"``, default
``"01"``). ``None`` (default) is byte-identical to today.
Validation + the valid-member enums live in the weather impl;
this wrapper passes ``member`` straight through (issue #74).
client: Reuse an ``httpx.Client`` for connection pooling.

Returns:
Expand All @@ -157,7 +164,9 @@ def forecast_nwp(
Raises:
ValueError: ``model`` not in :data:`SUPPORTED_NWP_MODELS` and
not a reserved ECMWF id; ``fxx`` is negative; ``cycle`` is
naive; ``mirror`` outside the supported set.
naive; ``mirror`` outside the supported set; ``member`` set
for a non-member model or not a valid member of the model's
ensemble (validated in the weather impl).
NwpModelNotAvailableError: ``model`` is a reserved ECMWF id.
SourceUnavailableError: ``[nwp]`` optional extra not installed.
NoLiveForNwpError: every wired NOAA BDP mirror failed.
Expand Down Expand Up @@ -261,6 +270,14 @@ def forecast_nwp(
source=f"nwp.{model}",
) from None

# Issue #74 cross-version skew guard: thread ``member`` only when the
# caller actually set it. Passing ``member=member`` unconditionally would
# make EVERY core-wrapper call TypeError against an older
# mostlyrightmd-weather whose impl predates the kwarg (core 1.7.0 +
# weather <=1.6.0 outside the [research] extra's floor). With the guard,
# default calls stay call-compatible across the skew; only explicit
# ``member=`` callers on an old weather see the loud TypeError.
member_kwargs: dict[str, str] = {} if member is None else {"member": member}
return _impl(
station,
model,
Expand All @@ -270,4 +287,5 @@ def forecast_nwp(
fxx=fxx,
mirror=mirror,
client=client,
**member_kwargs,
)
41 changes: 41 additions & 0 deletions packages/core/tests/test_forecast_nwp_schema_phase17.py
Original file line number Diff line number Diff line change
Expand Up @@ -126,3 +126,44 @@ def test_rtma_default_call_does_not_raise_analysis_guard() -> None:
)
except Exception:
pass


# ---------------------------------------------------------------------------
# Issue #74 — core wrapper forwards member= to the weather impl
# ---------------------------------------------------------------------------
def test_core_wrapper_forwards_member_to_impl() -> None:
"""The public ``mostlyright.forecasts.forecast_nwp`` wrapper must accept
``member=`` and forward it verbatim to the weather impl. Patch the
delegation target (``mostlyright.weather.forecast_nwp.forecast_nwp``)
with a Mock so no network / [nwp] extra is required."""
from unittest.mock import Mock, patch

from mostlyright.forecasts import forecast_nwp

fake_impl = Mock(return_value="sentinel-df")
with patch("mostlyright.weather.forecast_nwp.forecast_nwp", fake_impl):
result = forecast_nwp("KNYC", "gefs", member="p05")

assert result == "sentinel-df"
fake_impl.assert_called_once()
_, kwargs = fake_impl.call_args
assert kwargs.get("member") == "p05"


def test_core_wrapper_omits_member_kwarg_by_default() -> None:
"""Cross-version skew guard (#74): a default call (no ``member=``) must
NOT pass a ``member`` kwarg to the weather impl, so core stays
call-compatible with a pre-1.7.0 mostlyrightmd-weather whose impl lacks
the parameter. Mirrors the weather-side Test D at the delegation layer."""
from unittest.mock import Mock, patch

from mostlyright.forecasts import forecast_nwp

fake_impl = Mock(return_value="sentinel-df")
with patch("mostlyright.weather.forecast_nwp.forecast_nwp", fake_impl):
result = forecast_nwp("KNYC", "gefs")

assert result == "sentinel-df"
fake_impl.assert_called_once()
_, kwargs = fake_impl.call_args
assert "member" not in kwargs
54 changes: 52 additions & 2 deletions packages/weather/src/mostlyright/weather/forecast_nwp.py
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,13 @@
_RESERVED_MODELS: frozenset[str] = frozenset(NWP_MODEL_VALUES) - _WIRED_NWP_MODELS


#: Issue #74: models whose ensemble ``member=`` is wired into their path
#: builder (``_gefs_path`` / ``_cfs_path``). RRFS declares members but its
#: member is NOT threaded into the path builder, so it is excluded here —
#: passing ``member=`` for any non-listed model raises ``ValueError``.
_MEMBER_CAPABLE_MODELS: frozenset[str] = frozenset({"gefs", "cfs"})


#: cfgrib's canonical short-name for each GRIB2 ``(variable, level)`` pair
#: in the mostlyright variable maps. Lifted from cfgrib's CF / GRIB2 short-
#: name table; used to project per-record xarray datasets back to the
Expand Down Expand Up @@ -324,14 +331,24 @@ def _try_fetch_records_for_mirror(
fxx: int,
variable_map: dict[str, tuple[str, str]],
client: httpx.Client,
member: str | None = None,
) -> tuple[NwpFetchPlan, list[IdxRecord], int] | None:
"""Resolve a fetch plan + ``.idx`` + content length on one mirror.

Returns ``None`` if the mirror failed (caller falls back to the next
one in the chain). Returns a tuple ``(plan, records, content_length)``
on success — caller does the per-record byte-range fetch.

Issue #74: ``member`` is threaded to ``build_fetch_plan`` (and on to
the GEFS/CFS path builder) ONLY when non-None — passing ``member=None``
would override the path-builder default (``c00`` / ``01``) and crash
f-string formatting.
"""
plan = build_fetch_plan(model=model, mirror=mirror, cycle=cycle, fxx=fxx)
# Build per-model kwargs, including ``member`` only when explicitly set.
per_model_kwargs: dict[str, Any] = {}
if member is not None:
per_model_kwargs["member"] = member
plan = build_fetch_plan(model=model, mirror=mirror, cycle=cycle, fxx=fxx, **per_model_kwargs)
try:
idx_text = fetch_idx_text(plan, client=client)
# Phase 17 FORECAST-04: dispatch idx parser style per model.
Expand Down Expand Up @@ -583,6 +600,7 @@ def forecast_nwp(
cycle_range_end: datetime | None = None,
fxx: int | None = None,
mirror: str | None = None,
member: str | None = None,
client: httpx.Client | None = None,
backend: str = "pandas",
return_type: str = "dataframe",
Expand All @@ -601,6 +619,12 @@ def forecast_nwp(
fxx: Forecast hour ahead of ``cycle``. Default ``1`` (next hour).
mirror: Force a specific mirror (``"aws_bdp"`` or ``"nomads"``).
Default: try AWS first then NOMADS.
member: Ensemble member id — only valid for the member-capable
models GEFS (e.g. ``"p05"``, default ``"c00"``) and CFS
(``"01"``..``"04"``, default ``"01"``). ``None`` (the default)
keeps the path-builder default and is byte-identical to
pre-issue-#74 behavior. Threaded to the GEFS/CFS path builder
only when non-None.
client: Reuse an ``httpx.Client`` for connection pooling. A
fresh client is created (and closed) per call if omitted.

Expand All @@ -614,7 +638,9 @@ def forecast_nwp(
Raises:
NwpModelNotAvailableError: ``model`` is reserved (ECMWF Tier-2).
ValueError: ``model`` or ``mirror`` is not in the supported set;
``fxx`` is negative; ``cycle`` is naive.
``fxx`` is negative; ``cycle`` is naive; ``member`` is set for
a non-member model, or is not a valid member of the requested
model's ensemble (GEFS/CFS).
SourceUnavailableError: the ``[nwp]`` optional extra
(``cfgrib`` + ``xarray`` + ``sklearn``) is not installed.
NoLiveForNwpError: every wired mirror failed (typically while
Expand Down Expand Up @@ -688,6 +714,28 @@ def forecast_nwp(
f"got {mirror!r}"
)

# Issue #74: validate the ``member=`` ensemble selector EARLY — before
# the lazy ``[nwp]`` imports below — so callers without cfgrib still get
# the right ValueError. Only GEFS / CFS have their member wired into the
# path builder; RRFS member is NOT wired and is intentionally excluded.
# The member enums are plain frozensets (gefs.py / cfs.py import no
# cfgrib/xarray at module level), so importing them here is cheap.
if member is not None:
if model not in _MEMBER_CAPABLE_MODELS:
raise ValueError(
f"member= is only supported for models "
f"{sorted(_MEMBER_CAPABLE_MODELS)}; got model={model!r}"
)
if model == "gefs":
from ._fetchers._nwp_grids.gefs import GEFS_MEMBERS as _MEMBERS
else: # model == "cfs"
from ._fetchers._nwp_grids.cfs import CFS_MEMBERS as _MEMBERS
if member not in _MEMBERS:
raise ValueError(
f"member={member!r} is not a valid {model} member; "
f"valid members are {sorted(_MEMBERS)}"
)

# Phase 17 Wave-2 iter-3: model-aware fxx default. RTMA / URMA are
# analysis products with no forecast hour -- default to 0. All other
# models default to fxx=1. The None sentinel lets us distinguish an
Expand Down Expand Up @@ -751,6 +799,7 @@ def _fetch_cycle(_c: datetime) -> pd.DataFrame | None:
cycle=_c,
fxx=fxx if fxx is not None else (0 if model in {"rtma", "urma"} else 1),
mirror=mirror,
member=member,
client=client,
backend="pandas",
return_type="dataframe",
Expand Down Expand Up @@ -919,6 +968,7 @@ def _fetch_cycle(_c: datetime) -> pd.DataFrame | None:
fxx=fxx,
variable_map=variable_map,
client=client,
member=member,
)
if attempt is None:
continue
Expand Down
Loading
Loading