Skip to content

fix(weather): resolve GFS precip twin crash + expose cloud_cover_pct, visibility_m, cloud_ceiling_m (#63)#68

Merged
helloiamvu merged 3 commits into
mostlyrightmd:mainfrom
zax0rz:fix/63-nwp-cloud-cover-precip
Jun 6, 2026
Merged

fix(weather): resolve GFS precip twin crash + expose cloud_cover_pct, visibility_m, cloud_ceiling_m (#63)#68
helloiamvu merged 3 commits into
mostlyrightmd:mainfrom
zax0rz:fix/63-nwp-cloud-cover-precip

Conversation

@zax0rz

@zax0rz zax0rz commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #63. Resolves a latent GribIntegrityError crash that affects every
forecast_nwp(station, "gfs") call at the default fxx=1, and adds three
new NWP forecast columns: cloud_cover_pct, visibility_m, and
cloud_ceiling_m for both HRRR and GFS models.

The latent GFS precip bug

GFS GRIB2 files contain duplicate APCP:surface records at fxx >= 1
(two identical accumulation windows with different record numbers). The SDK's
ambiguity guard in _extract_records() raises GribIntegrityError on any
duplicate, completely aborting the fetch. Both mirrors carry the same twins, so
mirror fallback doesn't recover. At fxx=0 the bug is masked because GFS omits
APCP entirely at the analysis hour.

This was invisible because CI has no live GFS test — only HRRR.

Fix: _pick_record heuristic

Replaces the fatal GribIntegrityError with a deterministic picker that:

  1. Prefers instantaneous records over window-aggregated (ave/acc/max/min)
  2. Breaks ties by lowest record_no

This simultaneously fixes the APCP crash and unblocks cloud_cover_pct on GFS,
where (TCDC, entire atmosphere) also produces two records (instantaneous vs
time-averaged). HRRR is unaffected — all fields are single-record.

New columns

Column GRIB2 Key Level cfgrib short-name Units
cloud_cover_pct TCDC entire atmosphere tcc %
visibility_m VIS surface vis m
cloud_ceiling_m HGT cloud ceiling gh m

Naming uses cloud_ceiling_m (not ceiling_m) for cross-source consistency
with IEM adapters.

Supporting evidence

Academic and operational research in .briefs/ demonstrates why these fields
matter for prediction-market temperature modeling:

  • Cloud cover modulates DTR by 50%+ — overcast vs clear-sky shifts daily
    highs by 7–11°F and lows by 8–10°F. That's the exact margin around Kalshi
    strike prices.
  • NWS MOS uses cloud cover as a primary predictor in its temperature
    regression equations. The SDK was missing what MOS considers essential.
  • BC-Unet (NOAA's own ML bias correction for GFS) ingests total cloud
    cover as a core 2D input field for image-to-image temperature correction.
  • HRRRv4 systematically underestimates cloud fraction — exposing TCDC
    enables quants to detect and correct for this known radiative bias.

See .briefs/cloud-cover-deep-research.md (64 cited sources) and
.briefs/github-issue-63-nwp-fields-review.md for full analysis.

Changes

Bug fix

  • forecast_nwp.py: Add _pick_record() disambiguation helper
  • forecast_nwp.py: Replace raise GribIntegrityError with warning + picker

New fields

  • schemas/forecast_nwp.py (core): Register 3 nullable float64 columns
  • _nwp_grids/hrrr.py, _nwp_grids/gfs.py: Add TCDC, VIS, HGT mappings
  • forecast_nwp.py: Add cfgrib short-name lookups (tcc, vis, gh)
  • forecast_nwp.py: Register in nullable_numeric_cols and _empty_dataframe
  • qc/rules_nwp.py: Add physics-bounds QC rules (cloud 0–100%, vis/ceiling ≥ 0)
  • schemas/json/schema.forecast_nwp.v1.json: Regenerated

Tests

  • TestDisambiguationHeuristics: synthetic duplicate .idx fixtures, window vs
    instantaneous preference, record_no tiebreak
  • test_qc_rules_nwp.py: updated base rule count (7 → 10)
  • test_forecast_nwp_multi_cycle.py: mock row structures updated

Test plan

  • 1459 passed, 1 skipped, 23 deselected (live excluded) — uv run pytest packages/weather/tests -m "not live"
  • 2938 passed, 4 skipped, 48 deselected — full suite uv run pytest -m "not live"
  • Ruff check + format clean
  • Committed on branch fix/63-nwp-cloud-cover-precip from upstream/main

Cross-SDK Sync (see .planning/CROSS-SDK-SYNC.md)

python_only: true — external contributor PR; TS parity ticket will be filed
by a maintainer after merge. The schema column additions and .idx
disambiguation heuristic both need TS ports (documented in PR body and
.briefs/github-issue-63-nwp-fields-review.md §6).

zach added 3 commits June 5, 2026 09:35
…stlyrightmd#63)

- Resolve the latent GFS precipitation twin bug by adding a `_pick_record` helper to forecast_nwp.py.
  It filters duplicates by prioritizing instantaneous records and breaking ties using lowest `record_no`.
- Implement `cloud_cover_pct`, `visibility_m`, and `cloud_ceiling_m` for GFS and HRRR.
- Define and integrate physics-bounds QC rules for the three new fields.
- Regenerate schema.forecast_nwp.v1 JSON schemas.
- Add comprehensive test coverage in test_qc_rules_nwp.py, test_forecast_nwp.py, and test_forecast_nwp_multi_cycle.py.
- Include planning artifacts and research briefs in .briefs/ directory.
Reverted accidental inclusion of all extras in root dependency list.
The [nwp] extra pulls eccodes which is not available in CI without the
full extra install, causing test detection guards to pass while the
actual eccodes binary fails to load.
@helloiamvu

Copy link
Copy Markdown
Member

Codex review (gpt-5.5, medium reasoning) — 1 blocking finding (P2):

  • [P2] Exported NWP schema not wired into TS codegenscripts/export_schemas.py:97. schema.forecast_nwp.v1 is added to the exported Group A schemas, but the TS generator's schema list is hard-coded without schema.forecast_nwp.v1.json, so pnpm codegen won't emit a ForecastNwpV1 type/validator and TS consumers stay unaware of the new public columns. Per the CLAUDE.md dual-SDK rule, update the TS codegen schema list and regenerate the generated TS files alongside this export (or file the parity ticket the issue's TS-Parity section anticipates).

CI is green and the Python disambiguation logic + GFS-precip fix look solid; this is the one issue to resolve before merge. (Reviewed in the 2026-06-06 autonomous triage run — feature-scope, recommend a 1.6.0 minor.)

helloiamvu added a commit that referenced this pull request Jun 6, 2026
… P2)

Codex P2 on #68: scripts/export_schemas.py exported schema.forecast_nwp.v1
but the TS codegen SCHEMA_FILES list omitted it, so pnpm codegen never
emitted a ForecastNwpV1 type/validator and TS consumers stayed unaware of
the new public NWP columns (dual-SDK parity gap). Add the schema to the TS
codegen list and regenerate: new generated forecast_nwp.v1.ts + ajv
validator + barrel/format-map exports. Codegen + core validator tests green.
helloiamvu added a commit that referenced this pull request Jun 6, 2026
…-join fix (#67) + docs (#52)

Minor release (features added -> 1.6.0, not a patch). Bump all three
packages 1.5.3 -> 1.6.0; regenerate uv.lock (codex P2 on #71: lock was
stale at 1.5.2/1.5.3). CHANGELOG: fold the never-published 1.5.3 entry into
1.6.0 and add the #63 NWP-fields and #64 OM-cache/throttle feature entries.

Bundles PRs #67/#52 (already in merged-vision) + #66/#64 + #68/#63, each
with its codex P2 resolved on this integration branch.
@helloiamvu helloiamvu merged commit 1fcbfdf into mostlyrightmd:main Jun 6, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(weather): expose cloud_cover_pct / visibility_m / ceiling_m in forecast_nwp (HRRR/GFS); pressure_pa_* already ship

2 participants