Team Review: Data Center Plan

[spydmobile]

Open Nomad Data Center — Team Review

This proposes an open dataset standard (.ond) and a phased plan to make Nomad speak it, with an optional Data Center service to follow.

Everything proposed is additive and opt-in — Nomad without these features behaves exactly as it does today.

Review pack

Committed to branch docs/data-center-team-review (permalinks pinned to commit a2868da):

1. Briefing deck — start here.
   • ▶️ View live: https://htmlpreview.github.io/?https://github.com/WISE-Developers/project_nomad/blob/a2868da28d2b84c16933308e77318ef969edb633/Documentation/Nomad/data-center/team-briefing.html
   • 📄 Source / download: team-briefing.html (self-contained; arrow keys to navigate)
2. .ond standard, draft v0.1 — ond-standard-v0.1.md — the format contract; open items marked [TBD-v1.0].
3. Deliverable A design proposal — deliverableA-design-proposal.md — the Dataset aggregate + extension seams + honest effort/risk.

What we're asking for

• Review the Dataset aggregate design — the one real domain change: landscape → fire → run.
• Sanity-check the extension seams and the dormancy approach ("off stays off").
• Agree on scope & sequence: standard → A (Nomad speaks .ond) → B (Data Center).

Detailed review points: §9 of the design proposal, and the [TBD-v1.0] items in the standard.

Out of scope this round

The Data Center service itself (GeoServer, catalog, federation) follows once the standard and Deliverable A are settled. The deck sketches it for context only.

[BadgerOnABike]

First up, I really like where this is headed. Provides a great deal of consistency in a space where chaos reigns. I especially see the utility once the vision of Agency Centric Nomad is realized.

Do we want to limit ourselves to GeoTiffs in the datacenter? While the tools themselves cannot use other formats (yet) there are more efficient methods to store vast amounts of raster data. Thinking on the cloud side, I guess Zarr is a thing, more scientifically I encounter netCDF and HDF5 all the time as they are far more efficient than tif. This may be a path to superior transmission as well with a decompression stage prior to handing the file to the engine. All of which are OGC compliant.

The same can be said for our vector sets. Geopackages are wildly efficient, but Parquet and GeoParquet are also incredibly useful, OGC compliant and cloud ready. While this isn't true 'cloud' computing it's distributed enough that these may be good formats to consider.

The consideration of more efficient export pathways for WCS may allow that streamed connection to be more performant. It wouldn't be terribly difficult to enforce some kind of bounding box to return the WCS as the client wants to see it while providing the WMS around the edges, essentially making it look like the whole WCS rendered when in reality just the AOI + a buffer did. I only suggest the buffer as users all to often do the ol' gee i wonder what's over here (I don't have adhd at all). Beyond that, tiled cloud optimized geotiffs (COGs) are pretty snappy and that should provide much of the speed we're looking for.

I concur with the dataset aggregation suggestion - also improves searchability and overlap assessments.
Regarding off stays off - how do we reconcile when someone has been off, but now what's to be on and publish the work done while off, does it need to be more explicit than A6?
Scope and sequence make good sense.

Suffice it to say, I support this work and would like it see it continue.

If there is intent to use this material for other folks to consume I have some suggestions below:

Formatting on slide 2 is awkward

Formatting on slide 7 is also awkward

Slide 11 weird formatting

Slide 7
Engine-neutral is this preferred terminology to agnostic now? Arguably it doesn't need an engine at all right as it would be able to read and display without the intent of simulating?

Slide 15
Anyone who never enables it odd wording in my opinion. Perhaps it's unnecessary as the previous 2 bullets cover it? Unless you're after symmetry in which case perhaps Current comfortable workflow? Which does also say the same thing but is more positively framed.

Within: https://github.com/WISE-Developers/project_nomad/blob/a2868da28d2b84c16933308e77318ef969edb633/Documentation/Nomad/data-center/deliverableA-design-proposal.md

Can all § be hyperlinks?

[krowbar]

Zarr seems like a container format specification for gridded data, and might be a bit of an over kill for our needs. I see some similarities between zarr and .ond format -- would we get turtles all the way down if we implement it in .ond?!?

I like the idea and would like to see it continue.