0.9

.github/workflows/cicd.yml

@@ -18,10 +18,10 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
 
     - name: Set up Python 3.10
-      uses: actions/setup-python@v3
+      uses: actions/setup-python@v4
       with:
         python-version: "3.10"
         cache: "pip"

CHANGELOG.md

@@ -4,15 +4,41 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## Unreleased: 0.8.0
+## 0.9.2
+
+### Added
+- `--sites-only` flag to only retrieve site data
+- `--output-format` flag to write out sites/summary tables as csv or geojson.
+  - options are `csv` or `geojson`
+  - timeseries data is always written to a csv
+- NM OSE POD data for sites.
+  - can be removed from output with `--no-nmose-pod`
+- `--output-dir` to change the output directory to a location other than `.` (the current working directory)
+
+### Changed
+- `output` to `output-type` for CLI
+
+### Fixed
+- a bug with `--site-limit`. it now exports the number of sets requested by the user
+
+## 0.8.0
 
 ### Added
 - water level for WQP
+- `earliest_date`, `earliest_time`, `earliest_value`, and `earliest_units` to the summary table
+- `die wells` to get all wells for which the DIE reports observations
+- `die source {parameter}` to list sources that report a particular parameter
+- NM OSE PODs, though its information is only currently available for the invocation of `die wells`
 
 ### Changed
 - NM OSE Roswell data is now pulled from ST2 and not CKAN
+- renamed the column `location` to `name` in the summary table to match the format of the `sites` table when timeseries data are exported
+- renamed the columns `most_recent_date`, `most_recent_time`, `most_recent_value`, and `most_recent_units` to `latest_date`, `latest_time`, `latest_value`, and `latest_units` respectively for succinctness and juxtaposition with the newly added `earliest` columns.
+  - This naming schema also enables the development of datetime filters as the descriptor will apply to the latest datetime within the provided time frame filter, whereas most recent indicates np filters.
+- removed sites that are not in New Mexico  
 
 ### Fixed
+- removed records from USGS where the value is "-999999"
 
 
 ## 0.7.0

README.md

@@ -30,6 +30,8 @@ Data comes from the following sources. We are continuously adding new sources as
   - Available data: `water levels`, `water quality`
 - [New Mexico Environment Department Drinking Water Bureau (NMED DWB)](https://nmenv.newmexicowaterdata.org/FROST-Server/v1.1/)
   - Available data: `water quality`
+- [New Mexico Office of the State Engineer Points of Diversions (NMOSEPODs)](https://services2.arcgis.com/qXZbWTdPDbTjl7Dy/ArcGIS/rest/services/OSE_PODs/FeatureServer/0)
+  - Available data: `None`
 - [New Mexico Office of the State Engineer ISC Seven Rivers (NMOSE ISC Seven Rivers)](https://nmisc-wf.gladata.com/api/getMonitoringPoints.ashx)
   - Available data: `water levels`, `water quality`
 - [New Mexico Office of the State Engineer Roswell District Office (NMOSE Roswell)](https://catalog.newmexicowaterdata.org/dataset/pecos_region_manual_groundwater_levels)
@@ -63,27 +65,28 @@ where `{parameter}` is the name of the parameter whose data is to be retrieved,
 | **nmbgmr-amp**             | X           | X       | X           | X       | X         | X        | X        | X         | X       | X   | X         | X      | X      | X       | X   | X       |
 | **nmed-dwb**               | -           | X       | X           | X       | -         | X        | X        | X         | X       | X   | X         | X      | X      | X       | X   | X       |
 | **nmose-isc-seven-rivers** | X           | -       | X           | X       | -         | X        | X        | X         | X       | X   | X         | X      | X      | X       | X   | -       |
+| **nmose-pod**              | -           | -       | -           | -       | -         | -        | -        | -         | -       | -   | -         | -      | -      | -       | -   | -       |
 | **nmose-roswell**          | X           | -       | -           | -       | -         | -        | -        | -         | -       | -   | -         | -      | -      | -       | -   | -       |
 | **nwis**                   | X           | -       | -           | -       | -         | -        | -        | -         | -       | -   | -         | -      | -      | -       | -   | -       |
 | **pvacd**                  | X           | -       | -           | -       | -         | -        | -        | -         | -       | -   | -         | -      | -      | -       | -   | -       |
 | **wqp**                    | X           | X       | X           | X       | X         | X        | X        | X         | X       | X   | X         | X      | X      | X       | X   | X       |
 
-### Output
-The `--output` option is required and used to set the output type:
+### Output Type
+The `--output-type` option is required and used to set the output type:
 
 ```
---output summary
+--output-type summary
 ```
 - A summary table consisting of location information as well as summary statistics for the parameter of interest for every location that has observations.
 
 ```
---output timeseries_unified
+--output-type timeseries_unified
 ```
 - A single table consisting of time series data for all locations for the parameter of interest.
 - A single table of site data that contains information such as latitude, longitude, and elevation
 
 ```
---output timeseries_separated
+--output-type timeseries_separated
 ```
 - Separate time series tables for all locations for the parameter of interest.
 - A single table of site data that contains information such as latitude, longitude, and elevation
@@ -98,7 +101,7 @@ A log of the inputs and processes, called `die.log`, is also saved to the output
 | :----------- | :---------- | :-------- | :------------- |
 | source | the organization/source for the site | string | Y |
 | id | the id of the site. The id is used as the key to join the site and timeseries tables | string | Y |
-| location | the colloquial name for the site | string | Y |
+| name | the colloquial name for the site | string | Y |
 | usgs_site_id | USGS site id | string | N |
 | alternate_site_id | alternate site id | string | N | 
 | latitude | latitude in decimal degrees | float | Y |
@@ -114,10 +117,14 @@ A log of the inputs and processes, called `die.log`, is also saved to the output
 | min | the minimum observation | float | Y |
 | max | the maximum observation | float | Y |
 | mean | the mean value of the observations | float | Y |
-| most_recent_date| date of most recent record in YYYY-MM-DD | string | Y |
-| most_recent_time | time of most recent record in HH:MM:SS or HH:MM:SS.mmm | string | N |
-| most_recent_value | value of the most recent record  | float | Y |
-| most_recent_units | units of the most recent record | string | Y |
+| earliest_date| date of the earliest record in YYYY-MM-DD | string | Y |
+| earliest_time | time of the earliest record in HH:MM:SS or HH:MM:SS.mmm | string | N |
+| earliest_value | value of the earliest recent record  | float | Y |
+| earliest_units | units of the earliest record | string | Y |
+| latest_date| date of the latest record in YYYY-MM-DD | string | Y |
+| latest_time | time of the latest record in HH:MM:SS or HH:MM:SS.mmm | string | N |
+| latest_value | value of the latest recent record  | float | Y |
+| latest_units | units of the latest record | string | Y |
 
 <sup>*CABQ elevation is calculated as [elevation at top of casing] - [stickup height]; if stickup height < 0 the measuring point is assumed to be beneath the ground surface</sup>
 
@@ -139,6 +146,7 @@ A log of the inputs and processes, called `die.log`, is also saved to the output
 | formation | geologic formation in which the well terminates | string | N |
 | aquifer | aquifer from which the well draws water | string | N |
 | well_depth | depth of well | float | N |
+| well_depth_units | units of well depth. Defaults to ft | string | N |
 
 <sup>**CABQ elevation is calculated as [elevation at top of casing] - [stickup height]; if stickup height < 0 the measuring point is assumed to be beneath the ground surface</sup>
 
@@ -167,12 +175,13 @@ The Data Integration Engine enables the user to obtain groundwater level and gro
 - `--no-nmbgmr-amp` to exclude New Mexico Bureau of Geology and Mineral Resources (NMBGMR) Aquifer Mapping Program (AMP) data
 - `--no-nmed-dwb` to exclude New Mexico Environment Department (NMED) Drinking Water Bureau (DWB) data
 - `--no-nmose-isc-seven-rivers` to exclude New Mexico Office of State Engineer (NMOSE) Interstate Stream Commission (ISC) Seven Rivers data
+- `--no-nmose-pod` to exclude New Mexico Office of State Engineer (NMOSE) Point of Diversion (POD) data (though none except for well information is currently available)
 - `--no-nmose-roswell` to exclude New Mexico Office of State Engineer (NMOSE) Roswell data
 - `--no-nwis` to exclude USGS NWIS data
 - `--no-pvacd` to exclude Pecos Valley Artesian Convservancy District (PVACD) data
 - `--no-wqp` to exclude Water Quality Portal (WQP) data
 
-### Geographic Filters
+### Geographic Filters [In Development]
 
 The following flags can be used to geographically filter data:
 
@@ -184,7 +193,11 @@ The following flags can be used to geographically filter data:
 -- bbox 'x1 y1, x2 y2'
 ```
 
-### Date Filters
+```
+-- wkt {wkt polygon or multipolygon}
+```
+
+### Date Filters [In Development]
 
 The following flags can be used to filter by dates:
 
@@ -206,12 +219,12 @@ die sources {parameter}
 
 to print the sources that report that parameter to the terminal.
 
-### Wells [In Development]
+### Sites
 
 Use
 
 ```
-die wells
+die sites
 ```
 
-to print wells to the terminal.
+to export site information only

auto_worker_requirements.txt

@@ -0,0 +1,11 @@
+flask
+gunicorn
+httpx
+pandas
+geopandas
+frost_sta_client
+google-cloud-storage
+pytest
+urllib3>=2.2.0,<3.0.0
+Geoalchemy2
+sqlalchemy

backend/__init__.py

@@ -0,0 +1,16 @@
+from enum import Enum
+from os import environ
+
+
+class OutputFormat(str, Enum):
+    GEOJSON = "geojson"
+    CSV = "csv"
+    GEOSERVER = "geoserver"
+
+
+def get_bool_env_variable(var: str) -> bool:
+    env_var = environ.get(var, None)
+    if env_var is None or env_var.strip().lower() not in ["true", "1", "yes"]:
+        return False
+    else:
+        return True

backend/bounding_polygons.py

@@ -15,12 +15,15 @@
 # ===============================================================================
 import json
 import os
+from pprint import pprint
 
 import click
 import httpx
 from shapely import Polygon, box
 from shapely.geometry import shape
 
+from backend.geo_utils import transform_srid, SRID_WGS84, SRID_UTM_ZONE_13N
+
 
 # polygon retrivial functions
 # multiple polygons
@@ -159,21 +162,28 @@ def get_county_polygon(name, as_wkt=True):
         _warning(f"Invalid state. {state}")
 
 
-def get_state_polygon(state):
+def get_state_polygon(state: str, buffer: int | None = None):
     statefp = _statelookup(state)
     if statefp:
         obj = _get_cached_object(
             f"{state}.state",
             f"{state} state",
             f"https://reference.geoconnex.us/collections/states/items/{statefp}?&f=json",
         )
+        geom_gcs = shape(obj["features"][0]["geometry"])
+
+        if buffer:
+            geom_utm = transform_srid(geom_gcs, SRID_WGS84, SRID_UTM_ZONE_13N)
+            geom_utm = geom_utm.buffer(buffer)
+            geom_gcs = transform_srid(geom_utm, SRID_UTM_ZONE_13N, SRID_WGS84)
 
-        return shape(obj["geometry"])
+        return geom_gcs
 
 
 # private helpers ============================
 def _make_shape(obj, as_wkt):
     poly = shape(obj["geometry"])
+    poly = poly.simplify(0.1)
     if as_wkt:
         return poly.wkt
     return poly
@@ -231,8 +241,9 @@ def _get_cached_object(name, msg, url):
     return obj
 
 
+NM_BOUNDARY_BUFFERED = get_state_polygon("NM", 25000)
+
+
 if __name__ == "__main__":
-    # w = get_huc_polygon('0101000201')
-    # print(w)
-    print(get_state_hucs_boundaries(state="CO", level=4))
+    print(get_state_polygon("NM"))
 # ============= EOF =============================================