Skip to content

feat(tests): relax validation rules and expand enum coverage in well-inventory-csv feature#548

Merged
jirhiker merged 3 commits intostagingfrom
marissa
Mar 3, 2026
Merged

feat(tests): relax validation rules and expand enum coverage in well-inventory-csv feature#548
jirhiker merged 3 commits intostagingfrom
marissa

Conversation

@marissafichera
Copy link
Copy Markdown
Contributor

BDMS-608: Relax well inventory CSV validation rules and expand enum coverage

@BDMS-608

Why

  • Validation rules in the feature file were stricter than necessary, causing real user-submitted CSVs to fail ingestion
  • Required fields did not reflect actual field usage in the field — elevation and measuring point data is not always available at time of entry
  • Contact and water level rules were all-or-nothing, blocking valid partial submissions
  • Several enum fields (address_type, state, well_hole_status, monitoring_status, well_pump_type) lacked explicit validation coverage

How

  • Moved site_name, elevation_ft, elevation_method, and measuring_point_height_ft from required to optional
  • Replaced "both contact_name and contact_organization required" rule with "at least one of the two required"
  • Removed all-or-nothing water level validation; water_level_date_time is now required only when depth_to_water_ft is provided, all other water level fields are independent and optional
  • Added negative scenarios for invalid address_type, state abbreviation, well_hole_status, monitoring_status, and well_pump_type with allowed values specified in each Given step
  • Added well_notes, well_measuring_notes, water_notes, historical_notes, well_hole_status, and monitoring_status to optional fields

Notes

  • Corresponding changes to the Pydantic schema (WellInventoryRow) are required for these rules to take effect in the backend — this PR covers the feature spec only
  • Allowed values for new enum scenarios are based on the current lexicon and should be kept in sync if the lexicon changes

…inventory-csv feature

- Move site_name, elevation_ft, elevation_method, and measuring_point_height_ft from required to optional
- Replace "both contact_name and contact_organization required" rule with "at least one required" rule
- Remove all-or-nothing water level rule; water_level_date_time now required only when depth_to_water_ft is provided, all other water level fields are independent and optional
- Add negative scenarios for invalid address_type, state abbreviation, well_hole_status, monitoring_status, and well_pump_type with allowed values specified
- Add well_notes, well_measuring_notes, water_notes, historical_notes, well_hole_status, and monitoring_status to optional fields
@marissafichera marissafichera requested review from Copilot and ksmuczynski and removed request for Copilot February 26, 2026 19:39
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 56f6cbf678

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@negative @validation @BDMS-TBD
Scenario: Upload fails when a row has an invalid postal code format
Given my CSV file contains a row that has an invalid postal code format in contact_1_address_1_postal_code
Given my CSV file contains a row that has an invalid postal code format in contact_1_address_1_postal_code
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Restore matching Given text for invalid postal code step

This edit changes the step text from row that has... to row that has..., but the registered Given step in tests/features/steps/well-inventory-csv-given.py still uses the original double-space phrase, so this scenario no longer matches any step definition. In a behave run, that makes the scenario undefined and prevents the validation assertion from executing.

Useful? React with 👍 / 👎.


@negative @validation @BDMS-TBD
Scenario: Upload fails when a row has a contact with an invalid "address_type"
Given my CSV file contains a row with an address_type value that is not one of: Work, Personal, Mailing, Physical
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add step implementations for newly added validation scenarios

The new enum/contact/water-level scenarios introduced in this section rely on Given/Then phrases that currently have no matching step decorators in the well-inventory step files (tests/features/steps/well-inventory-csv-given.py and tests/features/steps/well-inventory-csv-validation-error.py). Because these steps are undefined, the added scenarios fail at step resolution time and do not provide executable coverage.

Useful? React with 👍 / 👎.

@ksmuczynski
Copy link
Copy Markdown
Contributor

@marissafichera Two comments below related to potential future work, but I agree overall with the updates. Approved!

Regarding point no. 1 in the HOW section of the PR:

  1. Moved site_name, elevation_ft, elevation_method, and measuring_point_height_ft from required to optional
  • site_name: Yes
  • elevation_ft: Yes for now.
    • The Ocotillo codebase includes functionality to retrieve elevation data from the USGS National Map Elevation Point Query Service (EPQS). While this feature exists, it is not currently used during the ingestion of the well inventory CSV.
    • The ingestion logic uses the user-provided values and does not automatically query or verify them against the USGS DEM.
    • In the interest of getting new data into Ocotillo, let's relax the requirement. In the future, we can implement functionality to retrieve elevation data from the USGS National Map Elevation Point Query Service (EPQS), and assign elevation_method appropriately.
  • elevation_method: Yes for now
    • In the interest of getting new data into Ocotillo, let's relax the requirement. In the future, if we implement functionality to retrieve elevation data from the USGS National Map Elevation Point Query Service (EPQS), we can have this be assigned automatically
  • measuring_point_height_ft: Yes

@marissafichera marissafichera marked this pull request as draft March 3, 2026 17:47
@marissafichera marissafichera requested a review from jirhiker March 3, 2026 17:47
@marissafichera marissafichera marked this pull request as ready for review March 3, 2026 18:45
Copilot AI review requested due to automatic review settings March 3, 2026 18:45
Copy link
Copy Markdown
Member

@jirhiker jirhiker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Remove instances of " And the system should return a response in JSON format" and this should be ready to proceed to implementation.

Since this updates the spec without updating the implementation, there are (at least) two options for keeping the tests passing on staging. 1) do not merge into staging. Whoever is working on the implementation should create a new branch from marissa, make the tests pass, then open a PR to staging 2) remove the @production tag from this file so that CI skips this test. once implementation is complete @production tag can be re-added

…inventory-csv feature

- removed production tag
- removed instances of returning in JSON format
- Move site_name, elevation_ft, elevation_method, and measuring_point_height_ft from required to optional
- Replace "both contact_name and contact_organization required" rule with "at least one required" rule
- Remove all-or-nothing water level rule; water_level_date_time now required only when depth_to_water_ft is provided, all other water level fields are independent and optional
- Add negative scenarios for invalid address_type, state abbreviation, well_hole_status, monitoring_status, and well_pump_type with allowed values specified
- Add well_notes, well_measuring_notes, water_notes, historical_notes, well_hole_status, and monitoring_status to optional fields
@marissafichera marissafichera requested a review from jirhiker March 3, 2026 19:16
…yle field requirements

- Update water-level feature scenarios to mirror well-inventory behavior for requested headers
- Keep only base required fields (`field_staff`, `well_name_point_id`, `field_event_date_time`)
- Move water-level measurement fields to optional set (`water_level_date_time`, `measuring_person`, `sample_method`, `mp_height`, `level_status`, `depth_to_water_ft`, `data_quality`, `water_level_notes`)
- Add timezone-naive ISO 8601 expectation for `field_event_date_time` and `water_level_date_time`
- Fix required-field examples and column-order scenario to match new required set
- Normalize numeric validation wording to `mp_height` (not `mp_height_ft`)`
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@jirhiker jirhiker merged commit 4382fd5 into staging Mar 3, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants