diff --git a/CHANGELOG.md b/CHANGELOG.md index 99c0117..dd3295c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] - Nil +## [0.0.2] - 2026-04-27 + +### 🎉 vowl is now an official ODCS vendor! + +We're thrilled to announce that **vowl** has been recognised as an official [Open Data Contract Standard (ODCS)](https://bitol.io/open-data-contract-standard/) vendor. This is a proud milestone for the project and a testament to the community's commitment to open, interoperable data contracts. + +### Fixed +- Getting outputs no longer crashes when a contract contains a list-valued `mustBe` field (#23). +- Broken link on the Known Issues documentation page (#19). +- Git link sanitisation issue in documentation (#15). + +### Changed (Breaking) +- Check metadata now uses `check_definition` and `contract_definition` (#21, #26). This replaces several top-level metadata fields: + - **Renamed fields:** `schema` → `schema_name`, `rule` → `rendered_implementation`. + - **Removed from top-level:** `dimension`, `type`, `description`, `severity`, `unit` — these are no longer top-level keys in `CheckResultMetadata`. + - **`check_definition`** carries the resolved/generated check definition dict. Auto-generated checks are tagged with `vowl_generated_check`. + - **`contract_definition`** carries the raw ODCS contract content at the check's JSONPath. + - **Output DataFrame:** `get_check_results_df()` no longer flattens definitions into top-level columns. Pass `include_check_definition=True` and/or `include_contract_definition=True` to include them as JSON-serialised columns. `save()` accepts the same keyword arguments. + - Code that accesses `metadata["schema"]`, `metadata["rule"]`, `metadata["unit"]`, etc. must be updated to use the new key names or read from `metadata["check_definition"]` / `metadata["contract_definition"]`. + +### Added +- Logical type `options.format` validation — contracts can now specify format constraints (e.g. date formats) on logical types, and vowl will auto-generate format checks. See the [Format Checks](https://govtech-data-practice.github.io/vowl/contracts/#format-checks) docs for details (#25). +- DuckDB attach example demonstrating cross-database validation (#18). +- Open Graph and SEO metadata for the documentation site (#14). +- Google site verification meta tag (#12). +- Updated Jupyter notebook examples (#16). +- README badge and data contract example link (#13, #22). + +### Dependencies +- Bumped `pytest` from 9.0.2 to 9.0.3 (#20). +- Bumped `cryptography` (#17). +- Updated Python Docker tag to 3.14.3 (#11). + ## [0.0.1] - 2026-04-02 ### 🎉 Celebrating Open Source @@ -34,5 +67,6 @@ Initial public release of **vowl**. - `THIRD_PARTY_NOTICES` and `LICENSE_AUDIT_REPORT.md`. - `CONTRIBUTING.md` with development setup and release workflow. -[Unreleased]: https://github.com/govtech-data-practice/vowl/compare/v0.0.1...HEAD +[Unreleased]: https://github.com/govtech-data-practice/vowl/compare/v0.0.2...HEAD +[0.0.2]: https://github.com/govtech-data-practice/vowl/compare/v0.0.1...v0.0.2 [0.0.1]: https://github.com/govtech-data-practice/vowl/releases/tag/v0.0.1 diff --git a/README.md b/README.md index 9a259cd..eaa0154 100644 --- a/README.md +++ b/README.md @@ -855,8 +855,8 @@ result.display_full_report() | Capability | Description | Status | |------------|-------------|--------| | � **Alternative Check Engines** | Support for dqx, dbt, Soda, Great Expectations (subject to licensing review) | Planned | -| 📅 **Parallel Check Execution** | Run checks in parallel for faster validation across large contracts | Planned | - +| 📅 **Parallel Check Execution** | Run checks in parallel for faster validation across large contracts | Planned || 📅 **CLI Interface** | Command-line interface for running validations directly from the terminal | Planned | +| 📅 **vowl-ui** | Web-based validation interface for vowl | Planned | --- ## Contributing diff --git a/docs/roadmap.md b/docs/roadmap.md index d842e4b..7cb13b9 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -28,3 +28,5 @@ description: >- |------------|-------------|--------| | **Alternative Check Engines** | Support for dqx, dbt, Soda, Great Expectations (subject to licensing review) | Planned | | **Parallel Check Execution** | Run checks in parallel for faster validation across large contracts | Planned | +| **CLI Interface** | Command-line interface for running validations directly from the terminal | Planned | +| **vowl-ui** | Web-based validation interface for vowl | Planned | diff --git a/examples/vowl_demo_HDB_results_check_results.csv b/examples/vowl_demo_HDB_results_check_results.csv index 3861b25..7daeac4 100644 --- a/examples/vowl_demo_HDB_results_check_results.csv +++ b/examples/vowl_demo_HDB_results_check_results.csv @@ -1,21 +1,21 @@ -"check_name","target","schema","engine","type","dimension","description","status","severity","operator","actual_value","expected_value","failed_rows_count","aggregation_type","message","rule","tables_in_query","check_path","check_ref_type","logical_type","is_generated","execution_time_ms" -"month_column_exists_check","hdb_resale_prices.month","hdb_resale_prices","sql","sql","conformity","Column 'month' must exist in 'hdb_resale_prices'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""month"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[0].name","DeclaredColumnExistsCheckReference","string",true,5.757582999649458 -"month_logical_type_check","hdb_resale_prices.month","hdb_resale_prices","sql","sql","conformity","Values in 'month' must be valid string","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""hdb_resale_prices"" WHERE NOT ""month"" IS NULL AND TRY_CAST(""month"" AS TEXT) IS NULL","['hdb_resale_prices']","$.schema[0].properties[0].logicalType","LogicalTypeCheckReference","string",true,9.819083992624655 -"town_column_exists_check","hdb_resale_prices.town","hdb_resale_prices","sql","sql","conformity","Column 'town' must exist in 'hdb_resale_prices'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""town"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[1].name","DeclaredColumnExistsCheckReference",,true,5.337791008059867 -"flat_type_column_exists_check","hdb_resale_prices.flat_type","hdb_resale_prices","sql","sql","conformity","Column 'flat_type' must exist in 'hdb_resale_prices'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""flat_type"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[2].name","DeclaredColumnExistsCheckReference",,true,5.779207989689894 -"block_column_exists_check","hdb_resale_prices.block","hdb_resale_prices","sql","sql","conformity","Column 'block' must exist in 'hdb_resale_prices'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""block"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[3].name","DeclaredColumnExistsCheckReference",,true,5.379000009270385 -"street_name_column_exists_check","hdb_resale_prices.street_name","hdb_resale_prices","sql","sql","conformity","Column 'street_name' must exist in 'hdb_resale_prices'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""street_name"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[4].name","DeclaredColumnExistsCheckReference",,true,5.304541002260521 -"storey_range_column_exists_check","hdb_resale_prices.storey_range","hdb_resale_prices","sql","sql","conformity","Column 'storey_range' must exist in 'hdb_resale_prices'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""storey_range"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[5].name","DeclaredColumnExistsCheckReference",,true,5.266625012154691 -"floor_area_sqm_column_exists_check","hdb_resale_prices.floor_area_sqm","hdb_resale_prices","sql","sql","conformity","Column 'floor_area_sqm' must exist in 'hdb_resale_prices'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""floor_area_sqm"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[6].name","DeclaredColumnExistsCheckReference",,true,5.168709001736715 -"flat_model_column_exists_check","hdb_resale_prices.flat_model","hdb_resale_prices","sql","sql","conformity","Column 'flat_model' must exist in 'hdb_resale_prices'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""flat_model"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[7].name","DeclaredColumnExistsCheckReference",,true,5.213583004660904 -"lease_commence_date_column_exists_check","hdb_resale_prices.lease_commence_date","hdb_resale_prices","sql","sql","conformity","Column 'lease_commence_date' must exist in 'hdb_resale_prices'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""lease_commence_date"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[8].name","DeclaredColumnExistsCheckReference",,true,4.999249998945743 -"remaining_lease_column_exists_check","hdb_resale_prices.remaining_lease","hdb_resale_prices","sql","sql","conformity","Column 'remaining_lease' must exist in 'hdb_resale_prices'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""remaining_lease"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[9].name","DeclaredColumnExistsCheckReference",,true,5.113916995469481 -"resale_price_column_exists_check","hdb_resale_prices.resale_price","hdb_resale_prices","sql","sql","conformity","Column 'resale_price' must exist in 'hdb_resale_prices'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""resale_price"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[10].name","DeclaredColumnExistsCheckReference",,true,5.010666005546227 -"hdb_resale_prices_rowCount","hdb_resale_prices","hdb_resale_prices","sql","library","completeness","Total row count for 'hdb_resale_prices'","PASSED",,"mustBeBetween","201879","[0, 30000000]",0,"count","","SELECT COUNT(*) FROM ""hdb_resale_prices""","['hdb_resale_prices']","$.schema[0].quality[0]","RowCountCheckReference",,true,7.587959000375122 -"Month","hdb_resale_prices.month","hdb_resale_prices","sql","sql","conformity","Based on ISO 8601, assumed to be in UTC +8 | YYYY-MM","FAILED",,"mustBe","2","0",2,"count","","SELECT COUNT(*) FROM ""hdb_resale_prices"" WHERE NOT REGEXP_MATCHES(TRY_CAST(month AS TEXT), '^[0-9]{4}-(0[1-9]|1[0-2])$')","['hdb_resale_prices']","$.schema[0].properties[0].quality[0]","SQLColumnCheckReference","string",false,21.381875005317852 -"town_nullValues","hdb_resale_prices.town","hdb_resale_prices","sql","library","completeness","Count of NULL values in 'town' of 'hdb_resale_prices'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""hdb_resale_prices"" WHERE ""town"" IS NULL","['hdb_resale_prices']","$.schema[0].properties[1].quality[0]","NullValuesCheckReference",,true,14.130416995612904 -"flat_type_invalidValues","hdb_resale_prices.flat_type","hdb_resale_prices","sql","library","conformity","Count of invalid values in 'flat_type' of 'hdb_resale_prices'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""hdb_resale_prices"" WHERE NOT ""flat_type"" IS NULL AND NOT TRY_CAST(""flat_type"" AS TEXT) IN ('1 ROOM', '2 ROOM', '3 ROOM', '4 ROOM', '5 ROOM', 'EXECUTIVE', 'MULTI-GENERATION')","['hdb_resale_prices']","$.schema[0].properties[2].quality[0]","InvalidValuesCheckReference",,true,21.572665995336138 -"AddressBlockHouseNumber","hdb_resale_prices.block","hdb_resale_prices","sql","sql","conformity","House number component of a street address | Alphanumeric up to 10 characters (e.g. 12, 535, 358C)","FAILED",,"mustBe","1","0",1,"count","","SELECT COUNT(*) FROM ""hdb_resale_prices"" WHERE NOT REGEXP_MATCHES(TRY_CAST(block AS TEXT), '^[A-Za-z0-9]{1,10}$')","['hdb_resale_prices']","$.schema[0].properties[3].quality[0]","SQLColumnCheckReference",,false,20.722290995763615 -"floor_area_must_be_less_than_200","hdb_resale_prices.floor_area_sqm","hdb_resale_prices","sql","sql","consistency","Validates that floor area must be less than 200","FAILED","error","mustBe","12","0",12,"count","","SELECT COUNT(*) FROM ""hdb_resale_prices"" WHERE TRY_CAST(floor_area_sqm AS BIGINT) >= 200","['hdb_resale_prices']","$.schema[0].properties[6].quality[0]","SQLColumnCheckReference",,false,17.006708003464155 -"Year","hdb_resale_prices.lease_commence_date","hdb_resale_prices","sql","sql","conformity","Based on ISO 8601, assumed to be in UTC +8 | YYYY","FAILED",,"mustBe","3","0",3,"count","","SELECT COUNT(*) FROM ""hdb_resale_prices"" WHERE NOT REGEXP_MATCHES(TRY_CAST(lease_commence_date AS TEXT), '^[0-9]{4}$')","['hdb_resale_prices']","$.schema[0].properties[8].quality[0]","SQLColumnCheckReference",,false,20.985333001590334 -"resale_price_must_not_exceed_2m","hdb_resale_prices.resale_price","hdb_resale_prices","sql","sql","conformity","Resale price must not be more than 2 million SGD","PASSED","error","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""hdb_resale_prices"" WHERE TRY_CAST(resale_price AS BIGINT) > 2000000","['hdb_resale_prices']","$.schema[0].properties[10].quality[0]","SQLColumnCheckReference",,false,18.29566700325813 +"check_name","target","schema_name","engine","status","operator","actual_value","expected_value","failed_rows_count","aggregation_type","message","rendered_implementation","tables_in_query","check_path","check_ref_type","logical_type","is_generated","execution_time_ms" +"month_column_exists_check","hdb_resale_prices.month","hdb_resale_prices","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""month"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[0].name","DeclaredColumnExistsCheckReference","string","True",9.511541997198947 +"month_logical_type_check","hdb_resale_prices.month","hdb_resale_prices","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""hdb_resale_prices"" WHERE NOT ""month"" IS NULL AND TRY_CAST(""month"" AS TEXT) IS NULL","['hdb_resale_prices']","$.schema[0].properties[0].logicalType","LogicalTypeCheckReference","string","True",15.60741598950699 +"town_column_exists_check","hdb_resale_prices.town","hdb_resale_prices","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""town"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[1].name","DeclaredColumnExistsCheckReference",,"True",7.495708996430039 +"flat_type_column_exists_check","hdb_resale_prices.flat_type","hdb_resale_prices","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""flat_type"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[2].name","DeclaredColumnExistsCheckReference",,"True",7.0631249982398 +"block_column_exists_check","hdb_resale_prices.block","hdb_resale_prices","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""block"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[3].name","DeclaredColumnExistsCheckReference",,"True",8.010416000615805 +"street_name_column_exists_check","hdb_resale_prices.street_name","hdb_resale_prices","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""street_name"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[4].name","DeclaredColumnExistsCheckReference",,"True",6.567249991348945 +"storey_range_column_exists_check","hdb_resale_prices.storey_range","hdb_resale_prices","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""storey_range"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[5].name","DeclaredColumnExistsCheckReference",,"True",7.874708011513576 +"floor_area_sqm_column_exists_check","hdb_resale_prices.floor_area_sqm","hdb_resale_prices","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""floor_area_sqm"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[6].name","DeclaredColumnExistsCheckReference",,"True",7.521874998928979 +"flat_model_column_exists_check","hdb_resale_prices.flat_model","hdb_resale_prices","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""flat_model"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[7].name","DeclaredColumnExistsCheckReference",,"True",7.332666995353065 +"lease_commence_date_column_exists_check","hdb_resale_prices.lease_commence_date","hdb_resale_prices","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""lease_commence_date"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[8].name","DeclaredColumnExistsCheckReference",,"True",8.957708007073961 +"remaining_lease_column_exists_check","hdb_resale_prices.remaining_lease","hdb_resale_prices","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""remaining_lease"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[9].name","DeclaredColumnExistsCheckReference",,"True",7.871249996242113 +"resale_price_column_exists_check","hdb_resale_prices.resale_price","hdb_resale_prices","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""resale_price"" FROM ""hdb_resale_prices"" LIMIT 0) AS _vowl_column_exists","['hdb_resale_prices']","$.schema[0].properties[10].name","DeclaredColumnExistsCheckReference",,"True",8.052582998061553 +"hdb_resale_prices_rowCount","hdb_resale_prices","hdb_resale_prices","sql","PASSED","mustBeBetween","201879","[0, 30000000]",0,"count","","SELECT COUNT(*) FROM ""hdb_resale_prices""","['hdb_resale_prices']","$.schema[0].quality[0]","RowCountCheckReference",,"True",10.504459001822397 +"Month","hdb_resale_prices.month","hdb_resale_prices","sql","FAILED","mustBe","2","0",2,"count","","SELECT COUNT(*) FROM ""hdb_resale_prices"" WHERE NOT REGEXP_MATCHES(TRY_CAST(month AS TEXT), '^[0-9]{4}-(0[1-9]|1[0-2])$')","['hdb_resale_prices']","$.schema[0].properties[0].quality[0]","SQLColumnCheckReference","string","False",31.510332992183976 +"town_nullValues","hdb_resale_prices.town","hdb_resale_prices","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""hdb_resale_prices"" WHERE ""town"" IS NULL","['hdb_resale_prices']","$.schema[0].properties[1].quality[0]","NullValuesCheckReference",,"True",17.8000419982709 +"flat_type_invalidValues","hdb_resale_prices.flat_type","hdb_resale_prices","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""hdb_resale_prices"" WHERE NOT ""flat_type"" IS NULL AND NOT TRY_CAST(""flat_type"" AS TEXT) IN ('1 ROOM', '2 ROOM', '3 ROOM', '4 ROOM', '5 ROOM', 'EXECUTIVE', 'MULTI-GENERATION')","['hdb_resale_prices']","$.schema[0].properties[2].quality[0]","InvalidValuesCheckReference",,"True",75.48116600082722 +"AddressBlockHouseNumber","hdb_resale_prices.block","hdb_resale_prices","sql","FAILED","mustBe","1","0",1,"count","","SELECT COUNT(*) FROM ""hdb_resale_prices"" WHERE NOT REGEXP_MATCHES(TRY_CAST(block AS TEXT), '^[A-Za-z0-9]{1,10}$')","['hdb_resale_prices']","$.schema[0].properties[3].quality[0]","SQLColumnCheckReference",,"False",62.309749991982244 +"floor_area_must_be_less_than_200","hdb_resale_prices.floor_area_sqm","hdb_resale_prices","sql","FAILED","mustBe","12","0",12,"count","","SELECT COUNT(*) FROM ""hdb_resale_prices"" WHERE TRY_CAST(floor_area_sqm AS BIGINT) >= 200","['hdb_resale_prices']","$.schema[0].properties[6].quality[0]","SQLColumnCheckReference",,"False",65.37941700662486 +"Year","hdb_resale_prices.lease_commence_date","hdb_resale_prices","sql","FAILED","mustBe","3","0",3,"count","","SELECT COUNT(*) FROM ""hdb_resale_prices"" WHERE NOT REGEXP_MATCHES(TRY_CAST(lease_commence_date AS TEXT), '^[0-9]{4}$')","['hdb_resale_prices']","$.schema[0].properties[8].quality[0]","SQLColumnCheckReference",,"False",27.749708999181166 +"resale_price_must_not_exceed_2m","hdb_resale_prices.resale_price","hdb_resale_prices","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""hdb_resale_prices"" WHERE TRY_CAST(resale_price AS BIGINT) > 2000000","['hdb_resale_prices']","$.schema[0].properties[10].quality[0]","SQLColumnCheckReference",,"False",25.50208300817758 diff --git a/examples/vowl_demo_HDB_results_summary.json b/examples/vowl_demo_HDB_results_summary.json index db9e98a..26d74fa 100644 --- a/examples/vowl_demo_HDB_results_summary.json +++ b/examples/vowl_demo_HDB_results_summary.json @@ -14,7 +14,7 @@ "use_try_cast": true }, "failed_rows": 18, - "total_execution_time_ms": 209.8328730207868, + "total_execution_time_ms": 418.1032079795841, "success_rate": 80.0, "connection_results": { "hdb_resale_prices": { @@ -30,24 +30,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.757582999649458, + "execution_time_ms": 9.511541997198947, "check_path": "$.schema[0].properties[0].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'month' must exist in 'hdb_resale_prices'", - "severity": null, - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "month_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'month' must exist in 'hdb_resale_prices'", + "query": "SELECT COUNT(*) FROM (SELECT \"month\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "month" + }, "target": "hdb_resale_prices.month", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"month\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"month\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" }, { "name": "month_logical_type_check", @@ -56,24 +66,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 9.819083992624655, + "execution_time_ms": 15.60741598950699, "check_path": "$.schema[0].properties[0].logicalType", "check_ref_type": "LogicalTypeCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Values in 'month' must be valid string", - "severity": null, - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "month_logical_type_check", + "type": "sql", + "dimension": "conformity", + "description": "Values in 'month' must be valid string", + "query": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE NOT \"month\" IS NULL AND CAST(\"month\" AS VARCHAR) IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "string" + }, "target": "hdb_resale_prices.month", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE NOT \"month\" IS NULL AND TRY_CAST(\"month\" AS TEXT) IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE NOT \"month\" IS NULL AND TRY_CAST(\"month\" AS TEXT) IS NULL" }, { "name": "town_column_exists_check", @@ -82,23 +102,33 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.337791008059867, + "execution_time_ms": 7.495708996430039, "check_path": "$.schema[0].properties[1].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'town' must exist in 'hdb_resale_prices'", - "severity": null, - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "town_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'town' must exist in 'hdb_resale_prices'", + "query": "SELECT COUNT(*) FROM (SELECT \"town\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "town" + }, "target": "hdb_resale_prices.town", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"town\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"town\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" }, { "name": "flat_type_column_exists_check", @@ -107,23 +137,33 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.779207989689894, + "execution_time_ms": 7.0631249982398, "check_path": "$.schema[0].properties[2].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'flat_type' must exist in 'hdb_resale_prices'", - "severity": null, - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "flat_type_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'flat_type' must exist in 'hdb_resale_prices'", + "query": "SELECT COUNT(*) FROM (SELECT \"flat_type\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "flat_type" + }, "target": "hdb_resale_prices.flat_type", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"flat_type\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"flat_type\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" }, { "name": "block_column_exists_check", @@ -132,23 +172,33 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.379000009270385, + "execution_time_ms": 8.010416000615805, "check_path": "$.schema[0].properties[3].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'block' must exist in 'hdb_resale_prices'", - "severity": null, - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "block_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'block' must exist in 'hdb_resale_prices'", + "query": "SELECT COUNT(*) FROM (SELECT \"block\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "block" + }, "target": "hdb_resale_prices.block", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"block\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"block\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" }, { "name": "street_name_column_exists_check", @@ -157,23 +207,33 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.304541002260521, + "execution_time_ms": 6.567249991348945, "check_path": "$.schema[0].properties[4].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'street_name' must exist in 'hdb_resale_prices'", - "severity": null, - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "street_name_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'street_name' must exist in 'hdb_resale_prices'", + "query": "SELECT COUNT(*) FROM (SELECT \"street_name\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "street_name" + }, "target": "hdb_resale_prices.street_name", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"street_name\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"street_name\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" }, { "name": "storey_range_column_exists_check", @@ -182,23 +242,33 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.266625012154691, + "execution_time_ms": 7.874708011513576, "check_path": "$.schema[0].properties[5].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'storey_range' must exist in 'hdb_resale_prices'", - "severity": null, - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "storey_range_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'storey_range' must exist in 'hdb_resale_prices'", + "query": "SELECT COUNT(*) FROM (SELECT \"storey_range\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "storey_range" + }, "target": "hdb_resale_prices.storey_range", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"storey_range\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"storey_range\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" }, { "name": "floor_area_sqm_column_exists_check", @@ -207,23 +277,33 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.168709001736715, + "execution_time_ms": 7.521874998928979, "check_path": "$.schema[0].properties[6].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'floor_area_sqm' must exist in 'hdb_resale_prices'", - "severity": null, - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "floor_area_sqm_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'floor_area_sqm' must exist in 'hdb_resale_prices'", + "query": "SELECT COUNT(*) FROM (SELECT \"floor_area_sqm\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "floor_area_sqm" + }, "target": "hdb_resale_prices.floor_area_sqm", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"floor_area_sqm\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"floor_area_sqm\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" }, { "name": "flat_model_column_exists_check", @@ -232,23 +312,33 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.213583004660904, + "execution_time_ms": 7.332666995353065, "check_path": "$.schema[0].properties[7].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'flat_model' must exist in 'hdb_resale_prices'", - "severity": null, - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "flat_model_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'flat_model' must exist in 'hdb_resale_prices'", + "query": "SELECT COUNT(*) FROM (SELECT \"flat_model\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "flat_model" + }, "target": "hdb_resale_prices.flat_model", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"flat_model\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"flat_model\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" }, { "name": "lease_commence_date_column_exists_check", @@ -257,23 +347,33 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 4.999249998945743, + "execution_time_ms": 8.957708007073961, "check_path": "$.schema[0].properties[8].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'lease_commence_date' must exist in 'hdb_resale_prices'", - "severity": null, - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "lease_commence_date_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'lease_commence_date' must exist in 'hdb_resale_prices'", + "query": "SELECT COUNT(*) FROM (SELECT \"lease_commence_date\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "lease_commence_date" + }, "target": "hdb_resale_prices.lease_commence_date", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"lease_commence_date\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"lease_commence_date\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" }, { "name": "remaining_lease_column_exists_check", @@ -282,23 +382,33 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.113916995469481, + "execution_time_ms": 7.871249996242113, "check_path": "$.schema[0].properties[9].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'remaining_lease' must exist in 'hdb_resale_prices'", - "severity": null, - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "remaining_lease_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'remaining_lease' must exist in 'hdb_resale_prices'", + "query": "SELECT COUNT(*) FROM (SELECT \"remaining_lease\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "remaining_lease" + }, "target": "hdb_resale_prices.remaining_lease", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"remaining_lease\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"remaining_lease\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" }, { "name": "resale_price_column_exists_check", @@ -307,23 +417,33 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.010666005546227, + "execution_time_ms": 8.052582998061553, "check_path": "$.schema[0].properties[10].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'resale_price' must exist in 'hdb_resale_prices'", - "severity": null, - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "resale_price_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'resale_price' must exist in 'hdb_resale_prices'", + "query": "SELECT COUNT(*) FROM (SELECT \"resale_price\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "resale_price" + }, "target": "hdb_resale_prices.resale_price", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"resale_price\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"resale_price\" FROM \"hdb_resale_prices\" LIMIT 0) AS _vowl_column_exists" }, { "name": "hdb_resale_prices_rowCount", @@ -335,23 +455,41 @@ ], "actual_value": 201879, "failed_rows_count": 0, - "execution_time_ms": 7.587959000375122, + "execution_time_ms": 10.504459001822397, "check_path": "$.schema[0].quality[0]", "check_ref_type": "RowCountCheckReference", - "dimension": "completeness", - "type": "library", - "description": "Total row count for 'hdb_resale_prices'", - "severity": null, - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBeBetween", "is_generated": true, "engine": "sql", + "check_definition": { + "type": "library", + "metric": "rowCount", + "mustBeBetween": [ + 0, + 30000000 + ], + "dimension": "completeness", + "tags": [ + "vowl_generated_check" + ], + "description": "Total row count for 'hdb_resale_prices'" + }, + "contract_definition": { + "type": "library", + "metric": "rowCount", + "mustBeBetween": [ + 0, + 30000000 + ], + "dimension": "completeness" + }, "target": "hdb_resale_prices", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM \"hdb_resale_prices\"" + "rendered_implementation": "SELECT COUNT(*) FROM \"hdb_resale_prices\"" }, { "name": "Month", @@ -360,24 +498,42 @@ "expected_value": 0, "actual_value": 2, "failed_rows_count": 2, - "execution_time_ms": 21.381875005317852, + "execution_time_ms": 31.510332992183976, "check_path": "$.schema[0].properties[0].quality[0]", "check_ref_type": "SQLColumnCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Based on ISO 8601, assumed to be in UTC +8 | YYYY-MM", - "severity": null, - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": false, "engine": "sql", + "check_definition": { + "type": "sql", + "name": "Month", + "description": "Based on ISO 8601, assumed to be in UTC +8 | YYYY-MM", + "mustBe": 0, + "query": "SELECT COUNT(*)\nFROM \"hdb_resale_prices\"\nWHERE CAST(month AS TEXT) !~ '^[0-9]{4}-(0[1-9]|1[0-2])$';", + "tags": [ + "SG-DRM v5.0" + ], + "dimension": "conformity" + }, + "contract_definition": { + "type": "sql", + "name": "Month", + "description": "Based on ISO 8601, assumed to be in UTC +8 | YYYY-MM", + "mustBe": 0, + "query": "SELECT COUNT(*)\nFROM \"hdb_resale_prices\"\nWHERE CAST(month AS TEXT) !~ '^[0-9]{4}-(0[1-9]|1[0-2])$';", + "tags": [ + "SG-DRM v5.0" + ], + "dimension": "conformity" + }, "target": "hdb_resale_prices.month", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE NOT REGEXP_MATCHES(TRY_CAST(month AS TEXT), '^[0-9]{4}-(0[1-9]|1[0-2])$')" + "rendered_implementation": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE NOT REGEXP_MATCHES(TRY_CAST(month AS TEXT), '^[0-9]{4}-(0[1-9]|1[0-2])$')" }, { "name": "town_nullValues", @@ -386,23 +542,35 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 14.130416995612904, + "execution_time_ms": 17.8000419982709, "check_path": "$.schema[0].properties[1].quality[0]", "check_ref_type": "NullValuesCheckReference", - "dimension": "completeness", - "type": "library", - "description": "Count of NULL values in 'town' of 'hdb_resale_prices'", - "severity": null, - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "type": "library", + "metric": "nullValues", + "mustBe": 0, + "dimension": "completeness", + "tags": [ + "vowl_generated_check" + ], + "description": "Count of NULL values in 'town' of 'hdb_resale_prices'" + }, + "contract_definition": { + "type": "library", + "metric": "nullValues", + "mustBe": 0, + "dimension": "completeness" + }, "target": "hdb_resale_prices.town", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE \"town\" IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE \"town\" IS NULL" }, { "name": "flat_type_invalidValues", @@ -411,23 +579,57 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 21.572665995336138, + "execution_time_ms": 75.48116600082722, "check_path": "$.schema[0].properties[2].quality[0]", "check_ref_type": "InvalidValuesCheckReference", - "dimension": "conformity", - "type": "library", - "description": "Count of invalid values in 'flat_type' of 'hdb_resale_prices'", - "severity": null, - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "type": "library", + "metric": "invalidValues", + "mustBe": 0, + "dimension": "conformity", + "arguments": { + "validValues": [ + "1 ROOM", + "2 ROOM", + "3 ROOM", + "4 ROOM", + "5 ROOM", + "EXECUTIVE", + "MULTI-GENERATION" + ] + }, + "tags": [ + "vowl_generated_check" + ], + "description": "Count of invalid values in 'flat_type' of 'hdb_resale_prices'" + }, + "contract_definition": { + "type": "library", + "metric": "invalidValues", + "mustBe": 0, + "dimension": "conformity", + "arguments": { + "validValues": [ + "1 ROOM", + "2 ROOM", + "3 ROOM", + "4 ROOM", + "5 ROOM", + "EXECUTIVE", + "MULTI-GENERATION" + ] + } + }, "target": "hdb_resale_prices.flat_type", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE NOT \"flat_type\" IS NULL AND NOT TRY_CAST(\"flat_type\" AS TEXT) IN ('1 ROOM', '2 ROOM', '3 ROOM', '4 ROOM', '5 ROOM', 'EXECUTIVE', 'MULTI-GENERATION')" + "rendered_implementation": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE NOT \"flat_type\" IS NULL AND NOT TRY_CAST(\"flat_type\" AS TEXT) IN ('1 ROOM', '2 ROOM', '3 ROOM', '4 ROOM', '5 ROOM', 'EXECUTIVE', 'MULTI-GENERATION')" }, { "name": "AddressBlockHouseNumber", @@ -436,23 +638,41 @@ "expected_value": 0, "actual_value": 1, "failed_rows_count": 1, - "execution_time_ms": 20.722290995763615, + "execution_time_ms": 62.309749991982244, "check_path": "$.schema[0].properties[3].quality[0]", "check_ref_type": "SQLColumnCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "House number component of a street address | Alphanumeric up to 10 characters (e.g. 12, 535, 358C)", - "severity": null, - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": false, "engine": "sql", + "check_definition": { + "type": "sql", + "name": "AddressBlockHouseNumber", + "description": "House number component of a street address | Alphanumeric up to 10 characters (e.g. 12, 535, 358C)", + "mustBe": 0, + "query": "SELECT COUNT(*)\nFROM \"hdb_resale_prices\"\nWHERE CAST(block AS TEXT) !~ '^[A-Za-z0-9]{1,10}$';", + "tags": [ + "SG-DRM v5.0" + ], + "dimension": "conformity" + }, + "contract_definition": { + "type": "sql", + "name": "AddressBlockHouseNumber", + "description": "House number component of a street address | Alphanumeric up to 10 characters (e.g. 12, 535, 358C)", + "mustBe": 0, + "query": "SELECT COUNT(*)\nFROM \"hdb_resale_prices\"\nWHERE CAST(block AS TEXT) !~ '^[A-Za-z0-9]{1,10}$';", + "tags": [ + "SG-DRM v5.0" + ], + "dimension": "conformity" + }, "target": "hdb_resale_prices.block", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE NOT REGEXP_MATCHES(TRY_CAST(block AS TEXT), '^[A-Za-z0-9]{1,10}$')" + "rendered_implementation": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE NOT REGEXP_MATCHES(TRY_CAST(block AS TEXT), '^[A-Za-z0-9]{1,10}$')" }, { "name": "floor_area_must_be_less_than_200", @@ -461,23 +681,43 @@ "expected_value": 0, "actual_value": 12, "failed_rows_count": 12, - "execution_time_ms": 17.006708003464155, + "execution_time_ms": 65.37941700662486, "check_path": "$.schema[0].properties[6].quality[0]", "check_ref_type": "SQLColumnCheckReference", - "dimension": "consistency", - "type": "sql", - "description": "Validates that floor area must be less than 200", - "severity": "error", - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": false, "engine": "sql", + "check_definition": { + "name": "floor_area_must_be_less_than_200", + "description": "Validates that floor area must be less than 200", + "type": "sql", + "dimension": "consistency", + "severity": "error", + "query": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE floor_area_sqm >= 200", + "mustBe": 0, + "tags": [ + "User-Defined Quality Rule" + ] + }, + "contract_definition": { + "name": "floor_area_must_be_less_than_200", + "description": "Validates that floor area must be less than 200", + "type": "sql", + "dimension": "consistency", + "severity": "error", + "query": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE floor_area_sqm >= 200", + "mustBe": 0, + "tags": [ + "User-Defined Quality Rule" + ] + }, "target": "hdb_resale_prices.floor_area_sqm", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE TRY_CAST(floor_area_sqm AS BIGINT) >= 200" + "rendered_implementation": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE TRY_CAST(floor_area_sqm AS BIGINT) >= 200" }, { "name": "Year", @@ -486,23 +726,41 @@ "expected_value": 0, "actual_value": 3, "failed_rows_count": 3, - "execution_time_ms": 20.985333001590334, + "execution_time_ms": 27.749708999181166, "check_path": "$.schema[0].properties[8].quality[0]", "check_ref_type": "SQLColumnCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Based on ISO 8601, assumed to be in UTC +8 | YYYY", - "severity": null, - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": false, "engine": "sql", + "check_definition": { + "type": "sql", + "name": "Year", + "description": "Based on ISO 8601, assumed to be in UTC +8 | YYYY", + "mustBe": 0, + "query": "SELECT COUNT(*)\nFROM \"hdb_resale_prices\"\nWHERE CAST(lease_commence_date AS TEXT) !~ '^[0-9]{4}$';", + "tags": [ + "SG-DRM v5.0" + ], + "dimension": "conformity" + }, + "contract_definition": { + "type": "sql", + "name": "Year", + "description": "Based on ISO 8601, assumed to be in UTC +8 | YYYY", + "mustBe": 0, + "query": "SELECT COUNT(*)\nFROM \"hdb_resale_prices\"\nWHERE CAST(lease_commence_date AS TEXT) !~ '^[0-9]{4}$';", + "tags": [ + "SG-DRM v5.0" + ], + "dimension": "conformity" + }, "target": "hdb_resale_prices.lease_commence_date", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE NOT REGEXP_MATCHES(TRY_CAST(lease_commence_date AS TEXT), '^[0-9]{4}$')" + "rendered_implementation": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE NOT REGEXP_MATCHES(TRY_CAST(lease_commence_date AS TEXT), '^[0-9]{4}$')" }, { "name": "resale_price_must_not_exceed_2m", @@ -511,23 +769,43 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 18.29566700325813, + "execution_time_ms": 25.50208300817758, "check_path": "$.schema[0].properties[10].quality[0]", "check_ref_type": "SQLColumnCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Resale price must not be more than 2 million SGD", - "severity": "error", - "schema": "hdb_resale_prices", + "schema_name": "hdb_resale_prices", "operator": "mustBe", "is_generated": false, "engine": "sql", + "check_definition": { + "name": "resale_price_must_not_exceed_2m", + "description": "Resale price must not be more than 2 million SGD", + "type": "sql", + "dimension": "conformity", + "severity": "error", + "query": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE resale_price > 2000000", + "mustBe": 0, + "tags": [ + "Business Metadata Enrichment" + ] + }, + "contract_definition": { + "name": "resale_price_must_not_exceed_2m", + "description": "Resale price must not be more than 2 million SGD", + "type": "sql", + "dimension": "conformity", + "severity": "error", + "query": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE resale_price > 2000000", + "mustBe": 0, + "tags": [ + "Business Metadata Enrichment" + ] + }, "target": "hdb_resale_prices.resale_price", "aggregation_type": "count", "tables_in_query": [ "hdb_resale_prices" ], - "rule": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE TRY_CAST(resale_price AS BIGINT) > 2000000" + "rendered_implementation": "SELECT COUNT(*) FROM \"hdb_resale_prices\" WHERE TRY_CAST(resale_price AS BIGINT) > 2000000" } ], "contract_metadata": { diff --git a/examples/vowl_demo_multi_table_results_check_results.csv b/examples/vowl_demo_multi_table_results_check_results.csv index 457b601..f73ae8c 100644 --- a/examples/vowl_demo_multi_table_results_check_results.csv +++ b/examples/vowl_demo_multi_table_results_check_results.csv @@ -1,77 +1,77 @@ -"check_name","target","schema","engine","type","dimension","description","status","severity","operator","actual_value","expected_value","failed_rows_count","aggregation_type","message","rule","tables_in_query","check_path","check_ref_type","logical_type","is_generated","execution_time_ms" -"employee_id_column_exists_check","demo_employee_payroll.employee_id","demo_employee_payroll","sql","sql","conformity","Column 'employee_id' must exist in 'demo_employee_payroll'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""employee_id"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[0].name","DeclaredColumnExistsCheckReference","string",true,6.11975000356324 -"employee_id_logical_type_check","demo_employee_payroll.employee_id","demo_employee_payroll","sql","sql","conformity","Values in 'employee_id' must be valid string","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""employee_id"" IS NULL AND TRY_CAST(""employee_id"" AS TEXT) IS NULL","['demo_employee_payroll']","$.schema[0].properties[0].logicalType","LogicalTypeCheckReference","string",true,8.546666009351611 -"employee_id_logical_type_options_minLength_check","demo_employee_payroll.employee_id","demo_employee_payroll","sql","sql","conformity","Column 'employee_id' must have minimum length of 7","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""employee_id"" IS NULL AND LENGTH(TRY_CAST(""employee_id"" AS TEXT)) < 7","['demo_employee_payroll']","$.schema[0].properties[0].logicalTypeOptions.minLength","LogicalTypeOptionsCheckReference","string",true,25.609999996959232 -"employee_id_logical_type_options_maxLength_check","demo_employee_payroll.employee_id","demo_employee_payroll","sql","sql","conformity","Column 'employee_id' must have maximum length of 7","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""employee_id"" IS NULL AND LENGTH(TRY_CAST(""employee_id"" AS TEXT)) > 7","['demo_employee_payroll']","$.schema[0].properties[0].logicalTypeOptions.maxLength","LogicalTypeOptionsCheckReference","string",true,25.552249993779697 -"employee_id_logical_type_options_pattern_check","demo_employee_payroll.employee_id","demo_employee_payroll","sql","sql","conformity","Column 'employee_id' must match pattern '^e[0-9]{6}$'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""employee_id"" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(""employee_id"" AS TEXT), '^e[0-9]{6}$')","['demo_employee_payroll']","$.schema[0].properties[0].logicalTypeOptions.pattern","LogicalTypeOptionsCheckReference","string",true,25.541707989759743 -"employee_id_required_check","demo_employee_payroll.employee_id","demo_employee_payroll","sql","sql","completeness","Column 'employee_id' must not contain NULL values","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""employee_id"" IS NULL","['demo_employee_payroll']","$.schema[0].properties[0].required","RequiredCheckReference","string",true,5.14758299686946 -"employee_id_unique_check","demo_employee_payroll.employee_id","demo_employee_payroll","sql","sql","consistency","Column 'employee_id' must contain unique values","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""employee_id"" FROM ""demo_employee_payroll"" WHERE NOT ""employee_id"" IS NULL GROUP BY ""employee_id"" HAVING COUNT(*) > 1)","['demo_employee_payroll']","$.schema[0].properties[0].unique","UniqueCheckReference","string",true,6.010749988490716 -"employee_id_primary_key_check","demo_employee_payroll.employee_id","demo_employee_payroll","sql","sql","consistency","Primary key column 'employee_id' must be unique and not null","PASSED",,"mustBe","0","0",0,"custom","","SELECT ((SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""employee_id"" IS NULL)) + ((SELECT COUNT(*) FROM (SELECT ""employee_id"" FROM ""demo_employee_payroll"" WHERE NOT ""employee_id"" IS NULL GROUP BY ""employee_id"" HAVING COUNT(*) > 1)))","['demo_employee_payroll']","$.schema[0].properties[0].primaryKey","PrimaryKeyCheckReference","string",true,4.767832986544818 -"payroll_id_column_exists_check","demo_employee_payroll.payroll_id","demo_employee_payroll","sql","sql","conformity","Column 'payroll_id' must exist in 'demo_employee_payroll'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""payroll_id"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[1].name","DeclaredColumnExistsCheckReference","string",true,5.433333994005807 -"payroll_id_logical_type_check","demo_employee_payroll.payroll_id","demo_employee_payroll","sql","sql","conformity","Values in 'payroll_id' must be valid string","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""payroll_id"" IS NULL AND TRY_CAST(""payroll_id"" AS TEXT) IS NULL","['demo_employee_payroll']","$.schema[0].properties[1].logicalType","LogicalTypeCheckReference","string",true,8.019708999199793 -"payroll_id_logical_type_options_minLength_check","demo_employee_payroll.payroll_id","demo_employee_payroll","sql","sql","conformity","Column 'payroll_id' must have minimum length of 36","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""payroll_id"" IS NULL AND LENGTH(TRY_CAST(""payroll_id"" AS TEXT)) < 36","['demo_employee_payroll']","$.schema[0].properties[1].logicalTypeOptions.minLength","LogicalTypeOptionsCheckReference","string",true,25.0603340100497 -"payroll_id_logical_type_options_maxLength_check","demo_employee_payroll.payroll_id","demo_employee_payroll","sql","sql","conformity","Column 'payroll_id' must have maximum length of 36","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""payroll_id"" IS NULL AND LENGTH(TRY_CAST(""payroll_id"" AS TEXT)) > 36","['demo_employee_payroll']","$.schema[0].properties[1].logicalTypeOptions.maxLength","LogicalTypeOptionsCheckReference","string",true,25.716958989505656 -"payroll_id_logical_type_options_pattern_check","demo_employee_payroll.payroll_id","demo_employee_payroll","sql","sql","conformity","Column 'payroll_id' must match pattern '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""payroll_id"" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(""payroll_id"" AS TEXT), '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$')","['demo_employee_payroll']","$.schema[0].properties[1].logicalTypeOptions.pattern","LogicalTypeOptionsCheckReference","string",true,25.72074999625329 -"payroll_id_required_check","demo_employee_payroll.payroll_id","demo_employee_payroll","sql","sql","completeness","Column 'payroll_id' must not contain NULL values","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""payroll_id"" IS NULL","['demo_employee_payroll']","$.schema[0].properties[1].required","RequiredCheckReference","string",true,5.311874992912635 -"payroll_id_unique_check","demo_employee_payroll.payroll_id","demo_employee_payroll","sql","sql","consistency","Column 'payroll_id' must contain unique values","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""payroll_id"" FROM ""demo_employee_payroll"" WHERE NOT ""payroll_id"" IS NULL GROUP BY ""payroll_id"" HAVING COUNT(*) > 1)","['demo_employee_payroll']","$.schema[0].properties[1].unique","UniqueCheckReference","string",true,5.644874996505678 -"payroll_id_primary_key_check","demo_employee_payroll.payroll_id","demo_employee_payroll","sql","sql","consistency","Primary key column 'payroll_id' must be unique and not null","PASSED",,"mustBe","0","0",0,"custom","","SELECT ((SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""payroll_id"" IS NULL)) + ((SELECT COUNT(*) FROM (SELECT ""payroll_id"" FROM ""demo_employee_payroll"" WHERE NOT ""payroll_id"" IS NULL GROUP BY ""payroll_id"" HAVING COUNT(*) > 1)))","['demo_employee_payroll']","$.schema[0].properties[1].primaryKey","PrimaryKeyCheckReference","string",true,4.597375009325333 -"month_column_exists_check","demo_employee_payroll.month","demo_employee_payroll","sql","sql","conformity","Column 'month' must exist in 'demo_employee_payroll'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""month"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[2].name","DeclaredColumnExistsCheckReference","string",true,3.79183299082797 -"month_logical_type_check","demo_employee_payroll.month","demo_employee_payroll","sql","sql","conformity","Values in 'month' must be valid string","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""month"" IS NULL AND TRY_CAST(""month"" AS TEXT) IS NULL","['demo_employee_payroll']","$.schema[0].properties[2].logicalType","LogicalTypeCheckReference","string",true,5.933833002927713 -"month_logical_type_options_pattern_check","demo_employee_payroll.month","demo_employee_payroll","sql","sql","conformity","Column 'month' must match pattern '^\d{4}-\d{2}$'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""month"" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(""month"" AS TEXT), '^\d{4}-\d{2}$')","['demo_employee_payroll']","$.schema[0].properties[2].logicalTypeOptions.pattern","LogicalTypeOptionsCheckReference","string",true,22.392333004972897 -"month_required_check","demo_employee_payroll.month","demo_employee_payroll","sql","sql","completeness","Column 'month' must not contain NULL values","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""month"" IS NULL","['demo_employee_payroll']","$.schema[0].properties[2].required","RequiredCheckReference","string",true,3.6305000103311613 -"payroll_start_dt_column_exists_check","demo_employee_payroll.payroll_start_dt","demo_employee_payroll","sql","sql","conformity","Column 'payroll_start_dt' must exist in 'demo_employee_payroll'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""payroll_start_dt"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[3].name","DeclaredColumnExistsCheckReference","string",true,3.7929579993942752 -"payroll_start_dt_logical_type_check","demo_employee_payroll.payroll_start_dt","demo_employee_payroll","sql","sql","conformity","Values in 'payroll_start_dt' must be valid string","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""payroll_start_dt"" IS NULL AND TRY_CAST(""payroll_start_dt"" AS TEXT) IS NULL","['demo_employee_payroll']","$.schema[0].properties[3].logicalType","LogicalTypeCheckReference","string",true,6.0861669917358086 -"payroll_end_dt_column_exists_check","demo_employee_payroll.payroll_end_dt","demo_employee_payroll","sql","sql","conformity","Column 'payroll_end_dt' must exist in 'demo_employee_payroll'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""payroll_end_dt"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[4].name","DeclaredColumnExistsCheckReference","date",true,3.690333993290551 -"payroll_end_dt_logical_type_check","demo_employee_payroll.payroll_end_dt","demo_employee_payroll","sql","sql","conformity","Values in 'payroll_end_dt' must be valid date","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""payroll_end_dt"" IS NULL AND TRY_CAST(""payroll_end_dt"" AS DATE) IS NULL","['demo_employee_payroll']","$.schema[0].properties[4].logicalType","LogicalTypeCheckReference","date",true,5.9707079926738515 -"total_amt_column_exists_check","demo_employee_payroll.total_amt","demo_employee_payroll","sql","sql","conformity","Column 'total_amt' must exist in 'demo_employee_payroll'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""total_amt"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[5].name","DeclaredColumnExistsCheckReference","integer",true,3.760708001209423 -"total_amt_logical_type_check","demo_employee_payroll.total_amt","demo_employee_payroll","sql","sql","conformity","Values in 'total_amt' must be valid integer","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""total_amt"" IS NULL AND (TRY_CAST(""total_amt"" AS DOUBLE) IS NULL OR TRY_CAST(""total_amt"" AS DOUBLE) <> TRY_CAST(""total_amt"" AS BIGINT))","['demo_employee_payroll']","$.schema[0].properties[5].logicalType","LogicalTypeCheckReference","integer",true,6.376332996296696 -"employer_cpf_amt_column_exists_check","demo_employee_payroll.employer_cpf_amt","demo_employee_payroll","sql","sql","conformity","Column 'employer_cpf_amt' must exist in 'demo_employee_payroll'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""employer_cpf_amt"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[6].name","DeclaredColumnExistsCheckReference","number",true,3.631291998317465 -"employer_cpf_amt_logical_type_check","demo_employee_payroll.employer_cpf_amt","demo_employee_payroll","sql","sql","conformity","Values in 'employer_cpf_amt' must be valid number","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""employer_cpf_amt"" IS NULL AND TRY_CAST(""employer_cpf_amt"" AS DOUBLE) IS NULL","['demo_employee_payroll']","$.schema[0].properties[6].logicalType","LogicalTypeCheckReference","number",true,20.88683401234448 -"total_amt_employee_column_exists_check","demo_employee_payroll.total_amt_employee","demo_employee_payroll","sql","sql","conformity","Column 'total_amt_employee' must exist in 'demo_employee_payroll'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""total_amt_employee"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[7].name","DeclaredColumnExistsCheckReference","number",true,3.625041004852392 -"total_amt_employee_logical_type_check","demo_employee_payroll.total_amt_employee","demo_employee_payroll","sql","sql","conformity","Values in 'total_amt_employee' must be valid number","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""total_amt_employee"" IS NULL AND TRY_CAST(""total_amt_employee"" AS DOUBLE) IS NULL","['demo_employee_payroll']","$.schema[0].properties[7].logicalType","LogicalTypeCheckReference","number",true,5.843083010404371 -"total_amt_employee_required_check","demo_employee_payroll.total_amt_employee","demo_employee_payroll","sql","sql","completeness","Column 'total_amt_employee' must not contain NULL values","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""total_amt_employee"" IS NULL","['demo_employee_payroll']","$.schema[0].properties[7].required","RequiredCheckReference","number",true,3.5080829984508455 -"employee_cpf_amt_column_exists_check","demo_employee_payroll.employee_cpf_amt","demo_employee_payroll","sql","sql","conformity","Column 'employee_cpf_amt' must exist in 'demo_employee_payroll'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""employee_cpf_amt"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[8].name","DeclaredColumnExistsCheckReference","number",true,3.5907499986933544 -"employee_cpf_amt_logical_type_check","demo_employee_payroll.employee_cpf_amt","demo_employee_payroll","sql","sql","conformity","Values in 'employee_cpf_amt' must be valid number","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""employee_cpf_amt"" IS NULL AND TRY_CAST(""employee_cpf_amt"" AS DOUBLE) IS NULL","['demo_employee_payroll']","$.schema[0].properties[8].logicalType","LogicalTypeCheckReference","number",true,8.018790991627611 -"employee_cpf_amt_required_check","demo_employee_payroll.employee_cpf_amt","demo_employee_payroll","sql","sql","completeness","Column 'employee_cpf_amt' must not contain NULL values","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""employee_cpf_amt"" IS NULL","['demo_employee_payroll']","$.schema[0].properties[8].required","RequiredCheckReference","number",true,4.540499998256564 -"employee_gross_amt_column_exists_check","demo_employee_payroll.employee_gross_amt","demo_employee_payroll","sql","sql","conformity","Column 'employee_gross_amt' must exist in 'demo_employee_payroll'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""employee_gross_amt"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[9].name","DeclaredColumnExistsCheckReference","number",true,5.041374999564141 -"employee_gross_amt_logical_type_check","demo_employee_payroll.employee_gross_amt","demo_employee_payroll","sql","sql","conformity","Values in 'employee_gross_amt' must be valid number","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""employee_gross_amt"" IS NULL AND TRY_CAST(""employee_gross_amt"" AS DOUBLE) IS NULL","['demo_employee_payroll']","$.schema[0].properties[9].logicalType","LogicalTypeCheckReference","number",true,7.9375420027645305 -"employee_gross_amt_required_check","demo_employee_payroll.employee_gross_amt","demo_employee_payroll","sql","sql","completeness","Column 'employee_gross_amt' must not contain NULL values","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""employee_gross_amt"" IS NULL","['demo_employee_payroll']","$.schema[0].properties[9].required","RequiredCheckReference","number",true,4.815750013221987 -"phone_number_column_exists_check","demo_employee_payroll.phone_number","demo_employee_payroll","sql","sql","conformity","Column 'phone_number' must exist in 'demo_employee_payroll'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""phone_number"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[10].name","DeclaredColumnExistsCheckReference","string",true,5.021249991841614 -"phone_number_logical_type_check","demo_employee_payroll.phone_number","demo_employee_payroll","sql","sql","conformity","Values in 'phone_number' must be valid string","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""phone_number"" IS NULL AND TRY_CAST(""phone_number"" AS TEXT) IS NULL","['demo_employee_payroll']","$.schema[0].properties[10].logicalType","LogicalTypeCheckReference","string",true,8.666124995215796 -"phone_number_logical_type_options_minLength_check","demo_employee_payroll.phone_number","demo_employee_payroll","sql","sql","conformity","Column 'phone_number' must have minimum length of 8","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""phone_number"" IS NULL AND LENGTH(TRY_CAST(""phone_number"" AS TEXT)) < 8","['demo_employee_payroll']","$.schema[0].properties[10].logicalTypeOptions.minLength","LogicalTypeOptionsCheckReference","string",true,25.763958008610643 -"phone_number_logical_type_options_maxLength_check","demo_employee_payroll.phone_number","demo_employee_payroll","sql","sql","conformity","Column 'phone_number' must have maximum length of 11","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""phone_number"" IS NULL AND LENGTH(TRY_CAST(""phone_number"" AS TEXT)) > 11","['demo_employee_payroll']","$.schema[0].properties[10].logicalTypeOptions.maxLength","LogicalTypeOptionsCheckReference","string",true,30.281333994935267 -"phone_number_logical_type_options_pattern_check","demo_employee_payroll.phone_number","demo_employee_payroll","sql","sql","conformity","Column 'phone_number' must match pattern '^\+65[0-9]{8}$'","FAILED",,"mustBe","3","0",3,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""phone_number"" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(""phone_number"" AS TEXT), '^\+65[0-9]{8}$')","['demo_employee_payroll']","$.schema[0].properties[10].logicalTypeOptions.pattern","LogicalTypeOptionsCheckReference","string",true,26.03491699846927 -"total_amt_employee_equals_gross_plus_cpf","demo_employee_payroll","demo_employee_payroll","sql","sql","consistency","Validates that total_amt_employee equals the sum of employee_gross_amt and employee_cpf_amt for consistency","PASSED","error","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM demo_employee_payroll WHERE total_amt_employee <> (employee_gross_amt + employee_cpf_amt)","['demo_employee_payroll']","$.schema[0].quality[0]","SQLTableCheckReference",,false,15.561499996692874 -"demo_employee_payroll_rowCount","demo_employee_payroll","demo_employee_payroll","sql","library","completeness","Total row count for 'demo_employee_payroll'","PASSED",,"mustBeGreaterThan","3","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll""","['demo_employee_payroll']","$.schema[0].quality[3]","RowCountCheckReference",,true,7.83212500391528 -"employee_id_nullValues","demo_employee_payroll.employee_id","demo_employee_payroll","sql","library","completeness","Count of NULL values in 'employee_id' of 'demo_employee_payroll'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""employee_id"" IS NULL","['demo_employee_payroll']","$.schema[0].properties[0].quality[0]","NullValuesCheckReference","string",true,13.46854200528469 -"Month","demo_employee_payroll.month","demo_employee_payroll","sql","sql","conformity","Based on ISO 8601, assumed to be in UTC +8 | YYYY-MM","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM demo_employee_payroll WHERE NOT REGEXP_MATCHES(TRY_CAST(month AS TEXT), '^[0-9]{4}-(0[1-9]|1[0-2])$')","['demo_employee_payroll']","$.schema[0].properties[2].quality[0]","SQLColumnCheckReference","string",false,18.365625001024455 -"month_invalidValues","demo_employee_payroll.month","demo_employee_payroll","sql","library","conformity","Count of invalid values in 'month' of 'demo_employee_payroll'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""month"" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(""month"" AS TEXT), '^\d{4}-(0[1-9]|1[0-2])$')","['demo_employee_payroll']","$.schema[0].properties[2].quality[1]","InvalidValuesCheckReference","string",true,19.918834004784003 -"Date With Timezone","demo_employee_payroll.payroll_start_dt","demo_employee_payroll","sql","sql","conformity","If not in UTC +8 | YYYY-MM-DD[T]hh:mm+hh:mm e.g. 2015-01-01T12:00+02:00 for UTC +2","FAILED",,"mustBe","3","0",3,"count","","SELECT COUNT(*) FROM demo_employee_payroll WHERE NOT REGEXP_MATCHES(TRY_CAST(payroll_start_dt AS TEXT), '^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])T([01][0-9]|2[0-3]):[0-5][0-9][+-]([01][0-9]|2[0-3]):[0-5][0-9]$') OR LENGTH(TRY_CAST(payroll_start_dt AS TEXT)) > 22","['demo_employee_payroll']","$.schema[0].properties[3].quality[0]","SQLColumnCheckReference","string",false,17.863624991150573 -"Date With Timezone","demo_employee_payroll.payroll_end_dt","demo_employee_payroll","sql","sql","conformity","If not in UTC +8 | YYYY-MM-DD[T]hh:mm+hh:mm e.g. 2015-01-01T12:00+02:00 for UTC +2","FAILED",,"mustBe","3","0",3,"count","","SELECT COUNT(*) FROM demo_employee_payroll WHERE NOT REGEXP_MATCHES(TRY_CAST(payroll_end_dt AS TEXT), '^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])T([01][0-9]|2[0-3]):[0-5][0-9][+-]([01][0-9]|2[0-3]):[0-5][0-9]$') OR LENGTH(TRY_CAST(payroll_end_dt AS TEXT)) > 22","['demo_employee_payroll']","$.schema[0].properties[4].quality[0]","SQLColumnCheckReference","date",false,17.999208997935057 -"total_amt_equals_employer_cpf_plus_total_amt_employee","demo_employee_payroll.total_amt","demo_employee_payroll","sql","sql","consistency","Validates that total_amt equals the sum of employer_cpf_amt and total_amt_employee for consistency","PASSED","error","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM demo_employee_payroll WHERE total_amt <> (employer_cpf_amt + total_amt_employee) OR (NOT total_amt IS NULL AND (employer_cpf_amt IS NULL OR total_amt_employee IS NULL))","['demo_employee_payroll']","$.schema[0].properties[5].quality[0]","SQLColumnCheckReference","integer",false,14.988583003287204 -"total_amt_missingValues","demo_employee_payroll.total_amt","demo_employee_payroll","sql","library","completeness","Count of missing values in 'total_amt' of 'demo_employee_payroll'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""total_amt"" IS NULL OR TRY_CAST(""total_amt"" AS TEXT) IN ('')","['demo_employee_payroll']","$.schema[0].properties[5].quality[1]","MissingValuesCheckReference","integer",true,19.144125006278045 -"employee_id_column_exists_check","demo_employee_list.employee_id","demo_employee_list","sql","sql","conformity","Column 'employee_id' must exist in 'demo_employee_list'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""employee_id"" FROM ""demo_employee_list"" LIMIT 0) AS _vowl_column_exists","['demo_employee_list']","$.schema[1].properties[0].name","DeclaredColumnExistsCheckReference","string",true,5.450958997244015 -"employee_id_logical_type_check","demo_employee_list.employee_id","demo_employee_list","sql","sql","conformity","Values in 'employee_id' must be valid string","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE NOT ""employee_id"" IS NULL AND TRY_CAST(""employee_id"" AS TEXT) IS NULL","['demo_employee_list']","$.schema[1].properties[0].logicalType","LogicalTypeCheckReference","string",true,8.749791988520883 -"employee_id_logical_type_options_pattern_check","demo_employee_list.employee_id","demo_employee_list","sql","sql","conformity","Column 'employee_id' must match pattern '^e[0-9]+$'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE NOT ""employee_id"" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(""employee_id"" AS TEXT), '^e[0-9]+$')","['demo_employee_list']","$.schema[1].properties[0].logicalTypeOptions.pattern","LogicalTypeOptionsCheckReference","string",true,25.176167007884942 -"employee_id_logical_type_options_minLength_check","demo_employee_list.employee_id","demo_employee_list","sql","sql","conformity","Column 'employee_id' must have minimum length of 7","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE NOT ""employee_id"" IS NULL AND LENGTH(TRY_CAST(""employee_id"" AS TEXT)) < 7","['demo_employee_list']","$.schema[1].properties[0].logicalTypeOptions.minLength","LogicalTypeOptionsCheckReference","string",true,25.518875001580454 -"employee_id_logical_type_options_maxLength_check","demo_employee_list.employee_id","demo_employee_list","sql","sql","conformity","Column 'employee_id' must have maximum length of 7","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE NOT ""employee_id"" IS NULL AND LENGTH(TRY_CAST(""employee_id"" AS TEXT)) > 7","['demo_employee_list']","$.schema[1].properties[0].logicalTypeOptions.maxLength","LogicalTypeOptionsCheckReference","string",true,27.36829199420754 -"employee_id_required_check","demo_employee_list.employee_id","demo_employee_list","sql","sql","completeness","Column 'employee_id' must not contain NULL values","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE ""employee_id"" IS NULL","['demo_employee_list']","$.schema[1].properties[0].required","RequiredCheckReference","string",true,5.373791995225474 -"employee_id_unique_check","demo_employee_list.employee_id","demo_employee_list","sql","sql","consistency","Column 'employee_id' must contain unique values","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""employee_id"" FROM ""demo_employee_list"" WHERE NOT ""employee_id"" IS NULL GROUP BY ""employee_id"" HAVING COUNT(*) > 1)","['demo_employee_list']","$.schema[1].properties[0].unique","UniqueCheckReference","string",true,6.475375004811212 -"employee_id_primary_key_check","demo_employee_list.employee_id","demo_employee_list","sql","sql","consistency","Primary key column 'employee_id' must be unique and not null","PASSED",,"mustBe","0","0",0,"custom","","SELECT ((SELECT COUNT(*) FROM ""demo_employee_list"" WHERE ""employee_id"" IS NULL)) + ((SELECT COUNT(*) FROM (SELECT ""employee_id"" FROM ""demo_employee_list"" WHERE NOT ""employee_id"" IS NULL GROUP BY ""employee_id"" HAVING COUNT(*) > 1)))","['demo_employee_list']","$.schema[1].properties[0].primaryKey","PrimaryKeyCheckReference","string",true,5.115499996463768 -"employee_last_name_column_exists_check","demo_employee_list.employee_last_name","demo_employee_list","sql","sql","conformity","Column 'employee_last_name' must exist in 'demo_employee_list'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""employee_last_name"" FROM ""demo_employee_list"" LIMIT 0) AS _vowl_column_exists","['demo_employee_list']","$.schema[1].properties[1].name","DeclaredColumnExistsCheckReference","string",true,6.529666992719285 -"employee_last_name_logical_type_check","demo_employee_list.employee_last_name","demo_employee_list","sql","sql","conformity","Values in 'employee_last_name' must be valid string","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE NOT ""employee_last_name"" IS NULL AND TRY_CAST(""employee_last_name"" AS TEXT) IS NULL","['demo_employee_list']","$.schema[1].properties[1].logicalType","LogicalTypeCheckReference","string",true,10.200917007750832 -"employee_last_name_required_check","demo_employee_list.employee_last_name","demo_employee_list","sql","sql","completeness","Column 'employee_last_name' must not contain NULL values","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE ""employee_last_name"" IS NULL","['demo_employee_list']","$.schema[1].properties[1].required","RequiredCheckReference","string",true,5.171458004042506 -"employee_first_name_column_exists_check","demo_employee_list.employee_first_name","demo_employee_list","sql","sql","conformity","Column 'employee_first_name' must exist in 'demo_employee_list'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""employee_first_name"" FROM ""demo_employee_list"" LIMIT 0) AS _vowl_column_exists","['demo_employee_list']","$.schema[1].properties[2].name","DeclaredColumnExistsCheckReference","string",true,5.425541996373795 -"employee_first_name_logical_type_check","demo_employee_list.employee_first_name","demo_employee_list","sql","sql","conformity","Values in 'employee_first_name' must be valid string","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE NOT ""employee_first_name"" IS NULL AND TRY_CAST(""employee_first_name"" AS TEXT) IS NULL","['demo_employee_list']","$.schema[1].properties[2].logicalType","LogicalTypeCheckReference","string",true,8.74150000163354 -"employee_first_name_required_check","demo_employee_list.employee_first_name","demo_employee_list","sql","sql","completeness","Column 'employee_first_name' must not contain NULL values","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE ""employee_first_name"" IS NULL","['demo_employee_list']","$.schema[1].properties[2].required","RequiredCheckReference","string",true,6.4498749998165295 -"current_title_column_exists_check","demo_employee_list.current_title","demo_employee_list","sql","sql","conformity","Column 'current_title' must exist in 'demo_employee_list'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""current_title"" FROM ""demo_employee_list"" LIMIT 0) AS _vowl_column_exists","['demo_employee_list']","$.schema[1].properties[3].name","DeclaredColumnExistsCheckReference","string",true,7.187124996562488 -"current_title_logical_type_check","demo_employee_list.current_title","demo_employee_list","sql","sql","conformity","Values in 'current_title' must be valid string","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE NOT ""current_title"" IS NULL AND TRY_CAST(""current_title"" AS TEXT) IS NULL","['demo_employee_list']","$.schema[1].properties[3].logicalType","LogicalTypeCheckReference","string",true,9.6572080074111 -"phone_number_column_exists_check","demo_employee_list.phone_number","demo_employee_list","sql","sql","conformity","Column 'phone_number' must exist in 'demo_employee_list'","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""phone_number"" FROM ""demo_employee_list"" LIMIT 0) AS _vowl_column_exists","['demo_employee_list']","$.schema[1].properties[4].name","DeclaredColumnExistsCheckReference","string",true,5.327791994204745 -"phone_number_logical_type_check","demo_employee_list.phone_number","demo_employee_list","sql","sql","conformity","Values in 'phone_number' must be valid string","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE NOT ""phone_number"" IS NULL AND TRY_CAST(""phone_number"" AS TEXT) IS NULL","['demo_employee_list']","$.schema[1].properties[4].logicalType","LogicalTypeCheckReference","string",true,8.912875011446886 -"phone_number_logical_type_options_pattern_check","demo_employee_list.phone_number","demo_employee_list","sql","sql","conformity","Column 'phone_number' must match pattern '^\+[0-9]+$'","FAILED",,"mustBe","2","0",2,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE NOT ""phone_number"" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(""phone_number"" AS TEXT), '^\+[0-9]+$')","['demo_employee_list']","$.schema[1].properties[4].logicalTypeOptions.pattern","LogicalTypeOptionsCheckReference","string",true,44.16029200365301 -"phone_number_unique_check","demo_employee_list.phone_number","demo_employee_list","sql","sql","consistency","Column 'phone_number' must contain unique values","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""phone_number"" FROM ""demo_employee_list"" WHERE NOT ""phone_number"" IS NULL GROUP BY ""phone_number"" HAVING COUNT(*) > 1)","['demo_employee_list']","$.schema[1].properties[4].unique","UniqueCheckReference","string",true,4.733582987682894 -"PersonName","demo_employee_list.employee_last_name","demo_employee_list","sql","sql","conformity","PersonName may include various name types for UIN/FIN holders including legal names, abbreviated names, Hanyu Pinyin names, alias names, married names, and changed names | X(66)","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM demo_employee_list WHERE NOT employee_last_name IS NULL AND LENGTH(TRY_CAST(employee_last_name AS TEXT)) > 66","['demo_employee_list']","$.schema[1].properties[1].quality[0]","SQLColumnCheckReference","string",false,14.657958003226668 -"PersonName","demo_employee_list.employee_first_name","demo_employee_list","sql","sql","conformity","PersonName may include various name types for UIN/FIN holders including legal names, abbreviated names, Hanyu Pinyin names, alias names, married names, and changed names | X(66)","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM demo_employee_list WHERE NOT employee_first_name IS NULL AND LENGTH(TRY_CAST(employee_first_name AS TEXT)) > 66","['demo_employee_list']","$.schema[1].properties[2].quality[0]","SQLColumnCheckReference","string",false,14.439665988902561 -"TelephoneNumber","demo_employee_list.phone_number","demo_employee_list","sql","sql","conformity","Based on ITU T E.164 | e.g. 095236","PASSED",,"mustBe","0","0",0,"count","","SELECT COUNT(*) FROM demo_employee_list WHERE NOT REGEXP_MATCHES(TRY_CAST(phone_number AS TEXT), '^[0-9]{1,12}$')","['demo_employee_list']","$.schema[1].properties[4].quality[0]","SQLColumnCheckReference","string",false,14.721458006533794 -"employee_id_exists_in_master_list","demo_employee_payroll","demo_employee_payroll","sql","sql","consistency","Referential integrity check: Ensures all employee IDs in the payroll table exist in the reference employee list.","FAILED",,"mustBe","1","0",1,"count","","SELECT COUNT(*) FROM demo_employee_payroll AS payroll LEFT JOIN demo_employee_list AS ref ON payroll.employee_id = ref.employee_id WHERE ref.employee_id IS NULL","['demo_employee_list', 'demo_employee_payroll']","$.schema[0].quality[1]","SQLTableCheckReference",,false,47.93762500048615 -"phone_number_exists_in_master_list","demo_employee_payroll","demo_employee_payroll","sql","sql","consistency","Ensures the phone numbers recorded in payroll matches an entry in the master employee list.","FAILED",,"mustBe","2","0",2,"count","","SELECT COUNT(*) FROM demo_employee_payroll AS payroll LEFT JOIN demo_employee_list AS ref ON payroll.phone_number = ref.phone_number WHERE NOT payroll.phone_number IS NULL AND ref.phone_number IS NULL","['demo_employee_list', 'demo_employee_payroll']","$.schema[0].quality[2]","SQLTableCheckReference",,false,14.986333000706509 +"check_name","target","schema_name","engine","status","operator","actual_value","expected_value","failed_rows_count","aggregation_type","message","rendered_implementation","tables_in_query","check_path","check_ref_type","logical_type","is_generated","execution_time_ms" +"employee_id_column_exists_check","demo_employee_payroll.employee_id","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""employee_id"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[0].name","DeclaredColumnExistsCheckReference","string","True",7.536083998274989 +"employee_id_logical_type_check","demo_employee_payroll.employee_id","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""employee_id"" IS NULL AND TRY_CAST(""employee_id"" AS TEXT) IS NULL","['demo_employee_payroll']","$.schema[0].properties[0].logicalType","LogicalTypeCheckReference","string","True",11.358792005921714 +"employee_id_logical_type_options_minLength_check","demo_employee_payroll.employee_id","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""employee_id"" IS NULL AND LENGTH(TRY_CAST(""employee_id"" AS TEXT)) < 7","['demo_employee_payroll']","$.schema[0].properties[0].logicalTypeOptions.minLength","LogicalTypeOptionsCheckReference","string","True",33.94816699437797 +"employee_id_logical_type_options_maxLength_check","demo_employee_payroll.employee_id","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""employee_id"" IS NULL AND LENGTH(TRY_CAST(""employee_id"" AS TEXT)) > 7","['demo_employee_payroll']","$.schema[0].properties[0].logicalTypeOptions.maxLength","LogicalTypeOptionsCheckReference","string","True",33.068207994801924 +"employee_id_logical_type_options_pattern_check","demo_employee_payroll.employee_id","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""employee_id"" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(""employee_id"" AS TEXT), '^e[0-9]{6}$')","['demo_employee_payroll']","$.schema[0].properties[0].logicalTypeOptions.pattern","LogicalTypeOptionsCheckReference","string","True",29.56283300591167 +"employee_id_required_check","demo_employee_payroll.employee_id","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""employee_id"" IS NULL","['demo_employee_payroll']","$.schema[0].properties[0].required","RequiredCheckReference","string","True",5.534250012715347 +"employee_id_unique_check","demo_employee_payroll.employee_id","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""employee_id"" FROM ""demo_employee_payroll"" WHERE NOT ""employee_id"" IS NULL GROUP BY ""employee_id"" HAVING COUNT(*) > 1)","['demo_employee_payroll']","$.schema[0].properties[0].unique","UniqueCheckReference","string","True",6.91154100059066 +"employee_id_primary_key_check","demo_employee_payroll.employee_id","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"custom","","SELECT ((SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""employee_id"" IS NULL)) + ((SELECT COUNT(*) FROM (SELECT ""employee_id"" FROM ""demo_employee_payroll"" WHERE NOT ""employee_id"" IS NULL GROUP BY ""employee_id"" HAVING COUNT(*) > 1)))","['demo_employee_payroll']","$.schema[0].properties[0].primaryKey","PrimaryKeyCheckReference","string","True",6.103708001319319 +"payroll_id_column_exists_check","demo_employee_payroll.payroll_id","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""payroll_id"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[1].name","DeclaredColumnExistsCheckReference","string","True",5.827750006574206 +"payroll_id_logical_type_check","demo_employee_payroll.payroll_id","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""payroll_id"" IS NULL AND TRY_CAST(""payroll_id"" AS TEXT) IS NULL","['demo_employee_payroll']","$.schema[0].properties[1].logicalType","LogicalTypeCheckReference","string","True",9.204667003359646 +"payroll_id_logical_type_options_minLength_check","demo_employee_payroll.payroll_id","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""payroll_id"" IS NULL AND LENGTH(TRY_CAST(""payroll_id"" AS TEXT)) < 36","['demo_employee_payroll']","$.schema[0].properties[1].logicalTypeOptions.minLength","LogicalTypeOptionsCheckReference","string","True",28.868458000943065 +"payroll_id_logical_type_options_maxLength_check","demo_employee_payroll.payroll_id","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""payroll_id"" IS NULL AND LENGTH(TRY_CAST(""payroll_id"" AS TEXT)) > 36","['demo_employee_payroll']","$.schema[0].properties[1].logicalTypeOptions.maxLength","LogicalTypeOptionsCheckReference","string","True",30.009207999682985 +"payroll_id_logical_type_options_pattern_check","demo_employee_payroll.payroll_id","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""payroll_id"" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(""payroll_id"" AS TEXT), '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$')","['demo_employee_payroll']","$.schema[0].properties[1].logicalTypeOptions.pattern","LogicalTypeOptionsCheckReference","string","True",34.06583299511112 +"payroll_id_required_check","demo_employee_payroll.payroll_id","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""payroll_id"" IS NULL","['demo_employee_payroll']","$.schema[0].properties[1].required","RequiredCheckReference","string","True",33.69458299130201 +"payroll_id_unique_check","demo_employee_payroll.payroll_id","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""payroll_id"" FROM ""demo_employee_payroll"" WHERE NOT ""payroll_id"" IS NULL GROUP BY ""payroll_id"" HAVING COUNT(*) > 1)","['demo_employee_payroll']","$.schema[0].properties[1].unique","UniqueCheckReference","string","True",11.5525409928523 +"payroll_id_primary_key_check","demo_employee_payroll.payroll_id","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"custom","","SELECT ((SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""payroll_id"" IS NULL)) + ((SELECT COUNT(*) FROM (SELECT ""payroll_id"" FROM ""demo_employee_payroll"" WHERE NOT ""payroll_id"" IS NULL GROUP BY ""payroll_id"" HAVING COUNT(*) > 1)))","['demo_employee_payroll']","$.schema[0].properties[1].primaryKey","PrimaryKeyCheckReference","string","True",9.977291003451683 +"month_column_exists_check","demo_employee_payroll.month","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""month"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[2].name","DeclaredColumnExistsCheckReference","string","True",11.00429099460598 +"month_logical_type_check","demo_employee_payroll.month","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""month"" IS NULL AND TRY_CAST(""month"" AS TEXT) IS NULL","['demo_employee_payroll']","$.schema[0].properties[2].logicalType","LogicalTypeCheckReference","string","True",14.883666997775435 +"month_logical_type_options_pattern_check","demo_employee_payroll.month","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""month"" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(""month"" AS TEXT), '^\d{4}-\d{2}$')","['demo_employee_payroll']","$.schema[0].properties[2].logicalTypeOptions.pattern","LogicalTypeOptionsCheckReference","string","True",50.78175000380725 +"month_required_check","demo_employee_payroll.month","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""month"" IS NULL","['demo_employee_payroll']","$.schema[0].properties[2].required","RequiredCheckReference","string","True",14.663374997326173 +"payroll_start_dt_column_exists_check","demo_employee_payroll.payroll_start_dt","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""payroll_start_dt"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[3].name","DeclaredColumnExistsCheckReference","string","True",7.284459003130905 +"payroll_start_dt_logical_type_check","demo_employee_payroll.payroll_start_dt","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""payroll_start_dt"" IS NULL AND TRY_CAST(""payroll_start_dt"" AS TEXT) IS NULL","['demo_employee_payroll']","$.schema[0].properties[3].logicalType","LogicalTypeCheckReference","string","True",8.672416995977983 +"payroll_end_dt_column_exists_check","demo_employee_payroll.payroll_end_dt","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""payroll_end_dt"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[4].name","DeclaredColumnExistsCheckReference","date","True",5.804250002256595 +"payroll_end_dt_logical_type_check","demo_employee_payroll.payroll_end_dt","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""payroll_end_dt"" IS NULL AND TRY_CAST(""payroll_end_dt"" AS DATE) IS NULL","['demo_employee_payroll']","$.schema[0].properties[4].logicalType","LogicalTypeCheckReference","date","True",9.189958000206389 +"total_amt_column_exists_check","demo_employee_payroll.total_amt","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""total_amt"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[5].name","DeclaredColumnExistsCheckReference","integer","True",4.887916991719976 +"total_amt_logical_type_check","demo_employee_payroll.total_amt","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""total_amt"" IS NULL AND (TRY_CAST(""total_amt"" AS DOUBLE) IS NULL OR TRY_CAST(""total_amt"" AS DOUBLE) <> TRY_CAST(""total_amt"" AS BIGINT))","['demo_employee_payroll']","$.schema[0].properties[5].logicalType","LogicalTypeCheckReference","integer","True",39.02799999923445 +"employer_cpf_amt_column_exists_check","demo_employee_payroll.employer_cpf_amt","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""employer_cpf_amt"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[6].name","DeclaredColumnExistsCheckReference","number","True",5.7217919966205955 +"employer_cpf_amt_logical_type_check","demo_employee_payroll.employer_cpf_amt","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""employer_cpf_amt"" IS NULL AND TRY_CAST(""employer_cpf_amt"" AS DOUBLE) IS NULL","['demo_employee_payroll']","$.schema[0].properties[6].logicalType","LogicalTypeCheckReference","number","True",9.278917001211084 +"total_amt_employee_column_exists_check","demo_employee_payroll.total_amt_employee","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""total_amt_employee"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[7].name","DeclaredColumnExistsCheckReference","number","True",12.201416000607423 +"total_amt_employee_logical_type_check","demo_employee_payroll.total_amt_employee","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""total_amt_employee"" IS NULL AND TRY_CAST(""total_amt_employee"" AS DOUBLE) IS NULL","['demo_employee_payroll']","$.schema[0].properties[7].logicalType","LogicalTypeCheckReference","number","True",11.42637500015553 +"total_amt_employee_required_check","demo_employee_payroll.total_amt_employee","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""total_amt_employee"" IS NULL","['demo_employee_payroll']","$.schema[0].properties[7].required","RequiredCheckReference","number","True",6.754499991075136 +"employee_cpf_amt_column_exists_check","demo_employee_payroll.employee_cpf_amt","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""employee_cpf_amt"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[8].name","DeclaredColumnExistsCheckReference","number","True",7.811667004716583 +"employee_cpf_amt_logical_type_check","demo_employee_payroll.employee_cpf_amt","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""employee_cpf_amt"" IS NULL AND TRY_CAST(""employee_cpf_amt"" AS DOUBLE) IS NULL","['demo_employee_payroll']","$.schema[0].properties[8].logicalType","LogicalTypeCheckReference","number","True",10.776291994261555 +"employee_cpf_amt_required_check","demo_employee_payroll.employee_cpf_amt","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""employee_cpf_amt"" IS NULL","['demo_employee_payroll']","$.schema[0].properties[8].required","RequiredCheckReference","number","True",7.453291007550433 +"employee_gross_amt_column_exists_check","demo_employee_payroll.employee_gross_amt","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""employee_gross_amt"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[9].name","DeclaredColumnExistsCheckReference","number","True",6.992291993810795 +"employee_gross_amt_logical_type_check","demo_employee_payroll.employee_gross_amt","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""employee_gross_amt"" IS NULL AND TRY_CAST(""employee_gross_amt"" AS DOUBLE) IS NULL","['demo_employee_payroll']","$.schema[0].properties[9].logicalType","LogicalTypeCheckReference","number","True",11.974291992373765 +"employee_gross_amt_required_check","demo_employee_payroll.employee_gross_amt","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""employee_gross_amt"" IS NULL","['demo_employee_payroll']","$.schema[0].properties[9].required","RequiredCheckReference","number","True",6.4171249978244305 +"phone_number_column_exists_check","demo_employee_payroll.phone_number","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""phone_number"" FROM ""demo_employee_payroll"" LIMIT 0) AS _vowl_column_exists","['demo_employee_payroll']","$.schema[0].properties[10].name","DeclaredColumnExistsCheckReference","string","True",9.263290994567797 +"phone_number_logical_type_check","demo_employee_payroll.phone_number","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""phone_number"" IS NULL AND TRY_CAST(""phone_number"" AS TEXT) IS NULL","['demo_employee_payroll']","$.schema[0].properties[10].logicalType","LogicalTypeCheckReference","string","True",17.175583998323418 +"phone_number_logical_type_options_minLength_check","demo_employee_payroll.phone_number","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""phone_number"" IS NULL AND LENGTH(TRY_CAST(""phone_number"" AS TEXT)) < 8","['demo_employee_payroll']","$.schema[0].properties[10].logicalTypeOptions.minLength","LogicalTypeOptionsCheckReference","string","True",43.42120799992699 +"phone_number_logical_type_options_maxLength_check","demo_employee_payroll.phone_number","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""phone_number"" IS NULL AND LENGTH(TRY_CAST(""phone_number"" AS TEXT)) > 11","['demo_employee_payroll']","$.schema[0].properties[10].logicalTypeOptions.maxLength","LogicalTypeOptionsCheckReference","string","True",45.03570799715817 +"phone_number_logical_type_options_pattern_check","demo_employee_payroll.phone_number","demo_employee_payroll","sql","FAILED","mustBe","3","0",3,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""phone_number"" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(""phone_number"" AS TEXT), '^\+65[0-9]{8}$')","['demo_employee_payroll']","$.schema[0].properties[10].logicalTypeOptions.pattern","LogicalTypeOptionsCheckReference","string","True",43.62716700416058 +"total_amt_employee_equals_gross_plus_cpf","demo_employee_payroll","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM demo_employee_payroll WHERE total_amt_employee <> (employee_gross_amt + employee_cpf_amt)","['demo_employee_payroll']","$.schema[0].quality[0]","SQLTableCheckReference",,"False",21.877167004277 +"demo_employee_payroll_rowCount","demo_employee_payroll","demo_employee_payroll","sql","PASSED","mustBeGreaterThan","3","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll""","['demo_employee_payroll']","$.schema[0].quality[3]","RowCountCheckReference",,"True",42.15633300191257 +"employee_id_nullValues","demo_employee_payroll.employee_id","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""employee_id"" IS NULL","['demo_employee_payroll']","$.schema[0].properties[0].quality[0]","NullValuesCheckReference","string","True",27.207833001739345 +"Month","demo_employee_payroll.month","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM demo_employee_payroll WHERE NOT REGEXP_MATCHES(TRY_CAST(month AS TEXT), '^[0-9]{4}-(0[1-9]|1[0-2])$')","['demo_employee_payroll']","$.schema[0].properties[2].quality[0]","SQLColumnCheckReference","string","False",32.52749999228399 +"month_invalidValues","demo_employee_payroll.month","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE NOT ""month"" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(""month"" AS TEXT), '^\d{4}-(0[1-9]|1[0-2])$')","['demo_employee_payroll']","$.schema[0].properties[2].quality[1]","InvalidValuesCheckReference","string","True",26.39941700908821 +"Date With Timezone","demo_employee_payroll.payroll_start_dt","demo_employee_payroll","sql","FAILED","mustBe","3","0",3,"count","","SELECT COUNT(*) FROM demo_employee_payroll WHERE NOT REGEXP_MATCHES(TRY_CAST(payroll_start_dt AS TEXT), '^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])T([01][0-9]|2[0-3]):[0-5][0-9][+-]([01][0-9]|2[0-3]):[0-5][0-9]$') OR LENGTH(TRY_CAST(payroll_start_dt AS TEXT)) > 22","['demo_employee_payroll']","$.schema[0].properties[3].quality[0]","SQLColumnCheckReference","string","False",21.720541000831872 +"Date With Timezone","demo_employee_payroll.payroll_end_dt","demo_employee_payroll","sql","FAILED","mustBe","3","0",3,"count","","SELECT COUNT(*) FROM demo_employee_payroll WHERE NOT REGEXP_MATCHES(TRY_CAST(payroll_end_dt AS TEXT), '^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])T([01][0-9]|2[0-3]):[0-5][0-9][+-]([01][0-9]|2[0-3]):[0-5][0-9]$') OR LENGTH(TRY_CAST(payroll_end_dt AS TEXT)) > 22","['demo_employee_payroll']","$.schema[0].properties[4].quality[0]","SQLColumnCheckReference","date","False",21.855708007933572 +"total_amt_equals_employer_cpf_plus_total_amt_employee","demo_employee_payroll.total_amt","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM demo_employee_payroll WHERE total_amt <> (employer_cpf_amt + total_amt_employee) OR (NOT total_amt IS NULL AND (employer_cpf_amt IS NULL OR total_amt_employee IS NULL))","['demo_employee_payroll']","$.schema[0].properties[5].quality[0]","SQLColumnCheckReference","integer","False",18.529374996433035 +"total_amt_missingValues","demo_employee_payroll.total_amt","demo_employee_payroll","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_payroll"" WHERE ""total_amt"" IS NULL OR TRY_CAST(""total_amt"" AS TEXT) IN ('')","['demo_employee_payroll']","$.schema[0].properties[5].quality[1]","MissingValuesCheckReference","integer","True",33.28112499730196 +"employee_id_column_exists_check","demo_employee_list.employee_id","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""employee_id"" FROM ""demo_employee_list"" LIMIT 0) AS _vowl_column_exists","['demo_employee_list']","$.schema[1].properties[0].name","DeclaredColumnExistsCheckReference","string","True",10.498125004232861 +"employee_id_logical_type_check","demo_employee_list.employee_id","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE NOT ""employee_id"" IS NULL AND TRY_CAST(""employee_id"" AS TEXT) IS NULL","['demo_employee_list']","$.schema[1].properties[0].logicalType","LogicalTypeCheckReference","string","True",20.03816599608399 +"employee_id_logical_type_options_pattern_check","demo_employee_list.employee_id","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE NOT ""employee_id"" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(""employee_id"" AS TEXT), '^e[0-9]+$')","['demo_employee_list']","$.schema[1].properties[0].logicalTypeOptions.pattern","LogicalTypeOptionsCheckReference","string","True",42.80774999642745 +"employee_id_logical_type_options_minLength_check","demo_employee_list.employee_id","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE NOT ""employee_id"" IS NULL AND LENGTH(TRY_CAST(""employee_id"" AS TEXT)) < 7","['demo_employee_list']","$.schema[1].properties[0].logicalTypeOptions.minLength","LogicalTypeOptionsCheckReference","string","True",48.060291999718174 +"employee_id_logical_type_options_maxLength_check","demo_employee_list.employee_id","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE NOT ""employee_id"" IS NULL AND LENGTH(TRY_CAST(""employee_id"" AS TEXT)) > 7","['demo_employee_list']","$.schema[1].properties[0].logicalTypeOptions.maxLength","LogicalTypeOptionsCheckReference","string","True",34.422624987200834 +"employee_id_required_check","demo_employee_list.employee_id","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE ""employee_id"" IS NULL","['demo_employee_list']","$.schema[1].properties[0].required","RequiredCheckReference","string","True",8.225458004744723 +"employee_id_unique_check","demo_employee_list.employee_id","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""employee_id"" FROM ""demo_employee_list"" WHERE NOT ""employee_id"" IS NULL GROUP BY ""employee_id"" HAVING COUNT(*) > 1)","['demo_employee_list']","$.schema[1].properties[0].unique","UniqueCheckReference","string","True",8.616082996013574 +"employee_id_primary_key_check","demo_employee_list.employee_id","demo_employee_list","sql","PASSED","mustBe","0","0",0,"custom","","SELECT ((SELECT COUNT(*) FROM ""demo_employee_list"" WHERE ""employee_id"" IS NULL)) + ((SELECT COUNT(*) FROM (SELECT ""employee_id"" FROM ""demo_employee_list"" WHERE NOT ""employee_id"" IS NULL GROUP BY ""employee_id"" HAVING COUNT(*) > 1)))","['demo_employee_list']","$.schema[1].properties[0].primaryKey","PrimaryKeyCheckReference","string","True",6.8277919926913455 +"employee_last_name_column_exists_check","demo_employee_list.employee_last_name","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""employee_last_name"" FROM ""demo_employee_list"" LIMIT 0) AS _vowl_column_exists","['demo_employee_list']","$.schema[1].properties[1].name","DeclaredColumnExistsCheckReference","string","True",17.24595800624229 +"employee_last_name_logical_type_check","demo_employee_list.employee_last_name","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE NOT ""employee_last_name"" IS NULL AND TRY_CAST(""employee_last_name"" AS TEXT) IS NULL","['demo_employee_list']","$.schema[1].properties[1].logicalType","LogicalTypeCheckReference","string","True",14.31154199235607 +"employee_last_name_required_check","demo_employee_list.employee_last_name","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE ""employee_last_name"" IS NULL","['demo_employee_list']","$.schema[1].properties[1].required","RequiredCheckReference","string","True",6.7928750067949295 +"employee_first_name_column_exists_check","demo_employee_list.employee_first_name","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""employee_first_name"" FROM ""demo_employee_list"" LIMIT 0) AS _vowl_column_exists","['demo_employee_list']","$.schema[1].properties[2].name","DeclaredColumnExistsCheckReference","string","True",7.681333998334594 +"employee_first_name_logical_type_check","demo_employee_list.employee_first_name","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE NOT ""employee_first_name"" IS NULL AND TRY_CAST(""employee_first_name"" AS TEXT) IS NULL","['demo_employee_list']","$.schema[1].properties[2].logicalType","LogicalTypeCheckReference","string","True",11.540167004568502 +"employee_first_name_required_check","demo_employee_list.employee_first_name","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE ""employee_first_name"" IS NULL","['demo_employee_list']","$.schema[1].properties[2].required","RequiredCheckReference","string","True",7.362165997619741 +"current_title_column_exists_check","demo_employee_list.current_title","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""current_title"" FROM ""demo_employee_list"" LIMIT 0) AS _vowl_column_exists","['demo_employee_list']","$.schema[1].properties[3].name","DeclaredColumnExistsCheckReference","string","True",7.876791001763195 +"current_title_logical_type_check","demo_employee_list.current_title","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE NOT ""current_title"" IS NULL AND TRY_CAST(""current_title"" AS TEXT) IS NULL","['demo_employee_list']","$.schema[1].properties[3].logicalType","LogicalTypeCheckReference","string","True",12.056583000230603 +"phone_number_column_exists_check","demo_employee_list.phone_number","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""phone_number"" FROM ""demo_employee_list"" LIMIT 0) AS _vowl_column_exists","['demo_employee_list']","$.schema[1].properties[4].name","DeclaredColumnExistsCheckReference","string","True",13.484708993928507 +"phone_number_logical_type_check","demo_employee_list.phone_number","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE NOT ""phone_number"" IS NULL AND TRY_CAST(""phone_number"" AS TEXT) IS NULL","['demo_employee_list']","$.schema[1].properties[4].logicalType","LogicalTypeCheckReference","string","True",11.268125002970919 +"phone_number_logical_type_options_pattern_check","demo_employee_list.phone_number","demo_employee_list","sql","FAILED","mustBe","2","0",2,"count","","SELECT COUNT(*) FROM ""demo_employee_list"" WHERE NOT ""phone_number"" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(""phone_number"" AS TEXT), '^\+[0-9]+$')","['demo_employee_list']","$.schema[1].properties[4].logicalTypeOptions.pattern","LogicalTypeOptionsCheckReference","string","True",32.583833002718166 +"phone_number_unique_check","demo_employee_list.phone_number","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM (SELECT ""phone_number"" FROM ""demo_employee_list"" WHERE NOT ""phone_number"" IS NULL GROUP BY ""phone_number"" HAVING COUNT(*) > 1)","['demo_employee_list']","$.schema[1].properties[4].unique","UniqueCheckReference","string","True",6.587625000975095 +"PersonName","demo_employee_list.employee_last_name","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM demo_employee_list WHERE NOT employee_last_name IS NULL AND LENGTH(TRY_CAST(employee_last_name AS TEXT)) > 66","['demo_employee_list']","$.schema[1].properties[1].quality[0]","SQLColumnCheckReference","string","False",21.21066600375343 +"PersonName","demo_employee_list.employee_first_name","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM demo_employee_list WHERE NOT employee_first_name IS NULL AND LENGTH(TRY_CAST(employee_first_name AS TEXT)) > 66","['demo_employee_list']","$.schema[1].properties[2].quality[0]","SQLColumnCheckReference","string","False",25.908499999786727 +"TelephoneNumber","demo_employee_list.phone_number","demo_employee_list","sql","PASSED","mustBe","0","0",0,"count","","SELECT COUNT(*) FROM demo_employee_list WHERE NOT REGEXP_MATCHES(TRY_CAST(phone_number AS TEXT), '^[0-9]{1,12}$')","['demo_employee_list']","$.schema[1].properties[4].quality[0]","SQLColumnCheckReference","string","False",23.390207992633805 +"employee_id_exists_in_master_list","demo_employee_payroll","demo_employee_payroll","sql","FAILED","mustBe","1","0",1,"count","","SELECT COUNT(*) FROM demo_employee_payroll AS payroll LEFT JOIN demo_employee_list AS ref ON payroll.employee_id = ref.employee_id WHERE ref.employee_id IS NULL","['demo_employee_list', 'demo_employee_payroll']","$.schema[0].quality[1]","SQLTableCheckReference",,"False",121.92870900616981 +"phone_number_exists_in_master_list","demo_employee_payroll","demo_employee_payroll","sql","FAILED","mustBe","2","0",2,"count","","SELECT COUNT(*) FROM demo_employee_payroll AS payroll LEFT JOIN demo_employee_list AS ref ON payroll.phone_number = ref.phone_number WHERE NOT payroll.phone_number IS NULL AND ref.phone_number IS NULL","['demo_employee_list', 'demo_employee_payroll']","$.schema[0].quality[2]","SQLTableCheckReference",,"False",29.520874988520518 diff --git a/examples/vowl_demo_multi_table_results_summary.json b/examples/vowl_demo_multi_table_results_summary.json index d553b49..6d78d8f 100644 --- a/examples/vowl_demo_multi_table_results_summary.json +++ b/examples/vowl_demo_multi_table_results_summary.json @@ -15,7 +15,7 @@ "use_try_cast": true }, "failed_rows": 14, - "total_execution_time_ms": 929.0458769537508, + "total_execution_time_ms": 1514.5568709558574, "success_rate": 92.10526315789474, "connection_results": { "demo_employee_payroll": { @@ -35,24 +35,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 6.11975000356324, + "execution_time_ms": 7.536083998274989, "check_path": "$.schema[0].properties[0].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'employee_id' must exist in 'demo_employee_payroll'", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_id_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'employee_id' must exist in 'demo_employee_payroll'", + "query": "SELECT COUNT(*) FROM (SELECT \"employee_id\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "employee_id" + }, "target": "demo_employee_payroll.employee_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"employee_id\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"employee_id\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" }, { "name": "employee_id_logical_type_check", @@ -61,24 +71,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 8.546666009351611, + "execution_time_ms": 11.358792005921714, "check_path": "$.schema[0].properties[0].logicalType", "check_ref_type": "LogicalTypeCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Values in 'employee_id' must be valid string", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_id_logical_type_check", + "type": "sql", + "dimension": "conformity", + "description": "Values in 'employee_id' must be valid string", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employee_id\" IS NULL AND CAST(\"employee_id\" AS VARCHAR) IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "string" + }, "target": "demo_employee_payroll.employee_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employee_id\" IS NULL AND TRY_CAST(\"employee_id\" AS TEXT) IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employee_id\" IS NULL AND TRY_CAST(\"employee_id\" AS TEXT) IS NULL" }, { "name": "employee_id_logical_type_options_minLength_check", @@ -87,24 +107,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 25.609999996959232, + "execution_time_ms": 33.94816699437797, "check_path": "$.schema[0].properties[0].logicalTypeOptions.minLength", "check_ref_type": "LogicalTypeOptionsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'employee_id' must have minimum length of 7", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_id_logical_type_options_minLength_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'employee_id' must have minimum length of 7", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employee_id\" IS NULL AND LENGTH(CAST(\"employee_id\" AS VARCHAR)) < 7", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": 7 + }, "target": "demo_employee_payroll.employee_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employee_id\" IS NULL AND LENGTH(TRY_CAST(\"employee_id\" AS TEXT)) < 7" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employee_id\" IS NULL AND LENGTH(TRY_CAST(\"employee_id\" AS TEXT)) < 7" }, { "name": "employee_id_logical_type_options_maxLength_check", @@ -113,24 +143,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 25.552249993779697, + "execution_time_ms": 33.068207994801924, "check_path": "$.schema[0].properties[0].logicalTypeOptions.maxLength", "check_ref_type": "LogicalTypeOptionsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'employee_id' must have maximum length of 7", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_id_logical_type_options_maxLength_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'employee_id' must have maximum length of 7", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employee_id\" IS NULL AND LENGTH(CAST(\"employee_id\" AS VARCHAR)) > 7", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": 7 + }, "target": "demo_employee_payroll.employee_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employee_id\" IS NULL AND LENGTH(TRY_CAST(\"employee_id\" AS TEXT)) > 7" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employee_id\" IS NULL AND LENGTH(TRY_CAST(\"employee_id\" AS TEXT)) > 7" }, { "name": "employee_id_logical_type_options_pattern_check", @@ -139,24 +179,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 25.541707989759743, + "execution_time_ms": 29.56283300591167, "check_path": "$.schema[0].properties[0].logicalTypeOptions.pattern", "check_ref_type": "LogicalTypeOptionsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'employee_id' must match pattern '^e[0-9]{6}$'", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_id_logical_type_options_pattern_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'employee_id' must match pattern '^e[0-9]{6}$'", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employee_id\" IS NULL AND NOT CAST(\"employee_id\" AS VARCHAR) ~ '^e[0-9]{6}$'", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "^e[0-9]{6}$" + }, "target": "demo_employee_payroll.employee_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employee_id\" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(\"employee_id\" AS TEXT), '^e[0-9]{6}$')" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employee_id\" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(\"employee_id\" AS TEXT), '^e[0-9]{6}$')" }, { "name": "employee_id_required_check", @@ -165,24 +215,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.14758299686946, + "execution_time_ms": 5.534250012715347, "check_path": "$.schema[0].properties[0].required", "check_ref_type": "RequiredCheckReference", - "dimension": "completeness", - "type": "sql", - "description": "Column 'employee_id' must not contain NULL values", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_id_required_check", + "type": "sql", + "dimension": "completeness", + "description": "Column 'employee_id' must not contain NULL values", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"employee_id\" IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": true + }, "target": "demo_employee_payroll.employee_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"employee_id\" IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"employee_id\" IS NULL" }, { "name": "employee_id_unique_check", @@ -191,24 +251,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 6.010749988490716, + "execution_time_ms": 6.91154100059066, "check_path": "$.schema[0].properties[0].unique", "check_ref_type": "UniqueCheckReference", - "dimension": "consistency", - "type": "sql", - "description": "Column 'employee_id' must contain unique values", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_id_unique_check", + "type": "sql", + "dimension": "consistency", + "description": "Column 'employee_id' must contain unique values", + "query": "SELECT COUNT(*) FROM (SELECT \"employee_id\" FROM \"demo_employee_payroll\" WHERE NOT \"employee_id\" IS NULL GROUP BY \"employee_id\" HAVING COUNT(*) > 1)", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": true + }, "target": "demo_employee_payroll.employee_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"employee_id\" FROM \"demo_employee_payroll\" WHERE NOT \"employee_id\" IS NULL GROUP BY \"employee_id\" HAVING COUNT(*) > 1)" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"employee_id\" FROM \"demo_employee_payroll\" WHERE NOT \"employee_id\" IS NULL GROUP BY \"employee_id\" HAVING COUNT(*) > 1)" }, { "name": "employee_id_primary_key_check", @@ -217,24 +287,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 4.767832986544818, + "execution_time_ms": 6.103708001319319, "check_path": "$.schema[0].properties[0].primaryKey", "check_ref_type": "PrimaryKeyCheckReference", - "dimension": "consistency", - "type": "sql", - "description": "Primary key column 'employee_id' must be unique and not null", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_id_primary_key_check", + "type": "sql", + "dimension": "consistency", + "description": "Primary key column 'employee_id' must be unique and not null", + "query": "SELECT ((SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"employee_id\" IS NULL)) + ((SELECT COUNT(*) FROM (SELECT \"employee_id\" FROM \"demo_employee_payroll\" WHERE NOT \"employee_id\" IS NULL GROUP BY \"employee_id\" HAVING COUNT(*) > 1)))", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": true + }, "target": "demo_employee_payroll.employee_id", "logical_type": "string", "aggregation_type": "custom", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT ((SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"employee_id\" IS NULL)) + ((SELECT COUNT(*) FROM (SELECT \"employee_id\" FROM \"demo_employee_payroll\" WHERE NOT \"employee_id\" IS NULL GROUP BY \"employee_id\" HAVING COUNT(*) > 1)))" + "rendered_implementation": "SELECT ((SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"employee_id\" IS NULL)) + ((SELECT COUNT(*) FROM (SELECT \"employee_id\" FROM \"demo_employee_payroll\" WHERE NOT \"employee_id\" IS NULL GROUP BY \"employee_id\" HAVING COUNT(*) > 1)))" }, { "name": "payroll_id_column_exists_check", @@ -243,24 +323,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.433333994005807, + "execution_time_ms": 5.827750006574206, "check_path": "$.schema[0].properties[1].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'payroll_id' must exist in 'demo_employee_payroll'", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "payroll_id_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'payroll_id' must exist in 'demo_employee_payroll'", + "query": "SELECT COUNT(*) FROM (SELECT \"payroll_id\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "payroll_id" + }, "target": "demo_employee_payroll.payroll_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"payroll_id\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"payroll_id\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" }, { "name": "payroll_id_logical_type_check", @@ -269,24 +359,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 8.019708999199793, + "execution_time_ms": 9.204667003359646, "check_path": "$.schema[0].properties[1].logicalType", "check_ref_type": "LogicalTypeCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Values in 'payroll_id' must be valid string", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "payroll_id_logical_type_check", + "type": "sql", + "dimension": "conformity", + "description": "Values in 'payroll_id' must be valid string", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"payroll_id\" IS NULL AND CAST(\"payroll_id\" AS VARCHAR) IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "string" + }, "target": "demo_employee_payroll.payroll_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"payroll_id\" IS NULL AND TRY_CAST(\"payroll_id\" AS TEXT) IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"payroll_id\" IS NULL AND TRY_CAST(\"payroll_id\" AS TEXT) IS NULL" }, { "name": "payroll_id_logical_type_options_minLength_check", @@ -295,24 +395,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 25.0603340100497, + "execution_time_ms": 28.868458000943065, "check_path": "$.schema[0].properties[1].logicalTypeOptions.minLength", "check_ref_type": "LogicalTypeOptionsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'payroll_id' must have minimum length of 36", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "payroll_id_logical_type_options_minLength_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'payroll_id' must have minimum length of 36", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"payroll_id\" IS NULL AND LENGTH(CAST(\"payroll_id\" AS VARCHAR)) < 36", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": 36 + }, "target": "demo_employee_payroll.payroll_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"payroll_id\" IS NULL AND LENGTH(TRY_CAST(\"payroll_id\" AS TEXT)) < 36" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"payroll_id\" IS NULL AND LENGTH(TRY_CAST(\"payroll_id\" AS TEXT)) < 36" }, { "name": "payroll_id_logical_type_options_maxLength_check", @@ -321,24 +431,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 25.716958989505656, + "execution_time_ms": 30.009207999682985, "check_path": "$.schema[0].properties[1].logicalTypeOptions.maxLength", "check_ref_type": "LogicalTypeOptionsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'payroll_id' must have maximum length of 36", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "payroll_id_logical_type_options_maxLength_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'payroll_id' must have maximum length of 36", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"payroll_id\" IS NULL AND LENGTH(CAST(\"payroll_id\" AS VARCHAR)) > 36", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": 36 + }, "target": "demo_employee_payroll.payroll_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"payroll_id\" IS NULL AND LENGTH(TRY_CAST(\"payroll_id\" AS TEXT)) > 36" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"payroll_id\" IS NULL AND LENGTH(TRY_CAST(\"payroll_id\" AS TEXT)) > 36" }, { "name": "payroll_id_logical_type_options_pattern_check", @@ -347,24 +467,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 25.72074999625329, + "execution_time_ms": 34.06583299511112, "check_path": "$.schema[0].properties[1].logicalTypeOptions.pattern", "check_ref_type": "LogicalTypeOptionsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'payroll_id' must match pattern '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "payroll_id_logical_type_options_pattern_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'payroll_id' must match pattern '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"payroll_id\" IS NULL AND NOT CAST(\"payroll_id\" AS VARCHAR) ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$" + }, "target": "demo_employee_payroll.payroll_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"payroll_id\" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(\"payroll_id\" AS TEXT), '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$')" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"payroll_id\" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(\"payroll_id\" AS TEXT), '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$')" }, { "name": "payroll_id_required_check", @@ -373,24 +503,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.311874992912635, + "execution_time_ms": 33.69458299130201, "check_path": "$.schema[0].properties[1].required", "check_ref_type": "RequiredCheckReference", - "dimension": "completeness", - "type": "sql", - "description": "Column 'payroll_id' must not contain NULL values", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "payroll_id_required_check", + "type": "sql", + "dimension": "completeness", + "description": "Column 'payroll_id' must not contain NULL values", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"payroll_id\" IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": true + }, "target": "demo_employee_payroll.payroll_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"payroll_id\" IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"payroll_id\" IS NULL" }, { "name": "payroll_id_unique_check", @@ -399,24 +539,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.644874996505678, + "execution_time_ms": 11.5525409928523, "check_path": "$.schema[0].properties[1].unique", "check_ref_type": "UniqueCheckReference", - "dimension": "consistency", - "type": "sql", - "description": "Column 'payroll_id' must contain unique values", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "payroll_id_unique_check", + "type": "sql", + "dimension": "consistency", + "description": "Column 'payroll_id' must contain unique values", + "query": "SELECT COUNT(*) FROM (SELECT \"payroll_id\" FROM \"demo_employee_payroll\" WHERE NOT \"payroll_id\" IS NULL GROUP BY \"payroll_id\" HAVING COUNT(*) > 1)", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": true + }, "target": "demo_employee_payroll.payroll_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"payroll_id\" FROM \"demo_employee_payroll\" WHERE NOT \"payroll_id\" IS NULL GROUP BY \"payroll_id\" HAVING COUNT(*) > 1)" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"payroll_id\" FROM \"demo_employee_payroll\" WHERE NOT \"payroll_id\" IS NULL GROUP BY \"payroll_id\" HAVING COUNT(*) > 1)" }, { "name": "payroll_id_primary_key_check", @@ -425,24 +575,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 4.597375009325333, + "execution_time_ms": 9.977291003451683, "check_path": "$.schema[0].properties[1].primaryKey", "check_ref_type": "PrimaryKeyCheckReference", - "dimension": "consistency", - "type": "sql", - "description": "Primary key column 'payroll_id' must be unique and not null", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "payroll_id_primary_key_check", + "type": "sql", + "dimension": "consistency", + "description": "Primary key column 'payroll_id' must be unique and not null", + "query": "SELECT ((SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"payroll_id\" IS NULL)) + ((SELECT COUNT(*) FROM (SELECT \"payroll_id\" FROM \"demo_employee_payroll\" WHERE NOT \"payroll_id\" IS NULL GROUP BY \"payroll_id\" HAVING COUNT(*) > 1)))", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": true + }, "target": "demo_employee_payroll.payroll_id", "logical_type": "string", "aggregation_type": "custom", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT ((SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"payroll_id\" IS NULL)) + ((SELECT COUNT(*) FROM (SELECT \"payroll_id\" FROM \"demo_employee_payroll\" WHERE NOT \"payroll_id\" IS NULL GROUP BY \"payroll_id\" HAVING COUNT(*) > 1)))" + "rendered_implementation": "SELECT ((SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"payroll_id\" IS NULL)) + ((SELECT COUNT(*) FROM (SELECT \"payroll_id\" FROM \"demo_employee_payroll\" WHERE NOT \"payroll_id\" IS NULL GROUP BY \"payroll_id\" HAVING COUNT(*) > 1)))" }, { "name": "month_column_exists_check", @@ -451,24 +611,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 3.79183299082797, + "execution_time_ms": 11.00429099460598, "check_path": "$.schema[0].properties[2].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'month' must exist in 'demo_employee_payroll'", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "month_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'month' must exist in 'demo_employee_payroll'", + "query": "SELECT COUNT(*) FROM (SELECT \"month\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "month" + }, "target": "demo_employee_payroll.month", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"month\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"month\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" }, { "name": "month_logical_type_check", @@ -477,24 +647,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.933833002927713, + "execution_time_ms": 14.883666997775435, "check_path": "$.schema[0].properties[2].logicalType", "check_ref_type": "LogicalTypeCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Values in 'month' must be valid string", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "month_logical_type_check", + "type": "sql", + "dimension": "conformity", + "description": "Values in 'month' must be valid string", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"month\" IS NULL AND CAST(\"month\" AS VARCHAR) IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "string" + }, "target": "demo_employee_payroll.month", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"month\" IS NULL AND TRY_CAST(\"month\" AS TEXT) IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"month\" IS NULL AND TRY_CAST(\"month\" AS TEXT) IS NULL" }, { "name": "month_logical_type_options_pattern_check", @@ -503,24 +683,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 22.392333004972897, + "execution_time_ms": 50.78175000380725, "check_path": "$.schema[0].properties[2].logicalTypeOptions.pattern", "check_ref_type": "LogicalTypeOptionsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'month' must match pattern '^\\d{4}-\\d{2}$'", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "month_logical_type_options_pattern_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'month' must match pattern '^\\d{4}-\\d{2}$'", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"month\" IS NULL AND NOT CAST(\"month\" AS VARCHAR) ~ '^\\d{4}-\\d{2}$'", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "^\\d{4}-\\d{2}$" + }, "target": "demo_employee_payroll.month", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"month\" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(\"month\" AS TEXT), '^\\d{4}-\\d{2}$')" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"month\" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(\"month\" AS TEXT), '^\\d{4}-\\d{2}$')" }, { "name": "month_required_check", @@ -529,24 +719,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 3.6305000103311613, + "execution_time_ms": 14.663374997326173, "check_path": "$.schema[0].properties[2].required", "check_ref_type": "RequiredCheckReference", - "dimension": "completeness", - "type": "sql", - "description": "Column 'month' must not contain NULL values", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "month_required_check", + "type": "sql", + "dimension": "completeness", + "description": "Column 'month' must not contain NULL values", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"month\" IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": true + }, "target": "demo_employee_payroll.month", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"month\" IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"month\" IS NULL" }, { "name": "payroll_start_dt_column_exists_check", @@ -555,24 +755,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 3.7929579993942752, + "execution_time_ms": 7.284459003130905, "check_path": "$.schema[0].properties[3].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'payroll_start_dt' must exist in 'demo_employee_payroll'", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "payroll_start_dt_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'payroll_start_dt' must exist in 'demo_employee_payroll'", + "query": "SELECT COUNT(*) FROM (SELECT \"payroll_start_dt\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "payroll_start_dt" + }, "target": "demo_employee_payroll.payroll_start_dt", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"payroll_start_dt\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"payroll_start_dt\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" }, { "name": "payroll_start_dt_logical_type_check", @@ -581,24 +791,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 6.0861669917358086, + "execution_time_ms": 8.672416995977983, "check_path": "$.schema[0].properties[3].logicalType", "check_ref_type": "LogicalTypeCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Values in 'payroll_start_dt' must be valid string", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "payroll_start_dt_logical_type_check", + "type": "sql", + "dimension": "conformity", + "description": "Values in 'payroll_start_dt' must be valid string", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"payroll_start_dt\" IS NULL AND CAST(\"payroll_start_dt\" AS VARCHAR) IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "string" + }, "target": "demo_employee_payroll.payroll_start_dt", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"payroll_start_dt\" IS NULL AND TRY_CAST(\"payroll_start_dt\" AS TEXT) IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"payroll_start_dt\" IS NULL AND TRY_CAST(\"payroll_start_dt\" AS TEXT) IS NULL" }, { "name": "payroll_end_dt_column_exists_check", @@ -607,24 +827,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 3.690333993290551, + "execution_time_ms": 5.804250002256595, "check_path": "$.schema[0].properties[4].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'payroll_end_dt' must exist in 'demo_employee_payroll'", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "payroll_end_dt_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'payroll_end_dt' must exist in 'demo_employee_payroll'", + "query": "SELECT COUNT(*) FROM (SELECT \"payroll_end_dt\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "payroll_end_dt" + }, "target": "demo_employee_payroll.payroll_end_dt", "logical_type": "date", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"payroll_end_dt\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"payroll_end_dt\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" }, { "name": "payroll_end_dt_logical_type_check", @@ -633,24 +863,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.9707079926738515, + "execution_time_ms": 9.189958000206389, "check_path": "$.schema[0].properties[4].logicalType", "check_ref_type": "LogicalTypeCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Values in 'payroll_end_dt' must be valid date", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "payroll_end_dt_logical_type_check", + "type": "sql", + "dimension": "conformity", + "description": "Values in 'payroll_end_dt' must be valid date", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"payroll_end_dt\" IS NULL AND CAST(\"payroll_end_dt\" AS DATE) IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "date" + }, "target": "demo_employee_payroll.payroll_end_dt", "logical_type": "date", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"payroll_end_dt\" IS NULL AND TRY_CAST(\"payroll_end_dt\" AS DATE) IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"payroll_end_dt\" IS NULL AND TRY_CAST(\"payroll_end_dt\" AS DATE) IS NULL" }, { "name": "total_amt_column_exists_check", @@ -659,24 +899,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 3.760708001209423, + "execution_time_ms": 4.887916991719976, "check_path": "$.schema[0].properties[5].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'total_amt' must exist in 'demo_employee_payroll'", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "total_amt_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'total_amt' must exist in 'demo_employee_payroll'", + "query": "SELECT COUNT(*) FROM (SELECT \"total_amt\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "total_amt" + }, "target": "demo_employee_payroll.total_amt", "logical_type": "integer", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"total_amt\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"total_amt\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" }, { "name": "total_amt_logical_type_check", @@ -685,24 +935,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 6.376332996296696, + "execution_time_ms": 39.02799999923445, "check_path": "$.schema[0].properties[5].logicalType", "check_ref_type": "LogicalTypeCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Values in 'total_amt' must be valid integer", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "total_amt_logical_type_check", + "type": "sql", + "dimension": "conformity", + "description": "Values in 'total_amt' must be valid integer", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"total_amt\" IS NULL AND (CAST(\"total_amt\" AS DOUBLE PRECISION) IS NULL OR CAST(\"total_amt\" AS DOUBLE PRECISION) <> CAST(\"total_amt\" AS BIGINT))", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "integer" + }, "target": "demo_employee_payroll.total_amt", "logical_type": "integer", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"total_amt\" IS NULL AND (TRY_CAST(\"total_amt\" AS DOUBLE) IS NULL OR TRY_CAST(\"total_amt\" AS DOUBLE) <> TRY_CAST(\"total_amt\" AS BIGINT))" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"total_amt\" IS NULL AND (TRY_CAST(\"total_amt\" AS DOUBLE) IS NULL OR TRY_CAST(\"total_amt\" AS DOUBLE) <> TRY_CAST(\"total_amt\" AS BIGINT))" }, { "name": "employer_cpf_amt_column_exists_check", @@ -711,24 +971,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 3.631291998317465, + "execution_time_ms": 5.7217919966205955, "check_path": "$.schema[0].properties[6].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'employer_cpf_amt' must exist in 'demo_employee_payroll'", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employer_cpf_amt_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'employer_cpf_amt' must exist in 'demo_employee_payroll'", + "query": "SELECT COUNT(*) FROM (SELECT \"employer_cpf_amt\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "employer_cpf_amt" + }, "target": "demo_employee_payroll.employer_cpf_amt", "logical_type": "number", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"employer_cpf_amt\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"employer_cpf_amt\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" }, { "name": "employer_cpf_amt_logical_type_check", @@ -737,24 +1007,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 20.88683401234448, + "execution_time_ms": 9.278917001211084, "check_path": "$.schema[0].properties[6].logicalType", "check_ref_type": "LogicalTypeCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Values in 'employer_cpf_amt' must be valid number", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employer_cpf_amt_logical_type_check", + "type": "sql", + "dimension": "conformity", + "description": "Values in 'employer_cpf_amt' must be valid number", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employer_cpf_amt\" IS NULL AND CAST(\"employer_cpf_amt\" AS DOUBLE PRECISION) IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "number" + }, "target": "demo_employee_payroll.employer_cpf_amt", "logical_type": "number", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employer_cpf_amt\" IS NULL AND TRY_CAST(\"employer_cpf_amt\" AS DOUBLE) IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employer_cpf_amt\" IS NULL AND TRY_CAST(\"employer_cpf_amt\" AS DOUBLE) IS NULL" }, { "name": "total_amt_employee_column_exists_check", @@ -763,24 +1043,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 3.625041004852392, + "execution_time_ms": 12.201416000607423, "check_path": "$.schema[0].properties[7].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'total_amt_employee' must exist in 'demo_employee_payroll'", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "total_amt_employee_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'total_amt_employee' must exist in 'demo_employee_payroll'", + "query": "SELECT COUNT(*) FROM (SELECT \"total_amt_employee\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "total_amt_employee" + }, "target": "demo_employee_payroll.total_amt_employee", "logical_type": "number", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"total_amt_employee\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"total_amt_employee\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" }, { "name": "total_amt_employee_logical_type_check", @@ -789,24 +1079,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.843083010404371, + "execution_time_ms": 11.42637500015553, "check_path": "$.schema[0].properties[7].logicalType", "check_ref_type": "LogicalTypeCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Values in 'total_amt_employee' must be valid number", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "total_amt_employee_logical_type_check", + "type": "sql", + "dimension": "conformity", + "description": "Values in 'total_amt_employee' must be valid number", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"total_amt_employee\" IS NULL AND CAST(\"total_amt_employee\" AS DOUBLE PRECISION) IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "number" + }, "target": "demo_employee_payroll.total_amt_employee", "logical_type": "number", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"total_amt_employee\" IS NULL AND TRY_CAST(\"total_amt_employee\" AS DOUBLE) IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"total_amt_employee\" IS NULL AND TRY_CAST(\"total_amt_employee\" AS DOUBLE) IS NULL" }, { "name": "total_amt_employee_required_check", @@ -815,24 +1115,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 3.5080829984508455, + "execution_time_ms": 6.754499991075136, "check_path": "$.schema[0].properties[7].required", "check_ref_type": "RequiredCheckReference", - "dimension": "completeness", - "type": "sql", - "description": "Column 'total_amt_employee' must not contain NULL values", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "total_amt_employee_required_check", + "type": "sql", + "dimension": "completeness", + "description": "Column 'total_amt_employee' must not contain NULL values", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"total_amt_employee\" IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": true + }, "target": "demo_employee_payroll.total_amt_employee", "logical_type": "number", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"total_amt_employee\" IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"total_amt_employee\" IS NULL" }, { "name": "employee_cpf_amt_column_exists_check", @@ -841,24 +1151,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 3.5907499986933544, + "execution_time_ms": 7.811667004716583, "check_path": "$.schema[0].properties[8].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'employee_cpf_amt' must exist in 'demo_employee_payroll'", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_cpf_amt_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'employee_cpf_amt' must exist in 'demo_employee_payroll'", + "query": "SELECT COUNT(*) FROM (SELECT \"employee_cpf_amt\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "employee_cpf_amt" + }, "target": "demo_employee_payroll.employee_cpf_amt", "logical_type": "number", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"employee_cpf_amt\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"employee_cpf_amt\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" }, { "name": "employee_cpf_amt_logical_type_check", @@ -867,24 +1187,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 8.018790991627611, + "execution_time_ms": 10.776291994261555, "check_path": "$.schema[0].properties[8].logicalType", "check_ref_type": "LogicalTypeCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Values in 'employee_cpf_amt' must be valid number", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_cpf_amt_logical_type_check", + "type": "sql", + "dimension": "conformity", + "description": "Values in 'employee_cpf_amt' must be valid number", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employee_cpf_amt\" IS NULL AND CAST(\"employee_cpf_amt\" AS DOUBLE PRECISION) IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "number" + }, "target": "demo_employee_payroll.employee_cpf_amt", "logical_type": "number", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employee_cpf_amt\" IS NULL AND TRY_CAST(\"employee_cpf_amt\" AS DOUBLE) IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employee_cpf_amt\" IS NULL AND TRY_CAST(\"employee_cpf_amt\" AS DOUBLE) IS NULL" }, { "name": "employee_cpf_amt_required_check", @@ -893,24 +1223,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 4.540499998256564, + "execution_time_ms": 7.453291007550433, "check_path": "$.schema[0].properties[8].required", "check_ref_type": "RequiredCheckReference", - "dimension": "completeness", - "type": "sql", - "description": "Column 'employee_cpf_amt' must not contain NULL values", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_cpf_amt_required_check", + "type": "sql", + "dimension": "completeness", + "description": "Column 'employee_cpf_amt' must not contain NULL values", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"employee_cpf_amt\" IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": true + }, "target": "demo_employee_payroll.employee_cpf_amt", "logical_type": "number", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"employee_cpf_amt\" IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"employee_cpf_amt\" IS NULL" }, { "name": "employee_gross_amt_column_exists_check", @@ -919,24 +1259,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.041374999564141, + "execution_time_ms": 6.992291993810795, "check_path": "$.schema[0].properties[9].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'employee_gross_amt' must exist in 'demo_employee_payroll'", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_gross_amt_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'employee_gross_amt' must exist in 'demo_employee_payroll'", + "query": "SELECT COUNT(*) FROM (SELECT \"employee_gross_amt\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "employee_gross_amt" + }, "target": "demo_employee_payroll.employee_gross_amt", "logical_type": "number", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"employee_gross_amt\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"employee_gross_amt\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" }, { "name": "employee_gross_amt_logical_type_check", @@ -945,24 +1295,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 7.9375420027645305, + "execution_time_ms": 11.974291992373765, "check_path": "$.schema[0].properties[9].logicalType", "check_ref_type": "LogicalTypeCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Values in 'employee_gross_amt' must be valid number", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_gross_amt_logical_type_check", + "type": "sql", + "dimension": "conformity", + "description": "Values in 'employee_gross_amt' must be valid number", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employee_gross_amt\" IS NULL AND CAST(\"employee_gross_amt\" AS DOUBLE PRECISION) IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "number" + }, "target": "demo_employee_payroll.employee_gross_amt", "logical_type": "number", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employee_gross_amt\" IS NULL AND TRY_CAST(\"employee_gross_amt\" AS DOUBLE) IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"employee_gross_amt\" IS NULL AND TRY_CAST(\"employee_gross_amt\" AS DOUBLE) IS NULL" }, { "name": "employee_gross_amt_required_check", @@ -971,24 +1331,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 4.815750013221987, + "execution_time_ms": 6.4171249978244305, "check_path": "$.schema[0].properties[9].required", "check_ref_type": "RequiredCheckReference", - "dimension": "completeness", - "type": "sql", - "description": "Column 'employee_gross_amt' must not contain NULL values", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_gross_amt_required_check", + "type": "sql", + "dimension": "completeness", + "description": "Column 'employee_gross_amt' must not contain NULL values", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"employee_gross_amt\" IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": true + }, "target": "demo_employee_payroll.employee_gross_amt", "logical_type": "number", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"employee_gross_amt\" IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"employee_gross_amt\" IS NULL" }, { "name": "phone_number_column_exists_check", @@ -997,24 +1367,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.021249991841614, + "execution_time_ms": 9.263290994567797, "check_path": "$.schema[0].properties[10].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'phone_number' must exist in 'demo_employee_payroll'", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "phone_number_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'phone_number' must exist in 'demo_employee_payroll'", + "query": "SELECT COUNT(*) FROM (SELECT \"phone_number\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "phone_number" + }, "target": "demo_employee_payroll.phone_number", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"phone_number\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"phone_number\" FROM \"demo_employee_payroll\" LIMIT 0) AS _vowl_column_exists" }, { "name": "phone_number_logical_type_check", @@ -1023,24 +1403,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 8.666124995215796, + "execution_time_ms": 17.175583998323418, "check_path": "$.schema[0].properties[10].logicalType", "check_ref_type": "LogicalTypeCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Values in 'phone_number' must be valid string", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "phone_number_logical_type_check", + "type": "sql", + "dimension": "conformity", + "description": "Values in 'phone_number' must be valid string", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"phone_number\" IS NULL AND CAST(\"phone_number\" AS VARCHAR) IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "string" + }, "target": "demo_employee_payroll.phone_number", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"phone_number\" IS NULL AND TRY_CAST(\"phone_number\" AS TEXT) IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"phone_number\" IS NULL AND TRY_CAST(\"phone_number\" AS TEXT) IS NULL" }, { "name": "phone_number_logical_type_options_minLength_check", @@ -1049,24 +1439,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 25.763958008610643, + "execution_time_ms": 43.42120799992699, "check_path": "$.schema[0].properties[10].logicalTypeOptions.minLength", "check_ref_type": "LogicalTypeOptionsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'phone_number' must have minimum length of 8", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "phone_number_logical_type_options_minLength_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'phone_number' must have minimum length of 8", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"phone_number\" IS NULL AND LENGTH(CAST(\"phone_number\" AS VARCHAR)) < 8", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": 8 + }, "target": "demo_employee_payroll.phone_number", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"phone_number\" IS NULL AND LENGTH(TRY_CAST(\"phone_number\" AS TEXT)) < 8" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"phone_number\" IS NULL AND LENGTH(TRY_CAST(\"phone_number\" AS TEXT)) < 8" }, { "name": "phone_number_logical_type_options_maxLength_check", @@ -1075,24 +1475,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 30.281333994935267, + "execution_time_ms": 45.03570799715817, "check_path": "$.schema[0].properties[10].logicalTypeOptions.maxLength", "check_ref_type": "LogicalTypeOptionsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'phone_number' must have maximum length of 11", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "phone_number_logical_type_options_maxLength_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'phone_number' must have maximum length of 11", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"phone_number\" IS NULL AND LENGTH(CAST(\"phone_number\" AS VARCHAR)) > 11", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": 11 + }, "target": "demo_employee_payroll.phone_number", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"phone_number\" IS NULL AND LENGTH(TRY_CAST(\"phone_number\" AS TEXT)) > 11" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"phone_number\" IS NULL AND LENGTH(TRY_CAST(\"phone_number\" AS TEXT)) > 11" }, { "name": "phone_number_logical_type_options_pattern_check", @@ -1101,24 +1511,34 @@ "expected_value": 0, "actual_value": 3, "failed_rows_count": 3, - "execution_time_ms": 26.03491699846927, + "execution_time_ms": 43.62716700416058, "check_path": "$.schema[0].properties[10].logicalTypeOptions.pattern", "check_ref_type": "LogicalTypeOptionsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'phone_number' must match pattern '^\\+65[0-9]{8}$'", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "phone_number_logical_type_options_pattern_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'phone_number' must match pattern '^\\+65[0-9]{8}$'", + "query": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"phone_number\" IS NULL AND NOT CAST(\"phone_number\" AS VARCHAR) ~ '^\\+65[0-9]{8}$'", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "^\\+65[0-9]{8}$" + }, "target": "demo_employee_payroll.phone_number", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"phone_number\" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(\"phone_number\" AS TEXT), '^\\+65[0-9]{8}$')" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"phone_number\" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(\"phone_number\" AS TEXT), '^\\+65[0-9]{8}$')" }, { "name": "total_amt_employee_equals_gross_plus_cpf", @@ -1127,23 +1547,43 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 15.561499996692874, + "execution_time_ms": 21.877167004277, "check_path": "$.schema[0].quality[0]", "check_ref_type": "SQLTableCheckReference", - "dimension": "consistency", - "type": "sql", - "description": "Validates that total_amt_employee equals the sum of employee_gross_amt and employee_cpf_amt for consistency", - "severity": "error", - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": false, "engine": "sql", + "check_definition": { + "name": "total_amt_employee_equals_gross_plus_cpf", + "description": "Validates that total_amt_employee equals the sum of employee_gross_amt and employee_cpf_amt for consistency", + "type": "sql", + "dimension": "consistency", + "severity": "error", + "query": "SELECT COUNT(*) FROM demo_employee_payroll WHERE total_amt_employee != (employee_gross_amt + employee_cpf_amt)", + "mustBe": 0, + "tags": [ + "User-Defined Quality Rule" + ] + }, + "contract_definition": { + "name": "total_amt_employee_equals_gross_plus_cpf", + "description": "Validates that total_amt_employee equals the sum of employee_gross_amt and employee_cpf_amt for consistency", + "type": "sql", + "dimension": "consistency", + "severity": "error", + "query": "SELECT COUNT(*) FROM demo_employee_payroll WHERE total_amt_employee != (employee_gross_amt + employee_cpf_amt)", + "mustBe": 0, + "tags": [ + "User-Defined Quality Rule" + ] + }, "target": "demo_employee_payroll", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM demo_employee_payroll WHERE total_amt_employee <> (employee_gross_amt + employee_cpf_amt)" + "rendered_implementation": "SELECT COUNT(*) FROM demo_employee_payroll WHERE total_amt_employee <> (employee_gross_amt + employee_cpf_amt)" }, { "name": "demo_employee_payroll_rowCount", @@ -1152,23 +1592,35 @@ "expected_value": 0, "actual_value": 3, "failed_rows_count": 0, - "execution_time_ms": 7.83212500391528, + "execution_time_ms": 42.15633300191257, "check_path": "$.schema[0].quality[3]", "check_ref_type": "RowCountCheckReference", - "dimension": "completeness", - "type": "library", - "description": "Total row count for 'demo_employee_payroll'", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBeGreaterThan", "is_generated": true, "engine": "sql", + "check_definition": { + "type": "library", + "metric": "rowCount", + "mustBeGreaterThan": 0, + "dimension": "completeness", + "tags": [ + "vowl_generated_check" + ], + "description": "Total row count for 'demo_employee_payroll'" + }, + "contract_definition": { + "type": "library", + "metric": "rowCount", + "mustBeGreaterThan": 0, + "dimension": "completeness" + }, "target": "demo_employee_payroll", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\"" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\"" }, { "name": "employee_id_nullValues", @@ -1177,24 +1629,36 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 13.46854200528469, + "execution_time_ms": 27.207833001739345, "check_path": "$.schema[0].properties[0].quality[0]", "check_ref_type": "NullValuesCheckReference", - "dimension": "completeness", - "type": "library", - "description": "Count of NULL values in 'employee_id' of 'demo_employee_payroll'", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "type": "library", + "metric": "nullValues", + "mustBe": 0, + "dimension": "completeness", + "tags": [ + "vowl_generated_check" + ], + "description": "Count of NULL values in 'employee_id' of 'demo_employee_payroll'" + }, + "contract_definition": { + "type": "library", + "metric": "nullValues", + "mustBe": 0, + "dimension": "completeness" + }, "target": "demo_employee_payroll.employee_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"employee_id\" IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"employee_id\" IS NULL" }, { "name": "Month", @@ -1203,24 +1667,42 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 18.365625001024455, + "execution_time_ms": 32.52749999228399, "check_path": "$.schema[0].properties[2].quality[0]", "check_ref_type": "SQLColumnCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Based on ISO 8601, assumed to be in UTC +8 | YYYY-MM", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": false, "engine": "sql", + "check_definition": { + "type": "sql", + "name": "Month", + "description": "Based on ISO 8601, assumed to be in UTC +8 | YYYY-MM", + "mustBe": 0, + "query": "SELECT COUNT(*)\nFROM demo_employee_payroll\nWHERE CAST(month AS TEXT) !~ '^[0-9]{4}-(0[1-9]|1[0-2])$';", + "tags": [ + "SG-DRM v5.0" + ], + "dimension": "conformity" + }, + "contract_definition": { + "type": "sql", + "name": "Month", + "description": "Based on ISO 8601, assumed to be in UTC +8 | YYYY-MM", + "mustBe": 0, + "query": "SELECT COUNT(*)\nFROM demo_employee_payroll\nWHERE CAST(month AS TEXT) !~ '^[0-9]{4}-(0[1-9]|1[0-2])$';", + "tags": [ + "SG-DRM v5.0" + ], + "dimension": "conformity" + }, "target": "demo_employee_payroll.month", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM demo_employee_payroll WHERE NOT REGEXP_MATCHES(TRY_CAST(month AS TEXT), '^[0-9]{4}-(0[1-9]|1[0-2])$')" + "rendered_implementation": "SELECT COUNT(*) FROM demo_employee_payroll WHERE NOT REGEXP_MATCHES(TRY_CAST(month AS TEXT), '^[0-9]{4}-(0[1-9]|1[0-2])$')" }, { "name": "month_invalidValues", @@ -1229,24 +1711,42 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 19.918834004784003, + "execution_time_ms": 26.39941700908821, "check_path": "$.schema[0].properties[2].quality[1]", "check_ref_type": "InvalidValuesCheckReference", - "dimension": "conformity", - "type": "library", - "description": "Count of invalid values in 'month' of 'demo_employee_payroll'", - "severity": null, - "schema": "demo_employee_payroll", - "operator": "mustBe", - "is_generated": true, - "engine": "sql", + "schema_name": "demo_employee_payroll", + "operator": "mustBe", + "is_generated": true, + "engine": "sql", + "check_definition": { + "type": "library", + "metric": "invalidValues", + "mustBe": 0, + "dimension": "conformity", + "arguments": { + "pattern": "^\\d{4}-(0[1-9]|1[0-2])$" + }, + "tags": [ + "vowl_generated_check" + ], + "description": "Count of invalid values in 'month' of 'demo_employee_payroll'" + }, + "contract_definition": { + "type": "library", + "metric": "invalidValues", + "mustBe": 0, + "dimension": "conformity", + "arguments": { + "pattern": "^\\d{4}-(0[1-9]|1[0-2])$" + } + }, "target": "demo_employee_payroll.month", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"month\" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(\"month\" AS TEXT), '^\\d{4}-(0[1-9]|1[0-2])$')" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE NOT \"month\" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(\"month\" AS TEXT), '^\\d{4}-(0[1-9]|1[0-2])$')" }, { "name": "Date With Timezone", @@ -1255,24 +1755,42 @@ "expected_value": 0, "actual_value": 3, "failed_rows_count": 3, - "execution_time_ms": 17.863624991150573, + "execution_time_ms": 21.720541000831872, "check_path": "$.schema[0].properties[3].quality[0]", "check_ref_type": "SQLColumnCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "If not in UTC +8 | YYYY-MM-DD[T]hh:mm+hh:mm e.g. 2015-01-01T12:00+02:00 for UTC +2", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": false, "engine": "sql", + "check_definition": { + "type": "sql", + "name": "Date With Timezone", + "description": "If not in UTC +8 | YYYY-MM-DD[T]hh:mm+hh:mm e.g. 2015-01-01T12:00+02:00 for UTC +2", + "mustBe": 0, + "query": "SELECT COUNT(*)\nFROM demo_employee_payroll\nWHERE CAST(payroll_start_dt AS TEXT) !~ '^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])T([01][0-9]|2[0-3]):[0-5][0-9][+-]([01][0-9]|2[0-3]):[0-5][0-9]$'\n OR LENGTH(CAST(payroll_start_dt AS TEXT)) > 22;", + "tags": [ + "SG-DRM v5.0" + ], + "dimension": "conformity" + }, + "contract_definition": { + "type": "sql", + "name": "Date With Timezone", + "description": "If not in UTC +8 | YYYY-MM-DD[T]hh:mm+hh:mm e.g. 2015-01-01T12:00+02:00 for UTC +2", + "mustBe": 0, + "query": "SELECT COUNT(*)\nFROM demo_employee_payroll\nWHERE CAST(payroll_start_dt AS TEXT) !~ '^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])T([01][0-9]|2[0-3]):[0-5][0-9][+-]([01][0-9]|2[0-3]):[0-5][0-9]$'\n OR LENGTH(CAST(payroll_start_dt AS TEXT)) > 22;", + "tags": [ + "SG-DRM v5.0" + ], + "dimension": "conformity" + }, "target": "demo_employee_payroll.payroll_start_dt", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM demo_employee_payroll WHERE NOT REGEXP_MATCHES(TRY_CAST(payroll_start_dt AS TEXT), '^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])T([01][0-9]|2[0-3]):[0-5][0-9][+-]([01][0-9]|2[0-3]):[0-5][0-9]$') OR LENGTH(TRY_CAST(payroll_start_dt AS TEXT)) > 22" + "rendered_implementation": "SELECT COUNT(*) FROM demo_employee_payroll WHERE NOT REGEXP_MATCHES(TRY_CAST(payroll_start_dt AS TEXT), '^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])T([01][0-9]|2[0-3]):[0-5][0-9][+-]([01][0-9]|2[0-3]):[0-5][0-9]$') OR LENGTH(TRY_CAST(payroll_start_dt AS TEXT)) > 22" }, { "name": "Date With Timezone", @@ -1281,24 +1799,42 @@ "expected_value": 0, "actual_value": 3, "failed_rows_count": 3, - "execution_time_ms": 17.999208997935057, + "execution_time_ms": 21.855708007933572, "check_path": "$.schema[0].properties[4].quality[0]", "check_ref_type": "SQLColumnCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "If not in UTC +8 | YYYY-MM-DD[T]hh:mm+hh:mm e.g. 2015-01-01T12:00+02:00 for UTC +2", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": false, "engine": "sql", + "check_definition": { + "type": "sql", + "name": "Date With Timezone", + "description": "If not in UTC +8 | YYYY-MM-DD[T]hh:mm+hh:mm e.g. 2015-01-01T12:00+02:00 for UTC +2", + "mustBe": 0, + "query": "SELECT COUNT(*)\nFROM demo_employee_payroll\nWHERE CAST(payroll_end_dt AS TEXT) !~ '^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])T([01][0-9]|2[0-3]):[0-5][0-9][+-]([01][0-9]|2[0-3]):[0-5][0-9]$'\n OR LENGTH(CAST(payroll_end_dt AS TEXT)) > 22;", + "tags": [ + "SG-DRM v5.0" + ], + "dimension": "conformity" + }, + "contract_definition": { + "type": "sql", + "name": "Date With Timezone", + "description": "If not in UTC +8 | YYYY-MM-DD[T]hh:mm+hh:mm e.g. 2015-01-01T12:00+02:00 for UTC +2", + "mustBe": 0, + "query": "SELECT COUNT(*)\nFROM demo_employee_payroll\nWHERE CAST(payroll_end_dt AS TEXT) !~ '^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])T([01][0-9]|2[0-3]):[0-5][0-9][+-]([01][0-9]|2[0-3]):[0-5][0-9]$'\n OR LENGTH(CAST(payroll_end_dt AS TEXT)) > 22;", + "tags": [ + "SG-DRM v5.0" + ], + "dimension": "conformity" + }, "target": "demo_employee_payroll.payroll_end_dt", "logical_type": "date", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM demo_employee_payroll WHERE NOT REGEXP_MATCHES(TRY_CAST(payroll_end_dt AS TEXT), '^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])T([01][0-9]|2[0-3]):[0-5][0-9][+-]([01][0-9]|2[0-3]):[0-5][0-9]$') OR LENGTH(TRY_CAST(payroll_end_dt AS TEXT)) > 22" + "rendered_implementation": "SELECT COUNT(*) FROM demo_employee_payroll WHERE NOT REGEXP_MATCHES(TRY_CAST(payroll_end_dt AS TEXT), '^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])T([01][0-9]|2[0-3]):[0-5][0-9][+-]([01][0-9]|2[0-3]):[0-5][0-9]$') OR LENGTH(TRY_CAST(payroll_end_dt AS TEXT)) > 22" }, { "name": "total_amt_equals_employer_cpf_plus_total_amt_employee", @@ -1307,24 +1843,44 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 14.988583003287204, + "execution_time_ms": 18.529374996433035, "check_path": "$.schema[0].properties[5].quality[0]", "check_ref_type": "SQLColumnCheckReference", - "dimension": "consistency", - "type": "sql", - "description": "Validates that total_amt equals the sum of employer_cpf_amt and total_amt_employee for consistency", - "severity": "error", - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": false, "engine": "sql", + "check_definition": { + "name": "total_amt_equals_employer_cpf_plus_total_amt_employee", + "description": "Validates that total_amt equals the sum of employer_cpf_amt and total_amt_employee for consistency", + "type": "sql", + "dimension": "consistency", + "severity": "error", + "query": "SELECT COUNT(*) FROM demo_employee_payroll WHERE total_amt != (employer_cpf_amt + total_amt_employee) OR (total_amt IS NOT NULL AND (employer_cpf_amt IS NULL OR total_amt_employee IS NULL))", + "mustBe": 0, + "tags": [ + "User-Defined Quality Rule" + ] + }, + "contract_definition": { + "name": "total_amt_equals_employer_cpf_plus_total_amt_employee", + "description": "Validates that total_amt equals the sum of employer_cpf_amt and total_amt_employee for consistency", + "type": "sql", + "dimension": "consistency", + "severity": "error", + "query": "SELECT COUNT(*) FROM demo_employee_payroll WHERE total_amt != (employer_cpf_amt + total_amt_employee) OR (total_amt IS NOT NULL AND (employer_cpf_amt IS NULL OR total_amt_employee IS NULL))", + "mustBe": 0, + "tags": [ + "User-Defined Quality Rule" + ] + }, "target": "demo_employee_payroll.total_amt", "logical_type": "integer", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM demo_employee_payroll WHERE total_amt <> (employer_cpf_amt + total_amt_employee) OR (NOT total_amt IS NULL AND (employer_cpf_amt IS NULL OR total_amt_employee IS NULL))" + "rendered_implementation": "SELECT COUNT(*) FROM demo_employee_payroll WHERE total_amt <> (employer_cpf_amt + total_amt_employee) OR (NOT total_amt IS NULL AND (employer_cpf_amt IS NULL OR total_amt_employee IS NULL))" }, { "name": "total_amt_missingValues", @@ -1333,24 +1889,48 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 19.144125006278045, + "execution_time_ms": 33.28112499730196, "check_path": "$.schema[0].properties[5].quality[1]", "check_ref_type": "MissingValuesCheckReference", - "dimension": "completeness", - "type": "library", - "description": "Count of missing values in 'total_amt' of 'demo_employee_payroll'", - "severity": null, - "schema": "demo_employee_payroll", - "operator": "mustBe", - "is_generated": true, - "engine": "sql", + "schema_name": "demo_employee_payroll", + "operator": "mustBe", + "is_generated": true, + "engine": "sql", + "check_definition": { + "type": "library", + "metric": "missingValues", + "mustBe": 0, + "dimension": "completeness", + "arguments": { + "missingValues": [ + null, + "" + ] + }, + "tags": [ + "vowl_generated_check" + ], + "description": "Count of missing values in 'total_amt' of 'demo_employee_payroll'" + }, + "contract_definition": { + "type": "library", + "metric": "missingValues", + "mustBe": 0, + "dimension": "completeness", + "arguments": { + "missingValues": [ + null, + "" + ] + } + }, "target": "demo_employee_payroll.total_amt", "logical_type": "integer", "aggregation_type": "count", "tables_in_query": [ "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"total_amt\" IS NULL OR TRY_CAST(\"total_amt\" AS TEXT) IN ('')" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_payroll\" WHERE \"total_amt\" IS NULL OR TRY_CAST(\"total_amt\" AS TEXT) IN ('')" }, { "name": "employee_id_column_exists_check", @@ -1359,24 +1939,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.450958997244015, + "execution_time_ms": 10.498125004232861, "check_path": "$.schema[1].properties[0].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'employee_id' must exist in 'demo_employee_list'", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_id_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'employee_id' must exist in 'demo_employee_list'", + "query": "SELECT COUNT(*) FROM (SELECT \"employee_id\" FROM \"demo_employee_list\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "employee_id" + }, "target": "demo_employee_list.employee_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"employee_id\" FROM \"demo_employee_list\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"employee_id\" FROM \"demo_employee_list\" LIMIT 0) AS _vowl_column_exists" }, { "name": "employee_id_logical_type_check", @@ -1385,24 +1975,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 8.749791988520883, + "execution_time_ms": 20.03816599608399, "check_path": "$.schema[1].properties[0].logicalType", "check_ref_type": "LogicalTypeCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Values in 'employee_id' must be valid string", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_id_logical_type_check", + "type": "sql", + "dimension": "conformity", + "description": "Values in 'employee_id' must be valid string", + "query": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"employee_id\" IS NULL AND CAST(\"employee_id\" AS VARCHAR) IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "string" + }, "target": "demo_employee_list.employee_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"employee_id\" IS NULL AND TRY_CAST(\"employee_id\" AS TEXT) IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"employee_id\" IS NULL AND TRY_CAST(\"employee_id\" AS TEXT) IS NULL" }, { "name": "employee_id_logical_type_options_pattern_check", @@ -1411,24 +2011,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 25.176167007884942, + "execution_time_ms": 42.80774999642745, "check_path": "$.schema[1].properties[0].logicalTypeOptions.pattern", "check_ref_type": "LogicalTypeOptionsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'employee_id' must match pattern '^e[0-9]+$'", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_id_logical_type_options_pattern_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'employee_id' must match pattern '^e[0-9]+$'", + "query": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"employee_id\" IS NULL AND NOT CAST(\"employee_id\" AS VARCHAR) ~ '^e[0-9]+$'", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "^e[0-9]+$" + }, "target": "demo_employee_list.employee_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"employee_id\" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(\"employee_id\" AS TEXT), '^e[0-9]+$')" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"employee_id\" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(\"employee_id\" AS TEXT), '^e[0-9]+$')" }, { "name": "employee_id_logical_type_options_minLength_check", @@ -1437,24 +2047,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 25.518875001580454, + "execution_time_ms": 48.060291999718174, "check_path": "$.schema[1].properties[0].logicalTypeOptions.minLength", "check_ref_type": "LogicalTypeOptionsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'employee_id' must have minimum length of 7", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_id_logical_type_options_minLength_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'employee_id' must have minimum length of 7", + "query": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"employee_id\" IS NULL AND LENGTH(CAST(\"employee_id\" AS VARCHAR)) < 7", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": 7 + }, "target": "demo_employee_list.employee_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"employee_id\" IS NULL AND LENGTH(TRY_CAST(\"employee_id\" AS TEXT)) < 7" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"employee_id\" IS NULL AND LENGTH(TRY_CAST(\"employee_id\" AS TEXT)) < 7" }, { "name": "employee_id_logical_type_options_maxLength_check", @@ -1463,24 +2083,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 27.36829199420754, + "execution_time_ms": 34.422624987200834, "check_path": "$.schema[1].properties[0].logicalTypeOptions.maxLength", "check_ref_type": "LogicalTypeOptionsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'employee_id' must have maximum length of 7", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_id_logical_type_options_maxLength_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'employee_id' must have maximum length of 7", + "query": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"employee_id\" IS NULL AND LENGTH(CAST(\"employee_id\" AS VARCHAR)) > 7", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": 7 + }, "target": "demo_employee_list.employee_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"employee_id\" IS NULL AND LENGTH(TRY_CAST(\"employee_id\" AS TEXT)) > 7" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"employee_id\" IS NULL AND LENGTH(TRY_CAST(\"employee_id\" AS TEXT)) > 7" }, { "name": "employee_id_required_check", @@ -1489,24 +2119,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.373791995225474, + "execution_time_ms": 8.225458004744723, "check_path": "$.schema[1].properties[0].required", "check_ref_type": "RequiredCheckReference", - "dimension": "completeness", - "type": "sql", - "description": "Column 'employee_id' must not contain NULL values", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_id_required_check", + "type": "sql", + "dimension": "completeness", + "description": "Column 'employee_id' must not contain NULL values", + "query": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE \"employee_id\" IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": true + }, "target": "demo_employee_list.employee_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE \"employee_id\" IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE \"employee_id\" IS NULL" }, { "name": "employee_id_unique_check", @@ -1515,24 +2155,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 6.475375004811212, + "execution_time_ms": 8.616082996013574, "check_path": "$.schema[1].properties[0].unique", "check_ref_type": "UniqueCheckReference", - "dimension": "consistency", - "type": "sql", - "description": "Column 'employee_id' must contain unique values", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_id_unique_check", + "type": "sql", + "dimension": "consistency", + "description": "Column 'employee_id' must contain unique values", + "query": "SELECT COUNT(*) FROM (SELECT \"employee_id\" FROM \"demo_employee_list\" WHERE NOT \"employee_id\" IS NULL GROUP BY \"employee_id\" HAVING COUNT(*) > 1)", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": true + }, "target": "demo_employee_list.employee_id", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"employee_id\" FROM \"demo_employee_list\" WHERE NOT \"employee_id\" IS NULL GROUP BY \"employee_id\" HAVING COUNT(*) > 1)" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"employee_id\" FROM \"demo_employee_list\" WHERE NOT \"employee_id\" IS NULL GROUP BY \"employee_id\" HAVING COUNT(*) > 1)" }, { "name": "employee_id_primary_key_check", @@ -1541,24 +2191,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.115499996463768, + "execution_time_ms": 6.8277919926913455, "check_path": "$.schema[1].properties[0].primaryKey", "check_ref_type": "PrimaryKeyCheckReference", - "dimension": "consistency", - "type": "sql", - "description": "Primary key column 'employee_id' must be unique and not null", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_id_primary_key_check", + "type": "sql", + "dimension": "consistency", + "description": "Primary key column 'employee_id' must be unique and not null", + "query": "SELECT ((SELECT COUNT(*) FROM \"demo_employee_list\" WHERE \"employee_id\" IS NULL)) + ((SELECT COUNT(*) FROM (SELECT \"employee_id\" FROM \"demo_employee_list\" WHERE NOT \"employee_id\" IS NULL GROUP BY \"employee_id\" HAVING COUNT(*) > 1)))", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": true + }, "target": "demo_employee_list.employee_id", "logical_type": "string", "aggregation_type": "custom", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT ((SELECT COUNT(*) FROM \"demo_employee_list\" WHERE \"employee_id\" IS NULL)) + ((SELECT COUNT(*) FROM (SELECT \"employee_id\" FROM \"demo_employee_list\" WHERE NOT \"employee_id\" IS NULL GROUP BY \"employee_id\" HAVING COUNT(*) > 1)))" + "rendered_implementation": "SELECT ((SELECT COUNT(*) FROM \"demo_employee_list\" WHERE \"employee_id\" IS NULL)) + ((SELECT COUNT(*) FROM (SELECT \"employee_id\" FROM \"demo_employee_list\" WHERE NOT \"employee_id\" IS NULL GROUP BY \"employee_id\" HAVING COUNT(*) > 1)))" }, { "name": "employee_last_name_column_exists_check", @@ -1567,24 +2227,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 6.529666992719285, + "execution_time_ms": 17.24595800624229, "check_path": "$.schema[1].properties[1].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'employee_last_name' must exist in 'demo_employee_list'", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_last_name_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'employee_last_name' must exist in 'demo_employee_list'", + "query": "SELECT COUNT(*) FROM (SELECT \"employee_last_name\" FROM \"demo_employee_list\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "employee_last_name" + }, "target": "demo_employee_list.employee_last_name", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"employee_last_name\" FROM \"demo_employee_list\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"employee_last_name\" FROM \"demo_employee_list\" LIMIT 0) AS _vowl_column_exists" }, { "name": "employee_last_name_logical_type_check", @@ -1593,24 +2263,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 10.200917007750832, + "execution_time_ms": 14.31154199235607, "check_path": "$.schema[1].properties[1].logicalType", "check_ref_type": "LogicalTypeCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Values in 'employee_last_name' must be valid string", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_last_name_logical_type_check", + "type": "sql", + "dimension": "conformity", + "description": "Values in 'employee_last_name' must be valid string", + "query": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"employee_last_name\" IS NULL AND CAST(\"employee_last_name\" AS VARCHAR) IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "string" + }, "target": "demo_employee_list.employee_last_name", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"employee_last_name\" IS NULL AND TRY_CAST(\"employee_last_name\" AS TEXT) IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"employee_last_name\" IS NULL AND TRY_CAST(\"employee_last_name\" AS TEXT) IS NULL" }, { "name": "employee_last_name_required_check", @@ -1619,24 +2299,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.171458004042506, + "execution_time_ms": 6.7928750067949295, "check_path": "$.schema[1].properties[1].required", "check_ref_type": "RequiredCheckReference", - "dimension": "completeness", - "type": "sql", - "description": "Column 'employee_last_name' must not contain NULL values", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_last_name_required_check", + "type": "sql", + "dimension": "completeness", + "description": "Column 'employee_last_name' must not contain NULL values", + "query": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE \"employee_last_name\" IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": true + }, "target": "demo_employee_list.employee_last_name", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE \"employee_last_name\" IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE \"employee_last_name\" IS NULL" }, { "name": "employee_first_name_column_exists_check", @@ -1645,24 +2335,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.425541996373795, + "execution_time_ms": 7.681333998334594, "check_path": "$.schema[1].properties[2].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'employee_first_name' must exist in 'demo_employee_list'", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_first_name_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'employee_first_name' must exist in 'demo_employee_list'", + "query": "SELECT COUNT(*) FROM (SELECT \"employee_first_name\" FROM \"demo_employee_list\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "employee_first_name" + }, "target": "demo_employee_list.employee_first_name", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"employee_first_name\" FROM \"demo_employee_list\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"employee_first_name\" FROM \"demo_employee_list\" LIMIT 0) AS _vowl_column_exists" }, { "name": "employee_first_name_logical_type_check", @@ -1671,24 +2371,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 8.74150000163354, + "execution_time_ms": 11.540167004568502, "check_path": "$.schema[1].properties[2].logicalType", "check_ref_type": "LogicalTypeCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Values in 'employee_first_name' must be valid string", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_first_name_logical_type_check", + "type": "sql", + "dimension": "conformity", + "description": "Values in 'employee_first_name' must be valid string", + "query": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"employee_first_name\" IS NULL AND CAST(\"employee_first_name\" AS VARCHAR) IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "string" + }, "target": "demo_employee_list.employee_first_name", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"employee_first_name\" IS NULL AND TRY_CAST(\"employee_first_name\" AS TEXT) IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"employee_first_name\" IS NULL AND TRY_CAST(\"employee_first_name\" AS TEXT) IS NULL" }, { "name": "employee_first_name_required_check", @@ -1697,24 +2407,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 6.4498749998165295, + "execution_time_ms": 7.362165997619741, "check_path": "$.schema[1].properties[2].required", "check_ref_type": "RequiredCheckReference", - "dimension": "completeness", - "type": "sql", - "description": "Column 'employee_first_name' must not contain NULL values", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "employee_first_name_required_check", + "type": "sql", + "dimension": "completeness", + "description": "Column 'employee_first_name' must not contain NULL values", + "query": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE \"employee_first_name\" IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": true + }, "target": "demo_employee_list.employee_first_name", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE \"employee_first_name\" IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE \"employee_first_name\" IS NULL" }, { "name": "current_title_column_exists_check", @@ -1723,24 +2443,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 7.187124996562488, + "execution_time_ms": 7.876791001763195, "check_path": "$.schema[1].properties[3].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'current_title' must exist in 'demo_employee_list'", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "current_title_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'current_title' must exist in 'demo_employee_list'", + "query": "SELECT COUNT(*) FROM (SELECT \"current_title\" FROM \"demo_employee_list\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "current_title" + }, "target": "demo_employee_list.current_title", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"current_title\" FROM \"demo_employee_list\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"current_title\" FROM \"demo_employee_list\" LIMIT 0) AS _vowl_column_exists" }, { "name": "current_title_logical_type_check", @@ -1749,24 +2479,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 9.6572080074111, + "execution_time_ms": 12.056583000230603, "check_path": "$.schema[1].properties[3].logicalType", "check_ref_type": "LogicalTypeCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Values in 'current_title' must be valid string", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "current_title_logical_type_check", + "type": "sql", + "dimension": "conformity", + "description": "Values in 'current_title' must be valid string", + "query": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"current_title\" IS NULL AND CAST(\"current_title\" AS VARCHAR) IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "string" + }, "target": "demo_employee_list.current_title", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"current_title\" IS NULL AND TRY_CAST(\"current_title\" AS TEXT) IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"current_title\" IS NULL AND TRY_CAST(\"current_title\" AS TEXT) IS NULL" }, { "name": "phone_number_column_exists_check", @@ -1775,24 +2515,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 5.327791994204745, + "execution_time_ms": 13.484708993928507, "check_path": "$.schema[1].properties[4].name", "check_ref_type": "DeclaredColumnExistsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'phone_number' must exist in 'demo_employee_list'", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "phone_number_column_exists_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'phone_number' must exist in 'demo_employee_list'", + "query": "SELECT COUNT(*) FROM (SELECT \"phone_number\" FROM \"demo_employee_list\" LIMIT 0) AS _vowl_column_exists", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "phone_number" + }, "target": "demo_employee_list.phone_number", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"phone_number\" FROM \"demo_employee_list\" LIMIT 0) AS _vowl_column_exists" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"phone_number\" FROM \"demo_employee_list\" LIMIT 0) AS _vowl_column_exists" }, { "name": "phone_number_logical_type_check", @@ -1801,24 +2551,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 8.912875011446886, + "execution_time_ms": 11.268125002970919, "check_path": "$.schema[1].properties[4].logicalType", "check_ref_type": "LogicalTypeCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Values in 'phone_number' must be valid string", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "phone_number_logical_type_check", + "type": "sql", + "dimension": "conformity", + "description": "Values in 'phone_number' must be valid string", + "query": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"phone_number\" IS NULL AND CAST(\"phone_number\" AS VARCHAR) IS NULL", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "string" + }, "target": "demo_employee_list.phone_number", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"phone_number\" IS NULL AND TRY_CAST(\"phone_number\" AS TEXT) IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"phone_number\" IS NULL AND TRY_CAST(\"phone_number\" AS TEXT) IS NULL" }, { "name": "phone_number_logical_type_options_pattern_check", @@ -1827,24 +2587,34 @@ "expected_value": 0, "actual_value": 2, "failed_rows_count": 2, - "execution_time_ms": 44.16029200365301, + "execution_time_ms": 32.583833002718166, "check_path": "$.schema[1].properties[4].logicalTypeOptions.pattern", "check_ref_type": "LogicalTypeOptionsCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Column 'phone_number' must match pattern '^\\+[0-9]+$'", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "phone_number_logical_type_options_pattern_check", + "type": "sql", + "dimension": "conformity", + "description": "Column 'phone_number' must match pattern '^\\+[0-9]+$'", + "query": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"phone_number\" IS NULL AND NOT CAST(\"phone_number\" AS VARCHAR) ~ '^\\+[0-9]+$'", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": "^\\+[0-9]+$" + }, "target": "demo_employee_list.phone_number", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"phone_number\" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(\"phone_number\" AS TEXT), '^\\+[0-9]+$')" + "rendered_implementation": "SELECT COUNT(*) FROM \"demo_employee_list\" WHERE NOT \"phone_number\" IS NULL AND NOT REGEXP_MATCHES(TRY_CAST(\"phone_number\" AS TEXT), '^\\+[0-9]+$')" }, { "name": "phone_number_unique_check", @@ -1853,24 +2623,34 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 4.733582987682894, + "execution_time_ms": 6.587625000975095, "check_path": "$.schema[1].properties[4].unique", "check_ref_type": "UniqueCheckReference", - "dimension": "consistency", - "type": "sql", - "description": "Column 'phone_number' must contain unique values", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": true, "engine": "sql", + "check_definition": { + "name": "phone_number_unique_check", + "type": "sql", + "dimension": "consistency", + "description": "Column 'phone_number' must contain unique values", + "query": "SELECT COUNT(*) FROM (SELECT \"phone_number\" FROM \"demo_employee_list\" WHERE NOT \"phone_number\" IS NULL GROUP BY \"phone_number\" HAVING COUNT(*) > 1)", + "mustBe": 0, + "tags": [ + "vowl_generated_check" + ] + }, + "contract_definition": { + "value": true + }, "target": "demo_employee_list.phone_number", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM (SELECT \"phone_number\" FROM \"demo_employee_list\" WHERE NOT \"phone_number\" IS NULL GROUP BY \"phone_number\" HAVING COUNT(*) > 1)" + "rendered_implementation": "SELECT COUNT(*) FROM (SELECT \"phone_number\" FROM \"demo_employee_list\" WHERE NOT \"phone_number\" IS NULL GROUP BY \"phone_number\" HAVING COUNT(*) > 1)" }, { "name": "PersonName", @@ -1879,24 +2659,42 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 14.657958003226668, + "execution_time_ms": 21.21066600375343, "check_path": "$.schema[1].properties[1].quality[0]", "check_ref_type": "SQLColumnCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "PersonName may include various name types for UIN/FIN holders including legal names, abbreviated names, Hanyu Pinyin names, alias names, married names, and changed names | X(66)", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": false, "engine": "sql", + "check_definition": { + "type": "sql", + "name": "PersonName", + "description": "PersonName may include various name types for UIN/FIN holders including legal names, abbreviated names, Hanyu Pinyin names, alias names, married names, and changed names | X(66)", + "mustBe": 0, + "query": "SELECT COUNT(*)\nFROM demo_employee_list\nWHERE employee_last_name IS NOT NULL\n AND LENGTH(CAST(employee_last_name AS TEXT)) > 66;", + "tags": [ + "SG-DRM v5.0" + ], + "dimension": "conformity" + }, + "contract_definition": { + "type": "sql", + "name": "PersonName", + "description": "PersonName may include various name types for UIN/FIN holders including legal names, abbreviated names, Hanyu Pinyin names, alias names, married names, and changed names | X(66)", + "mustBe": 0, + "query": "SELECT COUNT(*)\nFROM demo_employee_list\nWHERE employee_last_name IS NOT NULL\n AND LENGTH(CAST(employee_last_name AS TEXT)) > 66;", + "tags": [ + "SG-DRM v5.0" + ], + "dimension": "conformity" + }, "target": "demo_employee_list.employee_last_name", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM demo_employee_list WHERE NOT employee_last_name IS NULL AND LENGTH(TRY_CAST(employee_last_name AS TEXT)) > 66" + "rendered_implementation": "SELECT COUNT(*) FROM demo_employee_list WHERE NOT employee_last_name IS NULL AND LENGTH(TRY_CAST(employee_last_name AS TEXT)) > 66" }, { "name": "PersonName", @@ -1905,24 +2703,42 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 14.439665988902561, + "execution_time_ms": 25.908499999786727, "check_path": "$.schema[1].properties[2].quality[0]", "check_ref_type": "SQLColumnCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "PersonName may include various name types for UIN/FIN holders including legal names, abbreviated names, Hanyu Pinyin names, alias names, married names, and changed names | X(66)", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": false, "engine": "sql", + "check_definition": { + "type": "sql", + "name": "PersonName", + "description": "PersonName may include various name types for UIN/FIN holders including legal names, abbreviated names, Hanyu Pinyin names, alias names, married names, and changed names | X(66)", + "mustBe": 0, + "query": "SELECT COUNT(*)\nFROM demo_employee_list\nWHERE employee_first_name IS NOT NULL\n AND LENGTH(CAST(employee_first_name AS TEXT)) > 66;", + "tags": [ + "SG-DRM v5.0" + ], + "dimension": "conformity" + }, + "contract_definition": { + "type": "sql", + "name": "PersonName", + "description": "PersonName may include various name types for UIN/FIN holders including legal names, abbreviated names, Hanyu Pinyin names, alias names, married names, and changed names | X(66)", + "mustBe": 0, + "query": "SELECT COUNT(*)\nFROM demo_employee_list\nWHERE employee_first_name IS NOT NULL\n AND LENGTH(CAST(employee_first_name AS TEXT)) > 66;", + "tags": [ + "SG-DRM v5.0" + ], + "dimension": "conformity" + }, "target": "demo_employee_list.employee_first_name", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM demo_employee_list WHERE NOT employee_first_name IS NULL AND LENGTH(TRY_CAST(employee_first_name AS TEXT)) > 66" + "rendered_implementation": "SELECT COUNT(*) FROM demo_employee_list WHERE NOT employee_first_name IS NULL AND LENGTH(TRY_CAST(employee_first_name AS TEXT)) > 66" }, { "name": "TelephoneNumber", @@ -1931,24 +2747,42 @@ "expected_value": 0, "actual_value": 0, "failed_rows_count": 0, - "execution_time_ms": 14.721458006533794, + "execution_time_ms": 23.390207992633805, "check_path": "$.schema[1].properties[4].quality[0]", "check_ref_type": "SQLColumnCheckReference", - "dimension": "conformity", - "type": "sql", - "description": "Based on ITU T E.164 | e.g. 095236", - "severity": null, - "schema": "demo_employee_list", + "schema_name": "demo_employee_list", "operator": "mustBe", "is_generated": false, "engine": "sql", + "check_definition": { + "type": "sql", + "name": "TelephoneNumber", + "description": "Based on ITU T E.164 | e.g. 095236", + "mustBe": 0, + "query": "SELECT COUNT(*)\nFROM demo_employee_list\nWHERE CAST(phone_number AS TEXT) !~ '^[0-9]{1,12}$';", + "tags": [ + "SG-DRM v5.0" + ], + "dimension": "conformity" + }, + "contract_definition": { + "type": "sql", + "name": "TelephoneNumber", + "description": "Based on ITU T E.164 | e.g. 095236", + "mustBe": 0, + "query": "SELECT COUNT(*)\nFROM demo_employee_list\nWHERE CAST(phone_number AS TEXT) !~ '^[0-9]{1,12}$';", + "tags": [ + "SG-DRM v5.0" + ], + "dimension": "conformity" + }, "target": "demo_employee_list.phone_number", "logical_type": "string", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list" ], - "rule": "SELECT COUNT(*) FROM demo_employee_list WHERE NOT REGEXP_MATCHES(TRY_CAST(phone_number AS TEXT), '^[0-9]{1,12}$')" + "rendered_implementation": "SELECT COUNT(*) FROM demo_employee_list WHERE NOT REGEXP_MATCHES(TRY_CAST(phone_number AS TEXT), '^[0-9]{1,12}$')" }, { "name": "employee_id_exists_in_master_list", @@ -1957,24 +2791,42 @@ "expected_value": 0, "actual_value": 1, "failed_rows_count": 1, - "execution_time_ms": 47.93762500048615, + "execution_time_ms": 121.92870900616981, "check_path": "$.schema[0].quality[1]", "check_ref_type": "SQLTableCheckReference", - "dimension": "consistency", - "type": "sql", - "description": "Referential integrity check: Ensures all employee IDs in the payroll table exist in the reference employee list.", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": false, "engine": "sql", + "check_definition": { + "name": "employee_id_exists_in_master_list", + "description": "Referential integrity check: Ensures all employee IDs in the payroll table exist in the reference employee list.", + "type": "sql", + "dimension": "consistency", + "query": "SELECT COUNT(*) FROM demo_employee_payroll payroll LEFT JOIN demo_employee_list ref \n ON payroll.employee_id = ref.employee_id\nWHERE ref.employee_id IS NULL", + "mustBe": 0, + "tags": [ + "Cross-Table Validation" + ] + }, + "contract_definition": { + "name": "employee_id_exists_in_master_list", + "description": "Referential integrity check: Ensures all employee IDs in the payroll table exist in the reference employee list.", + "type": "sql", + "dimension": "consistency", + "query": "SELECT COUNT(*) FROM demo_employee_payroll payroll LEFT JOIN demo_employee_list ref \n ON payroll.employee_id = ref.employee_id\nWHERE ref.employee_id IS NULL", + "mustBe": 0, + "tags": [ + "Cross-Table Validation" + ] + }, "target": "demo_employee_payroll", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list", "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM demo_employee_payroll AS payroll LEFT JOIN demo_employee_list AS ref ON payroll.employee_id = ref.employee_id WHERE ref.employee_id IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM demo_employee_payroll AS payroll LEFT JOIN demo_employee_list AS ref ON payroll.employee_id = ref.employee_id WHERE ref.employee_id IS NULL" }, { "name": "phone_number_exists_in_master_list", @@ -1983,24 +2835,42 @@ "expected_value": 0, "actual_value": 2, "failed_rows_count": 2, - "execution_time_ms": 14.986333000706509, + "execution_time_ms": 29.520874988520518, "check_path": "$.schema[0].quality[2]", "check_ref_type": "SQLTableCheckReference", - "dimension": "consistency", - "type": "sql", - "description": "Ensures the phone numbers recorded in payroll matches an entry in the master employee list.", - "severity": null, - "schema": "demo_employee_payroll", + "schema_name": "demo_employee_payroll", "operator": "mustBe", "is_generated": false, "engine": "sql", + "check_definition": { + "name": "phone_number_exists_in_master_list", + "description": "Ensures the phone numbers recorded in payroll matches an entry in the master employee list.", + "type": "sql", + "dimension": "consistency", + "query": "SELECT COUNT(*) FROM demo_employee_payroll payroll LEFT JOIN demo_employee_list ref \n ON payroll.phone_number = ref.phone_number\nWHERE payroll.phone_number IS NOT NULL \n AND ref.phone_number IS NULL", + "mustBe": 0, + "tags": [ + "Cross-Table Validation" + ] + }, + "contract_definition": { + "name": "phone_number_exists_in_master_list", + "description": "Ensures the phone numbers recorded in payroll matches an entry in the master employee list.", + "type": "sql", + "dimension": "consistency", + "query": "SELECT COUNT(*) FROM demo_employee_payroll payroll LEFT JOIN demo_employee_list ref \n ON payroll.phone_number = ref.phone_number\nWHERE payroll.phone_number IS NOT NULL \n AND ref.phone_number IS NULL", + "mustBe": 0, + "tags": [ + "Cross-Table Validation" + ] + }, "target": "demo_employee_payroll", "aggregation_type": "count", "tables_in_query": [ "demo_employee_list", "demo_employee_payroll" ], - "rule": "SELECT COUNT(*) FROM demo_employee_payroll AS payroll LEFT JOIN demo_employee_list AS ref ON payroll.phone_number = ref.phone_number WHERE NOT payroll.phone_number IS NULL AND ref.phone_number IS NULL" + "rendered_implementation": "SELECT COUNT(*) FROM demo_employee_payroll AS payroll LEFT JOIN demo_employee_list AS ref ON payroll.phone_number = ref.phone_number WHERE NOT payroll.phone_number IS NULL AND ref.phone_number IS NULL" } ], "contract_metadata": { diff --git a/examples/vowl_usage_patterns_demo.ipynb b/examples/vowl_usage_patterns_demo.ipynb index cb7f5d1..13be359 100644 --- a/examples/vowl_usage_patterns_demo.ipynb +++ b/examples/vowl_usage_patterns_demo.ipynb @@ -125,12 +125,153 @@ "name": "stderr", "output_type": "stream", "text": [ - "/var/folders/tg/0hbw1fbs5q1_5kdtp36klrtw0000gn/T/ipykernel_35770/1521035067.py:4: DtypeWarning: Columns (0: lease_commence_date) have mixed types. Specify dtype option on import or set low_memory=False.\n", + "/var/folders/tg/0hbw1fbs5q1_5kdtp36klrtw0000gn/T/ipykernel_25503/1521035067.py:4: DtypeWarning: Columns (0: lease_commence_date) have mixed types. Specify dtype option on import or set low_memory=False.\n", " df = pd.read_csv(HDB_CSV).fillna(\"\")\n" ] }, { "data": { + "application/vnd.microsoft.datawrangler.viewer.v0+json": { + "columns": [ + { + "name": "index", + "rawType": "int64", + "type": "integer" + }, + { + "name": "month", + "rawType": "str", + "type": "string" + }, + { + "name": "town", + "rawType": "str", + "type": "string" + }, + { + "name": "flat_type", + "rawType": "str", + "type": "string" + }, + { + "name": "block", + "rawType": "str", + "type": "string" + }, + { + "name": "street_name", + "rawType": "str", + "type": "string" + }, + { + "name": "storey_range", + "rawType": "str", + "type": "string" + }, + { + "name": "floor_area_sqm", + "rawType": "float64", + "type": "float" + }, + { + "name": "flat_model", + "rawType": "str", + "type": "string" + }, + { + "name": "lease_commence_date", + "rawType": "object", + "type": "string" + }, + { + "name": "remaining_lease", + "rawType": "str", + "type": "string" + }, + { + "name": "resale_price", + "rawType": "float64", + "type": "float" + } + ], + "ref": "ec1f9970-c371-41d3-836c-6ff2f09b92c8", + "rows": [ + [ + "0", + "2017-01", + "ANG MO KIO", + "2 ROOM", + "406", + "ANG MO KIO AVE 10", + "10 TO 12", + "44.0", + "Improved", + "1979", + "61 years 04 months", + "232000.0" + ], + [ + "1", + "2017-01", + "ANG MO KIO", + "3 ROOM", + "108", + "ANG MO KIO AVE 4", + "01 TO 03", + "67.0", + "New Generation", + "1978", + "60 years 07 months", + "250000.0" + ], + [ + "2", + "2017-01", + "ANG MO KIO", + "3 ROOM", + "602", + "ANG MO KIO AVE 5", + "01 TO 03", + "67.0", + "New Generation", + "1980", + "62 years 05 months", + "262000.0" + ], + [ + "3", + "2017-01", + "ANG MO KIO", + "3 ROOM", + "465", + "ANG MO KIO AVE 10", + "04 TO 06", + "68.0", + "New Generation", + "1980", + "62 years 01 month", + "265000.0" + ], + [ + "4", + "2017-01", + "ANG MO KIO", + "3 ROOM", + "601", + "ANG MO KIO AVE 5", + "01 TO 03", + "67.0", + "New Generation", + "1980", + "62 years 05 months", + "265000.0" + ] + ], + "shape": { + "columns": 11, + "rows": 5 + } + }, "text/html": [ "
\n", "