feat: experiment library schema changes, ingestion report with errors, json schema warnings

[v-rocheleau]

Ingestion report

The ingestion endpoint now returns an "ingestion report" in the response's JSON, with shape:

{
  "success": true,
  "errors": [],
  "warnings": []
}

Where,

• success indicates if the ingestion was successful (now http status 201, was 204)
• errors lists the errors that were caught during ingestion, causing a failed ingestion
• warnings lists custom warnings related to schema changes.

Ingestion WDLs

• Workflow files for experiments and phenopackets ingestion now store the ingestion report JSON in the run directory
• The ingestion report is now an output of the workflows

Schema Changes

Katsu can now keep track of JSON-schema modifications in a release.
The schema changes objects are used to perform data migrations in Django, they also allow the ingestion view to compile warnings if deprecated properties are found in the submitted data.

EXPERIMENT_SCHEMA_CHANGES = {
    "4.1.0": {
        "properties": {
            "library_strategy": [
                    ("WES", "WXS"),
                    ("Other", "OTHER"),
            ],
            "library_selection": [
                    ("Random", "RANDOM"),
                    ("Random PCR", "RANDOM PCR"),
                    ("Exome capture", "Hybrid Selection"),
                    ("Other", "other"),
            ]
        }
    }
}

EBI experiment schema

• XSD experiment file taken from EBI's schema database: http://ftp.ebi.ac.uk/pub/databases/ena/doc/xsd/sra_1_5/
• experiment's library_strategy and library_selection values are read from the file and used in the schema
• data migration to convert 'WES' -> 'WXS', etc

[codecov]

Codecov Report

Attention: 8 lines in your changes are missing coverage. Please review.

Comparison is base (6ab2ed8) 88.36% compared to head (dd0f66c) 86.60%.
Report is 98 commits behind head on develop.

❗ Current head dd0f66c differs from pull request most recent head bbef730. Consider uploading reports for the commit bbef730 to get more accurate results

Files	Patch %	Lines
chord_metadata_service/chord/ingest/exceptions.py	87.50%	1 Missing and 4 partials ⚠️
chord_metadata_service/chord/ingest/views.py	94.11%	1 Missing and 1 partial ⚠️
chord_metadata_service/chord/ingest/experiments.py	87.50%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #435      +/-   ##
===========================================
- Coverage    88.36%   86.60%   -1.77%     
===========================================
  Files          115      131      +16     
  Lines         4462     5247     +785     
  Branches       559      696     +137     
===========================================
+ Hits          3943     4544     +601     
- Misses         371      521     +150     
- Partials       148      182      +34     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[davidlougheed]

some notes

++ if we don't want to bother preserving backwards compatibility, we could just save this for katsu 5.0.0 instead

[davidlougheed]

two years out of date - I'm closing this.

[davidlougheed]

actually nevermind, some of this might still be needed for genovalia - re-opening haha