feat: dataset provenance form with view/edit/validation

[SanjeevLakhwani]

Summary

• Adds a new DatasetForm with tabbed UI covering all dataset metadata fields (core info, contacts, links, classification, study details, publications/funding, PCGL info)
• Adds a View Provenance button on the Dataset card (visible to all users) that opens a read-only modal of all dataset metadata
• Migrates all project.datasets references to project.datasets_v2 and removes the deprecated datasets field from the Project type

Pre-requisite

• Requires: Dataset provenance model update katsu#665
• Requires latest gohan

[gsfk]

First comment: wipes out existing datasets in web, although public seems fine.

Or perhaps I'm meant to migrate? It's not clear how to do that, however.

[gsfk]

First-pass attempt, I haven't tried everything yet.

• Existing datasets go missing, see comment above.

• Overall, the form is pretty long, so I suspect will be easier to submit as a document. I haven't tried this yet

• Because it's so long it should be a little more forgiving, in particular a mis-click outside of the modal should not close the modal and wipe all content.

• Does it need to be a modal?

• The "person" and "organization" roles are all together ("Site" doesn't make much sense as a role for a person)

• it wasn't immediately clear to me that "Primary Contact" had to be a single entity... I guess that's on me for not reading the word "primary", although possibly I was put off by "add name" which looks like you can add more contacts.

• "add Ontology resource" is bound to be filled in incorrectly without more controls, even in day-to-day bento operations you can find data with incorrect iri_prefix. This is more of a bento bookkeeping comment than a review of this particular PR. It refuses ontology additions without resources, so not sure if this will discourage ontology use rather than encourage it. I'm not sure how to make this easier.

• Validation errors produce the message "Zod validation error" which is a little odd sounding

• Are stakeholders really required? What if the primary contact is an organization and also the stakeholder?

• after all that, it actually failed to create a dataset, since katsu seems to require release_date but the form doesn't, so it failed (and I lost everything I typed in...)

[gsfk]

Coming along!

• can't actually create a dataset, even when form passes validation, although that now looks like a katsu error. It produces 400 Bad Request and this message: "Error validating discovery scope: project-dataset pair does not exist." which seems like a nonsensical error when trying to create a dataset
• some fields are required dynamically (license url is required only if some other license fields are filled in) but are never marked as required in the form
• I'm able to fill in keywords using both string and ontology even though the form says it should be one or the other
• submission by file seems to work well, although can't test because of first issue

less important stuff:

• validation errors could show which section of the form they're from (currently they show intenal form codes like root or taxa.0)
• validation errors are sometimes incomplete: if I add ontologies without a resource and a license without a url, it only shows the license error. If I correct the license, the ontology error appears

[gsfk]

Validation errors are missing... now only shows errors on the current tab, if any.

If you don't have errors on the current tab it just blocks submission without telling you anything.

[gsfk]

working as expected, pending migration away from "_v2"

[gsfk]

Re-testing after katsu updates:

• adding database Linked Field set is broken
• if you try to edit database info more than once, the information is often stale and behaves weirdly
• editing sometimes modifies the wrong dataset (I had dataset "A" and dataset "B".... editing the name of A changed B instead and left A untouched)

[gsfk]

Form data still looks stale, if I click through multiple datasets in turn, I get information for the previous dataset I clicked rather than the current one.

Also having trouble with a dataset creation features, lots of stuff not related to provenance is not working. Will this magically fix if we stop naming stuff "v2"?

• can't ingest data into a new dataset (katsu says error encountered while ingesting: dataset does not exist ). But ingestion does work for the v2 datasets that were created by the migration process.
• data summaries are broken (the modal when you click on "Clinical Data", "Experiments" or "Variants".. values are all zero or "N/A")

[gsfk]

does it do anything with the GeoJSON or just store it as a string?

[SanjeevLakhwani]

does it do anything with the GeoJSON or just store it as a string?

Nothing for now, it is just a possible input for a field in provenance

[gsfk]

bento_web search not working for new v2 datasets

[gsfk]

• shouldn't there be a field for data access codes?
• geoJson serializes weirdly:

[gsfk]

Preliminary approval since I'm out of issues for this latest iteration. No comments on code.

It's now possible to have ontology resource mismatches between the dataset info, and the data itself, by each having different versions, but:

• that's now easy to edit for the dataset info
• it's not clear that it even matters

For actually using the ontologies in phenopacket view in bento_public, it seems odd that we don't simply use the resources that are in the phenopacket itself. But this is a separate issue.

[davidlougheed]

lgtm