Skip to content

Marekhorst 1586 integrate lens.org patents mining#1613

Open
marekhorst wants to merge 10 commits into
masterfrom
marekhorst_1586_integrate_lens.org_patents_mining
Open

Marekhorst 1586 integrate lens.org patents mining#1613
marekhorst wants to merge 10 commits into
masterfrom
marekhorst_1586_integrate_lens.org_patents_mining

Conversation

@marekhorst

@marekhorst marekhorst commented Jul 1, 2026

Copy link
Copy Markdown
Member

Bunch of details which may help in the reviewing process:

  • in principle we are replacing an extremely complex, EPO-patents related, workflow involving importing patents from EPO, text mining (madis) supplemented with pantents entities generation, custom patent-related export procedure involving exporting of both document-patent relations and patent entities with the simplified version relying on predetermined set of patents coming from lens.org. Those patents are stored in an already constructed sqlite database (thus no need to import patents from EPO) and are used for text mining. There is no need to export patent entities because they will be introduced into the graph by relying on a specific workflow which is part of the openaire data provisioning system, so it happens outside of the IIS. IIS exporter module now focuses on relations only therefore it could be simplified and aligned with an already existing model (extending AbstractActionBuilderFactory) for exporting simple relations.
  • there are changes introduced around the madis script (patents.sql) and the core madis (text.py) which are bound to the text mining logic changes after EPO patents were replaced with patents provided by lens.org
  • DocumentToPatent#lensId is patent id provided by lens.org and it gets translated to the OpenAIRE id by the exporter module (DocumentToPatentActionBuilderModuleFactory)
  • PatentReferenceExtractionInput was modified and now it represents an avro schema with raw patents metadata fully compliant with sqlitedb patents table model. Since the dynamic patent import from the graph is going to be introduced in the future (now we simply rely on a predefined sqlitedb which resides on HDFS), which means there is no sqlite_builder script responsible for building sqlitedb dynamically, we had to introduce a way of creating sqlitedb just for the sake of running integration tests (to be able to create a small db on the fly instead of relying on a resource which needs to be pre-uploaded on HDFS). So the current version of the sqlite_builder script was prepared just to cover that requirement and its content is extremelly simplified (thus PatentReferenceExtractionInput mimics the internal sqlitedb model). sqlite_builder is used in the integration tests only, it relies on a small sqlitedb (as a kind of a template for building the final database with patents) at input including jurisdiction predefined table.
  • just realized in some commits there is Rewrite PIG transformers to spark and make them executable in the k8s environment #1596 referenced instead of Integrate lens.org patents mining #1586 - it needs to be fixed when sqashing

johnfouf and others added 10 commits June 3, 2026 12:57
@johnfouf @marekhorst
Update patents.sql - lens patent mining
06b9d5f
@marekhorst
Closes #1586: Integrate lens.org patents mining
9570a49 
Applying changes on the workflow level by removing actions related to handling patents metadata.
@johnfouf @marekhorst
Refactor SQL query for patent extraction
cbed7e1
@johnfouf @marekhorst
Add string_split_value function to text.py
45af05e 
Added a function to split a string from the last occurrence of a specified substring.
@marekhorst
Fix regex match in patents.sql
67030c9 
This is cherry-picked 20a6d06 commit from Mar 3, 2026 which got applied directly on Feb 9 commits without applying the commit from Feb 25 which was experimentation and was meant to be skipped (more in #10371#note-38).
@marekhorst
Closes #1596: Integrate lens.org patents mining
5d59b1f 
Propagating the change introduced by Yannis: "Update SQL query to select lens_id instead of appln_nr".
@marekhorst
Closes #1596: Integrate lens.org patents mining
25da2b1 
Replacing appln_nr with lens_id in the `DocumentToPatent` schema definition.
@marekhorst
Closes #1596: Integrate lens.org patents mining
057beb9 
Discontinuing the old way of handling EPO patents by removing:
* `ImportedPatent` and `Patent` avro records definitions
* EPO patents importer module along with tests suite
* patent entities export along with tests suite

New way of handling patents coming from lens.org was supplemented with:
* replacing Spark-based patent extraction module (handling EPO patent entities and caching) with plain Madis worfklow, currently relying on a predefined sqlite database with patents coming from lens.org
* exporting plain relations with the `ActionManager` sequence file model

IIS primary/processing workflow, along with the integration test expectations, was updated to reflect all the changes mentioned above.
@marekhorst
Closes #1596: Integrate lens.org patents mining
60fc871 
Adding simple <fs> action at the beggining of the primary/processing test in order to purge caches.
This solves test failure due to report expecatations mismatch when the test is rerun without being redeployed.
@marekhorst
Closes #1586: Integrate lens.org patents mining
76da3d2 
Replacing `lens_id` with `lensId` to keep this new field name aligned with the camel case naming convention.
@marekhorst marekhorst requested review from Copilot and mpol July 1, 2026 11:54
@marekhorst marekhorst self-assigned this Jul 1, 2026
Copilot AI reviewed Jul 1, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR migrates IIS patent reference extraction from the legacy EPO/PATSTAT-based import + remote metadata retrieval + entity export flow to a simplified Lens.org-based approach that mines against a pre-built (or test-built) SQLite patent DB and exports document→patent relations only.

Changes:

  • Replaces ImportedPatent/appln_nr inputs with Lens-oriented raw patent rows (PatentReferenceExtractionInput) and outputs DocumentToPatent.lensId.
  • Introduces/uses a simplified SQLite DB builder for integration tests and updates MadIS SQL + helper functions to match the new DB schema.
  • Removes obsolete EPO retrieval/extraction components, workflows, schemas, and associated tests/resources; adds a new ActionManager exporter module for document→patent relations.

Reviewed changes

Copilot reviewed 111 out of 112 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/main/sampletest/oozie_app/workflow.xml Updates patent sample workflow to use raw_patent + sqlite builder + new reporting.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/main/sampletest/oozie_app/import.txt Adds sqlite builder sub-workflow to sample test imports.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/main/sampletest_empty_text_input/oozie_app/workflow.xml Updates empty-text sample workflow for sqlite DB + Lens raw patent input.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/main/sampletest_empty_text_input/oozie_app/import.txt Adds sqlite builder to empty-text sample imports.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/main/sampletest_empty_patent_input/oozie_app/workflow.xml Updates empty-patent sample workflow to match new DB-based mining path.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/main/sampletest_empty_patent_input/oozie_app/import.txt Adds sqlite builder to empty-patent sample imports.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/WO.0042078.A1.xml Removes obsolete EPO XML fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/WO.0042078.A1.no_country_code.xml Removes obsolete EPO XML fixture variant.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/retriever/transformer/output.json Removes obsolete retriever transformer expected output.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/retriever/transformer/input_matched_patent.json Removes obsolete retriever transformer fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/retriever/transformer/input_imported_patent.json Removes obsolete imported-patent transformer fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/retriever/report.json Removes obsolete patent retrieval report fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/retriever/report_update.json Removes obsolete patent retrieval report fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/retriever/report_from_cache.json Removes obsolete cache report fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/retriever/output2.json Removes obsolete retriever output fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/retriever/output.json Removes obsolete retriever output fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/retriever/input2.json Removes obsolete retriever input fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/retriever/input.json Removes obsolete retriever input fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/retriever/cache_text2.json Removes obsolete cache text fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/retriever/cache_text12.json Removes obsolete cache text fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/retriever/cache_text1.json Removes obsolete cache text fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/report_patent.json Adds new patent reference counter report fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/report_patent_metadata_retrieval.json Removes obsolete retrieval report fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/report_patent_metadata_extraction.json Removes obsolete extraction report fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/raw_patent.json Adds Lens-style raw patent rows for test DB building.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/mock_facade_storage/WO.2013189646.A1.xml Removes obsolete facade XML storage fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/input_transformer/patent_transformed.json Removes obsolete input transformer expected output.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/input_transformer/imported_patent.json Removes obsolete imported-patent fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/imported_patent.json Removes obsolete imported-patent fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/extracted_patent_metadata.json Removes obsolete extracted patent metadata fixture.
iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/patent/data/document_to_patent.json Updates expected output to use lensId instead of appln_nr.
iis-wf/iis-wf-referenceextraction/src/test/java/eu/dnetlib/iis/wf/referenceextraction/patent/TestServiceFacadeFactories.java Removes obsolete test facade factories for EPO retrieval.
iis-wf/iis-wf-referenceextraction/src/test/java/eu/dnetlib/iis/wf/referenceextraction/patent/PatentMetadataExtractorJobTest.java Removes obsolete metadata extractor job tests.
iis-wf/iis-wf-referenceextraction/src/test/java/eu/dnetlib/iis/wf/referenceextraction/patent/OpenPatentWebServiceFacadeFactoryTest.java Removes obsolete EPO facade factory tests.
iis-wf/iis-wf-referenceextraction/src/test/java/eu/dnetlib/iis/wf/referenceextraction/patent/input/PatentReferenceExtractionInputTransformerJobTest.java Removes obsolete transformer job test.
iis-wf/iis-wf-referenceextraction/src/test/java/eu/dnetlib/iis/wf/referenceextraction/patent/input/PatentMetadataRetrieverInputTransformerJobTest.java Removes obsolete transformer job test.
iis-wf/iis-wf-referenceextraction/src/main/resources/eu/dnetlib/iis/wf/referenceextraction/patent/sqlite_builder/oozie_app/lib/scripts/buildpatentdb.sql Updates DB build SQL to persist the Lens raw-patent table shape.
iis-wf/iis-wf-referenceextraction/src/main/resources/eu/dnetlib/iis/wf/referenceextraction/patent/main/oozie_app/workflow.xml Simplifies patent extraction workflow to DB-based mining (no remote retrieval).
iis-wf/iis-wf-referenceextraction/src/main/resources/eu/dnetlib/iis/wf/referenceextraction/patent/main/oozie_app/import.txt Removes sqlite_builder import from main workflow imports (no longer invoked here).
iis-wf/iis-wf-referenceextraction/src/main/resources/eu/dnetlib/iis/wf/referenceextraction/patent/main_sqlite/oozie_app/workflow.xml Adds reporting to sqlite-based mining workflow and wires counters.
iis-wf/iis-wf-referenceextraction/src/main/resources/eu/dnetlib/iis/wf/referenceextraction/patent/main_sqlite/oozie_app/lib/scripts/patents.sql Reworks mining SQL to emit Lens IDs and new matching heuristics.
iis-wf/iis-wf-referenceextraction/src/main/java/eu/dnetlib/iis/wf/referenceextraction/patent/PatentWebServiceFacadeException.java Removes obsolete EPO facade exception.
iis-wf/iis-wf-referenceextraction/src/main/java/eu/dnetlib/iis/wf/referenceextraction/patent/PatentMetadataRetrieverJob.java Removes obsolete remote retrieval Spark job.
iis-wf/iis-wf-referenceextraction/src/main/java/eu/dnetlib/iis/wf/referenceextraction/patent/PatentMetadataExtractorJob.java Removes obsolete XML metadata extraction Spark job.
iis-wf/iis-wf-referenceextraction/src/main/java/eu/dnetlib/iis/wf/referenceextraction/patent/PatentDBBuilder.java Adjusts DB builder to start from a base Lens SQLite template DB.
iis-wf/iis-wf-referenceextraction/src/main/java/eu/dnetlib/iis/wf/referenceextraction/patent/parser/PatentMetadataParserException.java Removes obsolete parser exception type.
iis-wf/iis-wf-referenceextraction/src/main/java/eu/dnetlib/iis/wf/referenceextraction/patent/parser/PatentMetadataParser.java Removes obsolete parser interface.
iis-wf/iis-wf-referenceextraction/src/main/java/eu/dnetlib/iis/wf/referenceextraction/patent/parser/OpsPatentMetadataXPathBasedParser.java Removes obsolete OPS XML parser implementation.
iis-wf/iis-wf-referenceextraction/src/main/java/eu/dnetlib/iis/wf/referenceextraction/patent/OpenPatentWebServiceFacadeFactory.java Removes obsolete EPO webservice facade factory.
iis-wf/iis-wf-referenceextraction/src/main/java/eu/dnetlib/iis/wf/referenceextraction/patent/OpenPatentWebServiceFacade.java Removes obsolete EPO webservice facade implementation.
iis-wf/iis-wf-referenceextraction/src/main/java/eu/dnetlib/iis/wf/referenceextraction/patent/input/PatentReferenceExtractionInputTransformerJob.java Removes obsolete input transformer job.
iis-wf/iis-wf-referenceextraction/src/main/java/eu/dnetlib/iis/wf/referenceextraction/patent/input/PatentMetadataRetrieverInputTransformerJob.java Removes obsolete metadata retriever input transformer job.
iis-wf/iis-wf-referenceextraction/src/main/java/eu/dnetlib/iis/wf/referenceextraction/patent/ConnectionDetailsBuilder.java Removes obsolete EPO connection builder.
iis-wf/iis-wf-referenceextraction/src/main/java/eu/dnetlib/iis/wf/referenceextraction/patent/ConnectionDetails.java Removes obsolete EPO connection details model.
iis-wf/iis-wf-primary/src/test/resources/eu/dnetlib/iis/wf/primary/processing/sampledataproducer/output/report/pushgateway.json Updates expected metrics to new patent reference counter.
iis-wf/iis-wf-primary/src/test/resources/eu/dnetlib/iis/wf/primary/processing/sampledataproducer/output/report/document_to_patent.json Adds expected document→patent report output.
iis-wf/iis-wf-primary/src/test/resources/eu/dnetlib/iis/wf/primary/processing/sampledataproducer/output/report/doc_patent_metadata_retrieval.json Removes obsolete patent retrieval report output.
iis-wf/iis-wf-primary/src/test/resources/eu/dnetlib/iis/wf/primary/processing/sampledataproducer/output/report/doc_patent_metadata_extraction.json Removes obsolete patent extraction report output.
iis-wf/iis-wf-primary/src/test/resources/eu/dnetlib/iis/wf/primary/processing/sampledataproducer/output/extracted_patent_metadata.json Removes obsolete exported patent entity fixture.
iis-wf/iis-wf-primary/src/test/resources/eu/dnetlib/iis/wf/primary/processing/sampledataproducer/output/document_to_patent.json Updates expected document→patent output to Lens IDs.
iis-wf/iis-wf-primary/src/test/resources/eu/dnetlib/iis/wf/primary/processing/sampledataproducer/oozie_app/workflow.xml Updates integration sample workflow to build Lens DB and pass input_patent_db.
iis-wf/iis-wf-primary/src/test/resources/eu/dnetlib/iis/wf/primary/processing/sampledataproducer/oozie_app/import.txt Imports sqlite builder into primary processing sample test.
iis-wf/iis-wf-primary/src/test/resources/eu/dnetlib/iis/wf/primary/processing/sampledataproducer/input/text/id-50.txt Updates sample text to contain Lens-matching patent reference.
iis-wf/iis-wf-primary/src/test/resources/eu/dnetlib/iis/wf/primary/processing/sampledataproducer/input/meta/raw_patent.json Adds raw patent fixture for building test Lens DB.
iis-wf/iis-wf-primary/src/test/resources/eu/dnetlib/iis/wf/primary/processing/sampledataproducer/input/meta/patent.json Removes obsolete ImportedPatent fixture.
iis-wf/iis-wf-primary/src/test/java/eu/dnetlib/iis/wf/primary/processing/FileContentReturningPatentServiceFacadeFactory.java Removes obsolete patent service facade used for EPO-based tests.
iis-wf/iis-wf-primary/src/main/resources/eu/dnetlib/iis/wf/primary/processing/oozie_app/workflow.xml Switches primary processing input from input_patent to input_patent_db and drops patent entity output.
iis-wf/iis-wf-primary/src/main/resources/eu/dnetlib/iis/wf/primary/main/oozie_app/workflow.xml Switches primary workflow patent import from TSV to prebuilt Lens DB and removes patent entity export parameters.
iis-wf/iis-wf-import/src/test/resources/eu/dnetlib/iis/wf/importer/patent/sampletest/oozie_app/workflow.xml Removes obsolete patent TSV importer workflow test.
iis-wf/iis-wf-import/src/test/resources/eu/dnetlib/iis/wf/importer/patent/sampletest/oozie_app/input/patents_epo.tsv Removes obsolete EPO TSV fixture.
iis-wf/iis-wf-import/src/test/resources/eu/dnetlib/iis/wf/importer/patent/sampletest/oozie_app/import.txt Removes obsolete importer test import manifest.
iis-wf/iis-wf-import/src/test/resources/eu/dnetlib/iis/wf/importer/patent/data/output/report.json Removes obsolete importer report fixture.
iis-wf/iis-wf-import/src/test/resources/eu/dnetlib/iis/wf/importer/patent/data/output/patents_epo_output.json Removes obsolete ImportedPatent output fixture.
iis-wf/iis-wf-import/src/test/java/eu/dnetlib/iis/wf/importer/patent/WorkflowTest.java Removes obsolete importer workflow test class.
iis-wf/iis-wf-import/src/test/java/eu/dnetlib/iis/wf/importer/patent/PatentReaderJobTest.java Removes obsolete patent reader Spark job test.
iis-wf/iis-wf-import/src/main/java/eu/dnetlib/iis/wf/importer/patent/PatentReaderJob.java Removes obsolete patent TSV reader job.
iis-wf/iis-wf-export-actionmanager/src/test/resources/eu/dnetlib/iis/wf/export/actionmanager/sequencefile/sampledataproducer/output/document_to_patent_1.properties Updates expected action payload target to Lens-based ID.
iis-wf/iis-wf-export-actionmanager/src/test/resources/eu/dnetlib/iis/wf/export/actionmanager/sequencefile/sampledataproducer/oozie_app/workflow.xml Extends sequencefile exporter sample to include document→patent input/validation.
iis-wf/iis-wf-export-actionmanager/src/test/resources/eu/dnetlib/iis/wf/export/actionmanager/sequencefile/sampledataproducer/input/document_to_patent.json Adds document→patent input fixture for sequencefile exporter tests.
iis-wf/iis-wf-export-actionmanager/src/test/resources/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/default/output/report.json Removes obsolete patent entity export report expectations.
iis-wf/iis-wf-export-actionmanager/src/test/resources/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/default/output/patent2.expectations Removes obsolete patent entity expectations.
iis-wf/iis-wf-export-actionmanager/src/test/resources/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/default/output/patent2_to_document2.properties Removes obsolete entity-export relation expectations.
iis-wf/iis-wf-export-actionmanager/src/test/resources/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/default/output/patent1.expectations Removes obsolete patent entity expectations.
iis-wf/iis-wf-export-actionmanager/src/test/resources/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/default/output/patent1_to_document2.properties Removes obsolete entity-export relation expectations.
iis-wf/iis-wf-export-actionmanager/src/test/resources/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/default/output/patent1_to_document1.properties Removes obsolete entity-export relation expectations.
iis-wf/iis-wf-export-actionmanager/src/test/resources/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/default/output/document2_to_patent2.properties Removes obsolete entity-export relation expectations.
iis-wf/iis-wf-export-actionmanager/src/test/resources/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/default/output/document2_to_patent1.properties Removes obsolete entity-export relation expectations.
iis-wf/iis-wf-export-actionmanager/src/test/resources/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/default/oozie_app/workflow.xml Removes obsolete patent entity exporter workflow test definition.
iis-wf/iis-wf-export-actionmanager/src/test/resources/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/default/oozie_app/import.txt Removes obsolete entity exporter test import manifest.
iis-wf/iis-wf-export-actionmanager/src/test/resources/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/default/input/patent.json Removes obsolete patent entity input fixture.
iis-wf/iis-wf-export-actionmanager/src/test/resources/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/default/input/nullcheck/patent.json Removes obsolete nullcheck fixture for entity export.
iis-wf/iis-wf-export-actionmanager/src/test/resources/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/default/input/nullcheck/document_to_patent.json Removes obsolete nullcheck fixture for entity export.
iis-wf/iis-wf-export-actionmanager/src/test/resources/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/default/input/document_to_patent.json Removes obsolete document→patent fixture for entity export.
iis-wf/iis-wf-export-actionmanager/src/test/java/eu/dnetlib/iis/wf/export/actionmanager/module/DocumentToPatentActionBuilderModuleFactoryTest.java Adds unit tests for new document→patent action builder.
iis-wf/iis-wf-export-actionmanager/src/test/java/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/PatentExporterWorkflowTest.java Removes obsolete patent entity exporter workflow test class.
iis-wf/iis-wf-export-actionmanager/src/test/java/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/PatentExporterJobTest.java Removes obsolete patent entity exporter Spark job test.
iis-wf/iis-wf-export-actionmanager/src/test/java/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/PatentExportCounterReporterTest.java Removes obsolete patent entity export counter reporter test.
iis-wf/iis-wf-export-actionmanager/src/main/resources/eu/dnetlib/iis/wf/export/actionmanager/sequencefile/oozie_app/workflow.xml Adds document→patent export path, action set id, and trust threshold wiring.
iis-wf/iis-wf-export-actionmanager/src/main/java/eu/dnetlib/iis/wf/export/actionmanager/module/DocumentToPatentActionBuilderModuleFactory.java Introduces a new relation-only exporter module for Lens-based patents.
iis-wf/iis-wf-export-actionmanager/src/main/java/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/PatentExportMetadata.java Removes obsolete patent entity export metadata model.
iis-wf/iis-wf-export-actionmanager/src/main/java/eu/dnetlib/iis/wf/export/actionmanager/entity/patent/PatentExportCounterReporter.java Removes obsolete patent entity export counter reporter.
iis-schemas/src/main/avro/eu/dnetlib/iis/referenceextraction/patent/PatentReferenceExtractionInput.avdl Reworks the input schema to match Lens DB row model (c1,c2,normal,c4,c5).
iis-schemas/src/main/avro/eu/dnetlib/iis/referenceextraction/patent/Patent.avdl Removes obsolete patent entity schema.
iis-schemas/src/main/avro/eu/dnetlib/iis/referenceextraction/patent/ImportedPatent.avdl Removes obsolete imported patent schema.
iis-schemas/src/main/avro/eu/dnetlib/iis/referenceextraction/patent/DocumentToPatent.avdl Changes relation schema to emit lensId instead of appln_nr.
iis-common/src/main/java/eu/dnetlib/iis/common/InfoSpaceConstants.java Adds Lens entity prefix constant for consistent ID generation.
iis-3rdparty-madis/src/main/resources/eu/dnetlib/iis/3rdparty/scripts/madis/functions/row/text.py Adds helper function used by updated patent mining SQL.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .../src/main/resources/eu/dnetlib/iis/wf/referenceextraction/patent/main/oozie_app/workflow.xml
Comment on lines 9 to 13
<property>
<name>input_patent</name>
<description>input patent with eu.dnetlib.iis.referenceextraction.patent.schemas.ImportedPatent records</description>
<name>input_patent_db</name>
<value>com.cloudera.spark.lineage.NavigatorAppListener</value>
<description>pre-built sqlite database with patents</description>
</property>
Comment thread ...rdparty-madis/src/main/resources/eu/dnetlib/iis/3rdparty/scripts/madis/functions/row/text.py
Comment on lines +217 to +220
"""
Returns the substring of `s` starting from the last occurrence of `a` till the end.
If `a` is not found in `s`, returns an empty string.
"""
Comment thread ...u/dnetlib/iis/wf/export/actionmanager/module/DocumentToPatentActionBuilderModuleFactory.java
Comment on lines +67 to +74
/**
* Creates similarity related actions.
*/
private AtomicAction<Relation> createAction(String source, String target, float confidenceLevel,
String relClass) throws TrustLevelThresholdExceededException {
return createAtomicActionWithRelation(source, target, OafConstants.REL_TYPE_RESULT_RESULT,
OafConstants.SUBREL_TYPE_RELATIONSHIP, relClass, confidenceLevel);
}
@marekhorst

marekhorst commented Jul 2, 2026

Copy link
Copy Markdown
Member Author

Note to myself: we need to postpone the integration with the IIS master branch until having a green light from Claudio stating the remaining part of the provisioning pipeline is compliant with this change (both patents inclusion prior to IIS run and patents filtering after the mining is concluded are already in place).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
@mpol mpol Awaiting requested review from mpol

Assignees

@marekhorst marekhorst

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@marekhorst @johnfouf