Add evidence-checker extension by Franziska-Schallhorn · Pull Request #634 · open-component-model/odg-core

Franziska-Schallhorn · 2025-12-16T10:31:26Z

What this PR does / why we need it:
Adds a new ODG extension which creates findings for OCM resources without test-evidences.
Test-evidences are provided as additional OCM resources, whereas a set of defined OCM labels is used to correlate test-evidences to payload artefacts. Additionally, OCM resources can be flagged as "no tests required" which is useful for scenarios where no actual source code is built.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Release note:

A new "evidence-checker" ODG extension was added, automatically creating findings for OCM resources without test evidences.

Signed-off-by: Philipp Heil (zkdev) <philipp.heil@sap.com> Co-authored-by: Franziska Schallhorn <franziska.schallhorn@sap.com>

TuanAnh17N · 2026-02-24T07:56:16Z

+3. Test Scope Label
+~~~~~~~~~~~~~~~~~~~
+
+The test scope label is set in the component descriptor of a test artefact (aka an artefact whose purpose label has the value 'test) and it defines for which other artefacts it represents the test evidence.


missing closing quote in 'test)

TuanAnh17N · 2026-02-24T07:57:17Z

+Functionality
+-------------
+
+1. The extension scans each artefact of a component and first validates whether the currently scanned artefact is a test evidence itself or not. This is achieved with the use of the gardener.cloud/purposes label. An artefact is considered a test evidence provided the label's value is 'test'.


label value instead of label's value, no? 🤔

TuanAnh17N · 2026-02-24T07:57:28Z

+
+1. The extension scans each artefact of a component and first validates whether the currently scanned artefact is a test evidence itself or not. This is achieved with the use of the gardener.cloud/purposes label. An artefact is considered a test evidence provided the label's value is 'test'.
+
+2. Provided the gardener.cloud/purposes label's has NOT been given the value 'test' the extension further checks, whether the artefact requires a test or not. This is achieved with the gardener.cloud/test-policy label which is either 'true' or 'false.


TuanAnh17N · 2026-02-24T07:58:03Z

+
+1. The extension scans each artefact of a component and first validates whether the currently scanned artefact is a test evidence itself or not. This is achieved with the use of the gardener.cloud/purposes label. An artefact is considered a test evidence provided the label's value is 'test'.
+
+2. Provided the gardener.cloud/purposes label's has NOT been given the value 'test' the extension further checks, whether the artefact requires a test or not. This is achieved with the gardener.cloud/test-policy label which is either 'true' or 'false.


missing closing quote in 'false

TuanAnh17N · 2026-02-24T07:58:32Z

+
+3. In case an artefact has been identified as a test artefact (as described in step 1) the extension further checks, for which artefacts within the component the test-evidence is valid for. This can be defined with the help of the gardener.cloud/test-scope label.
+
+4. In case one of the artefacts identified in step 2 is not covered by the test artefacts(aka evidences) a finding will be created and test-evidences will have to be provided retrospectively.


missing space before parentheses in artefacts(aka evidences)

TuanAnh17N · 2026-02-24T08:14:44Z

+        self,
+        artefact_kind: odg.model.ArtefactKind,
+    ) -> bool:
+        if artefact_kind is not odg.model.ArtefactKind.RESOURCE:


is_supported() can be simplified to a single return statement

return artefact_kind is odg.model.ArtefactKind.RESOURCE

TuanAnh17N · 2026-02-24T08:24:33Z

+
+    return missing_test_evidence_finding(
+        component_resource_id=artefact,
+        resource_node=artefact_node,


missing_test_evidence_finding(...) is called with unexpected keyword arguments (component_resource_id and resource_node) but the function signature expects artefact and artefact_node

TuanAnh17N · 2026-02-24T08:30:05Z

+        test_scope_label: odg.labels.TestScopeLabel = odg.labels.deserialise_label(
+            label=test_scope_label,
+        )
+        artefact_names_with_test_evidence.append(test_scope_label.value)


according to the documentation above, gardener.cloud/test-scope contains a list of artefact names — should this use extend(...) instead of append(...) ?

I agree, we have an inconsistency here, the definition in odg.labels only allows a single artefact name (as it is also honoured here) but the documentation states differently.

TuanAnh17N · 2026-02-24T08:35:20Z

+            last_update=now,
+        ),
+        data=odg.model.TestEvidenceMissingFinding(
+            test_status=odg.model.TestStatus.NO_TEST_EVIDENCE,


odg.model.TestStatus.NO_TEST_EVIDENCE is already passed when you call it, you can use sub_type here instead of hardcoding the same value again

TuanAnh17N · 2026-02-24T08:40:35Z

+        resource
+        for resource in component.resources
+        if (
+            (label := resource.find_label(name=odg.labels.PurposeLabel.name))


can we also deserialize PurposeLabel via odg.labels.deserialise_label(...)?

8R0WNI3 · 2026-02-24T12:42:53Z

@@ -0,0 +1,123 @@
+{{- $podName := "test-evidence" }}


As discussed out-of-bands, please rename this consistently to evidence-checker (please don't forget file names as well 🥲).

8R0WNI3 · 2026-02-24T12:44:50Z

+      terminationGracePeriodSeconds: 300 # 5 min
+      containers:
+        - name: {{ $podName }}
+          image: {{ include "image" .Values.image }}


I'm not sure whether this is on purpose for the first iteration, but the Helm chart image mapping configuration is still missing in the .github/workflows/build.yaml workflow.

8R0WNI3 · 2026-02-24T12:52:39Z

Some comments to this image:

There should be no question mark after "Test Artefact" as it seems to be a state rather than a condition.

The "yes" and "no" captions are missing for the "Do they match?" condition

Instead of using a boolean value for the test policy, I'd suggest to use an enum instead because

it is more meaningful without an additional description (e.g. using the values test-required and no-test-required or the like)

it allows future extensions

(this does not only affect this image of course but also the implementation and documentation, but I won't comment there again)

8R0WNI3 · 2026-02-24T12:56:32Z

+  issues:
+    enable_assignees: False
+  categorisations:
+    - id: NONE


Since it is difficult to change the id afterwards, I'd propose to use a more meaningful id right from the beginning (e.g. no-tests-required and no-test-evidence or similar).

8R0WNI3 · 2026-02-24T12:59:06Z

            Datasource.SAST: (
                Datatype.SAST_FINDING,
            ),
+            Datasource.TEST_EVIDENCE: (Datatype.TEST_EVIDENCE_FINDING)


This does not return a tuple but instead only a single entry, instead use:

Datasource.TEST_EVIDENCE: ( Datatype.TEST_EVIDENCE_FINDING, # note the trailing comma )

8R0WNI3 · 2026-02-24T13:06:34Z

+    artefact_node: ocm.iter.ArtefactNode,
+    extensions_cfg: odg.extensions_cfg.TestEvidenceConfig,
+) -> bool:
+    artefact = artefact_node.artefact


nit: Rather pass in the artefact directly to simplify the function's interface (and don't require an unnecessary complex object although only the artefact is used).

8R0WNI3 · 2026-02-24T13:09:16Z

+def missing_test_evidence_finding(
+    artefact: odg.model.ComponentArtefactId,
+    artefact_node: ocm.iter.ArtefactNode,
+    sub_type: odg.model.TestStatus,


nit: Rather name this test_status rather than sub_type (I think this comes from the SAST extension where we have indeed a sub_type as well as a sast_status).

8R0WNI3 · 2026-02-24T13:09:49Z

+    findings_cfg: odg.findings.Finding,
+    extensions_cfg: odg.extensions_cfg.TestEvidenceConfig,
+) -> odg.model.ArtefactMetadata | None:
+    categorisation: odg.findings.FindingCategorisation = odg.findings.categorise_finding(


The categorisation might as well be None in which case we should not create a finding.

8R0WNI3 · 2026-02-24T13:10:58Z

+            test_status=odg.model.TestStatus.NO_TEST_EVIDENCE,
+            severity=categorisation.id
+        ),
+        discovery_date=now.date(),


We should also store the allowed_processing_time here in the finding (read from the categorisation). Rationale: We always want to apply the processing time which was valid at the time of discovering a finding, as opposed to always using the currently configured one (which might change).

8R0WNI3 · 2026-02-24T13:12:51Z

+    ):
+        return None
+
+    now = datetime.datetime.now()


nit: Please add timezone information (ditto further down below):

now = datetime.datetime.now(datetime.timezone.utc)

Franziska-Schallhorn requested a review from a team as a code owner December 16, 2025 10:31

Franziska-Schallhorn mentioned this pull request Dec 16, 2025

Missing test results extension #628

Closed

zkdev force-pushed the missing-test-results-extension branch from 40fb196 to ec1e0c4 Compare December 19, 2025 14:32

zkdev force-pushed the missing-test-results-extension branch from c507514 to ac5278a Compare January 28, 2026 15:55

zkdev changed the title ~~Missing test results extension - New PR~~ Test evidence extension Jan 28, 2026

zkdev force-pushed the missing-test-results-extension branch from ac5278a to c67037d Compare January 28, 2026 15:55

zkdev force-pushed the master branch from e41ee6c to d66ea84 Compare February 10, 2026 07:44

zkdev changed the title ~~Test evidence extension~~ Add evidence-checker extension Feb 19, 2026

zkdev and others added 2 commits February 19, 2026 16:59

feat(odg): Add test evidence extension

fd646d0

Signed-off-by: Philipp Heil (zkdev) <philipp.heil@sap.com> Co-authored-by: Franziska Schallhorn <franziska.schallhorn@sap.com>

docs: Add documentation on evidence-checker extension

b79a5b3

Signed-off-by: Philipp Heil (zkdev) <philipp.heil@sap.com> Co-authored-by: Franziska Schallhorn <franziska.schallhorn@sap.com>

zkdev force-pushed the missing-test-results-extension branch from 1e8057f to b79a5b3 Compare February 19, 2026 16:00

TuanAnh17N reviewed Feb 24, 2026

View reviewed changes

8R0WNI3 requested changes Feb 24, 2026

View reviewed changes

zkdev linked an issue Mar 27, 2026 that may be closed by this pull request

Evidence checking extension open-component-model/open-delivery-gear#14

Open


		1. The extension scans each artefact of a component and first validates whether the currently scanned artefact is a test evidence itself or not. This is achieved with the use of the gardener.cloud/purposes label. An artefact is considered a test evidence provided the label's value is 'test'.

		2. Provided the gardener.cloud/purposes label's has NOT been given the value 'test' the extension further checks, whether the artefact requires a test or not. This is achieved with the gardener.cloud/test-policy label which is either 'true' or 'false.


		3. In case an artefact has been identified as a test artefact (as described in step 1) the extension further checks, for which artefacts within the component the test-evidence is valid for. This can be defined with the help of the gardener.cloud/test-scope label.

		4. In case one of the artefacts identified in step 2 is not covered by the test artefacts(aka evidences) a finding will be created and test-evidences will have to be provided retrospectively.

Uh oh!

Conversation

Franziska-Schallhorn commented Dec 16, 2025 • edited by zkdev Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TuanAnh17N Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TuanAnh17N Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Franziska-Schallhorn commented Dec 16, 2025 •

edited by zkdev

Loading

TuanAnh17N Feb 24, 2026 •

edited

Loading

TuanAnh17N Feb 24, 2026 •

edited

Loading