SBOM generation

[rhubert]

One of the questions after the ESE Talk was "Can it generate a SBOM?". And since a SBOM is one of the requirements from CRA I think we should add some functionality for this.

At the moment I've a internal plugin for this, using the cyclone-dx python module, but maybe this can be a bob command as well? So I'm not sure if this is the correct place for the discussion at all...

However for the SBOM we use PKG_VERSION, PKG_LICENSE from the metaEnv. Other Required items for the SBOM, (e.g. as listed in the BSI-TR-03183-2_v2_1_0 Section 5 are just filled using the recipe name or other available information from the recipe.

But with the available information in the recipes not every required field can be filled. E.g. the BSI requires a Hash value of the deployable component and Executable|Archive|Structured property. ATM we need to build the packages first to get these infos using some find in the dist folders.

To avoid this I'd like to add PKG_COMPONENT_[EXECUTABLE|ARCHIVE|STRUCTURED] metaEnv variables with valid values as describes in https://github.com/BSI-Bund/tr-03183-cyclonedx-property-taxonomy. Maybe we should set them to executable by default...? Than we'd use the audit-file to get the result hash of each package and do not need to have the results of all dependencies when building the SBOM.

The Additional data fields lists Other unique identifiers, such as CPE and pURL, which are maybe(?) not a strong requirement? (BSI: Other identifiers that can be used, EO and NTIA: These other identifiers[...] should be used if they exist). To make it consistent I'd like to define some new metaEnvironment keywords.

A CPE-string has the following format: cpe:/<part>:<vendor>:<product>:<version>:<update>:<edition>:<language> . Therefor I'd like to define some new metaEnvironment variables like PKG_CPE_VENDOR, PKG_CPE_PRODUCT and so on.

We're not using pURLs ATM and I've no idea how to get the correct pURL identifier for the packages...so from my side we can skip them for now....

[jkloetzke]

Thanks for bringing this up.

How about gathering all this information at build time? Very much like we did it with the licenses? My gut feeling is that this post-processing is too disconnected. In principle, we could put this into the .bob directory and forward the information through the dependency tree at build time.

What makes it more complicated is how tooling that processes the SBOM records is available at build time. For the first components it sounds like a chicken-and-egg problem. But maybe that is not an issue because this affects only the earliest build tools.

I'll have to read into the BSI guideline to get a better understanding...

[rhubert]

* "Filename of the component - The actual filename of the component (i.e. not its file system path); see also section 3.2.1"
* "Hash value of the deployable component - Cryptographically secure checksum (hash value) of the deployed/deployable component (i.e. as a file on a mass storage device) as SHA-512; see also section 3.2."

We could add this to the audit trail using a custom hook :)

* we probably need the license texts of non-standard licenses there as well?

According to 6.1.-3 we just have to use the license-ref, not to embed the custom licenses. I'd argument, that we can provide them separate. But there is another item in 6.1-2:

the licence database “Scancode
LicenseDB AboutCode”16 MUST be consulted next. Identifiers from this database use the prefix
LicenseRef-scancode-[...] in their SPDX licence identifier to indicate their origin

Could this property be expressed by a formalized metaEnvironment property? Before thinking about a plugin interface, I would like to understand what is missing. Maybe it makes sense to include this in Bob directly.

Sure, but it might be easier to generate this property than to manually add it to ever recipe?

[jkloetzke]

I think there are two questions here:

1. How we can funnel additional information, that is created at build-time, into the audit trail? This involves things like file names and hash values. Maybe even custom license texts in the end.
2. Adding additional configuration-time properties. So far, metaEnvironment seems sufficient to me. I mean, this Executable/Archive/Structured nonsense does not seem to be reliably computable to me. Therefore somebody has to set this property manually. And if it turns out to be computable, it will fall into the case 1 above.

Maybe we add something like the following?

packageMetaEnvironment:
    BSI_HASHES: ".bob/bsi-hashes"

Bob will read the files and put them into the audit trail. Of course, there could be conditional files, required files, encoding settings, ... OTOH, a plugin for this thing seems problematic. In case of Jenkins build, Bob will have to get the plugin deployed into the Jenkins jobs. I would rather avoid this if possible.

[rhubert]

Thanks for reminding me to Jenkins - I always forget about it since we're not using it.

I like the idea of the packageMetaEnvironment. Together with the existing and maybe some further metaEnvironment it looks like we can get all the required information into the audit trail to build a sbom from it later. Do you have time to implement the packageMetaEnvironment feature? Ideally before 1.2.. Otherwise I can give it a try, but I'm unsure if this would take less time for you... ;)

[jkloetzke]

I should find some time in the coming days. Actually, this feature doesn't look too hard.

[jkloetzke]

Ok, here we go. I've uploaded BobBuildTool/bob#683...

[jkloetzke]

Adding a second thread here because I got the feeling we might need two additional features:

• A reliable way to execute things after the checkout-/build-/packageScript has run. This is probably important if we need to calculate hash sums. My current idea is to add something like {checkout,build,package}Finalize. The same inheritance logic of classes apply for these scripts but all Finalize scripts are executed after the regular scripts. Much like the Setup scripts run entirely before the regular scripts.
• Add something like inheritPrepend and inheritAppend to config.yaml to globally inherit classes by default. That way we could make sure the SBOM stuff always runs and nobody can forget it.

What do you think?

[rhubert]

* A reliable way to execute things _after_ the checkout-/build-/packageScript has run. This is probably important if we need to calculate hash sums. My current idea is to add something like `{checkout,build,package}Finalize`. The same inheritance logic of classes apply for these scripts but all `Finalize` scripts are executed after the regular scripts. Much like the `Setup` scripts run entirely before the regular scripts.

Interesting idea. For the setup it usually doesn't matter in which order the are executed since they only define functions or export variables, while the order of finalizeScripts might be important. So we might need some additional priority Setting here. 🤔

* Add something like `inheritPrepend` and `inheritAppend` to config.yaml to globally inherit classes by default. That way we could make sure the SBOM stuff always runs and nobody can forget it.

Hmmm...same here... How do we handle multiple inheritPr/Appends from different layers?

[jkloetzke]

The idea is to rely on the order of layers and inheritance as it is already existing.

For class inheritance, the rule is that they are inherited depth-first, left-to-right. That is, the (setup) script of the deepest class is run first and if multiple classes are inherited on a level, they run left-to-right. For the Finalize scripts, the order should be reversed. In contrast to the regular recipe script, which is run as the last script, the recipe Finalize script is run first and the deepest inherited class Finalize is run last.

The same pattern can be applied to inheritPrepend and inheritAppend. The root project has the highest precedence. The layers in config.yaml are named from highest to lowest precedence. This will also the the order in which inheritPrepend and inheritAppend are applied. As classes are only inherited once, forcing their inheritance by inheritPrepend gives them a either a very high priority resp. makes sure, the Finalize script runs last.

[jkloetzke]

I've made a prototype of the Finalize scripts in BobBuildTool/bob#687. Let me know what you think. Actually, I would only want to include the feature if we really need this. If it takes longer to find that out, I wouldn't mind making a 1.3 release shortly after 1.2...