Consider adopting OpenSSF Scorecard

[mpkorstanje]

From https://securityscorecards.dev/#the-problem:

The problem

By some estimates* 84% of all codebases have at least one vulnerability, with an average of 158 per codebase. The majority have been in the code for more than 2 years and have documented solutions available.

Even in large tech companies, the tedious process of reviewing code for vulnerabilities falls down the priority list, and there is little insight into known vulnerabilities and solutions that companies can draw on.

That’s where Security Scorecards [i.e., OpenSSF Scorecard] is helping. Its focus is to understand the security posture of a project and assess the risks that dependencies introduce.

*Open Source Security and Risk Analysis Report (Synopsys, 2021)

What is OpenSSF Scorecard?

Scorecard assesses open source projects for security risks through a series of automated checks.

It was created by OSS developers to help improve the health of critical projects that the community depends on.

You can use it to proactively assess and make informed decisions about accepting security risks within your codebase. You can also use the tool to evaluate other projects and dependencies, and work with maintainers to improve codebases you might want to integrate.

Scorecard helps you enforce best practices that can guard against:

• malicious maintainers
• build system compromises
• source code compromises
• malicious packages

[mpkorstanje]

Looking at the Scorecard, it is ultimately a rather expensive advertisement banner. So I would recommend against adopting it.

For the actual checks (https://securityscorecards.dev/#the-checks), we pretty much either pass with nearly full marks or fail the checks almost completely. For example see OpenSSF Scorecard report for Cucumber-JVM.

Given that this is a huge project with 3 core maintainers and only a handful of contributors changing our branch model, protections or code-review practices is unlikely. Not stripping these down to essentials would result in so much friction that this project would be impossible.

Furthermore, a large factor in failing the code review comes from automated Renovate updates. Due to the lack of maintainers, we either automate dependency updates (with a cool down period) or don't do them at all. Fuzzing might be worth while to adopt for Gherkin, Tag Expressions and Cucumber Expressions but not on every module.

For things that are likely to change, the passing checks on the Scorecard are already checked by GitHub, Dependabot, Renovate, Zizmor and Codeql. And for the failing checks I don't think we will make improvements. As such the automated checks from the Scorecard don't add any information for maintainers.

What remains is a nice advertising banner that lets other people known we're compliant with best practices to some degree. This does however come at the cost of adding a job to an already busy CI. I don't think that is worth the trade-off.

Nevertheless, the Scorecard project already scans some projects and some of Cucumbers repositories are included in that set. We could consider displaying the Scorecard badge on a select number of repositories without adding the CI job for it.

[mpkorstanje]

Repositories that have a scorecard provided by the OSSF:

• cucumber/cucumber-expressions
• cucumber/cucumber-parent
• cucumber/gherkin
• cucumber/cucumber-jvm-scala Add OpenSSF Scorecard badge cucumber-jvm-scala#432
• cucumber/aruba Add OpenSSF Scorecard badge aruba#1023
• cucumber/cucumber-js Add OpenSSF Scorecard badge cucumber-js#2814
• cucumber/screenplay.js
• cucumber/common
• cucumber/cucumber-ruby Add OpenSSF Scorecard cucumber-ruby#1874
• cucumber/messages
• cucumber/tag-expressions
• cucumber/godog Add OpenSSF Scorecard badge godog#746
• cucumber/cucumber-ruby-core
• cucumber/cucumber-rails
• cucumber/cucumber-jvm

[mpkorstanje]

With all the flag ships done. I think this is sufficient for now.