feat(scrapeconfig): add scrape_failure_log_file support

[slashpai]

Fixes #8425

Description

Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request.

Closes: #ISSUE-NUMBER

If you're contributing for the first-time, check our contribution guidelines.

Type of change

What type of changes does your code introduce to the Prometheus operator? Put an x in the box that apply.

• CHANGE (fix or feature that would cause existing functionality to not work as expected)
• FEATURE (non-breaking change which adds functionality)
• BUGFIX (non-breaking change which fixes an issue)
• ENHANCEMENT (non-breaking change which improves existing functionality)
• NONE (if none of the other choices apply. Example, tooling, build system, CI, docs, etc.)

Verification

Please check the Prometheus-Operator testing guidelines for recommendations about automated tests.

Changelog entry

Please put a one-line changelog entry below. This will be copied to the changelog file during the release process.

Add scrape_failure_log_file support in ScrapeConfig CRD

[heliapb]

lgtm

[simonpasquier]

What are the potential concerns for environments where Prometheus is provided as-a-service (e.g. a platform team manages the workload resources and application teams manage the configuration resources)?

Is there a risk for app teams to tamper the filesystem? Should there be an option for the workload owner to disallow scrape failure logs? Should the API rather be an on/off toggle and the operator generates a predictable file path from the resource namespace/name?

[slashpai]

What are the potential concerns for environments where Prometheus is provided as-a-service (e.g. a platform team manages the workload resources and application teams manage the configuration resources)?

Is there a risk for app teams to tamper the filesystem? Should there be an option for the workload owner to disallow scrape failure logs? Should the API rather be an on/off toggle and the operator generates a predictable file path from the resource namespace/name?

You are right. The global ScrapeFailureLogFile on PrometheusSpec doesn't have this problem, but the per-job version does. I liked the toggle based approach where operator can generate path. May be we could add disableScrapeFailureLogFile to let workload owners opt out of the feature entirely.