feat: Replace credentials-secret with db/broker connection(s)

[adwk67]

Description

Fixes #64
Decision: https://github.com/stackabletech/decisions/issues/72

Notes for reviewers

• This is large PR although most of the files that have been changed are due to tests and docs: the main changes are in database.rs, queue.rs and connection.rs.
• We don't want the operator to have to read the secrets, and so the connection code (which will eventually be moved to the framework) returns a connection string containing embedded variables.
• These can be resolved in one of the following ways:
  • by shell variable substitution (if the variables are presented as normal env-vars e.g. $FOO) by processes started from the shell
  • by config-utils templating: this assumes that the variable is presented in the form of ${env:...} and has been written to a config file
  • N.B. templating of env-vars themselves was considered but rejected as (re-)writing env-vars from Rust using the env library is not safe in a multithreaded environment
• The operator is responsible for passing env-var names in the form that matches one of the above.
• Some connections contain a driver-dependent prefix, depending on the context. The operator makes the decision which type of connection string to retrieve (e.g. postgresql/sqlalchemyuri or postgresql/celery).
• Queues/brokers could be considered as being a subset of backends but have been kept separate as they were not specifically covered by ADR 29.

Jenkins 🟢 https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/airflow-operator-it-custom/75/
Openshift 🟢 https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/airflow-operator-it-custom/76/

Important

The logging test has been deactivated temporarily while testing this PR. Revert the setting before final approval/merging.

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

• Changes are OpenShift compatible
• CRD changes approved
• CRD documentation for all fields, following the style guide.
• Helm chart can be installed and deployed operator works
• Integration tests passed (for non trivial changes)
• Changes need to be "offline" compatible
• Links to generated (nightly) docs added
• Release note snippet added

Reviewer

• Code contains useful comments
• Code contains useful logging statements
• (Integration-)Test cases added
• Documentation added or updated. Follows the style guide.
• Changelog updated
• Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

• Feature Tracker has been updated
• Proper release label has been added
• Links to generated (nightly) docs added
• Release note snippet added
• Add type/deprecation label & add to the deprecation schedule
• Add type/experimental label & add to the experimental features tracker

[maltesander]

First batch of comments.

[maltesander]

marker!

[maltesander]

I feel URI is very generic. I think connectionString is a better description?

[adwk67]

No strong feelings - I just went with what we had in the decision.

[maltesander]

Same, @sbernauer any opinions?

[maltesander]

Some more doc reviews. Tests look good.

[adwk67]

Re-ran tests after latest changes 🟢 https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/airflow-operator-it-custom/77/

[adwk67]

We (me, @sbernauer and @maltesander) discussed this today and decided to move it to tracking. The PR is "finished" in the sense that the implementation is complete, but there are improvements we can make to the code (e.g. using traits) to make it more flexible for other products that will use this.

• The ideal way to do this is to look at implementing the same changes for e.g. the hive-operator in parallel to make sure we have everything covered: Implement generic database connection hive-operator#677
• We also want to check whether we can use airflow.cfg as an alternative to using env-vars for everything as this would make templating easier (i.e. check whether airflow.cfg defaults all need to explicitly (re-)written or whether we only need to write deltas).

[adwk67]

Update to this:

We also want to check whether we can use airflow.cfg as an alternative to using env-vars for everything as this would make templating easier (i.e. check whether airflow.cfg defaults all need to explicitly (re-)written or whether we only need to write deltas).

Yes, we only need to write out explicit entries to airflow.cfg as the fallback are the defaults in code.
This means we can use the .cfg file to store database connections (will require changes to the oeprator code). See https://github.com/apache/airflow/blob/8064eb91d02deaa8e5d4cb8c9cf21e537fa78acf/shared/configuration/src/airflow_shared/configuration/parser.py#L210-L214