RabbitMQ transient_nonexcl_queues deprecation triggered by Celery pidbox — bump kombu to >= 5.7.0

[or-benjamin]

Under which category would you file this issue?

Airflow Core

Apache Airflow version

3.3.0

What happened and how to reproduce it?

With CeleryExecutor and a RabbitMQ 4.x broker, RabbitMQ reports usage of the
deprecated feature transient_nonexcl_queues:

Deprecated features are being used ... transient_nonexcl_queues

The offending queue is Celery's pidbox (remote-control mailbox), which kombu
declares as non-durable + non-exclusive — exactly the queue shape RabbitMQ 4.x
deprecates and plans to remove in a future major version.

Task queues themselves are fine (they can be quorum), but a quorum queue must be
durable, so the pidbox cannot be converted via configuration — it's a kombu-side
declaration.

What you think should happen instead?

The bundled/pinned kombu version should include the pidbox durability fix so the
deprecation is not triggered out of the box.

The fix landed in kombu 5.7.0 (kombu#2237 → PR celery/kombu#2531: pidbox declared
durable). However, Airflow's constraints still pin kombu 5.6.2:

Constraints	kombu	celery	amqp
constraints-3.2.2 (py3.12)	5.6.2	5.6.3	5.3.1
constraints-3.3.0 (py3.12)	5.6.2	5.6.3	5.3.1

Request: allow/bump kombu >= 5.7.0 (via apache-airflow-providers-celery
requirements / constraints) so the RabbitMQ transient_nonexcl_queues deprecation is
resolved without a workaround.

Operating System

No response

Deployment

None

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

No response

Official Helm Chart version

Not Applicable

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

How to reproduce

1. Deploy Airflow with CeleryExecutor against a RabbitMQ 4.x broker.
2. Start scheduler + a Celery worker (remote control enabled — the default).
3. Observe RabbitMQ logs (or rabbitmqctl list_deprecated_features --used): the
   transient_nonexcl_queues deprecated feature is flagged as used, caused by the
   *.pidbox queue.

Workarounds (for reference)

• RabbitMQ: deprecated_features.permit.transient_nonexcl_queues = true (temporary).
• Celery: worker_enable_remote_control = False (drops pidbox; disables remote
  control / Flower inspect / revoke).
  Neither is a real fix — kombu >= 5.7.0 is.

References

• kombu#2237 — pidbox durable flag: Change default value for Durable flag on Queue creation for pidbox (RabbitMQ deprecated feature transient_nonexcl_queues) celery/kombu#2237
• RabbitMQ transient_nonexcl_queues: [Other] Understanding `transient_nonexcl_queues` feature deprecation (queues that are both non-durable and non-exclusive) rabbitmq/rabbitmq-server#13161
• Celery quorum queues support: Is there a function that allows the celery worker provided by airflow to take DAG from the Rabbitmq Quorum queue? #35649

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[eladkal]

Where do you see that Airflow limits the versions?

airflow/providers/celery/pyproject.toml

Line 68 in d19a68a

"celery[redis]>=5.5.0,<6",

Latest release of Celery is 5.6.3 that is with the range of installable versions.
You don't have to use Airflow constraints. You can create your own.
Airflow constraints does not guarantee latest available versions. It guarantees only reproducible install.
PIP algorithm decides on the versions. Since you are likely not to use all the dependencies Airflow has sometimes generating your own constraints file just with the packages you need will give you a better result.

Unless there is a version which is broken completely and should be excluded entirely from our resolution options I don't see any task here for Airflow.