Scheduler Memory Leak

[VPatel5]

Under which category would you file this issue?

Helm chart

Apache Airflow version

3.2.2 (Helm 1.22.0)

What happened and how to reproduce it?

My airflow scheduler slowly increases in memory usage.

Below are the flamegraph files of the profiles I ran on the scheduler. The 2nd was a snapshot taken 12+ hours after.

leaks-flamegraph-1.html
leaks-flamegraph-2.html

What you think should happen instead?

Memory usage should stay stable.

Operating System

No response

Deployment

Official Apache Airflow Helm Chart

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

No response

Official Helm Chart version

1.22.0 (latest released)

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[arkadiuszbach]

its due to this: #67127, #68905

[VPatel5]

its due to this: #67127, #68905

Thanks! Was this fix released in 3.3.0, or is it targeted for a later version?

[Miretpl]

Thanks! Was this fix released in 3.3.0, or is it targeted for a later version?

Released in the Celery provider 3.21.0 version. Let us know if that resolves your issue or if you still observe a memory leak.

[VPatel5]

I have updated the celery version to 3.22.0.

However, it seems the memory is still slowly increasing. It might be too early to tell, so I will provide an update after more observations.