Cleared backfill DAG runs get stuck with "Task is in the 'None' state"

[Kjir]

Under which category would you file this issue?

Airflow Core

Apache Airflow version

3.3.0

What happened and how to reproduce it?

I cleared all tasks from a certain date onwards to re-execute my models. However, the execution stalled on a backfill dag run. It says that Dependencies are blocking the task from being executed, but the task has no dependecies — it's the first task in the DAG!

I can confirm that nothing is running as well by looking at the pool, which has 0 slots taken despite having many tasks to be executed still.
Depends on past is true for the task in question. The backfill job did run successfully once before, but not after clearing it and the tasks before and after (tasks before were re-executed correctly and are hence green).
There are no failed tasks relating to this DAG.

What you think should happen instead?

The tasks should be executed.

Operating System

Debian GNU/Linux 12 (bookworm)

Deployment

Official Apache Airflow Helm Chart

Apache Airflow Provider(s)

cncf-kubernetes

Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes==10.19.0

Official Helm Chart version

1.22.0 (latest released)

Kubernetes Version

v1.34.6+rke2r3

Helm Chart configuration

Nothing seems relevant to me. If needed I'll provide additional details

Docker Image customizations

None

Anything else?

The cluster does require a proxy to access external resources, and thus there are HTTP_PROXY, HTTPS_PROXY and NO_PROXY variables defined.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[Kjir]

I have several DAGs generated in a loop (over a list of countries), and all of them have the issue. I tried marking as success the backfill DAG runs, and once I do that the normal scheduled DAG runs resume the execution. However since these runs did not execute the data is not actually correct, and since the scheduled runs are missing for those days because that's when the upgrade to Airflow 3.x was performed (and the way the interval is calculated changed, resulting in a skipped run), I don't have a way to rerun my models for those dates.

[Kjir]

Even weirder: some DAGs (always same configuration, different countries) have had some task instances run, but then they got stuck:

[prayaagmahajan]

Is there any fix for it, I am also facing a similar kind of issue!

[rehan243]

oh yeah, that "Task is in the 'None' state" error's bitten me before, especially after clearing with depends_on_past set. Honestly, the thing is, Airflow gets a bit confused on backfill runs when depends_on_past is true and previous runs are cleared or missing. The scheduler can't always figure out what the "previous" is supposed to be, so it won't transition to scheduled/running.

I've had cases where I had to manually mark old runs as successful just to get things unstuck. Not gonna lie, the docs are super vague about this edge case, but it usually boils down to the metadata DB not reflecting a valid prior run state. Sometimes, restarting the scheduler or clearing just the specific task instance (instead of the whole DAG run) helps, but that's hit or miss.

Are you seeing this only on 3.3.0? Curious if you tried toggling depends_on_past off temporarily to unblock it, or running airflow jobs manually (like airflow tasks run) to force progression?

[Kjir]

I have always avoided backfill tasks before because it feels weird and doesn't seem to fit my use case. Which means I can't really tell if the issue would happen with earlier versions or not.

If I remove the depends_on_past bit it would run all the tasks out of order, and that wouldn't be correct. It also would be imply a lot of manual work, and thus is not an actual workaround. Also in the second screenshot, the tasks that aren't being executed do not depend on past.
Setting the previous singular tasks to success doesn't work because I get an error message saying that they already are in the success state.
There isn't an airflow tasks run command, so I'm not sure what you mean?

[tirkarthi]

Related

#49508
#58554

How many schedulers are running in your deployment and the active running dagruns could cause this. Please also check and increase the value of max_dagruns_per_loop_to_schedule as required to see if it helps.

[Kjir]

Indeed the issue is indeed the number of schedulers + max_dagruns_per_loop_to_schedule. It's a bit strange that darguns that cannot be executed currently hold a slot in the scheduler, IMO, especially when there is depends_on_past involved.

Maybe there's technical reasons I cannot see, but I would expect the dag runs with an older date would be queued before dag runs with a newer one.
I believe backfills are always queued after scheduled runs and that's why if the pool fills up they are left starved for resources?