Upgrade to 2.4.3 cause all jobs to run 10x longer than previously

[tomrwillis]

Apache Airflow version

2.4.3

What happened

We upgraded our Docker Airflow image to 2.4.3-python 3.10 from 2.2.5-python 3.9 and jobs suddenly started taking many times longer to finish. No other changes were made other than the Airflow version upgrade.

What you think should happen instead

Jobs should behave the same as before the upgrade

How to reproduce

Upgrade existing docker airflow instance
rerun jobs

Operating System

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)"

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==6.0.0
apache-airflow-providers-celery==3.0.0
apache-airflow-providers-cncf-kubernetes==4.4.0
apache-airflow-providers-common-sql==1.2.0
apache-airflow-providers-docker==3.2.0
apache-airflow-providers-elasticsearch==4.2.1
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-google==8.4.0
apache-airflow-providers-grpc==3.0.0
apache-airflow-providers-hashicorp==3.1.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-microsoft-azure==4.3.0
apache-airflow-providers-mysql==3.2.1
apache-airflow-providers-odbc==3.1.2
apache-airflow-providers-postgres==5.2.2
apache-airflow-providers-redis==3.0.0
apache-airflow-providers-sendgrid==3.0.0
apache-airflow-providers-sftp==4.1.0
apache-airflow-providers-slack==6.0.0
apache-airflow-providers-sqlite==3.2.1
apache-airflow-providers-ssh==3.2.0

Deployment

Other Docker-based deployment

Deployment details

This instance is deployed on a Consul/Nomad cluster

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

I think you need to be much more specific than this.

Show the log files, screenshots, every evidence of what you can provide of what shows the long runnning. While I understand that you might think we will be able to help you and that your problem is everyone's problem, it is not likely (but possible) and comes from some of your specific configuration - but what it is, it's impossible to guess from that description.

If you want to get help with your problem, you need to show enought evidences documenting the behaviour you see, - ideally some kind of comparisions, where you show screenshots / logs from before and after the migration. What kind of tasks get slower, which part of the tasks execution is slower (is it scheduling? or task execution?). Anything that you provide will help those who try to help peopel like you in their free time on this forum to help you with as little investment of their time as possible on asking extra information and possibly being able to actually reproduce your case.

There might be multiple reasons why things get slower - including some of the dependencies we upgrade and facts that they might be misused, or some of the buggy code written in DAGs that got exposed after upgrade and masked before. But it is impossible to help if we do not have information that will help us to analyse it. It might well be an airlfow issue - but without the evidences, there is no reasonablke way to expect that someone will try to guess what the problem might be.

Please spend some time on gathering the evidences if you want to get help on the issue.