why lowest ( DB , K8S , PYTHON ) versions are the default ?

[raphaelauv]

Hi, I can't find the documentation explaining the choice of using lowest versions for DB , K8S , PYTHON for airflow

example the container apache/airflow:slim-2.5.0 is using python 3.7 and not 3.10

#28168 (comment)

Thanks

[potiuk]

This is our "default" Python version across the board (and same with other components) as we have to make sure it works on 3.7. Usually python versions only add features, so making sure that things work on Python 3.7 is the default choice on our CI by default because it's more likely that if we use Python 3.8 or 3.9, new features will be used that will break 3.7. This is the main reason 3.7 is considered as default version across the board for everything - starting from CI, ending with images.

For images It is documented here: https://airflow.apache.org/docs/docker-stack/index.html - though this reasoning is not explained there.

There is also a ver down to earth practical side to it - it has also the advantage that anyone who does not care the version they will be automatically upgraded only when the "default" version ends its life and otherwise it is stable through the whole period when the old version has been still supported. So when you get upgrade, you know why if you care about stability and "long term running" version. And there is no debate about it. Python version is EOL, we cannot keep it with that version, so we should bump it.

If we make another choice, we would have to set some rules - when to bump the version. And this would inevitably spark some debate - should we update immediately when it is released or when we support it ? Should it be latest or latest -1, how do we explain it to the users who care about stability more than "bleeding edge".

If we choose "newest" as default, this is problematic

Usually there will be a period after Python version is released (like it is the case in 3.11) that a number of libraries do not support new Python version (because for example binary wheels have not been released yet). This is the case for now when we are waiting for some deps to upgrade to 3.11 #27264. So we could make latest the default only after all of them are complete. But for some users this might break things - becasue THEIR dependencies might not support latest version yet. So we would not know when it is "safe" to upgrade the default version. With 3.7 - this will be easy. We upgrede to 3.8 right after 3.7 EOL - and if it breaks user's workflow, then justification is undisputable. It's EOL, we don't support it any more. If we upgrade to "latest" version - as soon as we can (or even a bit later) the "complaint" will be on Airflow for breaking the workflow.

If we choose "middle" version - we have similar issue. If we pick 3.8 today - when do we upgrade it to 3.9? What's the criteria? There will be a lot of debate and opinions if we start this discussion.

So picking "lowest" version provide the most "stable" approach for those who did not explicitly choose their version. And still leaving a freedom for those who want to choose a specific version.

If you want to discuss another approach - feel free though :) . Devlist thread with justification should be a good start. But prepare for discussion unless you have very good reason (I will be presenting the above as good reasons to keep the default as is now).

Also If you feel like it needs documenting - maybe you can come up with a PR summarizing what I have just described :).

[raphaelauv]

thanks it's perfectly clear 👍