No venv isolation between zipped workflows

[vavasilev]

Apache Airflow version

2.0.1

Operating System

ubunto 20.04

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

What happened

I am using zipped workflows so that I can have code-level isolation between them. For example I would like to use different versions of python code in different workflows.
When I upload a workflow with newer version of the code the airflow-web has no issues, but when I try to backfill in the scheduler, it tries to load the python libraries from the other workflow.
Example: I have 2 workflows: xxx_workflow_1_0 (the old one) and xxx_stockout_correlation_workflow_1_0 (the new one). The new workflow uses slightly different version of one library.
The web UI renders successfully both workflow. However when I try to backfill the new workflow with the following command:

airflow dags backfill --start-date 2021-03-03 --end-date 2021-03-03 --conf '{"execId": "stock_correlation_20210303"}' xxx_stockout_correlation_workflow_1_0'

I receive the following error:

ERROR - Failed to import: /opt/airflow/dags/xxx_stockout_correlation_1_0.zip
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dagbag.py", line 345, in _load_modules_from_zip
current_module = importlib.import_module(mod_name)
File "/usr/local/lib/python3.8/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 655, in _load_unlocked
File "", line 618, in _load_backward_compatible
File "", line 259, in load_module
File "/opt/airflow/dags/xxx_stockout_correlation_1_0.zip/xxx_stockout_correlation_1_0.py", line 3, in
from a4e_retail.a4e.retail.dags import retail_stockout_correlation_dag
ImportError: cannot import name 'retail_stockout_correlation_dag' from 'a4e_retail.a4e.retail.dags' (/opt/airflow/dags/xxx_1_0.zip/a4e_retail/a4e/retail/dags/init.py)

You can see it attempts to import from the wrong zip - xxx_1_0.zip. It should import from xxx_stockout_correlation_1_0.zip

When I delete the old workflow everything works properly with the new one

What you expected to happen

Both workflows should coexist

How to reproduce

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Interesting interaction I think that one might not be super-easy to fix and might require some more changes in the underlying way how backfill is done. Might be related with #18428 and #18317

[potiuk]

For now the workaround will be to name different copy of the modules differently in the two zip files. I do not think we have a "guarantee" today that zipped workflows will be isolated from each other.

[potiuk]

Converted it into discussion - I am not sure if this is a "feature" that should be implemented.