airflow tasks are getting randomly terminated with no errors in UI and logs on worker shows module not found

[gurjit-sandhu]

Official Helm Chart version

1.6.0

Apache Airflow version

airflow 2

Kubernetes Version

1.22

Helm Chart configuration

i have set PYTHONPATH as below in extraENV and verified after login to worker pod
extraEnv: |
- name: PYTHONPATH
value: "/opt/airflow/dags:/opt/airflow"

also have setup PYTHONPATH in docker image

Setting python path for importing dag modules

ENV PYTHONPATH="/opt/airflow/dags:/opt/airflow"

Docker Image customisations

base docker image == FROM apache/airflow:2.3.2-python3.8

also have setup PYTHONPATH in docker image

Setting python path for importing dag modules

ENV PYTHONPATH="/opt/airflow/dags:/opt/airflow"

What happened

dags are showing up on UI however after executing it get terminated and below are logs from worker showing its not able to find python modules -- if we re-run same tasks it succeeds however randomly it fails with below error module not found

_execute_in_fork(command_to_exec, celery_task_id)

File "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 108, in _execute_in_fork
raise AirflowException(msg)
airflow.exceptions.AirflowException: Celery command failed on host: airflow-worker-1.airflow-worker.airflow.svc.cluster.local with celery_task_id f672c24c-d54c-4401-a1c1-8892d54f90f4
[2022-10-27 18:48:32,001: INFO/MainProcess] Task airflow.executors.celery_executor.execute_command[1fdb829f-755a-4ac0-98c5-362e6b6c8a44] received
[2022-10-27 18:48:32,011: INFO/ForkPoolWorker-15] [1fdb829f-755a-4ac0-98c5-362e6b6c8a44] Executing command in Celery: ['airflow', 'tasks', 'run', 'yipit_fsv_data_export_dag', 'verify_all_exports_were_successful', 'manual__2022-10-27T13:42:55-05:00', '--local', '--subdir', 'DAGS_FOLDER/partner_exports/data_export_dag_builder.py']
[2022-10-27 18:48:32,087: INFO/ForkPoolWorker-15] Filling up the DagBag from /opt/airflow/dags/partner_exports/data_export_dag_builder.py
[2022-10-27 18:48:32,144: ERROR/ForkPoolWorker-15] Failed to import: /opt/airflow/dags/partner_exports/data_export_dag_builder.py
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dagbag.py", line 317, in parse
loader.exec_module(new_module)
File "", line 843, in exec_module
File "", line 219, in _call_with_frames_removed
File "/opt/airflow/dags/partner_exports/data_export_dag_builder.py", line 20, in
from utils.generic_callbacks import on_failure_callback, on_success_callback
ModuleNotFoundError: No module named 'utils'
[2022-10-27 18:48:32,145: ERROR/ForkPoolWorker-15] [1fdb829f-755a-4ac0-98c5-362e6b6c8a44] Failed to execute task Dag 'yipit_fsv_data_export_dag' could not be found; either it does not exist or it failed to parse..
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 128, in _execute_in_fork
args.func(args)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 51, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 99, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 360, in task_run
dag = get_dag(args.subdir, args.dag_id)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 203, in get_dag
raise AirflowException(
airflow.exceptions.AirflowException: Dag 'yipit_fsv_data_export_dag' could not be found; either it does not exist or it failed to parse.

What you think should happen instead

since PYTHONPATH is set in both docker and helm chart environment variables - dag should be able to find modules

How to reproduce

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Do not set overlapping PYTHONPATH. It is likely the reason for the confiusion - you likely have several utils folder and they are sometimes found in one path, sometimes in others. Follow Best Practices and module management like described in our documentation: https://airflow.apache.org/docs/apache-airflow/stable/modules_management.html#typical-structure-of-packages

[potiuk]

This is not a bug, I converted it into discussion BTW.