DAG Source Code Update API

[csm10495]

Hey folks,

I'm wondering what folks think of an API to modify the source code for a given DAG?

Right now we have /dagSources/{file_token} with GET to get the file content. My logic is a POST to the same endpoint with a simple json blob of

{
  "content": "file contents"
}

should replace the file contents at that location on disk. The scheduler would eventually notice the change and update renderings, etc. If the disk is read only, it will fail when attempting to write.

I've had a couple cases where we've thought about having a quick way to remotely update a dag file for some testing. Looking for thoughts on this type of API. Thanks.

[Taragolis]

I don't think this is possible just by design and nature of Airflow. I could be wrong but DAG code which you get from the API or could see in the UI only stored in Airflow Metabase for this purpose (and updated by dag_processor).

In additional due to different use-cases DAG might also just bundled within docker image so it couldn't be changed in worker/dag_processor/scheduler service.

[potiuk]

Yep. Webserver/API does not even have access to DAG files (on purpose and for security reason). So it's not even question of Docker file - webserver in a properly secured deployment can only access the serialized version of DAG from DB - it cannot even read executable code.. This is also a reason why our API does not (and never will support submitting DAGs). You can use any other APIS (whatever your files are deployed wuth - NFS, GIT, S3, GCS - to submit DAGs - specific to your deployment).

Those are completely separated from Airflow and have their own security mechanism to submit new code.

We do NOT want to take the responsibility of modifying the executable code over Airflow REST API. This is a conscious, deliberate choice and is not likely to change.