# Deploying Airflow Data Pipelines with a DAG Synchronization Service

DevFeed: [Deploying Airflow Data Pipelines with a DAG Synchronization Service](<https://devfeed.tech/articles/a-smarter-way-to-deploy-data-pipelines-20021.md>)

Original publisher: [Read original article](<https://technology.doximity.com/articles/a-smarter-way-to-deploy-data-pipelines>)

Author: Doximity

Published: 2025-03-12T14:28:00Z

Content type: article

Language: en

Sources: [Doximity](<https://devfeed.tech/sources/doximity.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [data](<https://devfeed.tech/topics/data.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [ci-cd](<https://devfeed.tech/tags/ci-cd.md>), [container](<https://devfeed.tech/tags/container.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [s3](<https://devfeed.tech/tags/s3.md>)

## AI overview

Doximity describes replacing Airflow container image rebuilds and scheduled service restarts with a DAG synchronization service that continuously updates DAGs from Amazon S3. The approach reduced deployment times from about 30 minutes to under one minute and reduced UI disruption and downtime.

## Source excerpt

At Doximity, we have nearly 20 data teams responsible for the development of data pipelines to support product and business intelligence needs. These teams rely on Apache Airflow to orchestrate over 900 active DAGs (fancy word for data pipelines), with dozens of updates deployed daily. However, with the growth of our data platform team, the bottlenecks in our deployment process for data pipelines could no longer be ignored. Deploying new pipelines or updating existing ones required building, publishing, and deploying a new Airflow container image--a process that could take up to 20 minutes. To make matters worse, pulling in the latest updates meant restarting Airflow services, causing temporary UI disruptions and delaying workflows--the salt in the wound, if you will. In this article, we will discuss in further details the bottlenecks of our legacy deployment system and how we overcame them by developing a DAG synchronization service that continuously updates DAGs directly from Amazon S3 object storage. This new approach not only accelerated deployment times--reducing them from 30 minutes to under one--but also improved system reliability, reduced downtime, and enhanced operational flexibility. Problem Overview Prior to the sync service, DAGs were deployed by building them into the Airflow container image using a CI/CD pipeline. This was both time-consuming and resource-intensive, with each build and delivery process taking nearly 20 minutes to complete. With dozens of teams pushing updates daily, each triggering a full image rebuild, these delays were a clear bottleneck to our productivity. Pulling in these changes required scheduled restarts of the Airflow Scheduler and Web Server at regular intervals. These restarts, while necessary, caused temporary (and dare I say, annoying) disruptions to the Airflow Web UI. Refreshing your screen only to be greeted by a 503 error was the virtual equivalent of stubbing your toe--frustrating and painfully avoidable. Factoring in the