# Lessons Learned From Running Apache Airflow at Scale

DevFeed: [Lessons Learned From Running Apache Airflow at Scale](<https://devfeed.tech/articles/lessons-learned-from-running-apache-airflow-at-scale-1467.md>)

Original publisher: [Read original article](<https://shopify.engineering/lessons-learned-apache-airflow-scale>)

Author: Megan Parker

Published: 2022-05-23T14:00:01Z

Content type: article

Language: en

Sources: [Shopify Engineering](<https://devfeed.tech/sources/shopify-engineering.md>), [Shopify Engineering - Shopify Engineering](<https://devfeed.tech/sources/shopify-engineering-shopify-engineering.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [Orchestration](<https://devfeed.tech/topics/orchestration.md>), [Shopify](<https://devfeed.tech/topics/shopify.md>), [Kubernetes](<https://devfeed.tech/topics/kubernetes.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [MySQL](<https://devfeed.tech/topics/mysql.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>), [data-modeling](<https://devfeed.tech/topics/data-modeling.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [apache](<https://devfeed.tech/tags/apache.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-storage](<https://devfeed.tech/tags/cloud-storage.md>), [data-modeling](<https://devfeed.tech/tags/data-modeling.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [load](<https://devfeed.tech/tags/load.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [model-training](<https://devfeed.tech/tags/model-training.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [mount](<https://devfeed.tech/tags/mount.md>), [mysql](<https://devfeed.tech/tags/mysql.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [scale](<https://devfeed.tech/tags/scale.md>), [scheduler](<https://devfeed.tech/tags/scheduler.md>), [shopify](<https://devfeed.tech/tags/shopify.md>), [solutions](<https://devfeed.tech/tags/solutions.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

## AI overview

Shopify explains lessons from operating Apache Airflow at scale for more than two years. The deployment runs on Kubernetes with the Celery executor and MySQL, supporting data extraction, machine learning model training, Apache Iceberg table maintenance, and DBT-powered data modeling. The article describes scaling to more than 10,000 DAGs, over 400 concurrent tasks, and more than 150,000 daily runs, along with challenges involving file access, DAG capabilities, traffic variability, and resource contention.

## Source excerpt

Shopify shares some of the lessons we learned and solutions we built in order to run Airflow at scale.