# Apache Airflow on AWS ECS

DevFeed: [Apache Airflow on AWS ECS](<https://devfeed.tech/articles/apache-airflow-on-aws-ecs-19822.md>)

Original publisher: [Read original article](<https://tech.gc.com/apache-airflow-on-aws-ecs/>)

Author: GameChanger

Published: 2020-07-06T09:00:18Z

Content type: tutorial

Language: en

Sources: [GameChanger](<https://devfeed.tech/sources/gamechanger.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Containers](<https://devfeed.tech/topics/containers.md>), [Jenkins](<https://devfeed.tech/topics/jenkins.md>), [data](<https://devfeed.tech/topics/data.md>), [Pull Request](<https://devfeed.tech/topics/pull-request.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [aws](<https://devfeed.tech/tags/aws.md>), [containers](<https://devfeed.tech/tags/containers.md>), [data](<https://devfeed.tech/tags/data.md>), [ecs](<https://devfeed.tech/tags/ecs.md>), [jenkins](<https://devfeed.tech/tags/jenkins.md>), [jobs](<https://devfeed.tech/tags/jobs.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [pull-request](<https://devfeed.tech/tags/pull-request.md>), [workflow](<https://devfeed.tech/tags/workflow.md>)

## AI overview

This tutorial explains how a data team moved from Jenkins to Apache Airflow and ran Airflow and other containerized workloads on AWS ECS. It introduces ECS clusters, task definitions, services, and deployment choices including Fargate, EC2, and Kubernetes, while noting that Airflow infrastructure is straightforward but configuring the ECS operator requires substantial setup.

## Source excerpt

An overview of what AWS ECS is, how to run Apache Airflow and tasks on it for eased infrastructure maintenance, and what we've encountered so that you have an easier time getting up and running. Our data team recently made the transition in workflow systems from Jenkins to Apache Airflow. Airflow was a completely new system to us that we had no previous experience with but is the current industry standard for the sort of data-centric workflow jobs we were looking to run. While it hasn't been a perfect fit, we have been able to get a lot of benefits from it: jobs are defined in code, we've the history of each job, it goes through our normal pull request process, and everyone on the team is able to read and write jobs. Since our team is data focused, we wanted our Airflow setup to be as easy to maintain as possible, especially around infrastructure, so we have minimal distractions with high resiliency. This led us to using AWS ECS not only to run Airflow but for our bigger tasks that are already containerized. Not familiar with ECS? Or how to run Airflow or its tasks on it? Don't worry, we weren't either. This is what we've learned. What is ECS? ECS with Terraform Setting up Airflow infrastructure on ECS Setting up Airflow tasks on ECS Learnings from Airflow and ECS tl;dr Running Airflow's infrastructure on ECS is super easy but running the ECS operator needs hecka help setting up. What is ECS? ECS is AWS's Elastic Container Service, designed to let you run containers without worrying about servers. You do this by creating a cluster for your system, define task definitions for the tasks you want to run, and possibly group your tasks into services. You can also choose if you want to have your containers fully managed (ECS on Fargate), kind of managed but kind of not (ECS on EC2), or if you'd like to use Kubernetes. If all of that made sense to you, congratulations: you can skip to the next section! If you need a bit more of a breakdown, here's what you need to know: A