# Apache Airflow 2.4.0: That Data Aware Release

DevFeed: [Apache Airflow 2.4.0: That Data Aware Release](<https://devfeed.tech/articles/apache-airflow-2-4-0-that-data-aware-release-32534.md>)

Original publisher: [Read original article](<https://airflow.apache.org/blog/airflow-2.4.0/>)

Author: Apache Airflow

Published: 2022-09-19T00:00:00Z

Content type: release

Language: en

Sources: [Apache Airflow Blog](<https://devfeed.tech/sources/apache-airflow-blog.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [datasets](<https://devfeed.tech/topics/datasets.md>), [releases](<https://devfeed.tech/topics/releases.md>)

Tags: [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [data](<https://devfeed.tech/tags/data.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [new-features](<https://devfeed.tech/tags/new-features.md>), [release](<https://devfeed.tech/tags/release.md>)

## AI overview

Apache Airflow 2.4.0 is a release with more than 650 user-facing commits, including new features, improvements, bug fixes, and documentation changes. Its main feature is data-aware scheduling, which allows DAGs to be scheduled when tasks update datasets and supports smaller, connected workflows. The release also introduces ExternalPythonOperator for running tasks in preconfigured virtual environments to manage conflicting Python dependencies.

## Source excerpt

Apache Airflow 2.4.0 contains over 650 "user-facing" commits (excluding commits to providers or chart) and over 870 total. That includes 46 new features, 39 improvements, 52 bug fixes, and several documentation changes. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html 🐳 Docker Image: docker pull apache/airflow:2.4.0 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0 Data-aware scheduling (AIP-48) This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets. What does this mean, exactly? This is a great new feature that lets DAG authors create smaller, more self-contained DAGs, which chain together into a larger data-based workflow. If you are currently using ExternalTaskSensor or TriggerDagRunOperator you should take a look at datasets - in most cases you can replace them with something that will speed up the scheduling! But enough talking, lets have a short example. First lets write a simple DAG with a task called my_task that produces a dataset called my-dataset: from airflow import Dataset dataset = Dataset(uri='my-dataset') with DAG(dag_id='producer', ...) @task(outlets=[dataset]) def my_task(): ... Datasets are defined by a URI. Now, we can create a second DAG (consumer) that gets scheduled whenever this dataset changes: from airflow import Dataset dataset = Dataset(uri='my-dataset') with DAG(dag_id='dataset-consumer', schedule=[dataset]): ... With these two DAGs, the instant my_task finishes, Airflow will create the DAG run for the dataset-consumer workflow. We know that what exists right now won't fit all use cases that people might wish for datasets, and in the coming minor releases (2.5, 2.6, etc.) we will expand and improve upon this foundation. Datasets represent the abstract concept of a dataset, and (for now) do not have any dire