# Apache Airflow Blog Platform created by the community to programmatically author, schedule and monitor workflows. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Apache Airflow 3.3.0: Stateful Tasks and Multi-Language Support DevFeed: [Apache Airflow 3.3.0: Stateful Tasks and Multi-Language Support]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-07-06T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [SDKs](), [Go Language](), [Java]() Tags: [airflow](), [apache-airflow](), [go](), [java](), [release](), [release-notes](), [sdk](), [state]() ### AI overview Apache Airflow 3.3.0 introduces durable state stores for tasks and assets, a Language Task SDK for Java and Go, expanded asset partitioning, and pluggable retry policies. ### Source excerpt We're proud to announce the release of Apache Airflow 3.3.0! Where 3.2 brought precision to data with asset partitioning, 3.3 gives your tasks memory and multi-language support: a first-class state store for tasks and assets, a Language Task SDK for writing task logic in Java and Go, a major expansion of asset partitioning, and pluggable retry policies. 🎯 Release Highlights 📦 PyPI: https://pypi.org/project/apache-airflow/3.3.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.3.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.3.0/release_notes.html 🐳 Docker Image: docker pull apache/airflow:3.3.0 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.3.0 🗃 Task & Asset State Store (AIP-103): Tasks That Remember Until now, if a task needed to remember something across retries or runs -- a cursor, a checkpoint, a high-water mark -- you reached for XComs, an external store, or a clever Variable hack. Airflow 3.3 makes durable task state a first-class concept. Tasks can persist arbitrary key-value state that survives across retries and runs via a new task_state_store accessor, and assets can carry their own state via asset_state_store -- both available directly from the Task SDK. State lives in the metadata database by default, or in a custom worker-side backend ([workers] state_store_backend), supports per-key retention with periodic garbage collection and an optional clear_on_success, and is fully manageable through the Core API and Execution API. Task State Store Task state is scoped to a specific task instance and persists across retries. Use it to track coordination state like remote job IDs, cursors, or progress checkpoints: @task def extract_data(**context): task_state = context["task_state_store"] # Resume from where we left off on retry cursor = task_state.get("last_cursor", default=0) records = fetch_records(since=cursor) new_cursor = records[-1]["id"] task_state.set("last_cursor", new_cursor) return records Persisted st ## Agentic Workloads on Airflow: Observable, Retryable, and Auditable by Design DevFeed: [Agentic Workloads on Airflow: Observable, Retryable, and Auditable by Design]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-04-15T00:00:00Z Content type: tutorial Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Artificial Intelligence](), [Large Language Model](), [SQL](), [datasets](), [Agent Harness]() Tags: [agentic](), [ai](), [airflow](), [community](), [csv](), [llm](), [sql](), [survey](), [tutorial](), [workflow]() ### AI overview This article explains how to build agentic workloads on Airflow 3 using the 2025 Airflow Community Survey dataset and the apache-airflow-providers-common-ai provider. It distinguishes single-query LLM pipelines from workflows that query multiple independent dimensions and synthesize their results, mapping each sub-query to a named Airflow task and using Dynamic Task Mapping and XCom for orchestration and observability. ### Source excerpt A question like "How does AI tool usage vary across Airflow versions?" has a natural SQL shape: one cross-tabulation, one result. A question like "What does a typical Airflow deployment look like for practitioners who are actively using AI in their workflow?" does not. It requires querying executor type, deployment method, cloud provider, and Airflow version independently, each filtered to the same respondent group, then synthesizing the results into a coherent picture. No single query returns the answer. The answer emerges from the relationship between all of them. This is where Airflow's agentic pattern begins: not when you add an LLM to a workflow, but when the structure of the work itself depends on running multiple LLM calls whose outputs feed a synthesis step. This post builds that pattern using the 2025 Airflow Community Survey data set and the apache-airflow-providers-common-ai provider for Airflow 3. If you haven't read the introductory survey analysis post yet, start there for a walkthrough of the single-query interactive and scheduled pipelines. This post picks up where that one ends. The Agentic Gap in the Single-Query Pattern The interactive and scheduled survey DAGs from the introductory post each do one thing: translate a natural language question into SQL, execute it against the CSV, and return the result. The LLM is involved once. The structure of the pipeline does not change based on what that LLM call returns. That is not a limitation to fix. It is the right design for that class of question. For a large fraction of production AI workflows, a single well-structured LLM call with good context is sufficient and preferable. The pattern becomes agentic when two things are true simultaneously: The question requires querying multiple independent dimensions The synthesis step, the thing that produces the final answer, depends on all of those results In an agent harness framework, this would be handled inside a reasoning loop: the LLM decides to call a to ## Ask Your Survey Anything: Building AI Analysis Pipelines with Airflow 3 DevFeed: [Ask Your Survey Anything: Building AI Analysis Pipelines with Airflow 3]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-04-15T00:00:00Z Content type: tutorial Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Artificial Intelligence](), [Large Language Model](), [SQL](), [CSV](), [data]() Tags: [ai](), [airflow](), [apache-airflow](), [automated](), [community](), [configuration](), [csv](), [executor](), [llm](), [pipelines](), [providers](), [scheduled](), [sql](), [tutorial]() ### AI overview A tutorial showing how to build interactive and scheduled AI analysis pipelines with Airflow 3. The pipelines use survey CSV data, generate and run SQL queries, validate schemas, and either request human approval or email results automatically. ### Source excerpt The 2025 Airflow Community Survey collected responses from nearly 6,000 practitioners across 168 questions. You can open a spreadsheet and filter, or write SQL by hand. But what if you could just ask a question and have Airflow figure out the query, run it, and bring the result back for your approval? This post builds two pipelines that do exactly that, using the apache-airflow-providers-common-ai provider for Airflow 3. The first pipeline is interactive: a human reviews the question before it reaches the LLM and approves the result before the DAG finishes. The second is scheduled: it downloads fresh survey data, validates the schema, runs the query unattended, and emails the result. If you haven't seen the common.ai provider overview yet, start there for a tour of all the operators. This post goes deep on a concrete end-to-end example. Two Pipelines, One Example File Both DAGs live in example_llm_survey_analysis.py and share the same schema context and datasource configuration. example_llm_survey_interactive: trigger manually, review at both ends: prompt_confirmation -> generate_sql -> run_query -> extract_data -> result_confirmation (HITLEntryOperator) (LLMSQLQuery) (Analytics) (@task) (ApprovalOperator) example_llm_survey_scheduled: runs @monthly, no human in the loop: download_survey -> prepare_csv -> check_schema -> generate_sql -> run_query -> extract_data -> send_result (HttpOperator) (@task) (LLMSchema (LLMSQLQuery) (Analytics) (@task) (@task / Email) Compare) The Data The Airflow Community Survey 2025 CSV has 5,856 rows and 168 columns covering everything from Airflow version and executor type to cloud provider, company size, and AI tool usage. A few highlights from the data: 3,320 respondents identify as Data Engineers 2,032 use AWS as their primary cloud provider for Airflow 1,445 are already running Airflow 3 1,351 say they often use AI tools to write Airflow code Those last two numbers together are part of why this example exists: the people most likely to use th ## Introducing the Common AI Provider: LLM and AI Agent Support for Apache Airflow DevFeed: [Introducing the Common AI Provider: LLM and AI Agent Support for Apache Airflow]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-04-14T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Large Language Model](), [AI Agent](), [Pydantic](), [Artificial Intelligence]() Tags: [2025](), [ai](), [ai-agent](), [airflow](), [apache](), [apache-airflow](), [community](), [llm](), [release]() ### AI overview Apache Airflow is releasing the Common AI Provider 0.1.0, a provider package that adds LLM and AI agent capabilities directly to Airflow. Built on Pydantic AI, it supports more than 20 model providers through a single install and requires Apache Airflow 3.0 or newer. ### Source excerpt At Airflow Summit 2025, we previewed what native AI integration in Apache Airflow could look like. Today we're shipping it. apache-airflow-providers-common-ai 0.1.0 adds LLM and agent capabilities directly to Airflow. Not a wrapper around another framework, but a provider package that plugs into the orchestrator you already run. It's built on Pydantic AI and supports 20+ model providers (OpenAI, Anthropic, Google, Azure, Bedrock, Ollama, and more) through a single install. pip install 'apache-airflow-providers-common-ai' Requires Apache Airflow 3.0+. Note: This is a 0.x release. We're actively looking for feedback and iterating fast, so breaking changes are possible between minor versions. Try it, tell us what works and what doesn't. Your input directly shapes the API. By the Numbers 6 Operators 6 TaskFlow decorators 5 Toolsets 4 Connection types 20+ Supported model providers via Pydantic AI The Decorator Suite Every operator has a matching TaskFlow decorator. @task.llm: Single LLM Call Send a prompt, get text or structured output back. from pydantic import BaseModel from airflow.providers.common.compat.sdk import dag, task @dag def my_pipeline(): class Entities(BaseModel): names: list[str] locations: list[str] @task.llm( llm_conn_id="my_openai_conn", system_prompt="Extract named entities.", output_type=Entities, ) def extract(text: str): return f"Extract entities from: {text}" extract("Alice visited Paris and met Bob in London.") my_pipeline() The LLM returns a typed Entities object, not a string you have to parse. Downstream tasks get structured data through XCom. @task.agent: Multi-Step Agent with Tools When the LLM needs to query databases, call APIs, or read files across multiple steps, use @task.agent. The agent picks which tools to call and loops until it has an answer. from airflow.providers.common.ai.toolsets.sql import SQLToolset from airflow.providers.common.compat.sdk import dag, task @dag def sql_analyst(): @task.agent( llm_conn_id="my_openai_conn", sys ## Apache Airflow 3.2.0: Data-Aware Workflows at Scale DevFeed: [Apache Airflow 3.2.0: Data-Aware Workflows at Scale]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-04-07T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [data](), [Orchestration](), [Release notes](), [SDK](), [Amazon S3](), [BigQuery]() Tags: [airflow](), [apache](), [apache-airflow](), [bigquery](), [data](), [docker-image](), [orchestration](), [pypi](), [release](), [release-notes](), [s3](), [sdk](), [workflows]() ### AI overview Apache Airflow 3.2.0 introduces asset partitioning for more granular data-aware scheduling, along with multi-team deployments, synchronous deadline alert callbacks, and continued Task SDK separation. Partition-aware workflows trigger downstream DAGs only when the relevant partition is updated. ### Source excerpt We're proud to announce the release of Apache Airflow 3.2.0! Airflow 3.1 puts humans at the center of automated workflows. 3.2 brings that same precision to data: Asset partitioning for granular pipeline orchestration, multi-team deployments for enterprise scale, synchronous deadline alert callbacks, and continued progress toward full Task SDK separation. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/3.2.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.2.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.2.0/release_notes.html 🐳 Docker Image: docker pull apache/airflow:3.2.0 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.2.0 🗂 Asset Partitioning (AIP-76): Only the Right Work Gets Triggered Asset partitioning has been one of the most requested additions to data-aware scheduling. If you work with date-partitioned S3 paths, Hive table partitions, BigQuery partitions, or really any partitioned data store, you've dealt with this: An upstream task updates one partition, and every downstream Dag fires regardless of which slice actually changed. It's wasteful, and for large deployments it creates real operational noise. Asset partitioning in 3.2 makes this granular. Downstream Dags trigger only when the specific partition they care about gets updated. It's the biggest change to data-aware scheduling since Assets were introduced, and it turns partition-driven orchestration into something Airflow handles natively rather than something you work around. Key Capabilities Partition-driven scheduling: Dags trigger on specific partition updates, not every asset change CronPartitionTimetable: Schedule Dags against partitions using cron expressions. Also available in the Task SDK Backfill for partitioned Dags: Backfill historical partitions without re-triggering everything downstream (#61464) Multi-asset partitions: A single Dag can listen for partitions across multiple assets, which matters when your downstream work depe ## Introducing the Apache Airflow Registry DevFeed: [Introducing the Apache Airflow Registry]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-03-19T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Data pipelines](), [API](), [JSON]() Tags: [airflow](), [apache](), [apache-airflow](), [community](), [data-pipelines](), [messaging](), [notifications](), [openai](), [registry]() ### AI overview Apache Airflow launches the Airflow Registry, a searchable catalog of official providers and modules. It includes provider and module search, installation and compatibility details, connection generation in URI, JSON, and environment-variable formats, ecosystem statistics, and a structured JSON API. ### Source excerpt Today we're launching the Apache Airflow Registry -- a searchable catalog of every official Airflow provider and its modules, live at airflow.apache.org/registry/. Need an S3 operator? A Snowflake hook? An OpenAI sensor? The Registry helps you find, compare, and configure the right components for your data pipelines -- without digging through docs or PyPI pages. By the Numbers 98 Official providers 1,602 Modules (operators, hooks, sensors, triggers, transfers, and more) 329M+ Monthly PyPI downloads across all providers 125+ Integrations with cloud platforms, databases, ML tools, and messaging services Search Everything Hit Cmd+K from any page and start typing. Results show up instantly, grouped by Providers and Modules, with type badges so you can tell a hook from an operator at a glance. Provider Pages Each provider gets a dedicated page with everything in one place: install command with copy-to-clipboard, version selector, extras dropdown, compatibility info, connection types, and the full module listing organized by type. The Amazon provider, for example, has 372 modules across operators, hooks, sensors, triggers, transfers, and more. Module type tabs let you filter to exactly what you're looking for, and a category sidebar groups modules by AWS service (S3, Lambda, Glue, Step Functions, etc.). Connection Builder Click any connection type badge on a provider page, fill in the fields, and the builder generates the connection in three formats -- URI, JSON, and Env Var -- ready to copy into your configuration. No more guessing URI encoding or JSON structure. Explore by Category Not sure which provider you need? The Explore page organizes providers into categories: Cloud Platforms, Databases, Data Warehouses, Messaging & Notifications, AI & Machine Learning, Data Processing, and more. Statistics The Stats page breaks down the ecosystem: 848 operators, 298 hooks, 164 triggers, 157 sensors, 83 transfers, and more -- plus top providers by downloads and module count. JSON API ## Airflow Survey 2025 DevFeed: [Airflow Survey 2025]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-01-22T00:00:00Z Content type: article Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [data](), [CSV]() Tags: [2025](), [airflow](), [analysis](), [apache-airflow](), [community](), [csv](), [download](), [form](), [graphics](), [interactive](), [report](), [responses](), [share](), [survey](), [users](), [view]() ### AI overview This page presents the Airflow Survey 2025 interactive report, hosted by Astronomer and sponsored and supported by the Apache Airflow community. The supplied evidence does not include the survey findings. ### Source excerpt The interactive report is hosted by Astronomer. The Apache Airflow community thanks Astronomer for running this survey, for sponsoring it and providing the report in this form, and for their effort in marketing, analysis, and preparing the graphics. View raw data Download survey responses (CSV) ## Apache Airflow CTL aka airflowctl 0.1.0 DevFeed: [Apache Airflow CTL aka airflowctl 0.1.0]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2025-10-15T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Command-line interface](), [Release notes](), [API](), [REST API](), [releases](), [Authentication](), [audit]() Tags: [airflow](), [apache-airflow](), [audit-trail](), [authentication](), [cli](), [command-line](), [release](), [release-notes](), [rest-api]() ### AI overview Apache Airflow announces airflowctl 0.1.0, a command-line interface that interacts exclusively with the Airflow REST API. The release is intended to provide secure, auditable, remote management of Airflow deployments without direct metadata-database access, while remote commands gradually move from the existing Airflow CLI. ### Source excerpt We are thrilled to announce the first major release of airflowctl 0.1.0, the new secure, API-driven command-line interface (CLI) for Apache Airflow -- built under AIP-81. This release marks CLI to join the general posture on communicating through API. Airflow CLI joins the modern era of secure, auditable, and remote-first operations. Details: 📦 PyPI: https://pypi.org/project/apache-airflow-ctl/0.1.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow-ctl/stable/release_notes.html 🪶 Source Code: https://github.com/apache/airflow/tree/main/airflow-ctl 🎯 What is airflowctl? airflowctl is a new command-line interface for Apache Airflow that interacts exclusively with the Airflow REST API. It provides a secure, auditable, and consistent way to manage Airflow deployments -- without direct access to the metadata database. 🔄 Coexistence with Airflow CLI The Airflow CLI will continue as intended, primarily for admin tasks such as running Airflow components (airflow api-server, airflow scheduler) or managing the metadata database (airflow db init). airflowctl focuses on operational commands that interact with Airflow resources via the API (airflowctl dagrun trigger, airflowctl connection create, etc.). We defined the commands falls under two main categories: Remote Commands: Operations that can be provided via API (e.g., managing DAGs, connections, variables, triggering DAG runs) are now available in airflowctl and will be the recommended approach going forward. Local/Admin Commands: Operations that manage Airflow components or the metadata database will remain in the Airflow CLI. Of course, in the current state they will both have the remote commands. We are planning a zero-disruption migration path where Remote Commands will be gradually deprecated from the Airflow CLI as they achieve parity in airflowctl. 🔒 Why airflowctl? Until now, Airflow CLI connected directly to the metadata database, bypassing RBAC, authentication, and API logs. While convenient, this app ## Apache Airflow 3.1.0: Human-Centered Workflows DevFeed: [Apache Airflow 3.1.0: Human-Centered Workflows]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2025-09-25T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Automation](), [human review](), [Data Quality](), [Forms](), [Developer experience](), [Inference](), [React]() Tags: [apache-airflow](), [automation](), [data-quality](), [developer-experience](), [forms](), [human-review](), [inference](), [release](), [release-notes](), [ui]() ### AI overview Apache Airflow 3.1.0 introduces human-in-the-loop tasks for pausing automated workflows and collecting reviews through web forms. The release also adds internationalization support, developer experience improvements, restored Calendar and Gantt views, UI filtering, DAG pinning, and accessibility-focused theme updates. ### Source excerpt We are thrilled to announce the release of Apache Airflow 3.1.0, an update that puts humans at the center of data workflows. This release introduces powerful new capabilities for human decision-making in automated processes, comprehensive internationalization support, and significant developer experience improvements. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/3.1.0/ 📚 Core Airflow Docs: https://airflow.apache.org/docs/apache-airflow/3.1.0/ 📚 Task SDK Docs: https://airflow.apache.org/docs/task-sdk/1.1.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.1.0/release_notes.html 🪶 Sources: https://airflow.apache.org/docs/apache-airflow/3.1.0/installation/installing-from-sources.html 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.1.0 🤝 Human-in-the-Loop (HITL): When Automation Meets Human Judgment This powerful capability bridges the gap between automated processes and human expertise, making Airflow invaluable for: AI/ML Model Validation: Pause inference pipelines for human review of model outputs Content Moderation: Route content through human reviewers before publication Approval Workflows: Require manager approval for sensitive operations Data Quality Gates: Allow data stewards to validate critical datasets HITL tasks pause in a deferred state while presenting intuitive web forms in the Airflow UI. Users with appropriate roles can review context data, DAG parameters, and XCom values before making informed decisions. Example Code: from airflow.sdk import DAG from airflow.providers.standard.operators.hitl import HITLOperator with DAG("content_moderation", schedule="@daily") as dag: moderate_content = HITLOperator( task_id="review_content", message="Please review this content for publication", data_key="content_to_review" ) 📊 UI Enhancements & Performance Calendar and Gantt Views Make Their Comeback Remember those beloved Calendar and Gantt chart views from Airflow 2.x? They're back, completely rebuilt for the modern ## Apache Airflow® 3 is Generally Available! DevFeed: [Apache Airflow® 3 is Generally Available!]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2025-04-22T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [event driven](), [Architecture & Design](), [React](), [Messaging](), [Cloud]() Tags: [airflow](), [apache-airflow](), [architecture](), [event-driven](), [hybrid-cloud](), [messaging](), [react](), [release](), [releases]() ### AI overview Apache Airflow 3.0 has been released as a major version. The release introduces a React-based UI, DAG versioning, improved backfill support, event-driven scheduling, and a Task Execution Interface with Task SDKs supporting secure execution across multi-cloud, hybrid-cloud, and local data center deployments. ### Source excerpt We announced our intent to focus on Apache Airflow 3.0® as the next big milestone for the Airflow project at the Airflow Summit in September 2024. We are delighted to announce that Airflow 3.0 is now released! A Major Release, Four Years in the Making Airflow 3.0 is the biggest release in Airflow's history--2.0 was released in 2020, and the last 4 years have seen incremental updates and releases every quarter with version 2.10 released in Q4 2024. With over 30 million monthly downloads (up over 30x since 2020) and 80,000 organizations (up from 25,000 in 2020) now using Airflow, we've seen an incredible growth in popularity since 2.0. Over the last four years, Airflow has grown to power business critical data workflows within organizations of all sizes. We have seen an exponential increase in the use cases for Airflow from its beginnings with ETL, ELT, and Reverse ETL, with over 30% of Airflow users using it for MLOps, and 10% using it for GenAI workflows. Airflow 3 is a response to this use case expansion and is the standard for data application development across the enterprise. Here are some highlights: Airflow 3 is significantly easier to use for data practitioners and incorporates their key requests for critical changes to Airflow. Early user reactions to features such as the new React based UI, DAG Versioning, and improved Backfill support have been incredibly positive. I was ecstatic to see the reaction from data engineers when I demonstrated this at a recent Airflow meetup. The seamless UI transition of navigating between Asset-oriented workflows and Task-oriented workflows is beautiful. Once again, Airflow lets the developer choose how you want to develop and navigate without imposing any restrictions. Introduction of Event Driven Scheduling enables Airflow to seamlessly integrate with messaging providers and react to events happening and data assets being updated outside of Airflow. The big architecture change with the introduction of the Task Execution Inte ## Airflow Survey 2024 DevFeed: [Airflow Survey 2024]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2025-02-27T00:00:00Z Content type: article Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [data](), [CSV]() Tags: [airflow](), [apache](), [community](), [csv](), [data](), [download](), [graphics](), [interactive](), [report](), [survey](), [users]() ### AI overview Airflow Survey 2024 is an interactive report hosted by Astronomer, based on a survey run with support from the Apache Airflow community. The page provides access to raw data and survey responses in CSV format. ### Source excerpt The interactive report is hosted by Astronomer. The Apache Airflow community thanks Astronomer for running this survey, for sponsoring it and providing the report in this form, and for their effort in marketing, analysis, and preparing the graphics. View raw data Download survey responses (CSV) ## Apache Airflow 2.10.0 is here DevFeed: [Apache Airflow 2.10.0 is here]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2024-08-08T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [telemetry](), [configuration](), [data](), [PyPI](), [Docker Image]() Tags: [airflow](), [apache-airflow](), [configuration](), [docker-image](), [pypi](), [release](), [release-notes](), [releases](), [telemetry]() ### AI overview Apache Airflow 2.10.0 is available with default basic telemetry collection, support for multiple executors in one environment, and dynamic dataset scheduling through DatasetAlias. Deployments can opt out of telemetry collection. ### Source excerpt I'm happy to announce that Apache Airflow 2.10.0 is now available, bringing an array of noteworthy enhancements and new features that will greatly serve our community. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.10.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.10.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.10.0/release_notes.html 🐳 Docker Image: "docker pull apache/airflow:2.10.0" 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.10.0 Airflow now collects Telemetry data by default With the release of Airflow 2.10.0, we've introduced the collection of basic telemetry data, as outlined here. This data will play a crucial role in helping Airflow maintainers gain a deeper understanding of how Airflow is utilized across various deployments. The insights derived from this information are invaluable in guiding the prioritization of patches, minor releases, and security fixes. Moreover, this data will inform key decisions regarding the development roadmap, ensuring that Airflow continues to evolve in line with community needs. For those who prefer not to participate in data collection, deployments can easily opt out by setting the [usage_data_collection] enabled option to False or by using the SCARF_ANALYTICS=false environment variable. Multiple Executor Configuration (formerly "Hybrid Execution") Each executor comes with its unique set of strengths and weaknesses, typically balancing latency, isolation, and compute efficiency. Traditionally, an Airflow environment is limited to a single executor, requiring users to make trade-offs, as no single executor is perfectly suited for all types of tasks. We are introducing a new feature that allows for the concurrent use of multiple executors within a single Airflow environment. This flexibility enables users to take advantage of the specific strengths of different executors for various tasks, improving overall efficiency and mitigating weaknesses. Users can s ## Apache Airflow 2.9.0: Dataset and UI Improvements DevFeed: [Apache Airflow 2.9.0: Dataset and UI Improvements]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2024-04-08T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [dataset](), [datasets](), [releases](), [Release notes](), [ui](), [Python]() Tags: [2-9-0](), [airflow](), [apache-airflow](), [dataset](), [datasets](), [python](), [release](), [release-notes](), [scheduling](), [ui]() ### AI overview Apache Airflow 2.9.0 introduces expanded data-aware scheduling with logical OR and arbitrary AND/OR combinations, plus a timetable combining dataset events with time-based schedules. It also adds dataset event REST API endpoints, dataset UI enhancements, Python 3.12 support with Pendulum 3, and custom names for dynamically mapped tasks. ### Source excerpt I'm happy to announce that Apache Airflow 2.9.0 has been released! This time around we have new features for data-aware scheduling and a bunch of UI-related improvements. Apache Airflow 2.9.0 contains over 550 commits, which include 38 new features, 70 improvements, 31 bug fixes, and 18 documentation changes. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.9.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.9.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.9.0/release_notes.html 🐳 Docker Image: "docker pull apache/airflow:2.9.0" 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.9.0 Airflow 2.9.0 is also the first release that supports Python 3.12. However, Pendulum 2 does not support Python 3.12, so you'll need to use Pendulum 3 if you upgrade to Python 3.12. New data-aware scheduling options Logical operators and conditional expressions for DAG scheduling When Datasets were added in Airflow 2.4, DAGs only had scheduling support for logical AND combinations of Datasets. Simply, you could schedule against more than one Dataset, but a DAG run would only be created once all the Datasets were updated after the last run. Now in Airflow 2.9, we support logical OR and even arbitrary combinations of AND and OR. As an example, you can schedule a DAG whenever dataset_1 or dataset_2 are updated : with DAG(schedule=(dataset_1 | dataset_2), ...): ... You can have arbitrary combinations: with DAG(schedule=((dataset_1 | dataset_2) & dataset_3), ...): ... You can read more about this new functionality in the data-aware scheduling docs. Combining Dataset and Time-Based Schedules Airflow 2.9 comes with a new timetable, DatasetOrTimeSchedule, that allows you to schedule DAGs based on both dataset events and a timetable. Now you have the best of both worlds. For example, to run whenever dataset_1 updates and at midnight UTC: with DAG( schedule=DatasetOrTimeSchedule( timetable=CronTriggerTimetable("0 0 * * *", timezone="UTC"), ## Vulnerability in long deprecated OpenID authentication method in Flask AppBuilder DevFeed: [Vulnerability in long deprecated OpenID authentication method in Flask AppBuilder]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2024-02-26T00:00:00Z Content type: article Language: en Sources: [Apache Airflow Blog]() Topics: [vulnerability](), [Authentication](), [airflow](), [Flask](), [openid](), [deprecated](), [OpenID connect (OIDC)]() Tags: [airflow](), [authentication](), [deprecated](), [flask](), [openid](), [upgrade](), [version](), [vulnerabilities](), [vulnerability]() ### AI overview This article describes a vulnerability in Flask AppBuilder's long-deprecated OpenID authentication method. Users with AUTH_OID configured could be exposed to identity takeover through a forged request and an attacker-controlled OpenID service. The article recommends switching authentication methods or upgrading to Apache Airflow 2.8.2, which uses Flask AppBuilder 4.3.11. ### Source excerpt Vulnerability in long deprecated OpenID authentication method in Flask AppBuilder Recently Islam Rzayev made us aware of a vulnerability in the long deprecated OpenID authentication method in Flask AppBuilder. This vulnerability allowed a malicious user to take over the identity of any Airflow UI user by forging a specially crafted request and implementing their own OpenID service. While this is an old, deprecated and almost not used authentication method, we still took the issue seriously. This issue ONLY affects users who have AUTH_OID set in their webserver_config.py file as AUTH_TYPE. This is a very old and deprecated authentication method that is unlikely to be used by anyone. We would like to advise even the small number of our users that still use this authentication method to take an immediate action and either upgrade to Apache Airflow 2.8.2 or switch to another authentication method (or apply a workaround we provide if they cannot do either of the above immediately). Important to stress, because many of the users might get confused by the name, OpenID is NOT the same as OpenID Connect. Those are completely different protocols and while OpenID Connect (also known as OIDC) is a modern, widely used protocol, OpenID is a legacy protocol that has been deprecated more than 10 years ago and since then has been abandoned by almost everyone in the community, including all services in Flask AppBuilder example services that supported it, so it is highly unlikely someone is still using it. Due to this highly unlikely configuration the Flask AppBuilder CVE is just "Moderate" not "Critical". It affects a very small (if any) number of users and it's not likely to be a target for an attack. However, we still advise our users who still use AUTH_OID to apply remediation. This vulnerability is fixed in Flask Appbuilder 4.3.11 and Apache Airflow 2.8.2 uses that version of Flask Application Builder. We advise users who still use this authentication method to either switch to a ## Apache Airflow 2.8.0 is here DevFeed: [Apache Airflow 2.8.0 is here]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2023-12-15T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Python](), [Amazon S3](), [Azure]() Tags: [apache-airflow](), [azure](), [configure](), [docker](), [logs](), [messages](), [object-storage](), [pypi](), [python](), [release](), [release-notes](), [s3]() ### AI overview Apache Airflow 2.8.0 is a software release that adds experimental object storage abstractions for systems such as S3, GCS, and Azure Blob Storage, integrates messages from other components into task logs, and introduces dataset listener hooks. It also adds extra index URL support for PythonVirtualEnvOperator and its decorator, including caching for additional Python package repositories. ### Source excerpt I am thrilled to announce the release of Apache Airflow 2.8.0, featuring a host of significant enhancements and new features that will greatly benefit our community. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.8.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.8.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.8.0/release_notes.html 🐳 Docker Image: "docker pull apache/airflow:2.8.0" 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.8.0 Airflow Object Storage (AIP-58) This feature is experimental and subject to change. Airflow now offers a generic abstraction layer over various object stores like S3, GCS, and Azure Blob Storage, enabling the use of different storage systems in DAGs without code modification. In addition, it allows you to use most of the standard Python modules, like shutil, that can work with file-like objects. Here is an example of how to use the new feature to open a file: from airflow.io.path import ObjectStoragePath base = ObjectStoragePath("s3://my-bucket/", conn_id="aws_default") # conn_id is optional @task def read_file(path: ObjectStoragePath) -> str: with path.open() as f: return f.read() The above example is just the tip of the iceberg. The new feature allows you to configure an alternative backend for a scheme or protocol. Here is an example of how to configure a custom backend for the dbfs scheme: from airflow.io.path import ObjectStoragePath from airflow.io.store import attach from fsspec.implementations.dbfs import DBFSFileSystem attach(protocol="dbfs", fs=DBFSFileSystem(instance="myinstance", token="mytoken")) base = ObjectStoragePath("dbfs://my-location/") For more information: Airflow Object Storage The support for a specific object storage system depends on the installed providers, with out-of-the-box support for the file scheme. Ship logs from other components to Task logs This feature seamlessly integrates task-related messages from various Airflow components, includi ## Airflow Survey 2023 DevFeed: [Airflow Survey 2023]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2023-09-21T00:00:00Z Content type: article Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [data-engineering]() Tags: [airflow](), [apache-airflow](), [community](), [data](), [data-engineering](), [survey](), [users]() ### AI overview The article presents the Apache Airflow Survey 2023, an annual survey intended to provide insights into Airflow usage and guide future efforts. The supplied text reports conflicting figures for participation: more than 5,818 responses from 122 countries and more than 5,250 responses from 116 countries. ### Source excerpt View Raw Data ## Apache Airflow 2.7.0 is here DevFeed: [Apache Airflow 2.7.0 is here]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2023-08-18T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [Security](), [Release notes](), [Python 3.8](), [Docker Image]() Tags: [apache-airflow](), [docker](), [python](), [python-3-8](), [release](), [release-notes](), [security]() ### AI overview Apache Airflow 2.7.0 has been released with new features, improvements, bug fixes, and documentation changes. The release focuses on security, drops support for end-of-life Python 3.7, and adds capabilities including setup and teardown tasks, a Cluster Activity UI, configurable deferrable tasks, and built-in OpenLineage integration. ### Source excerpt I'm happy to announce that Apache Airflow 2.7.0 has been released! Some notable features have been added that we are excited for the community to use. Apache Airflow 2.7.0 contains over 500 commits, which include 40 new features, 49 improvements, 53 bug fixes, and 15 documentation changes. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.7.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.7.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.7.0/release_notes.html 🐳 Docker Image: "docker pull apache/airflow:2.7.0" 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.7.0 Airflow 2.7.0 is a release that focuses on security. The Airflow security team, working together with security researchers, identified a number of areas that required strengthening of security. This resulted in, among others things, an improved description of the Airflow security model, a better explanation of our security policy and the disabling of certain, potentially dangerous, features by default - like, for example, connection testing (#32052). Airflow 2.7.0 is also the first release that drops support for end-of-life Python 3.7. This allows Airflow users and maintainers to make use of features and improvements in Python 3.8, and unlocks newer versions of our dependencies. Setup and Teardown (AIP-52) Airflow now has first class support for the concept of setup and teardown tasks. These tasks have special behavior in that: Teardown tasks will still run, no matter what state the upstream tasks end up in Teardown tasks failing won't, by default, cause the DAG run to fail Automatically clear setup/teardown tasks when clearing a dependent task You can read more about setup and teardown in the Introducing Setup and Teardown tasks blog post, or in the setup and teardown docs. Cluster Activity UI There is a new top level page in Airflow, the Cluster Activity page. This gives an overview of the cluster, including component health, dag and task state co ## Introducing Setup and Teardown tasks DevFeed: [Introducing Setup and Teardown tasks]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2023-08-18T00:00:00Z Content type: article Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Data pipelines](), [GPU]() Tags: [airflow](), [blog-post](), [cleanup](), [data-pipelines](), [dependencies](), [setup](), [tasks]() ### AI overview This article introduces setup and teardown tasks in Airflow 2.7 for managing infrastructure around work in data pipelines. It explains their dependency semantics, cleanup behavior, DAG run state handling, and behavior within task groups. ### Source excerpt In data pipelines, commonly we need to create infrastructure resources, like a cluster or GPU nodes in an existing cluster, before doing the actual "work" and delete them after the work is done. Airflow 2.7 adds "setup" and "teardown" tasks to better support this type of pipeline. This blog post aims to highlight the key features so you know what's possible. For full documentation on how to use setup and teardown tasks, see the setup and teardown docs. Why setup and teardown? Before we dig into examples, let me state at high level what setup and teardown bring to the table. More expressive dependencies Before setup and teardown, upstream and downstream relationships could only mean one thing: "this comes before that". With setup and teardown, in effect we can say "this requires that". And what it means in practice is, if you clear your task, and it requires a setup, that setup will be cleared too. And if that setup has a teardown, that will run again as well. Separating the work from the infra Sometimes the part of the dag you care about is not, say, the cleanup task. For example, suppose you have a dag that loads some data and then deletes temp files. As long as the data loads, you want your dag to be marked successful. By default, this is how teardown tasks work; that is, they are ignored when determining dag run state. Simple case A simple example is one setup / teardown pair, and one normal or "work" task. Setups and teardowns are indicated by the up and down arrows, respectively. From that we can see that .create_cluster is a setup task and delete_cluster is a teardown. The link between a setup and a teardown is always dotted to highlight the special relationship. Some things to observe: If create_cluster fails, neither run_query nor delete_cluster will run. If create_cluster succeeds and run_query fails, then delete_cluster will still run. If create_cluster is skipped, run_query and delete_cluster will be skipped By default, if run_query succeeds, and delete_c ## what's new in Apache Airflow 2.6.0 DevFeed: [what's new in Apache Airflow 2.6.0]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2023-04-30T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [Release notes](), [Logging](), [ui](), [notifications](), [configuration]() Tags: [airflow](), [apache-airflow](), [bug](), [bug-fixes](), [docker-image](), [feature](), [features](), [logs](), [notifications](), [pypi](), [release](), [release-notes](), [ui]() ### AI overview Apache Airflow 2.6.0 has been released with 42 new features, 58 improvements, 38 bug fixes, and 17 documentation changes. Notable updates include trigger logs in the webserver, Grid view enhancements, a DAG-parameter trigger form, improved handling of stuck queued tasks, plugin-based cluster policy hooks, and notifications support for Slack. ### Source excerpt I am excited to announce that Apache Airflow 2.6.0 has been released, bringing many minor features and improvements to the community. Apache Airflow 2.6.0 contains over 500 commits, which include 42 new features, 58 improvements, 38 bug fixes, and 17 documentation changes. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.6.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.6.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.6.0/release_notes.html 🐳 Docker Image: "docker pull apache/airflow:2.6.0" 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.6.0 As the changelog is quite large, the following are some notable new features that shipped in this release. Trigger logs can now be viewed in webserver Trigger logs have now been added to task logs. They appear right alongside the rest of the logs from your task. Adding this feature required changes across the entire Airflow logging stack, so be sure to update your providers if you are using remote logging. Grid view improvements The grid view has received a number of minor improvements in this release. Most notably, there is now a graph tab in the grid view. This offers a more integrated graph representation of the DAG, where choosing a task in either the grid or graph will highlight the same task in both views. You can also filter upstream and downstream from a single task. For example, in the screenshot above, describe_integrity is the selected task. If you choose to filter downstream, this is the result: Trigger UI based on DAG level params A user-friendly form is now shown to users triggering runs for DAGs with DAG level params. See the Params docs for more details. Consolidation of handling stuck queued tasks Airflow now has a single configuration, [scheduler] task_queued_timeout, to handle tasks that get stuck in queued for too long. With a simpler implementation than the outgoing code handling these tasks, tasks stuck in queued will no longer slip through the ## Apache Airflow 2.5.0 Released with Dataset UI, DAG Testing, and Task Log Improvements DevFeed: [Apache Airflow 2.5.0 Released with Dataset UI, DAG Testing, and Task Log Improvements]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2022-12-02T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [dataset](), [debug](), [Docker Image]() Tags: [airflow](), [apache-airflow](), [console](), [dataset](), [debug](), [docker-image](), [release](), [release-notes](), [releases]() ### AI overview Apache Airflow 2.5.0 introduces usability improvements to the Datasets UI, a faster and easier-to-debug airflow dags test command, automatic task-log tailing in Grid view, and further improvements to Dynamic Task Mapping. The release also marks a shorter release cadence than previously used. ### Source excerpt Apache Airflow 2.5 has just been released, barely two and a half months after 2.4! Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.5.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.5.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.5.0/release_notes.html 🐳 Docker Image: docker pull apache/airflow:2.5.0 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.5.0 This quicker release cadence is a departure from our previous habit of releasing every five-to-seven months and was a deliberate effort to listen to you, our users, and get the changes and improvements into your workflows earlier. Usability improvements to the Datasets UI When we released Dataset aware scheduling in September we knew that the tools we gave to manage the Datasets were very much a Minimum Viable Product, and in the last two months the committers and contributors have been hard at work at making the UI much more usable when it comes to Datasets. But we aren't done yet - keep an eye out for more improvements coming over the next couple of releases too. Greatly improved airflow dags test command This airflow subcommand has been rethought and re-optimized to make it much easier to test your DAGs locally - the major changes are: a. Task logs are visible right there in the console, instead of hidden away inside the task log files b. It is about an order of magnitude quicker to run the tasks than before (i.e. it gets to running the task code so much quicker) c. Everything runs in one process, so you can put a breakpoint in your IDE, and configure it to run airflow dags test then debug code! Auto tailing task logs in the Grid view Hopefully the headline says enough. It's lovely, go check it out. More improvements to Dynamic-Task mapping In a similar vein to the improvements to the Dataset (UI), we have continued to iterate on and improve the feature we first added in Airflow 2.3, Dynamic Task Mapping, and 2.5 includes dozens of improvement ## Apache Airflow 2.4.0: That Data Aware Release DevFeed: [Apache Airflow 2.4.0: That Data Aware Release]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2022-09-19T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [datasets](), [releases]() Tags: [apache-airflow](), [data](), [datasets](), [new-features](), [release]() ### AI overview Apache Airflow 2.4.0 is a release with more than 650 user-facing commits, including new features, improvements, bug fixes, and documentation changes. Its main feature is data-aware scheduling, which allows DAGs to be scheduled when tasks update datasets and supports smaller, connected workflows. The release also introduces ExternalPythonOperator for running tasks in preconfigured virtual environments to manage conflicting Python dependencies. ### Source excerpt Apache Airflow 2.4.0 contains over 650 "user-facing" commits (excluding commits to providers or chart) and over 870 total. That includes 46 new features, 39 improvements, 52 bug fixes, and several documentation changes. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html 🐳 Docker Image: docker pull apache/airflow:2.4.0 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0 Data-aware scheduling (AIP-48) This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets. What does this mean, exactly? This is a great new feature that lets DAG authors create smaller, more self-contained DAGs, which chain together into a larger data-based workflow. If you are currently using ExternalTaskSensor or TriggerDagRunOperator you should take a look at datasets - in most cases you can replace them with something that will speed up the scheduling! But enough talking, lets have a short example. First lets write a simple DAG with a task called my_task that produces a dataset called my-dataset: from airflow import Dataset dataset = Dataset(uri='my-dataset') with DAG(dag_id='producer', ...) @task(outlets=[dataset]) def my_task(): ... Datasets are defined by a URI. Now, we can create a second DAG (consumer) that gets scheduled whenever this dataset changes: from airflow import Dataset dataset = Dataset(uri='my-dataset') with DAG(dag_id='dataset-consumer', schedule=[dataset]): ... With these two DAGs, the instant my_task finishes, Airflow will create the DAG run for the dataset-consumer workflow. We know that what exists right now won't fit all use cases that people might wish for datasets, and in the coming minor releases (2.5, 2.6, etc.) we will expand and improve upon this foundation. Datasets represent the abstract concept of a dataset, and (for now) do not have any dire ## Airflow Survey 2022 DevFeed: [Airflow Survey 2022]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2022-06-17T00:00:00Z Content type: article Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [data](), [Documentation](), [migration](), [Monitoring](), [Stack Overflow]() Tags: [airflow](), [apache-airflow](), [community](), [contributions](), [data](), [documentation](), [migration](), [monitoring](), [results](), [stack-overflow](), [survey](), [users]() ### AI overview The 2022 Airflow User Survey collected 210 responses over two weeks. Respondents were primarily data engineers, and Airflow was widely used in larger companies. Most participants were willing to recommend Apache Airflow, documentation and Stack Overflow were important information sources, and most deployments used Airflow 2. Migration from Airflow 1 was mainly constrained by resources and capacity. ### Source excerpt Airflow User Survey 2022 This year's survey has come and gone, and with it we've got a new batch of data for everyone! We collected 210 responses over two weeks. We continue to see growth in both contributions and downloads over the last two years, and expect that trend will continue through 2022. The raw response data will be made available here soon, in the meantime, feel free to email john.thomas@astronomer.io for a copy. TL;DR Overview of the user Like previous years, more than half of the Airflow users are Data Engineers (54%). Solutions Architects (13%), Developers (12%), DevOps (6%) and Data Scientists (4%) are also active Airflow users! There was a slight increase in the representation of Solutions Architect roles compared to results from 2020 and 2019 . Airflow is used and popular in bigger companies, 64% of Airflow users work for companies with 200+ employees which is an 11 percent increase compared to 2020. 62% of the survey participants have more than 6 Airflow users in their company. More Airflow users (65.9%) are willing to recommend Apache Airflow compared to the survey results in 2020 and 2019. There is a general positive trend in a willingness to recommend Airflow, 93% of surveyed Airflow users are willing to recommend Airflow ( 85.7% in 2019 and 92% in 2020 ), only 1% of users are not likely to recommend (3.6% in 2019 and 3.5% in 2020). Airflow documentation is a critical source of information, with more than 90% (15% increase compared to results from 2020) of survey participants using the documentation. Airflow documentation is also one of the top areas to improve! What's interesting, also Stack Overflow usage is critical, with about 60% users declaring to use it as a source of information (24% increase compared to results from 2020). Deployments 85% of the Airflow users have between 1 and 7 active Airflow instances. 62.5% of the Airflow users have between 11 and 250 DAGs in their largest Airflow instance. 75% of the surveyed Airflow users have be ## Airflow Summit 2022 DevFeed: [Airflow Summit 2022]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2022-05-16T00:00:00Z Content type: news Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Data pipelines](), [data-governance](), [Open Source](), [AI, ML & Data Engineering]() Tags: [airflow-summit](), [apache-airflow](), [community](), [data-governance](), [data-pipelines](), [events](), [machine-learning](), [network](), [open-source](), [summit]() ### AI overview Airflow Summit 2022 was scheduled for May 23-27 as a free conference for Apache Airflow practitioners and data leaders. The program covered Airflow practices, data pipelines, data governance, machine learning, the project's future, and non-code open-source contributions. ### Source excerpt The biggest Airflow Event of the Year returns May 23-27! Airflow Summit 2022 will bring together the global community of Apache Airflow practitioners and data leaders. What's on the Agenda During the free conference, you will hear about Apache Airflow best practices, trends in building data pipelines, data governance, Airflow and machine learning, and the future of Airflow. There will also be a series of presentations on non-code contributions driving the open-source project. How to Attend This year's edition will include a variety of online sessions across different time zones. Additionally, you can take part in local in-person events organized worldwide for data communities to watch the event and network. Interested? 🪶 Register for Airflow Summit 2022 today 🤝 Check out the in-person events planned for Airflow Summit 2022. ## Apache Airflow 2.3.0 is here DevFeed: [Apache Airflow 2.3.0 is here]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2022-04-30T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [Release notes](), [Command-line interface](), [Databases](), [Code](), [JSON](), [Docker]() Tags: [airflow](), [apache-airflow](), [bug-fixes](), [cli](), [database](), [docker](), [json](), [new-features](), [release](), [release-notes]() ### AI overview Apache Airflow 2.3.0 is a release containing over 700 commits since version 2.2.0, with 50 new features, 99 improvements, 85 bug fixes, and documentation changes. Highlighted features include Dynamic Task Mapping, a Grid View, metadata database cleanup, LocalKubernetesExecutor, standalone DAG processing, and JSON connection serialization. ### Source excerpt Apache Airflow 2.3.0 contains over 700 commits since 2.2.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.3.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.3.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.3.0/release_notes.html 🐳 Docker Image: docker pull apache/airflow:2.3.0 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.3.0 As the changelog is quite large, the following are some notable new features that shipped in this release. Dynamic Task Mapping(AIP-42) There's now first-class support for dynamic tasks in Airflow. What this means is that you can generate tasks dynamically at runtime. Much like using a for loop to create a list of tasks, here you can create the same tasks without having to know the exact number of tasks ahead of time. You can have a task generate the list to iterate over, which is not possible with a for loop. Here is an example: @task def make_list(): # This can also be from an API call, checking a database, -- almost anything you like, as long as the # resulting list/dictionary can be stored in the current XCom backend. return [1, 2, {"a": "b"}, "str"] @task def consumer(arg): print(list(arg)) with DAG(dag_id="dynamic-map", start_date=datetime(2022, 4, 2)) as dag: consumer.expand(arg=make_list()) More information can be found here: Dynamic Task Mapping Grid View replaces Tree View Grid view replaces tree view in Airflow 2.3.0. Screenshots: Purge history from metadata database Airflow 2.3.0 introduces a new airflow db clean command that can be used to purge old data from the metadata database. You would want to use this command if you want to reduce the size of the metadata database. More information can be found here: Purge history from metadata database LocalKubernetesExecutor There is a new executor named LocalKubernetesExecutor. This executor helps you run some tasks using LocalExecutor a [Next page]()