# airflow

Platform for programmatically authoring, scheduling, and monitoring workflows.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Airflow 3's Control Plane Shifts Resilience Risks to the API, UI, and Metadata Database

DevFeed: [Airflow 3's Control Plane Shifts Resilience Risks to the API, UI, and Metadata Database](<https://devfeed.tech/articles/airflow-3-is-not-a-scheduler-anymore-65413.md>)

Original publisher: [Read original article](<https://blog.devops.dev/airflow-3-is-not-a-scheduler-anymore-c7b5a16c7cab?source=rss----33f8b2d9a328---4>)

Author: Sendoa Moronta

Published: 2026-10-06T13:05:06Z

Content type: article

Language: en

Sources: [DevOps.dev](<https://devfeed.tech/sources/devops-dev.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [control-plane](<https://devfeed.tech/topics/control-plane.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [REST API](<https://devfeed.tech/topics/rest-api.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [availability](<https://devfeed.tech/tags/availability.md>), [control-plane](<https://devfeed.tech/tags/control-plane.md>), [data-architecture](<https://devfeed.tech/tags/data-architecture.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-orchestration](<https://devfeed.tech/tags/data-orchestration.md>), [data-platforms](<https://devfeed.tech/tags/data-platforms.md>), [database](<https://devfeed.tech/tags/database.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [high-availability](<https://devfeed.tech/tags/high-availability.md>), [rest-api](<https://devfeed.tech/tags/rest-api.md>), [retries](<https://devfeed.tech/tags/retries.md>), [robustness](<https://devfeed.tech/tags/robustness.md>), [scaling](<https://devfeed.tech/tags/scaling.md>)

### AI overview

The article argues that Airflow 3 should be operated as a distributed control plane, with resilience risks centered on its API, Web UI, and metadata database as well as the scheduler and execution plane. It describes how UI query load and API latency can contribute to database contention, scheduler lag, duplicate operations, and cascading failures, and recommends measures such as read replicas, graceful UI degradation, rate limiting, idempotency keys, and monitoring control-plane health.

### Source excerpt

Designing a Resilient Control Plane: WebUI, API, and the Hidden Failure Modes Nobody Talks About Airflow has always been introduced the same way: a platform to programmatically author, schedule, and monitor workflows. That description was never wrong. But it is increasingly incomplete. With the evolution towards Airflow 3, something more fundamental is happening not just new features, not just API improvements, but a shift in what Airflow actually is. Airflow is no longer just a scheduler. It is a distributed control plane. And once you see it that way, a lot of uncomfortable truths emerge -- especially around resilience, failure modes, and the often ignored components: the Web UI and the API. The Misconception: "The Scheduler Is the System" Most production discussions around Airflow resilience revolve around: Scheduler high availability Executor scaling (Celery, Kubernetes, etc.) Task retries and DAG robustness These are important -- but they focus almost entirely on the execution plane. What gets overlooked is the control plane, which includes: The Web UI The REST API The Scheduler (as a state coordinator, not just a trigger engine) The Metadata Database In practice, this control plane is what operators, systems, and automation actually interact with. And in Airflow 3, this becomes even more explicit: The API is no longer secondary -- it is a first-class interface The UI is no longer just for visualization -- it is an operational dependency External systems increasingly interact with Airflow programmatically So the real question becomes: What happens when the control plane itself is degraded? Airflow 3 as a Distributed Control Plane Thinking in distributed systems terms helps clarify the shift. Airflow now resembles systems like Kubernetes more than traditional schedulers: | Component | Role | | ----------------- | ------------------------------------------ | | Scheduler | State reconciliation + orchestration logic | | Metadata DB | Source of truth | | API | Control s

## The official ClickHouse provider for Apache Airflow is now available

DevFeed: [The official ClickHouse provider for Apache Airflow is now available](<https://devfeed.tech/articles/the-official-clickhouse-provider-for-apache-airflow-is-now-available-42157.md>)

Original publisher: [Read original article](<https://clickhouse.com/blog/clickhouse-airflow-provider>)

Author: Aditya Chidurala; Bentsi Leviav; Alex Francoeur

Published: 2026-09-17T18:06:01Z

Content type: release

Language: en

Sources: [ClickHouse Blog](<https://devfeed.tech/sources/clickhouse-blog.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [clickhouse](<https://devfeed.tech/topics/clickhouse.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [SQL](<https://devfeed.tech/topics/sql.md>), [Python](<https://devfeed.tech/topics/python.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [integration](<https://devfeed.tech/tags/integration.md>), [python](<https://devfeed.tech/tags/python.md>), [release](<https://devfeed.tech/tags/release.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

ClickHouse has released an officially maintained Apache Airflow provider for orchestrating ClickHouse data workflows. The provider uses ClickHouse Connect over HTTP(S), supports Airflow's common SQL operators, and includes a hook for bulk and client-specific operations.

### Source excerpt

The official ClickHouse provider for Apache Airflow simplifies data workflows with standard SQL operators, bulk inserts, and shared setup across self-managed Airflow and Astronomer.

## How dbt works, and why orchestrators shouldn't split it into tasks

DevFeed: [How dbt works, and why orchestrators shouldn't split it into tasks](<https://devfeed.tech/articles/how-dbt-works-and-why-orchestrators-shouldn-t-split-it-into-tasks-30714.md>)

Original publisher: [Read original article](<https://www.windmill.dev/blog/how-dbt-works-and-its-orchestrators>)

Author: Ruben Fiszel

Published: 2026-08-20T00:00:00Z

Content type: article

Language: en

Sources: [Windmill Blog](<https://devfeed.tech/sources/windmill-blog.md>)

Topics: [Compiler](<https://devfeed.tech/topics/compiler.md>), [Job](<https://devfeed.tech/topics/job.md>), [airflow](<https://devfeed.tech/topics/airflow.md>), [data](<https://devfeed.tech/topics/data.md>), [Command-line interface](<https://devfeed.tech/topics/cli.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [cli](<https://devfeed.tech/tags/cli.md>), [dagster](<https://devfeed.tech/tags/dagster.md>), [data](<https://devfeed.tech/tags/data.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [dbt-data-pipelines-airflow-dagster-orchestration](<https://devfeed.tech/tags/dbt-data-pipelines-airflow-dagster-orchestration.md>), [job](<https://devfeed.tech/tags/job.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [scheduler](<https://devfeed.tech/tags/scheduler.md>), [series](<https://devfeed.tech/tags/series.md>)

### AI overview

This primer explains dbt's architecture and examines five ways teams orchestrate it in production. It argues that running a project as one dbt command, while reading dbt's per-model execution state, is generally more efficient and less fragile than creating one orchestrator task per model. It discusses Dagster and astronomer-cosmos's convergence on this approach, including a reported cost comparison.

### Source excerpt

How is dbt actually orchestrated, and why does running it as one job beat one task per model? A primer on dbt as a compiler with a scheduler attached, the five ways teams wrap it, and why both Dagster and astronomer-cosmos converged on a single dbt invocation with per-model state projected out of it.

## Quasi-Agentic Pipelines with Databricks and Apache Airflow

DevFeed: [Quasi-Agentic Pipelines with Databricks and Apache Airflow](<https://devfeed.tech/articles/quasi-agentic-pipelines-with-databricks-and-apache-airflow-38713.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/quasi-agentic-pipelines-with-databricks>)

Author: Daniel Beach

Published: 2026-08-10T21:23:57Z

Content type: tutorial

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [airflow](<https://devfeed.tech/topics/airflow.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [llms](<https://devfeed.tech/tags/llms.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>)

### AI overview

A practical developer discussion of incorporating LLMs and agents into existing data workflows using Databricks and Apache Airflow. It also examines determinism in data pipelines and the gap between business requirements and engineering implementation.

### Source excerpt

the strange space in between

## Migrating from a Monolithic Orchestrator to Apache Airflow

DevFeed: [Migrating from a Monolithic Orchestrator to Apache Airflow](<https://devfeed.tech/articles/migrating-from-a-monolithic-orchestrator-to-apache-airflow-30518.md>)

Original publisher: [Read original article](<https://medium.com/helpshift-engineering/migrating-from-a-monolithic-orchestrator-to-apache-airflow-30fde94bcdc0?source=rss----3229f31ca4f4---4>)

Author: Poorva Patil

Published: 2026-06-03T09:20:17Z

Content type: article

Language: en

Sources: [Helpshift](<https://devfeed.tech/sources/helpshift.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>), [monolithic-architecture](<https://devfeed.tech/topics/monolithic-architecture.md>), [observability](<https://devfeed.tech/topics/observability.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [clojure](<https://devfeed.tech/tags/clojure.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [emr](<https://devfeed.tech/tags/emr.md>), [etl](<https://devfeed.tech/tags/etl.md>), [event](<https://devfeed.tech/tags/event.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [scheduler](<https://devfeed.tech/tags/scheduler.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>), [software-architecture](<https://devfeed.tech/tags/software-architecture.md>), [workflow](<https://devfeed.tech/tags/workflow.md>)

### AI overview

The team replaced a tightly coupled Clojure scheduler, which bundled pipelines into one EMR step and lacked step-level monitoring, with Apache Airflow. They mapped pipelines to DAGs, steps to tasks or operators, and dependencies to explicit DAG relationships. Airflow coordinates scheduling, retries, monitoring, alerts, and recovery, while Spark, Hive, and Presto workloads run on transient EMR clusters; EMR sensors track step status and Airflow can link task instances to logs in S3 or CloudWatch. The migration also separated orchestration from business logic and proceeded incrementally: low-risk workflows first, parallel runs and comparisons of correctness, runtime, resource use, completion, and failures, then gradual migration of critical pipelines. The excerpt describes these improvements but provides no numerical success metrics.

### Source excerpt

Photo by Corinne Kutz on UnsplashBefore we knew better Our orchestration system started as a simple internal solution to manage event pipelines and trigger downstream jobs. Over time, as more workflows and dependencies were added, it gradually evolved into a tightly coupled monolithic scheduler that became increasingly difficult to understand and maintain. Understanding how a workflow executed often meant looking through multiple files, configurations and database tables. For newer team members, onboarding into the system took time because much of the workflow context was distributed across different parts of the codebase. Even relatively small changes required careful coordination to ensure existing pipelines continued to work as expected. Similarly, debugging typically involved manually tracing logs and rerunning jobs to better understand execution behavior. Limitations of our legacy design We had a monolithic architecture written in Clojure that bundled all our event pipelines together, added dependencies between them and triggered a Lambda function. Legacy Workflow This Lambda function added a single monolithic step to the EMR cluster. If there was an issue in any one of the pipelines, the entire flow would fail due to the single step on the cluster. We did not have step-wise monitoring in the old design, so during on-call situations it became very difficult to identify which part of the pipeline was causing the issue. Photo by Tim Gouw on Unsplash There was no single place to answer basic questions like: What runs first? What happens if this step fails? How do I re-run just one part safely? The scheduler worked, but it was hard to understand, hard to maintain and even harder to explain. That's when we realized we needed a better way. What we actually needed Our aim was less about fancy scheduling features and more about making our daily work easier and more reliable. Simpler onboarding, less mental overhead Our existing step scheduler was built in Clojure and c

## From SSH to REST: A Security-Driven Modernization of Slack's EMR Data Pipelines

DevFeed: [From SSH to REST: A Security-Driven Modernization of Slack's EMR Data Pipelines](<https://devfeed.tech/articles/from-ssh-to-rest-a-security-driven-modernization-of-slack-s-emr-data-pipelines-146.md>)

Original publisher: [Read original article](<https://slack.engineering/from-ssh-to-rest-a-security-driven-modernization-of-slacks-emr-data-pipelines/>)

Author: Mahendran Vasagam

Published: 2026-05-05T14:00:01Z

Content type: article

Language: en

Sources: [Engineering at Slack](<https://devfeed.tech/sources/engineering-at-slack.md>)

Topics: [Security](<https://devfeed.tech/topics/security.md>), [OpenSSH](<https://devfeed.tech/topics/openssh.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [airflow](<https://devfeed.tech/topics/airflow.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [Kubernetes](<https://devfeed.tech/topics/kubernetes.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [data-analytics](<https://devfeed.tech/tags/data-analytics.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-infrastructure](<https://devfeed.tech/tags/data-infrastructure.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [modernization](<https://devfeed.tech/tags/modernization.md>), [security](<https://devfeed.tech/tags/security.md>), [ssh](<https://devfeed.tech/tags/ssh.md>), [uncategorized](<https://devfeed.tech/tags/uncategorized.md>)

### AI overview

Slack describes migrating more than 700 SSH-based data pipeline jobs to a REST-based architecture across eight data regions, eliminating SSH access to production AWS EMR clusters without downtime. The article explains the security and operational problems that motivated the modernization, including attack-surface exposure, key-management overhead, resource contention, broken connections, zombie jobs, and unreliable job-status detection.

### Source excerpt

Excerpt By 2024, Slack's data platform had accumulated 700+ SSH-based operators orchestrating critical data pipelines. We're talking daily search indexing that processed terabytes of data, analytics jobs powering business intelligence, the whole shebang. Every single one of these jobs required direct SSH access to production AWS Elastic MapReduce (EMR) clusters. We had a massive security...

## Agentic Workloads on Airflow: Observable, Retryable, and Auditable by Design

DevFeed: [Agentic Workloads on Airflow: Observable, Retryable, and Auditable by Design](<https://devfeed.tech/articles/agentic-workloads-on-airflow-observable-retryable-and-auditable-by-design-32526.md>)

Original publisher: [Read original article](<https://airflow.apache.org/blog/agentic-workloads-airflow-3/>)

Author: Apache Airflow

Published: 2026-04-15T00:00:00Z

Content type: tutorial

Language: en

Sources: [Apache Airflow Blog](<https://devfeed.tech/sources/apache-airflow-blog.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [SQL](<https://devfeed.tech/topics/sql.md>), [dataset](<https://devfeed.tech/topics/dataset.md>), [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [ai](<https://devfeed.tech/tags/ai.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [community](<https://devfeed.tech/tags/community.md>), [csv](<https://devfeed.tech/tags/csv.md>), [llm](<https://devfeed.tech/tags/llm.md>), [sql](<https://devfeed.tech/tags/sql.md>), [survey](<https://devfeed.tech/tags/survey.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>), [workflow](<https://devfeed.tech/tags/workflow.md>)

### AI overview

This article explains how to build agentic workloads on Airflow 3 using the 2025 Airflow Community Survey dataset and the apache-airflow-providers-common-ai provider. It distinguishes single-query LLM pipelines from workflows that query multiple independent dimensions and synthesize their results, mapping each sub-query to a named Airflow task and using Dynamic Task Mapping and XCom for orchestration and observability.

### Source excerpt

A question like "How does AI tool usage vary across Airflow versions?" has a natural SQL shape: one cross-tabulation, one result. A question like "What does a typical Airflow deployment look like for practitioners who are actively using AI in their workflow?" does not. It requires querying executor type, deployment method, cloud provider, and Airflow version independently, each filtered to the same respondent group, then synthesizing the results into a coherent picture. No single query returns the answer. The answer emerges from the relationship between all of them. This is where Airflow's agentic pattern begins: not when you add an LLM to a workflow, but when the structure of the work itself depends on running multiple LLM calls whose outputs feed a synthesis step. This post builds that pattern using the 2025 Airflow Community Survey data set and the apache-airflow-providers-common-ai provider for Airflow 3. If you haven't read the introductory survey analysis post yet, start there for a walkthrough of the single-query interactive and scheduled pipelines. This post picks up where that one ends. The Agentic Gap in the Single-Query Pattern The interactive and scheduled survey DAGs from the introductory post each do one thing: translate a natural language question into SQL, execute it against the CSV, and return the result. The LLM is involved once. The structure of the pipeline does not change based on what that LLM call returns. That is not a limitation to fix. It is the right design for that class of question. For a large fraction of production AI workflows, a single well-structured LLM call with good context is sufficient and preferable. The pattern becomes agentic when two things are true simultaneously: The question requires querying multiple independent dimensions The synthesis step, the thing that produces the final answer, depends on all of those results In an agent harness framework, this would be handled inside a reasoning loop: the LLM decides to call a to

## Ask Your Survey Anything: Building AI Analysis Pipelines with Airflow 3

DevFeed: [Ask Your Survey Anything: Building AI Analysis Pipelines with Airflow 3](<https://devfeed.tech/articles/ask-your-survey-anything-building-ai-analysis-pipelines-with-airflow-3-32527.md>)

Original publisher: [Read original article](<https://airflow.apache.org/blog/ai-survey-analysis-pipelines/>)

Author: Apache Airflow

Published: 2026-04-15T00:00:00Z

Content type: tutorial

Language: en

Sources: [Apache Airflow Blog](<https://devfeed.tech/sources/apache-airflow-blog.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [SQL](<https://devfeed.tech/topics/sql.md>), [CSV](<https://devfeed.tech/topics/csv.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [automated](<https://devfeed.tech/tags/automated.md>), [community](<https://devfeed.tech/tags/community.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [csv](<https://devfeed.tech/tags/csv.md>), [executor](<https://devfeed.tech/tags/executor.md>), [llm](<https://devfeed.tech/tags/llm.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [providers](<https://devfeed.tech/tags/providers.md>), [scheduled](<https://devfeed.tech/tags/scheduled.md>), [sql](<https://devfeed.tech/tags/sql.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>)

### AI overview

A tutorial showing how to build interactive and scheduled AI analysis pipelines with Airflow 3. The pipelines use survey CSV data, generate and run SQL queries, validate schemas, and either request human approval or email results automatically.

### Source excerpt

The 2025 Airflow Community Survey collected responses from nearly 6,000 practitioners across 168 questions. You can open a spreadsheet and filter, or write SQL by hand. But what if you could just ask a question and have Airflow figure out the query, run it, and bring the result back for your approval? This post builds two pipelines that do exactly that, using the apache-airflow-providers-common-ai provider for Airflow 3. The first pipeline is interactive: a human reviews the question before it reaches the LLM and approves the result before the DAG finishes. The second is scheduled: it downloads fresh survey data, validates the schema, runs the query unattended, and emails the result. If you haven't seen the common.ai provider overview yet, start there for a tour of all the operators. This post goes deep on a concrete end-to-end example. Two Pipelines, One Example File Both DAGs live in example_llm_survey_analysis.py and share the same schema context and datasource configuration. example_llm_survey_interactive: trigger manually, review at both ends: prompt_confirmation -> generate_sql -> run_query -> extract_data -> result_confirmation (HITLEntryOperator) (LLMSQLQuery) (Analytics) (@task) (ApprovalOperator) example_llm_survey_scheduled: runs @monthly, no human in the loop: download_survey -> prepare_csv -> check_schema -> generate_sql -> run_query -> extract_data -> send_result (HttpOperator) (@task) (LLMSchema (LLMSQLQuery) (Analytics) (@task) (@task / Email) Compare) The Data The Airflow Community Survey 2025 CSV has 5,856 rows and 168 columns covering everything from Airflow version and executor type to cloud provider, company size, and AI tool usage. A few highlights from the data: 3,320 respondents identify as Data Engineers 2,032 use AWS as their primary cloud provider for Airflow 1,445 are already running Airflow 3 1,351 say they often use AI tools to write Airflow code Those last two numbers together are part of why this example exists: the people most likely to use th

## Introducing the Common AI Provider: LLM and AI Agent Support for Apache Airflow

DevFeed: [Introducing the Common AI Provider: LLM and AI Agent Support for Apache Airflow](<https://devfeed.tech/articles/introducing-the-common-ai-provider-llm-and-ai-agent-support-for-apache-airflow-32558.md>)

Original publisher: [Read original article](<https://airflow.apache.org/blog/common-ai-provider/>)

Author: Apache Airflow

Published: 2026-04-14T00:00:00Z

Content type: release

Language: en

Sources: [Apache Airflow Blog](<https://devfeed.tech/sources/apache-airflow-blog.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Pydantic](<https://devfeed.tech/topics/pydantic.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [2025](<https://devfeed.tech/tags/2025.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [community](<https://devfeed.tech/tags/community.md>), [llm](<https://devfeed.tech/tags/llm.md>), [release](<https://devfeed.tech/tags/release.md>)

### AI overview

Apache Airflow is releasing the Common AI Provider 0.1.0, a provider package that adds LLM and AI agent capabilities directly to Airflow. Built on Pydantic AI, it supports more than 20 model providers through a single install and requires Apache Airflow 3.0 or newer.

### Source excerpt

At Airflow Summit 2025, we previewed what native AI integration in Apache Airflow could look like. Today we're shipping it. apache-airflow-providers-common-ai 0.1.0 adds LLM and agent capabilities directly to Airflow. Not a wrapper around another framework, but a provider package that plugs into the orchestrator you already run. It's built on Pydantic AI and supports 20+ model providers (OpenAI, Anthropic, Google, Azure, Bedrock, Ollama, and more) through a single install. pip install 'apache-airflow-providers-common-ai' Requires Apache Airflow 3.0+. Note: This is a 0.x release. We're actively looking for feedback and iterating fast, so breaking changes are possible between minor versions. Try it, tell us what works and what doesn't. Your input directly shapes the API. By the Numbers 6 Operators 6 TaskFlow decorators 5 Toolsets 4 Connection types 20+ Supported model providers via Pydantic AI The Decorator Suite Every operator has a matching TaskFlow decorator. @task.llm: Single LLM Call Send a prompt, get text or structured output back. from pydantic import BaseModel from airflow.providers.common.compat.sdk import dag, task @dag def my_pipeline(): class Entities(BaseModel): names: list[str] locations: list[str] @task.llm( llm_conn_id="my_openai_conn", system_prompt="Extract named entities.", output_type=Entities, ) def extract(text: str): return f"Extract entities from: {text}" extract("Alice visited Paris and met Bob in London.") my_pipeline() The LLM returns a typed Entities object, not a string you have to parse. Downstream tasks get structured data through XCom. @task.agent: Multi-Step Agent with Tools When the LLM needs to query databases, call APIs, or read files across multiple steps, use @task.agent. The agent picks which tools to call and loops until it has an answer. from airflow.providers.common.ai.toolsets.sql import SQLToolset from airflow.providers.common.compat.sdk import dag, task @dag def sql_analyst(): @task.agent( llm_conn_id="my_openai_conn", sys

## Apache Airflow 3.2.0: Data-Aware Workflows at Scale

DevFeed: [Apache Airflow 3.2.0: Data-Aware Workflows at Scale](<https://devfeed.tech/articles/apache-airflow-3-2-0-data-aware-workflows-at-scale-32541.md>)

Original publisher: [Read original article](<https://airflow.apache.org/blog/airflow-3.2.0/>)

Author: Apache Airflow

Published: 2026-04-07T00:00:00Z

Content type: release

Language: en

Sources: [Apache Airflow Blog](<https://devfeed.tech/sources/apache-airflow-blog.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [releases](<https://devfeed.tech/topics/releases.md>), [data](<https://devfeed.tech/topics/data.md>), [Orchestration](<https://devfeed.tech/topics/orchestration.md>), [Release notes](<https://devfeed.tech/topics/release-notes.md>), [SDK](<https://devfeed.tech/topics/sdk.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [data](<https://devfeed.tech/tags/data.md>), [docker-image](<https://devfeed.tech/tags/docker-image.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [pypi](<https://devfeed.tech/tags/pypi.md>), [release](<https://devfeed.tech/tags/release.md>), [release-notes](<https://devfeed.tech/tags/release-notes.md>), [s3](<https://devfeed.tech/tags/s3.md>), [sdk](<https://devfeed.tech/tags/sdk.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

### AI overview

Apache Airflow 3.2.0 introduces asset partitioning for more granular data-aware scheduling, along with multi-team deployments, synchronous deadline alert callbacks, and continued Task SDK separation. Partition-aware workflows trigger downstream DAGs only when the relevant partition is updated.

### Source excerpt

We're proud to announce the release of Apache Airflow 3.2.0! Airflow 3.1 puts humans at the center of automated workflows. 3.2 brings that same precision to data: Asset partitioning for granular pipeline orchestration, multi-team deployments for enterprise scale, synchronous deadline alert callbacks, and continued progress toward full Task SDK separation. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/3.2.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.2.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.2.0/release_notes.html 🐳 Docker Image: docker pull apache/airflow:3.2.0 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.2.0 🗂 Asset Partitioning (AIP-76): Only the Right Work Gets Triggered Asset partitioning has been one of the most requested additions to data-aware scheduling. If you work with date-partitioned S3 paths, Hive table partitions, BigQuery partitions, or really any partitioned data store, you've dealt with this: An upstream task updates one partition, and every downstream Dag fires regardless of which slice actually changed. It's wasteful, and for large deployments it creates real operational noise. Asset partitioning in 3.2 makes this granular. Downstream Dags trigger only when the specific partition they care about gets updated. It's the biggest change to data-aware scheduling since Assets were introduced, and it turns partition-driven orchestration into something Airflow handles natively rather than something you work around. Key Capabilities Partition-driven scheduling: Dags trigger on specific partition updates, not every asset change CronPartitionTimetable: Schedule Dags against partitions using cron expressions. Also available in the Task SDK Backfill for partitioned Dags: Backfill historical partitions without re-triggering everything downstream (#61464) Multi-asset partitions: A single Dag can listen for partitions across multiple assets, which matters when your downstream work depe

## Introducing the Apache Airflow Registry

DevFeed: [Introducing the Apache Airflow Registry](<https://devfeed.tech/articles/introducing-the-apache-airflow-registry-32543.md>)

Original publisher: [Read original article](<https://airflow.apache.org/blog/airflow-registry/>)

Author: Apache Airflow

Published: 2026-03-19T00:00:00Z

Content type: release

Language: en

Sources: [Apache Airflow Blog](<https://devfeed.tech/sources/apache-airflow-blog.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [API](<https://devfeed.tech/topics/api.md>), [JSON](<https://devfeed.tech/topics/json.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [community](<https://devfeed.tech/tags/community.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [messaging](<https://devfeed.tech/tags/messaging.md>), [notifications](<https://devfeed.tech/tags/notifications.md>), [openai](<https://devfeed.tech/tags/openai.md>), [registry](<https://devfeed.tech/tags/registry.md>)

### AI overview

Apache Airflow launches the Airflow Registry, a searchable catalog of official providers and modules. It includes provider and module search, installation and compatibility details, connection generation in URI, JSON, and environment-variable formats, ecosystem statistics, and a structured JSON API.

### Source excerpt

Today we're launching the Apache Airflow Registry -- a searchable catalog of every official Airflow provider and its modules, live at airflow.apache.org/registry/. Need an S3 operator? A Snowflake hook? An OpenAI sensor? The Registry helps you find, compare, and configure the right components for your data pipelines -- without digging through docs or PyPI pages. By the Numbers 98 Official providers 1,602 Modules (operators, hooks, sensors, triggers, transfers, and more) 329M+ Monthly PyPI downloads across all providers 125+ Integrations with cloud platforms, databases, ML tools, and messaging services Search Everything Hit Cmd+K from any page and start typing. Results show up instantly, grouped by Providers and Modules, with type badges so you can tell a hook from an operator at a glance. Provider Pages Each provider gets a dedicated page with everything in one place: install command with copy-to-clipboard, version selector, extras dropdown, compatibility info, connection types, and the full module listing organized by type. The Amazon provider, for example, has 372 modules across operators, hooks, sensors, triggers, transfers, and more. Module type tabs let you filter to exactly what you're looking for, and a category sidebar groups modules by AWS service (S3, Lambda, Glue, Step Functions, etc.). Connection Builder Click any connection type badge on a provider page, fill in the fields, and the builder generates the connection in three formats -- URI, JSON, and Env Var -- ready to copy into your configuration. No more guessing URI encoding or JSON structure. Explore by Category Not sure which provider you need? The Explore page organizes providers into categories: Cloud Platforms, Databases, Data Warehouses, Messaging & Notifications, AI & Machine Learning, Data Processing, and more. Statistics The Stats page breaks down the ecosystem: 848 operators, 298 hooks, 164 triggers, 157 sensors, 83 transfers, and more -- plus top providers by downloads and module count. JSON API

## How Snap Evolved Flowrida from a Monolithic Airflow Cluster to a Resilient Multi-Cluster Platform

DevFeed: [How Snap Evolved Flowrida from a Monolithic Airflow Cluster to a Resilient Multi-Cluster Platform](<https://devfeed.tech/articles/from-monolith-to-multi-cluster-56687.md>)

Original publisher: [Read original article](<https://eng.snap.com/flowrida>)

Published: 2026-03-17T07:00:00Z

Content type: article

Language: en

Sources: [Snap](<https://devfeed.tech/sources/snap.md>)

Topics: [Orchestration](<https://devfeed.tech/topics/orchestration.md>), [airflow](<https://devfeed.tech/topics/airflow.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Scalability](<https://devfeed.tech/topics/scalability.md>), [data](<https://devfeed.tech/topics/data.md>), [data-processing](<https://devfeed.tech/topics/data-processing.md>), [Availability](<https://devfeed.tech/topics/availability.md>), [Kubernetes](<https://devfeed.tech/topics/kubernetes.md>), [Disaster Recovery](<https://devfeed.tech/topics/disaster-recovery.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Database](<https://devfeed.tech/topics/database.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>)

Tags: [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [database](<https://devfeed.tech/tags/database.md>), [disaster-recovery](<https://devfeed.tech/tags/disaster-recovery.md>), [high-availability](<https://devfeed.tech/tags/high-availability.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [multicluster](<https://devfeed.tech/tags/multicluster.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [scale](<https://devfeed.tech/tags/scale.md>)

### AI overview

Snap describes how it evolved Flowrida, an Apache Airflow-based workflow orchestration platform, from a monolithic Kubernetes cluster to a multicluster architecture. The changes addressed availability, disaster recovery, deployment blast radius, recovery time, and scalability challenges as data-processing demands grew.

### Source excerpt

From Monolith to Multi-Cluster: Snap's Journey to a Resilient Data Orchestration Platform Flowrida, Snap's workflow orchestration tool, is built on Apache Airflow and is critical for managing data at scale. It supports over 1,000 users across more than 170 teams, handling 3,300 active data pipelines and over 180,000 task instances daily. A significant limitation of open-source Apache Airflow is its lack of multi-cluster resiliency.

## The 2026 AI Data Engineer Roadmap

DevFeed: [The 2026 AI Data Engineer Roadmap](<https://devfeed.tech/articles/the-2026-ai-data-engineer-roadmap-27256.md>)

Original publisher: [Read original article](<https://blog.dataexpert.io/p/the-2026-ai-data-engineer-roadmap>)

Author: Zach Wilson

Published: 2026-02-05T20:26:37Z

Content type: article

Language: en

Sources: [DataExpert.io Newsletter](<https://devfeed.tech/sources/dataexpert-io-newsletter.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [airflow](<https://devfeed.tech/topics/airflow.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-coding-agents](<https://devfeed.tech/tags/ai-coding-agents.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>)

### AI overview

This article presents a 2026 roadmap for data engineering as AI automates more pipeline, SQL, Spark, testing, migration, and operational work. It compares responsibilities that are becoming more automated with areas such as system design, deep technical debt, performance tradeoffs, and organizational constraints where human expertise remains important.

### Source excerpt

And how to avoid getting replaced

## Airflow Survey 2025

DevFeed: [Airflow Survey 2025](<https://devfeed.tech/articles/airflow-survey-2025-32548.md>)

Original publisher: [Read original article](<https://airflow.apache.org/blog/airflow-survey-2025/>)

Author: Apache Airflow

Published: 2026-01-22T00:00:00Z

Content type: article

Language: en

Sources: [Apache Airflow Blog](<https://devfeed.tech/sources/apache-airflow-blog.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [data](<https://devfeed.tech/topics/data.md>), [CSV](<https://devfeed.tech/topics/csv.md>)

Tags: [2025](<https://devfeed.tech/tags/2025.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [community](<https://devfeed.tech/tags/community.md>), [csv](<https://devfeed.tech/tags/csv.md>), [download](<https://devfeed.tech/tags/download.md>), [form](<https://devfeed.tech/tags/form.md>), [graphics](<https://devfeed.tech/tags/graphics.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [report](<https://devfeed.tech/tags/report.md>), [responses](<https://devfeed.tech/tags/responses.md>), [share](<https://devfeed.tech/tags/share.md>), [survey](<https://devfeed.tech/tags/survey.md>), [users](<https://devfeed.tech/tags/users.md>), [view](<https://devfeed.tech/tags/view.md>)

### AI overview

This page presents the Airflow Survey 2025 interactive report, hosted by Astronomer and sponsored and supported by the Apache Airflow community. The supplied evidence does not include the survey findings.

### Source excerpt

The interactive report is hosted by Astronomer. The Apache Airflow community thanks Astronomer for running this survey, for sponsoring it and providing the report in this form, and for their effort in marketing, analysis, and preparing the graphics. View raw data Download survey responses (CSV)

## The 2025 AI + Data Engineering Roadmap

DevFeed: [The 2025 AI + Data Engineering Roadmap](<https://devfeed.tech/articles/the-2025-ai-data-engineering-roadmap-27255.md>)

Original publisher: [Read original article](<https://blog.dataexpert.io/p/the-2025-breaking-into-data-engineering-roadmap>)

Author: Zach Wilson

Published: 2025-10-17T22:35:45Z

Content type: tutorial

Language: en

Sources: [DataExpert.io Newsletter](<https://devfeed.tech/sources/dataexpert-io-newsletter.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Python](<https://devfeed.tech/topics/python.md>), [SQL](<https://devfeed.tech/topics/sql.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [airflow](<https://devfeed.tech/topics/airflow.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [Orchestration](<https://devfeed.tech/topics/orchestration.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [data-modeling](<https://devfeed.tech/topics/data-modeling.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>)

Tags: [2025](<https://devfeed.tech/tags/2025.md>), [ai](<https://devfeed.tech/tags/ai.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [count](<https://devfeed.tech/tags/count.md>), [course](<https://devfeed.tech/tags/course.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-modeling](<https://devfeed.tech/tags/data-modeling.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [framer](<https://devfeed.tech/tags/framer.md>), [newsletter](<https://devfeed.tech/tags/newsletter.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [python](<https://devfeed.tech/tags/python.md>), [rag](<https://devfeed.tech/tags/rag.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [right-join](<https://devfeed.tech/tags/right-join.md>), [spark](<https://devfeed.tech/tags/spark.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

A 2025 roadmap for entering data engineering, covering foundational SQL and Python skills, distributed computing, orchestration, data modeling, data quality, AI and data integrations, portfolio projects, and personal branding.

### Source excerpt

Getting a data engineering job is complicated.

## Apache Airflow 3.1.0: Human-Centered Workflows

DevFeed: [Apache Airflow 3.1.0: Human-Centered Workflows](<https://devfeed.tech/articles/apache-airflow-3-1-0-human-centered-workflows-32540.md>)

Original publisher: [Read original article](<https://airflow.apache.org/blog/airflow-3.1.0/>)

Author: Apache Airflow

Published: 2025-09-25T00:00:00Z

Content type: release

Language: en

Sources: [Apache Airflow Blog](<https://devfeed.tech/sources/apache-airflow-blog.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [Automation](<https://devfeed.tech/topics/automation.md>), [human review](<https://devfeed.tech/topics/human-review.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [Forms](<https://devfeed.tech/topics/forms.md>), [Developer experience](<https://devfeed.tech/topics/developer-experience.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [React](<https://devfeed.tech/topics/react.md>)

Tags: [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [automation](<https://devfeed.tech/tags/automation.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [developer-experience](<https://devfeed.tech/tags/developer-experience.md>), [forms](<https://devfeed.tech/tags/forms.md>), [human-review](<https://devfeed.tech/tags/human-review.md>), [inference](<https://devfeed.tech/tags/inference.md>), [release](<https://devfeed.tech/tags/release.md>), [release-notes](<https://devfeed.tech/tags/release-notes.md>), [ui](<https://devfeed.tech/tags/ui.md>)

### AI overview

Apache Airflow 3.1.0 introduces human-in-the-loop tasks for pausing automated workflows and collecting reviews through web forms. The release also adds internationalization support, developer experience improvements, restored Calendar and Gantt views, UI filtering, DAG pinning, and accessibility-focused theme updates.

### Source excerpt

We are thrilled to announce the release of Apache Airflow 3.1.0, an update that puts humans at the center of data workflows. This release introduces powerful new capabilities for human decision-making in automated processes, comprehensive internationalization support, and significant developer experience improvements. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/3.1.0/ 📚 Core Airflow Docs: https://airflow.apache.org/docs/apache-airflow/3.1.0/ 📚 Task SDK Docs: https://airflow.apache.org/docs/task-sdk/1.1.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.1.0/release_notes.html 🪶 Sources: https://airflow.apache.org/docs/apache-airflow/3.1.0/installation/installing-from-sources.html 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.1.0 🤝 Human-in-the-Loop (HITL): When Automation Meets Human Judgment This powerful capability bridges the gap between automated processes and human expertise, making Airflow invaluable for: AI/ML Model Validation: Pause inference pipelines for human review of model outputs Content Moderation: Route content through human reviewers before publication Approval Workflows: Require manager approval for sensitive operations Data Quality Gates: Allow data stewards to validate critical datasets HITL tasks pause in a deferred state while presenting intuitive web forms in the Airflow UI. Users with appropriate roles can review context data, DAG parameters, and XCom values before making informed decisions. Example Code: from airflow.sdk import DAG from airflow.providers.standard.operators.hitl import HITLOperator with DAG("content_moderation", schedule="@daily") as dag: moderate_content = HITLOperator( task_id="review_content", message="Please review this content for publication", data_key="content_to_review" ) 📊 UI Enhancements & Performance Calendar and Gantt Views Make Their Comeback Remember those beloved Calendar and Gantt chart views from Airflow 2.x? They're back, completely rebuilt for the modern

## Apache Airflow® 3 is Generally Available!

DevFeed: [Apache Airflow® 3 is Generally Available!](<https://devfeed.tech/articles/apache-airflow-3-is-generally-available-32550.md>)

Original publisher: [Read original article](<https://airflow.apache.org/blog/airflow-three-point-oh-is-here/>)

Author: Apache Airflow

Published: 2025-04-22T00:00:00Z

Content type: release

Language: en

Sources: [Apache Airflow Blog](<https://devfeed.tech/sources/apache-airflow-blog.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [releases](<https://devfeed.tech/topics/releases.md>), [event driven](<https://devfeed.tech/topics/event-driven.md>), [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [React](<https://devfeed.tech/topics/react.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [hybrid-cloud](<https://devfeed.tech/tags/hybrid-cloud.md>), [messaging](<https://devfeed.tech/tags/messaging.md>), [react](<https://devfeed.tech/tags/react.md>), [release](<https://devfeed.tech/tags/release.md>), [releases](<https://devfeed.tech/tags/releases.md>)

### AI overview

Apache Airflow 3.0 has been released as a major version. The release introduces a React-based UI, DAG versioning, improved backfill support, event-driven scheduling, and a Task Execution Interface with Task SDKs supporting secure execution across multi-cloud, hybrid-cloud, and local data center deployments.

### Source excerpt

We announced our intent to focus on Apache Airflow 3.0® as the next big milestone for the Airflow project at the Airflow Summit in September 2024. We are delighted to announce that Airflow 3.0 is now released! A Major Release, Four Years in the Making Airflow 3.0 is the biggest release in Airflow's history--2.0 was released in 2020, and the last 4 years have seen incremental updates and releases every quarter with version 2.10 released in Q4 2024. With over 30 million monthly downloads (up over 30x since 2020) and 80,000 organizations (up from 25,000 in 2020) now using Airflow, we've seen an incredible growth in popularity since 2.0. Over the last four years, Airflow has grown to power business critical data workflows within organizations of all sizes. We have seen an exponential increase in the use cases for Airflow from its beginnings with ETL, ELT, and Reverse ETL, with over 30% of Airflow users using it for MLOps, and 10% using it for GenAI workflows. Airflow 3 is a response to this use case expansion and is the standard for data application development across the enterprise. Here are some highlights: Airflow 3 is significantly easier to use for data practitioners and incorporates their key requests for critical changes to Airflow. Early user reactions to features such as the new React based UI, DAG Versioning, and improved Backfill support have been incredibly positive. I was ecstatic to see the reaction from data engineers when I demonstrated this at a recent Airflow meetup. The seamless UI transition of navigating between Asset-oriented workflows and Task-oriented workflows is beautiful. Once again, Airflow lets the developer choose how you want to develop and navigate without imposing any restrictions. Introduction of Event Driven Scheduling enables Airflow to seamlessly integrate with messaging providers and react to events happening and data assets being updated outside of Airflow. The big architecture change with the introduction of the Task Execution Inte

## April 2025 Newsletter

DevFeed: [April 2025 Newsletter](<https://devfeed.tech/articles/april-2025-newsletter-4888.md>)

Original publisher: [Read original article](<https://clickhouse.com/blog/202504-newsletter>)

Author: Mark Needham

Published: 2025-04-16T08:29:42Z

Content type: article

Language: en

Sources: [ClickHouse Blog](<https://devfeed.tech/sources/clickhouse-blog.md>)

Topics: [clickhouse](<https://devfeed.tech/topics/clickhouse.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [Data Infrastructure](<https://devfeed.tech/topics/data-infrastructure.md>), [data observability](<https://devfeed.tech/topics/data-observability.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [airflow](<https://devfeed.tech/topics/airflow.md>)

Tags: [2025](<https://devfeed.tech/tags/2025.md>), [ai](<https://devfeed.tech/tags/ai.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [aws](<https://devfeed.tech/tags/aws.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [data](<https://devfeed.tech/tags/data.md>), [data-infrastructure](<https://devfeed.tech/tags/data-infrastructure.md>), [data-observability](<https://devfeed.tech/tags/data-observability.md>), [newsletter](<https://devfeed.tech/tags/newsletter.md>), [openai](<https://devfeed.tech/tags/openai.md>), [real-time](<https://devfeed.tech/tags/real-time.md>)

### AI overview

The April 2025 ClickHouse newsletter covers CloudQuery's experience with ClickHouse, the new query condition cache in version 25.3, ClickHouse's Rust development, the acquisition of HyperDX, community news, upcoming events, training, and ClickHouse's role in data observability and AI applications.

### Source excerpt

Welcome to the April ClickHouse newsletter, which will round up what's happened in real-time data warehouses over the last month.

## Trading Airflow + EMR for Temporal + Bauplan: The Mediaset tale

DevFeed: [Trading Airflow + EMR for Temporal + Bauplan: The Mediaset tale](<https://devfeed.tech/articles/trading-airflow-emr-for-temporal-bauplan-the-mediaset-tale-36078.md>)

Original publisher: [Read original article](<https://temporal.io/blog/trading-airflow-emr-temporal-bauplan-mediaset>)

Author: Stu Kendall

Published: 2025-04-10T00:00:00Z

Content type: article

Language: en

Sources: [Temporal](<https://devfeed.tech/sources/temporal-blog.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [Orchestration](<https://devfeed.tech/topics/orchestration.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Python](<https://devfeed.tech/topics/python.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [data](<https://devfeed.tech/topics/data.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [aws](<https://devfeed.tech/tags/aws.md>), [community](<https://devfeed.tech/tags/community.md>), [data-management](<https://devfeed.tech/tags/data-management.md>), [formats](<https://devfeed.tech/tags/formats.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [python](<https://devfeed.tech/tags/python.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [s3](<https://devfeed.tech/tags/s3.md>)

### AI overview

This article describes how Mediaset replaced an Airflow-based AWS data stack with Temporal and Bauplan. The change produced a near-real-time news dashboard, with refresh time reduced from one hour to five minutes according to the supplied summary.

### Source excerpt

Mediaset replaced Airflow and six AWS services with Temporal and Bauplan, cutting dashboard refresh time from 1 hour to 5 minutes in just 6 weeks.

## Deploying Airflow Data Pipelines with a DAG Synchronization Service

DevFeed: [Deploying Airflow Data Pipelines with a DAG Synchronization Service](<https://devfeed.tech/articles/a-smarter-way-to-deploy-data-pipelines-20021.md>)

Original publisher: [Read original article](<https://technology.doximity.com/articles/a-smarter-way-to-deploy-data-pipelines>)

Author: Doximity

Published: 2025-03-12T14:28:00Z

Content type: article

Language: en

Sources: [Doximity](<https://devfeed.tech/sources/doximity.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [data](<https://devfeed.tech/topics/data.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [ci-cd](<https://devfeed.tech/tags/ci-cd.md>), [container](<https://devfeed.tech/tags/container.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [s3](<https://devfeed.tech/tags/s3.md>)

### AI overview

Doximity describes replacing Airflow container image rebuilds and scheduled service restarts with a DAG synchronization service that continuously updates DAGs from Amazon S3. The approach reduced deployment times from about 30 minutes to under one minute and reduced UI disruption and downtime.

### Source excerpt

At Doximity, we have nearly 20 data teams responsible for the development of data pipelines to support product and business intelligence needs. These teams rely on Apache Airflow to orchestrate over 900 active DAGs (fancy word for data pipelines), with dozens of updates deployed daily. However, with the growth of our data platform team, the bottlenecks in our deployment process for data pipelines could no longer be ignored. Deploying new pipelines or updating existing ones required building, publishing, and deploying a new Airflow container image--a process that could take up to 20 minutes. To make matters worse, pulling in the latest updates meant restarting Airflow services, causing temporary UI disruptions and delaying workflows--the salt in the wound, if you will. In this article, we will discuss in further details the bottlenecks of our legacy deployment system and how we overcame them by developing a DAG synchronization service that continuously updates DAGs directly from Amazon S3 object storage. This new approach not only accelerated deployment times--reducing them from 30 minutes to under one--but also improved system reliability, reduced downtime, and enhanced operational flexibility. Problem Overview Prior to the sync service, DAGs were deployed by building them into the Airflow container image using a CI/CD pipeline. This was both time-consuming and resource-intensive, with each build and delivery process taking nearly 20 minutes to complete. With dozens of teams pushing updates daily, each triggering a full image rebuild, these delays were a clear bottleneck to our productivity. Pulling in these changes required scheduled restarts of the Airflow Scheduler and Web Server at regular intervals. These restarts, while necessary, caused temporary (and dare I say, annoying) disruptions to the Airflow Web UI. Refreshing your screen only to be greeted by a 503 error was the virtual equivalent of stubbing your toe--frustrating and painfully avoidable. Factoring in the

## Replay '25, day two: Big announcements, real-world wins, and Durable Execution

DevFeed: [Replay '25, day two: Big announcements, real-world wins, and Durable Execution](<https://devfeed.tech/articles/replay-25-day-two-big-announcements-real-world-wins-and-durable-execution-35953.md>)

Original publisher: [Read original article](<https://temporal.io/blog/replay-25-day-two>)

Author: Tim Imkin

Published: 2025-03-06T00:00:00Z

Content type: article

Language: en

Sources: [Temporal](<https://devfeed.tech/sources/temporal-blog.md>)

Topics: [SDK](<https://devfeed.tech/topics/sdk.md>), [Replication](<https://devfeed.tech/topics/replication.md>), [migration](<https://devfeed.tech/topics/migration.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [airflow](<https://devfeed.tech/topics/airflow.md>), [Self-hosted](<https://devfeed.tech/topics/self-hosted.md>), [Spring Boot](<https://devfeed.tech/topics/spring-boot.md>), [Java](<https://devfeed.tech/topics/java.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Software Testing](<https://devfeed.tech/topics/software-testing.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [community](<https://devfeed.tech/tags/community.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [java](<https://devfeed.tech/tags/java.md>), [migration](<https://devfeed.tech/tags/migration.md>), [observability](<https://devfeed.tech/tags/observability.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [technology](<https://devfeed.tech/tags/technology.md>)

### AI overview

Day two of Replay '25 covered Temporal product announcements, including a pre-release Ruby SDK, Nexus general availability, namespace replication, worker versioning APIs, migration tooling, Temporal Cloud on Google Cloud, and new security and operations tools. The article also reports case studies and technical sessions on workflow orchestration, self-hosting, Spring Boot integration, observability, scaling, and testing.

### Source excerpt

Catch up on day two of Replay '25 in London, featuring big announcements from Temporal, real-world success stories across telecom, finance, and technology sectors, and practical insights into durable system execution.

## Airflow Survey 2024

DevFeed: [Airflow Survey 2024](<https://devfeed.tech/articles/airflow-survey-2024-32547.md>)

Original publisher: [Read original article](<https://airflow.apache.org/blog/airflow-survey-2024/>)

Author: Apache Airflow

Published: 2025-02-27T00:00:00Z

Content type: article

Language: en

Sources: [Apache Airflow Blog](<https://devfeed.tech/sources/apache-airflow-blog.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [data](<https://devfeed.tech/topics/data.md>), [CSV](<https://devfeed.tech/topics/csv.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [apache](<https://devfeed.tech/tags/apache.md>), [community](<https://devfeed.tech/tags/community.md>), [csv](<https://devfeed.tech/tags/csv.md>), [data](<https://devfeed.tech/tags/data.md>), [download](<https://devfeed.tech/tags/download.md>), [graphics](<https://devfeed.tech/tags/graphics.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [report](<https://devfeed.tech/tags/report.md>), [survey](<https://devfeed.tech/tags/survey.md>), [users](<https://devfeed.tech/tags/users.md>)

### AI overview

Airflow Survey 2024 is an interactive report hosted by Astronomer, based on a survey run with support from the Apache Airflow community. The page provides access to raw data and survey responses in CSV format.

### Source excerpt

The interactive report is hosted by Astronomer. The Apache Airflow community thanks Astronomer for running this survey, for sponsoring it and providing the report in this form, and for their effort in marketing, analysis, and preparing the graphics. View raw data Download survey responses (CSV)

## From Airflow to Argo Workflows and dbt Python models

DevFeed: [From Airflow to Argo Workflows and dbt Python models](<https://devfeed.tech/articles/from-airflow-to-argo-workflows-and-dbt-python-models-56737.md>)

Original publisher: [Read original article](<https://engineering.mercari.com/en/blog/entry/20241214-from-airflow-to-argo-workflows-and-dbt-python-models/>)

Author: I-Kounakis

Published: 2024-12-15T10:00:12Z

Content type: article

Language: en

Sources: [Mercari Engineering](<https://devfeed.tech/sources/mercari-engineering.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>), [Orchestration](<https://devfeed.tech/topics/orchestration.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Python](<https://devfeed.tech/topics/python.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [argo](<https://devfeed.tech/tags/argo.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [data-lineage](<https://devfeed.tech/tags/data-lineage.md>), [data-management](<https://devfeed.tech/tags/data-management.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [dependency-graph](<https://devfeed.tech/tags/dependency-graph.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [python](<https://devfeed.tech/tags/python.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

### AI overview

This article describes Merpay's migration from Apache Airflow to Argo Workflows and dbt, including the criteria behind the choice, tool capabilities, and migration practices. It also discusses dbt SQL and Python models, workflow orchestration on Kubernetes, and execution through Google Cloud services.

### Source excerpt

This post is Merpay & Mercoin Advent Calendar 2024, brought to you by @Yani from the Merpay Data Management team. This article describes the journey of Merpay when migrating from Airflow to Argo Workflows and dbt, and the considerations that went into this choice. We will start with an introduction of each tool and the [...]

## Apache Airflow 2.10.0 is here

DevFeed: [Apache Airflow 2.10.0 is here](<https://devfeed.tech/articles/apache-airflow-2-10-0-is-here-32531.md>)

Original publisher: [Read original article](<https://airflow.apache.org/blog/airflow-2.10.0/>)

Author: Apache Airflow

Published: 2024-08-08T00:00:00Z

Content type: release

Language: en

Sources: [Apache Airflow Blog](<https://devfeed.tech/sources/apache-airflow-blog.md>)

Topics: [airflow](<https://devfeed.tech/topics/airflow.md>), [releases](<https://devfeed.tech/topics/releases.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [configuration](<https://devfeed.tech/topics/configuration.md>), [data](<https://devfeed.tech/topics/data.md>), [PyPI](<https://devfeed.tech/topics/pypi.md>), [Docker Image](<https://devfeed.tech/topics/docker-image.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [docker-image](<https://devfeed.tech/tags/docker-image.md>), [pypi](<https://devfeed.tech/tags/pypi.md>), [release](<https://devfeed.tech/tags/release.md>), [release-notes](<https://devfeed.tech/tags/release-notes.md>), [releases](<https://devfeed.tech/tags/releases.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>)

### AI overview

Apache Airflow 2.10.0 is available with default basic telemetry collection, support for multiple executors in one environment, and dynamic dataset scheduling through DatasetAlias. Deployments can opt out of telemetry collection.

### Source excerpt

I'm happy to announce that Apache Airflow 2.10.0 is now available, bringing an array of noteworthy enhancements and new features that will greatly serve our community. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.10.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.10.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.10.0/release_notes.html 🐳 Docker Image: "docker pull apache/airflow:2.10.0" 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.10.0 Airflow now collects Telemetry data by default With the release of Airflow 2.10.0, we've introduced the collection of basic telemetry data, as outlined here. This data will play a crucial role in helping Airflow maintainers gain a deeper understanding of how Airflow is utilized across various deployments. The insights derived from this information are invaluable in guiding the prioritization of patches, minor releases, and security fixes. Moreover, this data will inform key decisions regarding the development roadmap, ensuring that Airflow continues to evolve in line with community needs. For those who prefer not to participate in data collection, deployments can easily opt out by setting the [usage_data_collection] enabled option to False or by using the SCARF_ANALYTICS=false environment variable. Multiple Executor Configuration (formerly "Hybrid Execution") Each executor comes with its unique set of strengths and weaknesses, typically balancing latency, isolation, and compute efficiency. Traditionally, an Airflow environment is limited to a single executor, requiring users to make trade-offs, as no single executor is perfectly suited for all types of tasks. We are introducing a new feature that allows for the concurrent use of multiple executors within a single Airflow environment. This flexibility enables users to take advantage of the specific strengths of different executors for various tasks, improving overall efficiency and mitigating weaknesses. Users can s

[Next page](<https://devfeed.tech/topics/airflow.md?cursor=WyIyMDI0LTA4LTA4VDAwOjAwOjAwKzAwOjAwIiwgIjE2YmNlNThjLWZkZWEtNGU2NS1hN2M4LTM4MzM2OTM1ZTU4NiJd>)