# airflow Published articles for airflow. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## The official ClickHouse provider for Apache Airflow is now available DevFeed: [The official ClickHouse provider for Apache Airflow is now available]() Original publisher: [Read original article]() Author: Aditya Chidurala; Bentsi Leviav; Alex Francoeur Published: 2026-09-17T18:06:01Z Content type: release Language: en Sources: [ClickHouse Blog]() Topics: [airflow](), [clickhouse](), [Data pipelines](), [SQL](), [Python](), [Open Source]() Tags: [airflow](), [clickhouse](), [data-pipelines](), [integration](), [python](), [release](), [sql]() ### AI overview ClickHouse has released an officially maintained Apache Airflow provider for orchestrating ClickHouse data workflows. The provider uses ClickHouse Connect over HTTP(S), supports Airflow's common SQL operators, and includes a hook for bulk and client-specific operations. ### Source excerpt The official ClickHouse provider for Apache Airflow simplifies data workflows with standard SQL operators, bulk inserts, and shared setup across self-managed Airflow and Astronomer. ## От Kubernetes-платформы к управлению гибридной распределённой инфраструктурой: встречайте Deckhouse Platform DevFeed: [От Kubernetes-платформы к управлению гибридной распределённой инфраструктурой: встречайте Deckhouse Platform]() Original publisher: [Read original article]() Author: Manassian (Флант) Published: 2026-09-17T10:04:13Z Content type: news Language: ru Sources: [Tagir Valeev]() Topics: [Kubernetes](), [on-prem](), [AI Platform](), [cloud-infrastructure](), [control-plane](), [GPU](), [PostgreSQL](), [clickhouse](), [airflow]() Tags: [airflow](), [clickhouse](), [control-plane](), [deckhouse](), [devops](), [gpu](), [kubernetes](), [on-prem](), [platform](), [postgresql](), [tag-82a17eb6b53d](), [tag-8b566f363766](), [tag-d57ff84cbcb7](), [tag-f7895e49ab8b]() ### AI overview The article announces Deckhouse Platform, which combines containerization, virtualization, and centralized cluster management. It describes ready-made infrastructure solutions for Kubernetes, virtual machines, combined workloads, AI workloads, data services, and private cloud deployments across cloud, on-premises, and edge environments. ### Source excerpt Мы объединили возможности контейнеризации, виртуализации и централизованного управления парком кластеров в один продукт -- Deckhouse Platform. Теперь контейнерами, виртуальными машинами, ИИ-нагрузками и сервисами данных можно управлять из одной точки, независимо от того, где они работают: в облаке, on-prem или на edge. В платформе уже есть готовые решения под типовые сценарии использования инфраструктуры, а под нестандартные можно собрать свою конфигурацию. Читать далее ## Refreshing the Travel-Time Map Behind Lyft's Marketplace: Rebuilding Neighborhood Reachability... DevFeed: [Refreshing the Travel-Time Map Behind Lyft's Marketplace: Rebuilding Neighborhood Reachability...]() Original publisher: [Read original article]() Author: Manjunath Shettar Published: 2026-09-10T16:12:28Z Content type: article Language: en Sources: [Lyft Engineering - Medium]() Topics: [dataset](), [airflow]() Tags: [airflow](), [data-processing](), [feature-engineering](), [latency](), [lyft](), [offline](), [tea]() ### AI overview Lyft describes rebuilding its Neighborhood Reachability Signals: offline, regional travel-time matrices between geohash-6 cells and their associated neighborhood-center lists. The refresh replaces older static data and is intended to support marketplace pricing, driver guidance, and demand heatmaps, with future work aimed at time-aware travel times. ### Source excerpt Refreshing the Travel-Time Map Behind Lyft's Marketplace: Rebuilding Neighborhood Reachability Signals Every time Lyft calculates pricing to balance a market, nudges a driver toward an under-served pocket of a city, or paints a heatmap of where demand is building, there is a quiet lookup table doing work in the background. It answers a deceptively simple question: how long does it take to get from here to there?, for millions of pairs of places, across hundreds of regions. That lookup table is the Neighborhood Reachability Signal, and for years large parts of it were frozen in a snapshot of the world from 2018-2019. This is the story of how we rebuilt it, why a refresh substantial enough to be worth adopting was what finally moved Pricing to switch, the cleanly positive results that came out of that switch, and where we're taking it next, from one static file per region to time-aware travel times that change with the rhythm of the day. What is a Neighborhood Reachability Signal? A geohash is a compact way of carving the world into a grid of cells. At geohash-6 resolution, each cell is roughly the size of a few city blocks. Slice a region into geohash-6 cells and you get a clean, discrete coordinate system for "neighborhoods" that downstream systems can reason about. The Forecasting & Real-Time Optimization (FORTOP) team produces the Neighborhood Reachability Signals dataset, which consists of two companion files for each region: Neighborhood Reachability Matrix: the estimated travel time, in minutes, between the centers of pairs of geohash-6 cells. Think of it as a sparse origin-to-destination travel-time matrix for a region. Neighborhood Centers: the list of all geohashes that appear in the ETA files for that region, i.e. the "vocabulary" of cells that the marketplace is allowed to talk about. Both files are generated offline on a schedule by an Airflow DAG. They are static in the sense that they are precomputed and shipped, rather than queried live (which is exact ## The best workflow engine is a programming language DevFeed: [The best workflow engine is a programming language]() Original publisher: [Read original article]() Author: Pranay Prakash Published: 2026-08-27T07:00:00Z Content type: opinion Language: en Sources: [Vercel News]() Topics: [SDKs]() Tags: [airflow](), [encryption](), [infrastructure](), [kubernetes](), [programming-language](), [tls](), [typescript](), [workflow]() ### AI overview The article argues that programming languages can express workflow graphs more naturally than manually defined DAGs. It describes using TypeScript for durable execution and contrasts that approach with the operational setup required to run Temporal workers and infrastructure. ### Source excerpt The idea of orchestrating long-running, stateful logic on top of unreliable, stateless infrastructure isn't new. We've had message queues, job runners, microservice choreographies, and full-blown workflow engines for a long time. What we never had was a version of it that felt good to write. I'd spent about six months working on a fork of Temporal, mostly on weekends, trying to turn it into a serverless answer with DX that felt more Vercel-native. Eventually it dawned on me that to ship that experience, I'd need to own the execution environment too. So I dropped the fork, joined Vercel, and started hacking on a new framework from scratch alongside Nathan Rajlich. Code is already a DAG A workflow is a DAG, a directed acyclic graph. Before Temporal (and Cadence before it), nearly every workflow framework made you draw that DAG by hand. Apache Airflow is the canonical example. You describe your pipeline as an explicit graph of tasks and dependencies, and your actual logic gets buried inside the nodes." That always felt backwards to me. We already have a tool for expressing "do this, then that, and these in parallel, and branch here." It's called a programming language. An abstract syntax tree is a DAG. Software itself is a DAG. I knew that in the abstract, but it didn't truly hit me until I saw Temporal, where you write what looks like normal sequential code and the engine makes it durable underneath. It was the dream all along. What running Temporal taught me Temporal is great once your infrastructure exists. I was starting from scratch, and setting it up meant: Standing up Temporal Cloud (or self-hosting the server: Frontend, History, Matching, and Worker services, plus a Cassandra/Postgres/MySQL backend and sharding). Running your own worker fleet: Temporal never executes your code. Your workers poll the server and run your workflows and activities. In practice that's a Kubernetes cluster you own. Wiring it all together: task queues, activity registration, client co ## How dbt works, and why orchestrators shouldn't split it into tasks DevFeed: [How dbt works, and why orchestrators shouldn't split it into tasks]() Original publisher: [Read original article]() Author: Ruben Fiszel Published: 2026-08-20T00:00:00Z Content type: article Language: en Sources: [Windmill Blog]() Topics: [Compiler](), [Job](), [airflow](), [data](), [Command-line interface]() Tags: [airflow](), [bigquery](), [cli](), [dagster](), [data](), [data-pipelines](), [dbt](), [dbt-data-pipelines-airflow-dagster-orchestration](), [job](), [orchestration](), [scheduler](), [series]() ### AI overview This primer explains dbt's architecture and examines five ways teams orchestrate it in production. It argues that running a project as one dbt command, while reading dbt's per-model execution state, is generally more efficient and less fragile than creating one orchestrator task per model. It discusses Dagster and astronomer-cosmos's convergence on this approach, including a reported cost comparison. ### Source excerpt How is dbt actually orchestrated, and why does running it as one job beat one task per model? A primer on dbt as a compiler with a scheduler attached, the five ways teams wrap it, and why both Dagster and astronomer-cosmos converged on a single dbt invocation with per-model state projected out of it. ## Quasi-Agentic Pipelines with Databricks and Apache Airflow DevFeed: [Quasi-Agentic Pipelines with Databricks and Apache Airflow]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-08-10T21:23:57Z Content type: tutorial Language: en Sources: [Data Engineering Central]() Topics: [databricks](), [airflow](), [Data pipelines](), [Large Language Model](), [data-engineering]() Tags: [airflow](), [apache-airflow](), [data-engineering](), [databricks](), [llms](), [pipelines]() ### AI overview A practical developer discussion of incorporating LLMs and agents into existing data workflows using Databricks and Apache Airflow. It also examines determinism in data pipelines and the gap between business requirements and engineering implementation. ### Source excerpt the strange space in between ## How We Refresh Razorpay's Data Warehouse 10x Faster with Graphs and Indexes DevFeed: [How We Refresh Razorpay's Data Warehouse 10x Faster with Graphs and Indexes]() Original publisher: [Read original article]() Author: Amit Prabhu Published: 2026-07-14T14:06:16Z Content type: article Language: en Sources: [Razorpay Engineering - Medium]() Topics: [data](), [airflow](), [Apache Spark](), [Amazon S3](), [Apache Iceberg](), [microservices architecture](), [parquet](), [Architecture & Design]() Tags: [airflow](), [analytics](), [apache-iceberg](), [architecture](), [batch](), [data](), [graphs](), [infrastructure](), [parquet](), [pipelines](), [razorpay](), [spark](), [trino](), [warehouse]() ### AI overview Razorpay describes its data warehouse refresh pipeline, which builds wide denormalized Facts by joining data from multiple microservices. The article covers the original Airflow- and Spark-based full-refresh process, the underlying lake formats and query layer, and the scaling challenges that led the team to reconsider refresh strategy, data layout, and high-cardinality dimensions. ### Source excerpt Contributors: Utkarsh Koppikar Rohan Background Razorpay provides the payment infrastructure for millions of merchants globally. Behind every payment, settlement, and refund is a microservices architecture where each service owns its own database. While this keeps services independent and scalable, it creates a challenge for stakeholders who need to see across those boundaries. The Data Platform team manages the infrastructure that bridges this gap. Transactional data flows into the lake via CDC pipelines, ingested onto S3 in Delta Lake, Apache Iceberg, or plain Parquet formats. On top of the lake, we build domain-specific warehouse tables -- wide, pre-joined tables that co-locate all the data a consumer needs, queryable via Trino. These power two use cases: Analytics (internal dashboards on Tableau and Superset) and Reporting (merchants and regulated entities who download structured data exports; Razorpay generates nearly a million such reports per month). The warehouse tables that power both use cases are called Facts. A Fact is a flat denormalised table on S3, produced by joining 10 to 30 microservice tables and materialising the result once. A settlement Fact, for example, merges payments, refunds, adjustments, and card details into a single wide row so that a dashboard or report reads from a single table instead of joining across services in real time. It is closer to a domain-specific materialised view than a classical data warehouse fact table. We maintain over 50 such Facts, and approximately 40% of all merchant reports are served directly from them. As data volumes and the number of entities per fact grew, the batch generation pipeline began to show its limits, prompting us to rethink the refresh strategy, the data layout, and how to handle high-cardinality dimensions. The rest of this post covers that journey. The Full Refresh Pipeline: Our Baseline and the Pain The original full-refresh pipeline was straightforward. Schedule: Airflow schedules Spark jobs o ## 98% faster queries, 50% lower cloud costs: Verihubs' journey from Postgres to ClickHouse DevFeed: [98% faster queries, 50% lower cloud costs: Verihubs' journey from Postgres to ClickHouse]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-07-06T13:20:29Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [Database](), [Kafka](), [data]() Tags: [airflow](), [analytics](), [architecture](), [batch](), [clickhouse](), [dashboards](), [data](), [database](), [kafka](), [performance](), [postgres](), [real-time](), [scale](), [streaming]() ### AI overview Verihubs describes moving its data warehouse from batch-oriented Postgres extraction to a ClickHouse-based real-time OLAP architecture. Kafka streaming and related components improved data freshness and reliability, while the company reports faster queries and lower cloud costs. ### Source excerpt Verihubs rebuilt its Postgres data warehouse on ClickHouse with real-time Kafka streaming, cutting query times by up to 98% and cloud costs by up to 50%. ## Apache Airflow 3.3.0: Stateful Tasks and Multi-Language Support DevFeed: [Apache Airflow 3.3.0: Stateful Tasks and Multi-Language Support]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-07-06T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [SDKs](), [Go Language](), [Java]() Tags: [airflow](), [apache-airflow](), [go](), [java](), [release](), [release-notes](), [sdk](), [state]() ### AI overview Apache Airflow 3.3.0 introduces durable state stores for tasks and assets, a Language Task SDK for Java and Go, expanded asset partitioning, and pluggable retry policies. ### Source excerpt We're proud to announce the release of Apache Airflow 3.3.0! Where 3.2 brought precision to data with asset partitioning, 3.3 gives your tasks memory and multi-language support: a first-class state store for tasks and assets, a Language Task SDK for writing task logic in Java and Go, a major expansion of asset partitioning, and pluggable retry policies. 🎯 Release Highlights 📦 PyPI: https://pypi.org/project/apache-airflow/3.3.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.3.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.3.0/release_notes.html 🐳 Docker Image: docker pull apache/airflow:3.3.0 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.3.0 🗃 Task & Asset State Store (AIP-103): Tasks That Remember Until now, if a task needed to remember something across retries or runs -- a cursor, a checkpoint, a high-water mark -- you reached for XComs, an external store, or a clever Variable hack. Airflow 3.3 makes durable task state a first-class concept. Tasks can persist arbitrary key-value state that survives across retries and runs via a new task_state_store accessor, and assets can carry their own state via asset_state_store -- both available directly from the Task SDK. State lives in the metadata database by default, or in a custom worker-side backend ([workers] state_store_backend), supports per-key retention with periodic garbage collection and an optional clear_on_success, and is fully manageable through the Core API and Execution API. Task State Store Task state is scoped to a specific task instance and persists across retries. Use it to track coordination state like remote job IDs, cursors, or progress checkpoints: @task def extract_data(**context): task_state = context["task_state_store"] # Resume from where we left off on retry cursor = task_state.get("last_cursor", default=0) records = fetch_records(since=cursor) new_cursor = records[-1]["id"] task_state.set("last_cursor", new_cursor) return records Persisted st ## Appcues delivers personalized customer engagement with ClickHouse Cloud DevFeed: [Appcues delivers personalized customer engagement with ClickHouse Cloud]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-06-18T16:21:05Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [data](), [dashboards](), [SDKs]() Tags: [airflow](), [analytics](), [apis](), [aws](), [batch](), [clickhouse](), [cloud](), [dashboards](), [data](), [databases](), [latency](), [observability](), [product-analytics](), [real-time](), [saas](), [sdks](), [streams]() ### AI overview Appcues moved its real-time customer-facing analytics and segmentation workload to ClickHouse Cloud on AWS. The article reports lower query and ingestion latency alongside reduced analytics spend at a scale of 1.31 PB and 410 billion events. ### Source excerpt Appcues cut P95 query times 90%, ingestion latency 99%, and analytics spend 23% by migrating from Snowflake and Airflow to ClickHouse Cloud for real-time segmentation across 1.31 PB of data. ## From SSH to REST: A Security-Driven Modernization of Slack's EMR Data Pipelines DevFeed: [From SSH to REST: A Security-Driven Modernization of Slack's EMR Data Pipelines]() Original publisher: [Read original article]() Author: Mahendran Vasagam Published: 2026-05-05T14:00:01Z Content type: article Language: en Sources: [Engineering at Slack]() Topics: [Security](), [OpenSSH](), [Amazon Web Services](), [airflow](), [DataOps](), [Architecture & Design](), [Kubernetes]() Tags: [airflow](), [architecture](), [aws](), [big-data](), [data-analytics](), [data-engineering](), [data-infrastructure](), [kubernetes](), [modernization](), [security](), [ssh](), [uncategorized]() ### AI overview Slack describes migrating more than 700 SSH-based data pipeline jobs to a REST-based architecture across eight data regions, eliminating SSH access to production AWS EMR clusters without downtime. The article explains the security and operational problems that motivated the modernization, including attack-surface exposure, key-management overhead, resource contention, broken connections, zombie jobs, and unreliable job-status detection. ### Source excerpt Excerpt By 2024, Slack's data platform had accumulated 700+ SSH-based operators orchestrating critical data pipelines. We're talking daily search indexing that processed terabytes of data, analytics jobs powering business intelligence, the whole shebang. Every single one of these jobs required direct SSH access to production AWS Elastic MapReduce (EMR) clusters. We had a massive security... ## Agentic Workloads on Airflow: Observable, Retryable, and Auditable by Design DevFeed: [Agentic Workloads on Airflow: Observable, Retryable, and Auditable by Design]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-04-15T00:00:00Z Content type: tutorial Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Artificial Intelligence](), [Large Language Model](), [SQL](), [datasets](), [Agent Harness]() Tags: [agentic](), [ai](), [airflow](), [community](), [csv](), [llm](), [sql](), [survey](), [tutorial](), [workflow]() ### AI overview This article explains how to build agentic workloads on Airflow 3 using the 2025 Airflow Community Survey dataset and the apache-airflow-providers-common-ai provider. It distinguishes single-query LLM pipelines from workflows that query multiple independent dimensions and synthesize their results, mapping each sub-query to a named Airflow task and using Dynamic Task Mapping and XCom for orchestration and observability. ### Source excerpt A question like "How does AI tool usage vary across Airflow versions?" has a natural SQL shape: one cross-tabulation, one result. A question like "What does a typical Airflow deployment look like for practitioners who are actively using AI in their workflow?" does not. It requires querying executor type, deployment method, cloud provider, and Airflow version independently, each filtered to the same respondent group, then synthesizing the results into a coherent picture. No single query returns the answer. The answer emerges from the relationship between all of them. This is where Airflow's agentic pattern begins: not when you add an LLM to a workflow, but when the structure of the work itself depends on running multiple LLM calls whose outputs feed a synthesis step. This post builds that pattern using the 2025 Airflow Community Survey data set and the apache-airflow-providers-common-ai provider for Airflow 3. If you haven't read the introductory survey analysis post yet, start there for a walkthrough of the single-query interactive and scheduled pipelines. This post picks up where that one ends. The Agentic Gap in the Single-Query Pattern The interactive and scheduled survey DAGs from the introductory post each do one thing: translate a natural language question into SQL, execute it against the CSV, and return the result. The LLM is involved once. The structure of the pipeline does not change based on what that LLM call returns. That is not a limitation to fix. It is the right design for that class of question. For a large fraction of production AI workflows, a single well-structured LLM call with good context is sufficient and preferable. The pattern becomes agentic when two things are true simultaneously: The question requires querying multiple independent dimensions The synthesis step, the thing that produces the final answer, depends on all of those results In an agent harness framework, this would be handled inside a reasoning loop: the LLM decides to call a to ## Ask Your Survey Anything: Building AI Analysis Pipelines with Airflow 3 DevFeed: [Ask Your Survey Anything: Building AI Analysis Pipelines with Airflow 3]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-04-15T00:00:00Z Content type: tutorial Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Artificial Intelligence](), [Large Language Model](), [SQL](), [CSV](), [data]() Tags: [ai](), [airflow](), [apache-airflow](), [automated](), [community](), [configuration](), [csv](), [executor](), [llm](), [pipelines](), [providers](), [scheduled](), [sql](), [tutorial]() ### AI overview A tutorial showing how to build interactive and scheduled AI analysis pipelines with Airflow 3. The pipelines use survey CSV data, generate and run SQL queries, validate schemas, and either request human approval or email results automatically. ### Source excerpt The 2025 Airflow Community Survey collected responses from nearly 6,000 practitioners across 168 questions. You can open a spreadsheet and filter, or write SQL by hand. But what if you could just ask a question and have Airflow figure out the query, run it, and bring the result back for your approval? This post builds two pipelines that do exactly that, using the apache-airflow-providers-common-ai provider for Airflow 3. The first pipeline is interactive: a human reviews the question before it reaches the LLM and approves the result before the DAG finishes. The second is scheduled: it downloads fresh survey data, validates the schema, runs the query unattended, and emails the result. If you haven't seen the common.ai provider overview yet, start there for a tour of all the operators. This post goes deep on a concrete end-to-end example. Two Pipelines, One Example File Both DAGs live in example_llm_survey_analysis.py and share the same schema context and datasource configuration. example_llm_survey_interactive: trigger manually, review at both ends: prompt_confirmation -> generate_sql -> run_query -> extract_data -> result_confirmation (HITLEntryOperator) (LLMSQLQuery) (Analytics) (@task) (ApprovalOperator) example_llm_survey_scheduled: runs @monthly, no human in the loop: download_survey -> prepare_csv -> check_schema -> generate_sql -> run_query -> extract_data -> send_result (HttpOperator) (@task) (LLMSchema (LLMSQLQuery) (Analytics) (@task) (@task / Email) Compare) The Data The Airflow Community Survey 2025 CSV has 5,856 rows and 168 columns covering everything from Airflow version and executor type to cloud provider, company size, and AI tool usage. A few highlights from the data: 3,320 respondents identify as Data Engineers 2,032 use AWS as their primary cloud provider for Airflow 1,445 are already running Airflow 3 1,351 say they often use AI tools to write Airflow code Those last two numbers together are part of why this example exists: the people most likely to use th ## Introducing the Common AI Provider: LLM and AI Agent Support for Apache Airflow DevFeed: [Introducing the Common AI Provider: LLM and AI Agent Support for Apache Airflow]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-04-14T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Large Language Model](), [AI Agent](), [Pydantic](), [Artificial Intelligence]() Tags: [2025](), [ai](), [ai-agent](), [airflow](), [apache](), [apache-airflow](), [community](), [llm](), [release]() ### AI overview Apache Airflow is releasing the Common AI Provider 0.1.0, a provider package that adds LLM and AI agent capabilities directly to Airflow. Built on Pydantic AI, it supports more than 20 model providers through a single install and requires Apache Airflow 3.0 or newer. ### Source excerpt At Airflow Summit 2025, we previewed what native AI integration in Apache Airflow could look like. Today we're shipping it. apache-airflow-providers-common-ai 0.1.0 adds LLM and agent capabilities directly to Airflow. Not a wrapper around another framework, but a provider package that plugs into the orchestrator you already run. It's built on Pydantic AI and supports 20+ model providers (OpenAI, Anthropic, Google, Azure, Bedrock, Ollama, and more) through a single install. pip install 'apache-airflow-providers-common-ai' Requires Apache Airflow 3.0+. Note: This is a 0.x release. We're actively looking for feedback and iterating fast, so breaking changes are possible between minor versions. Try it, tell us what works and what doesn't. Your input directly shapes the API. By the Numbers 6 Operators 6 TaskFlow decorators 5 Toolsets 4 Connection types 20+ Supported model providers via Pydantic AI The Decorator Suite Every operator has a matching TaskFlow decorator. @task.llm: Single LLM Call Send a prompt, get text or structured output back. from pydantic import BaseModel from airflow.providers.common.compat.sdk import dag, task @dag def my_pipeline(): class Entities(BaseModel): names: list[str] locations: list[str] @task.llm( llm_conn_id="my_openai_conn", system_prompt="Extract named entities.", output_type=Entities, ) def extract(text: str): return f"Extract entities from: {text}" extract("Alice visited Paris and met Bob in London.") my_pipeline() The LLM returns a typed Entities object, not a string you have to parse. Downstream tasks get structured data through XCom. @task.agent: Multi-Step Agent with Tools When the LLM needs to query databases, call APIs, or read files across multiple steps, use @task.agent. The agent picks which tools to call and loops until it has an answer. from airflow.providers.common.ai.toolsets.sql import SQLToolset from airflow.providers.common.compat.sdk import dag, task @dag def sql_analyst(): @task.agent( llm_conn_id="my_openai_conn", sys ## Apache Airflow 3.2.0: Data-Aware Workflows at Scale DevFeed: [Apache Airflow 3.2.0: Data-Aware Workflows at Scale]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-04-07T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [data](), [Orchestration](), [Release notes](), [SDK](), [Amazon S3](), [BigQuery]() Tags: [airflow](), [apache](), [apache-airflow](), [bigquery](), [data](), [docker-image](), [orchestration](), [pypi](), [release](), [release-notes](), [s3](), [sdk](), [workflows]() ### AI overview Apache Airflow 3.2.0 introduces asset partitioning for more granular data-aware scheduling, along with multi-team deployments, synchronous deadline alert callbacks, and continued Task SDK separation. Partition-aware workflows trigger downstream DAGs only when the relevant partition is updated. ### Source excerpt We're proud to announce the release of Apache Airflow 3.2.0! Airflow 3.1 puts humans at the center of automated workflows. 3.2 brings that same precision to data: Asset partitioning for granular pipeline orchestration, multi-team deployments for enterprise scale, synchronous deadline alert callbacks, and continued progress toward full Task SDK separation. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/3.2.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.2.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.2.0/release_notes.html 🐳 Docker Image: docker pull apache/airflow:3.2.0 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.2.0 🗂 Asset Partitioning (AIP-76): Only the Right Work Gets Triggered Asset partitioning has been one of the most requested additions to data-aware scheduling. If you work with date-partitioned S3 paths, Hive table partitions, BigQuery partitions, or really any partitioned data store, you've dealt with this: An upstream task updates one partition, and every downstream Dag fires regardless of which slice actually changed. It's wasteful, and for large deployments it creates real operational noise. Asset partitioning in 3.2 makes this granular. Downstream Dags trigger only when the specific partition they care about gets updated. It's the biggest change to data-aware scheduling since Assets were introduced, and it turns partition-driven orchestration into something Airflow handles natively rather than something you work around. Key Capabilities Partition-driven scheduling: Dags trigger on specific partition updates, not every asset change CronPartitionTimetable: Schedule Dags against partitions using cron expressions. Also available in the Task SDK Backfill for partitioned Dags: Backfill historical partitions without re-triggering everything downstream (#61464) Multi-asset partitions: A single Dag can listen for partitions across multiple assets, which matters when your downstream work depe ## Introducing the Apache Airflow Registry DevFeed: [Introducing the Apache Airflow Registry]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-03-19T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Data pipelines](), [API](), [JSON]() Tags: [airflow](), [apache](), [apache-airflow](), [community](), [data-pipelines](), [messaging](), [notifications](), [openai](), [registry]() ### AI overview Apache Airflow launches the Airflow Registry, a searchable catalog of official providers and modules. It includes provider and module search, installation and compatibility details, connection generation in URI, JSON, and environment-variable formats, ecosystem statistics, and a structured JSON API. ### Source excerpt Today we're launching the Apache Airflow Registry -- a searchable catalog of every official Airflow provider and its modules, live at airflow.apache.org/registry/. Need an S3 operator? A Snowflake hook? An OpenAI sensor? The Registry helps you find, compare, and configure the right components for your data pipelines -- without digging through docs or PyPI pages. By the Numbers 98 Official providers 1,602 Modules (operators, hooks, sensors, triggers, transfers, and more) 329M+ Monthly PyPI downloads across all providers 125+ Integrations with cloud platforms, databases, ML tools, and messaging services Search Everything Hit Cmd+K from any page and start typing. Results show up instantly, grouped by Providers and Modules, with type badges so you can tell a hook from an operator at a glance. Provider Pages Each provider gets a dedicated page with everything in one place: install command with copy-to-clipboard, version selector, extras dropdown, compatibility info, connection types, and the full module listing organized by type. The Amazon provider, for example, has 372 modules across operators, hooks, sensors, triggers, transfers, and more. Module type tabs let you filter to exactly what you're looking for, and a category sidebar groups modules by AWS service (S3, Lambda, Glue, Step Functions, etc.). Connection Builder Click any connection type badge on a provider page, fill in the fields, and the builder generates the connection in three formats -- URI, JSON, and Env Var -- ready to copy into your configuration. No more guessing URI encoding or JSON structure. Explore by Category Not sure which provider you need? The Explore page organizes providers into categories: Cloud Platforms, Databases, Data Warehouses, Messaging & Notifications, AI & Machine Learning, Data Processing, and more. Statistics The Stats page breaks down the ecosystem: 848 operators, 298 hooks, 164 triggers, 157 sensors, 83 transfers, and more -- plus top providers by downloads and module count. JSON API ## Data-to-Production: Bridging the Gap Between Iceberg and Live Microservices DevFeed: [Data-to-Production: Bridging the Gap Between Iceberg and Live Microservices]() Original publisher: [Read original article]() Author: Wix Engineering Published: 2026-02-17T11:04:08Z Content type: article Language: en Sources: [Wix Engineering]() Topics: [Apache Iceberg](), [data observability](), [Microservice](), [clickhouse](), [Amazon S3](), [Architecture & Design](), [Apache Spark](), [airflow](), [Back end](), [API]() Tags: [airflow](), [amazon-s3](), [apache-iceberg](), [api](), [architecture](), [clickhouse](), [data](), [data-engineering](), [json](), [microservices](), [spark]() ### AI overview Wix describes Data-to-Production, a platform that activates data from Amazon S3 and Apache Iceberg for backend microservices. The system ingests Iceberg data into ClickHouse and serves it through a type-safe JSON API, using metadata governance and an Airflow and Python ingestion engine. ### Source excerpt At Wix, our Data Warehouse (DWH) is a massive repository of insights. Built on Amazon S3 using Apache Iceberg table formats, and populated by Trino and Spark jobs, it houses petabytes of data--from user segmentation and logs to AI chat analytics. However, storage is only half the battle. The real challenge--and the "holy grail" for many data engineering teams--is Activation : taking that petabyte-scale data and exposing it to backend microservices with millisecond latency, high availability, and... ## Airflow Survey 2025 DevFeed: [Airflow Survey 2025]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-01-22T00:00:00Z Content type: article Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [data](), [CSV]() Tags: [2025](), [airflow](), [analysis](), [apache-airflow](), [community](), [csv](), [download](), [form](), [graphics](), [interactive](), [report](), [responses](), [share](), [survey](), [users](), [view]() ### AI overview This page presents the Airflow Survey 2025 interactive report, hosted by Astronomer and sponsored and supported by the Apache Airflow community. The supplied evidence does not include the survey findings. ### Source excerpt The interactive report is hosted by Astronomer. The Apache Airflow community thanks Astronomer for running this survey, for sponsoring it and providing the report in this form, and for their effort in marketing, analysis, and preparing the graphics. View raw data Download survey responses (CSV) ## Wix's AirBot: An AI-Powered Slack Agent for Investigating Data Pipeline Failures DevFeed: [Wix's AirBot: An AI-Powered Slack Agent for Investigating Data Pipeline Failures]() Original publisher: [Read original article]() Author: Wix Engineering Published: 2026-01-15T07:30:31Z Content type: article Language: en Sources: [Wix Engineering]() Topics: [airflow](), [AI, ML & Data Engineering](), [data-engineering](), [Cloud](), [Slack]() Tags: [agent](), [ai](), [airflow](), [automation](), [bugs](), [data-engineering](), [engineering](), [http](), [process](), [processes](), [slack]() ### AI overview The article describes Wix's engineering environment, where thousands of Apache Airflow pipelines operate across a large cloud-based data platform. It presents AirBot, an AI-powered Slack agent intended to help engineers investigate pipeline failures across Airflow, Spark, Kubernetes, and related services. ### Source excerpt 1. Introduction: The Challenge Understanding the engineering need for AirBot starts with recognizing the scale of the ecosystem it operates within. Wix operates a massive cloud-based development platform serving 250 million users. This generates over 4 billion HTTP transactions daily, feeding a data lake that currently holds 7 petabytes of data. To manage this volume, Wix Data Engineering maintains over 3,500 Apache Airflow pipelines (DAGs) . These pipelines handle everything from ETL... ## Behind the music: How Chartmetric is scaling music analytics with ClickHouse DevFeed: [Behind the music: How Chartmetric is scaling music analytics with ClickHouse]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-01-08T14:17:18Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [data](), [Time Series](), [real-time](), [Streaming](), [API](), [dashboards](), [Web app](), [airflow](), [elasticsearch](), [Web]() Tags: [airflow](), [analytics](), [api](), [app](), [apple](), [cache](), [clickhouse](), [cloud](), [dashboards](), [data](), [elasticsearch](), [music](), [scale](), [search](), [series](), [speed](), [streaming](), [time](), [time-series](), [web]() ### AI overview Chartmetric uses ClickHouse Cloud to scale real-time music analytics across billions of rows from streaming services, social media, and music charts. The migration from Postgres and Snowflake improved query speed and reduced storage costs, while ClickHouse now supports LLM-facing queries and a playlist cache ingesting more than 15 million rows daily. ### Source excerpt "ClickHouse works very well as part of our multi-system data stack. It's excellent for time-series data, and the VersionedCollapsingMergeTree engine was a game-changer for us, speeding up queries from 20 seconds in Snowflake to 1.5 seconds in ClickHouse." ## How Vinted Standardizes Large-Scale Decentralized Data Pipelines DevFeed: [How Vinted Standardizes Large-Scale Decentralized Data Pipelines]() Original publisher: [Read original article]() Author: Oscar Ligthart Published: 2025-12-29T00:00:00Z Content type: article Language: en Sources: [Vinted]() Topics: [Data Infrastructure](), [Orchestration](), [airflow](), [Cloud]() Tags: [airflow](), [business-logic](), [cloud](), [data](), [dependencies](), [domain](), [gcp](), [infrastructure](), [orchestration](), [pipelines](), [scale]() ### AI overview Vinted describes its migration to cloud-based data infrastructure and the challenges of coordinating dependencies across decentralized teams and domains. The article explains why relying on teams to hand-craft Airflow DAGs would add unnecessary complexity, and discusses adapting the conventional dbt-with-Airflow approach to Vinted's scale and cost constraints. ### Source excerpt TL;DR: How Vinted standardizes large-scale decentralized data pipelines. ## The 2025 AI + Data Engineering Roadmap DevFeed: [The 2025 AI + Data Engineering Roadmap]() Original publisher: [Read original article]() Author: Zach Wilson Published: 2025-10-17T22:35:45Z Content type: tutorial Language: en Sources: [DataExpert.io Newsletter]() Topics: [data-engineering](), [Python](), [SQL](), [Artificial Intelligence](), [airflow](), [BigQuery](), [Orchestration](), [Apache Spark](), [data-modeling](), [Data Quality](), [Embeddings](), [Retrieval Augmented Generation (RAG)]() Tags: [2025](), [ai](), [airflow](), [bigquery](), [count](), [course](), [data-engineering](), [data-modeling](), [data-quality](), [embeddings](), [engineering](), [framer](), [newsletter](), [orchestration](), [python](), [rag](), [real-time](), [right-join](), [spark](), [sql]() ### AI overview A 2025 roadmap for entering data engineering, covering foundational SQL and Python skills, distributed computing, orchestration, data modeling, data quality, AI and data integrations, portfolio projects, and personal branding. ### Source excerpt Getting a data engineering job is complicated. ## Apache Airflow CTL aka airflowctl 0.1.0 DevFeed: [Apache Airflow CTL aka airflowctl 0.1.0]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2025-10-15T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Command-line interface](), [Release notes](), [API](), [REST API](), [releases](), [Authentication](), [audit]() Tags: [airflow](), [apache-airflow](), [audit-trail](), [authentication](), [cli](), [command-line](), [release](), [release-notes](), [rest-api]() ### AI overview Apache Airflow announces airflowctl 0.1.0, a command-line interface that interacts exclusively with the Airflow REST API. The release is intended to provide secure, auditable, remote management of Airflow deployments without direct metadata-database access, while remote commands gradually move from the existing Airflow CLI. ### Source excerpt We are thrilled to announce the first major release of airflowctl 0.1.0, the new secure, API-driven command-line interface (CLI) for Apache Airflow -- built under AIP-81. This release marks CLI to join the general posture on communicating through API. Airflow CLI joins the modern era of secure, auditable, and remote-first operations. Details: 📦 PyPI: https://pypi.org/project/apache-airflow-ctl/0.1.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow-ctl/stable/release_notes.html 🪶 Source Code: https://github.com/apache/airflow/tree/main/airflow-ctl 🎯 What is airflowctl? airflowctl is a new command-line interface for Apache Airflow that interacts exclusively with the Airflow REST API. It provides a secure, auditable, and consistent way to manage Airflow deployments -- without direct access to the metadata database. 🔄 Coexistence with Airflow CLI The Airflow CLI will continue as intended, primarily for admin tasks such as running Airflow components (airflow api-server, airflow scheduler) or managing the metadata database (airflow db init). airflowctl focuses on operational commands that interact with Airflow resources via the API (airflowctl dagrun trigger, airflowctl connection create, etc.). We defined the commands falls under two main categories: Remote Commands: Operations that can be provided via API (e.g., managing DAGs, connections, variables, triggering DAG runs) are now available in airflowctl and will be the recommended approach going forward. Local/Admin Commands: Operations that manage Airflow components or the metadata database will remain in the Airflow CLI. Of course, in the current state they will both have the remote commands. We are planning a zero-disruption migration path where Remote Commands will be gradually deprecated from the Airflow CLI as they achieve parity in airflowctl. 🔒 Why airflowctl? Until now, Airflow CLI connected directly to the metadata database, bypassing RBAC, authentication, and API logs. While convenient, this app ## Apache Airflow® 3 is Generally Available! DevFeed: [Apache Airflow® 3 is Generally Available!]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2025-04-22T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [event driven](), [Architecture & Design](), [React](), [Messaging](), [Cloud]() Tags: [airflow](), [apache-airflow](), [architecture](), [event-driven](), [hybrid-cloud](), [messaging](), [react](), [release](), [releases]() ### AI overview Apache Airflow 3.0 has been released as a major version. The release introduces a React-based UI, DAG versioning, improved backfill support, event-driven scheduling, and a Task Execution Interface with Task SDKs supporting secure execution across multi-cloud, hybrid-cloud, and local data center deployments. ### Source excerpt We announced our intent to focus on Apache Airflow 3.0® as the next big milestone for the Airflow project at the Airflow Summit in September 2024. We are delighted to announce that Airflow 3.0 is now released! A Major Release, Four Years in the Making Airflow 3.0 is the biggest release in Airflow's history--2.0 was released in 2020, and the last 4 years have seen incremental updates and releases every quarter with version 2.10 released in Q4 2024. With over 30 million monthly downloads (up over 30x since 2020) and 80,000 organizations (up from 25,000 in 2020) now using Airflow, we've seen an incredible growth in popularity since 2.0. Over the last four years, Airflow has grown to power business critical data workflows within organizations of all sizes. We have seen an exponential increase in the use cases for Airflow from its beginnings with ETL, ELT, and Reverse ETL, with over 30% of Airflow users using it for MLOps, and 10% using it for GenAI workflows. Airflow 3 is a response to this use case expansion and is the standard for data application development across the enterprise. Here are some highlights: Airflow 3 is significantly easier to use for data practitioners and incorporates their key requests for critical changes to Airflow. Early user reactions to features such as the new React based UI, DAG Versioning, and improved Backfill support have been incredibly positive. I was ecstatic to see the reaction from data engineers when I demonstrated this at a recent Airflow meetup. The seamless UI transition of navigating between Asset-oriented workflows and Task-oriented workflows is beautiful. Once again, Airflow lets the developer choose how you want to develop and navigate without imposing any restrictions. Introduction of Event Driven Scheduling enables Airflow to seamlessly integrate with messaging providers and react to events happening and data assets being updated outside of Airflow. The big architecture change with the introduction of the Task Execution Inte [Next page]()