# airflow Platform for programmatically authoring, scheduling, and monitoring workflows. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Refreshing the Travel-Time Map Behind Lyft's Marketplace: Rebuilding Neighborhood Reachability... DevFeed: [Refreshing the Travel-Time Map Behind Lyft's Marketplace: Rebuilding Neighborhood Reachability...]() Original publisher: [Read original article]() Author: Manjunath Shettar Published: 2026-09-10T16:12:28Z Content type: article Language: en Sources: [Lyft Engineering - Medium]() Topics: [dataset](), [airflow]() Tags: [airflow](), [data-processing](), [feature-engineering](), [latency](), [lyft](), [offline](), [tea]() ### AI overview Lyft describes rebuilding its Neighborhood Reachability Signals: offline, regional travel-time matrices between geohash-6 cells and their associated neighborhood-center lists. The refresh replaces older static data and is intended to support marketplace pricing, driver guidance, and demand heatmaps, with future work aimed at time-aware travel times. ### Source excerpt Refreshing the Travel-Time Map Behind Lyft's Marketplace: Rebuilding Neighborhood Reachability Signals Every time Lyft calculates pricing to balance a market, nudges a driver toward an under-served pocket of a city, or paints a heatmap of where demand is building, there is a quiet lookup table doing work in the background. It answers a deceptively simple question: how long does it take to get from here to there?, for millions of pairs of places, across hundreds of regions. That lookup table is the Neighborhood Reachability Signal, and for years large parts of it were frozen in a snapshot of the world from 2018-2019. This is the story of how we rebuilt it, why a refresh substantial enough to be worth adopting was what finally moved Pricing to switch, the cleanly positive results that came out of that switch, and where we're taking it next, from one static file per region to time-aware travel times that change with the rhythm of the day. What is a Neighborhood Reachability Signal? A geohash is a compact way of carving the world into a grid of cells. At geohash-6 resolution, each cell is roughly the size of a few city blocks. Slice a region into geohash-6 cells and you get a clean, discrete coordinate system for "neighborhoods" that downstream systems can reason about. The Forecasting & Real-Time Optimization (FORTOP) team produces the Neighborhood Reachability Signals dataset, which consists of two companion files for each region: Neighborhood Reachability Matrix: the estimated travel time, in minutes, between the centers of pairs of geohash-6 cells. Think of it as a sparse origin-to-destination travel-time matrix for a region. Neighborhood Centers: the list of all geohashes that appear in the ETA files for that region, i.e. the "vocabulary" of cells that the marketplace is allowed to talk about. Both files are generated offline on a schedule by an Airflow DAG. They are static in the sense that they are precomputed and shipped, rather than queried live (which is exact ## How dbt works, and why orchestrators shouldn't split it into tasks DevFeed: [How dbt works, and why orchestrators shouldn't split it into tasks]() Original publisher: [Read original article]() Author: Ruben Fiszel Published: 2026-08-20T00:00:00Z Content type: article Language: en Sources: [Windmill Blog]() Topics: [Compiler](), [Job](), [airflow](), [data](), [Command-line interface]() Tags: [airflow](), [bigquery](), [cli](), [dagster](), [data](), [data-pipelines](), [dbt](), [dbt-data-pipelines-airflow-dagster-orchestration](), [job](), [orchestration](), [scheduler](), [series]() ### AI overview This primer explains dbt's architecture and examines five ways teams orchestrate it in production. It argues that running a project as one dbt command, while reading dbt's per-model execution state, is generally more efficient and less fragile than creating one orchestrator task per model. It discusses Dagster and astronomer-cosmos's convergence on this approach, including a reported cost comparison. ### Source excerpt How is dbt actually orchestrated, and why does running it as one job beat one task per model? A primer on dbt as a compiler with a scheduler attached, the five ways teams wrap it, and why both Dagster and astronomer-cosmos converged on a single dbt invocation with per-model state projected out of it. ## Quasi-Agentic Pipelines with Databricks and Apache Airflow DevFeed: [Quasi-Agentic Pipelines with Databricks and Apache Airflow]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-08-10T21:23:57Z Content type: tutorial Language: en Sources: [Data Engineering Central]() Topics: [databricks](), [airflow](), [Data pipelines](), [Large Language Model](), [data-engineering]() Tags: [airflow](), [apache-airflow](), [data-engineering](), [databricks](), [llms](), [pipelines]() ### AI overview A practical developer discussion of incorporating LLMs and agents into existing data workflows using Databricks and Apache Airflow. It also examines determinism in data pipelines and the gap between business requirements and engineering implementation. ### Source excerpt the strange space in between ## How We Refresh Razorpay's Data Warehouse 10x Faster with Graphs and Indexes DevFeed: [How We Refresh Razorpay's Data Warehouse 10x Faster with Graphs and Indexes]() Original publisher: [Read original article]() Author: Amit Prabhu Published: 2026-07-14T14:06:16Z Content type: article Language: en Sources: [Razorpay Engineering - Medium]() Topics: [data](), [airflow](), [Apache Spark](), [Amazon S3](), [Apache Iceberg](), [microservices architecture](), [parquet](), [Architecture & Design]() Tags: [airflow](), [analytics](), [apache-iceberg](), [architecture](), [batch](), [data](), [graphs](), [infrastructure](), [parquet](), [pipelines](), [razorpay](), [spark](), [trino](), [warehouse]() ### AI overview Razorpay describes its data warehouse refresh pipeline, which builds wide denormalized Facts by joining data from multiple microservices. The article covers the original Airflow- and Spark-based full-refresh process, the underlying lake formats and query layer, and the scaling challenges that led the team to reconsider refresh strategy, data layout, and high-cardinality dimensions. ### Source excerpt Contributors: Utkarsh Koppikar Rohan Background Razorpay provides the payment infrastructure for millions of merchants globally. Behind every payment, settlement, and refund is a microservices architecture where each service owns its own database. While this keeps services independent and scalable, it creates a challenge for stakeholders who need to see across those boundaries. The Data Platform team manages the infrastructure that bridges this gap. Transactional data flows into the lake via CDC pipelines, ingested onto S3 in Delta Lake, Apache Iceberg, or plain Parquet formats. On top of the lake, we build domain-specific warehouse tables -- wide, pre-joined tables that co-locate all the data a consumer needs, queryable via Trino. These power two use cases: Analytics (internal dashboards on Tableau and Superset) and Reporting (merchants and regulated entities who download structured data exports; Razorpay generates nearly a million such reports per month). The warehouse tables that power both use cases are called Facts. A Fact is a flat denormalised table on S3, produced by joining 10 to 30 microservice tables and materialising the result once. A settlement Fact, for example, merges payments, refunds, adjustments, and card details into a single wide row so that a dashboard or report reads from a single table instead of joining across services in real time. It is closer to a domain-specific materialised view than a classical data warehouse fact table. We maintain over 50 such Facts, and approximately 40% of all merchant reports are served directly from them. As data volumes and the number of entities per fact grew, the batch generation pipeline began to show its limits, prompting us to rethink the refresh strategy, the data layout, and how to handle high-cardinality dimensions. The rest of this post covers that journey. The Full Refresh Pipeline: Our Baseline and the Pain The original full-refresh pipeline was straightforward. Schedule: Airflow schedules Spark jobs o ## Apache Airflow 3.3.0: Stateful Tasks and Multi-Language Support DevFeed: [Apache Airflow 3.3.0: Stateful Tasks and Multi-Language Support]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-07-06T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [SDKs](), [Go Language](), [Java]() Tags: [airflow](), [apache-airflow](), [go](), [java](), [release](), [release-notes](), [sdk](), [state]() ### AI overview Apache Airflow 3.3.0 introduces durable state stores for tasks and assets, a Language Task SDK for Java and Go, expanded asset partitioning, and pluggable retry policies. ### Source excerpt We're proud to announce the release of Apache Airflow 3.3.0! Where 3.2 brought precision to data with asset partitioning, 3.3 gives your tasks memory and multi-language support: a first-class state store for tasks and assets, a Language Task SDK for writing task logic in Java and Go, a major expansion of asset partitioning, and pluggable retry policies. 🎯 Release Highlights 📦 PyPI: https://pypi.org/project/apache-airflow/3.3.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.3.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.3.0/release_notes.html 🐳 Docker Image: docker pull apache/airflow:3.3.0 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.3.0 🗃 Task & Asset State Store (AIP-103): Tasks That Remember Until now, if a task needed to remember something across retries or runs -- a cursor, a checkpoint, a high-water mark -- you reached for XComs, an external store, or a clever Variable hack. Airflow 3.3 makes durable task state a first-class concept. Tasks can persist arbitrary key-value state that survives across retries and runs via a new task_state_store accessor, and assets can carry their own state via asset_state_store -- both available directly from the Task SDK. State lives in the metadata database by default, or in a custom worker-side backend ([workers] state_store_backend), supports per-key retention with periodic garbage collection and an optional clear_on_success, and is fully manageable through the Core API and Execution API. Task State Store Task state is scoped to a specific task instance and persists across retries. Use it to track coordination state like remote job IDs, cursors, or progress checkpoints: @task def extract_data(**context): task_state = context["task_state_store"] # Resume from where we left off on retry cursor = task_state.get("last_cursor", default=0) records = fetch_records(since=cursor) new_cursor = records[-1]["id"] task_state.set("last_cursor", new_cursor) return records Persisted st ## Migrating from a Monolithic Orchestrator to Apache Airflow DevFeed: [Migrating from a Monolithic Orchestrator to Apache Airflow]() Original publisher: [Read original article]() Author: Poorva Patil Published: 2026-06-03T09:20:17Z Content type: article Language: en Sources: [Helpshift]() Topics: [airflow](), [Orchestration](), [Clojure](), [Python](), [Monitoring]() Tags: [apache-airflow](), [architecture](), [aws](), [clojure](), [data-engineering](), [etl](), [monitoring](), [monolithic-architecture](), [orchestration](), [python](), [software-architecture]() ### AI overview The article describes a migration from a monolithic Clojure-based scheduler to Apache Airflow. The legacy system coupled event pipelines, dependencies, and an EMR step, making workflows difficult to understand, maintain, monitor, and debug. The authors report that Airflow's Python-based DAGs made dependencies, retries, and scheduling more readable and helped improve onboarding and maintainability. ### Source excerpt Photo by Corinne Kutz on UnsplashBefore we knew better Our orchestration system started as a simple internal solution to manage event pipelines and trigger downstream jobs. Over time, as more workflows and dependencies were added, it gradually evolved into a tightly coupled monolithic scheduler that became increasingly difficult to understand and maintain. Understanding how a workflow executed often meant looking through multiple files, configurations and database tables. For newer team members, onboarding into the system took time because much of the workflow context was distributed across different parts of the codebase. Even relatively small changes required careful coordination to ensure existing pipelines continued to work as expected. Similarly, debugging typically involved manually tracing logs and rerunning jobs to better understand execution behavior. Limitations of our legacy design We had a monolithic architecture written in Clojure that bundled all our event pipelines together, added dependencies between them and triggered a Lambda function. Legacy Workflow This Lambda function added a single monolithic step to the EMR cluster. If there was an issue in any one of the pipelines, the entire flow would fail due to the single step on the cluster. We did not have step-wise monitoring in the old design, so during on-call situations it became very difficult to identify which part of the pipeline was causing the issue. Photo by Tim Gouw on Unsplash There was no single place to answer basic questions like: What runs first? What happens if this step fails? How do I re-run just one part safely? The scheduler worked, but it was hard to understand, hard to maintain and even harder to explain. That's when we realized we needed a better way. What we actually needed Our aim was less about fancy scheduling features and more about making our daily work easier and more reliable. Simpler onboarding, less mental overhead Our existing step scheduler was built in Clojure and c ## From SSH to REST: A Security-Driven Modernization of Slack's EMR Data Pipelines DevFeed: [From SSH to REST: A Security-Driven Modernization of Slack's EMR Data Pipelines]() Original publisher: [Read original article]() Author: Mahendran Vasagam Published: 2026-05-05T14:00:01Z Content type: article Language: en Sources: [Engineering at Slack]() Topics: [Security](), [OpenSSH](), [Amazon Web Services](), [airflow](), [DataOps](), [Architecture & Design](), [Kubernetes]() Tags: [airflow](), [architecture](), [aws](), [big-data](), [data-analytics](), [data-engineering](), [data-infrastructure](), [kubernetes](), [modernization](), [security](), [ssh](), [uncategorized]() ### AI overview Slack describes migrating more than 700 SSH-based data pipeline jobs to a REST-based architecture across eight data regions, eliminating SSH access to production AWS EMR clusters without downtime. The article explains the security and operational problems that motivated the modernization, including attack-surface exposure, key-management overhead, resource contention, broken connections, zombie jobs, and unreliable job-status detection. ### Source excerpt Excerpt By 2024, Slack's data platform had accumulated 700+ SSH-based operators orchestrating critical data pipelines. We're talking daily search indexing that processed terabytes of data, analytics jobs powering business intelligence, the whole shebang. Every single one of these jobs required direct SSH access to production AWS Elastic MapReduce (EMR) clusters. We had a massive security... ## Agentic Workloads on Airflow: Observable, Retryable, and Auditable by Design DevFeed: [Agentic Workloads on Airflow: Observable, Retryable, and Auditable by Design]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-04-15T00:00:00Z Content type: tutorial Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Artificial Intelligence](), [Large Language Model](), [SQL](), [datasets](), [Agent Harness]() Tags: [agentic](), [ai](), [airflow](), [community](), [csv](), [llm](), [sql](), [survey](), [tutorial](), [workflow]() ### AI overview This article explains how to build agentic workloads on Airflow 3 using the 2025 Airflow Community Survey dataset and the apache-airflow-providers-common-ai provider. It distinguishes single-query LLM pipelines from workflows that query multiple independent dimensions and synthesize their results, mapping each sub-query to a named Airflow task and using Dynamic Task Mapping and XCom for orchestration and observability. ### Source excerpt A question like "How does AI tool usage vary across Airflow versions?" has a natural SQL shape: one cross-tabulation, one result. A question like "What does a typical Airflow deployment look like for practitioners who are actively using AI in their workflow?" does not. It requires querying executor type, deployment method, cloud provider, and Airflow version independently, each filtered to the same respondent group, then synthesizing the results into a coherent picture. No single query returns the answer. The answer emerges from the relationship between all of them. This is where Airflow's agentic pattern begins: not when you add an LLM to a workflow, but when the structure of the work itself depends on running multiple LLM calls whose outputs feed a synthesis step. This post builds that pattern using the 2025 Airflow Community Survey data set and the apache-airflow-providers-common-ai provider for Airflow 3. If you haven't read the introductory survey analysis post yet, start there for a walkthrough of the single-query interactive and scheduled pipelines. This post picks up where that one ends. The Agentic Gap in the Single-Query Pattern The interactive and scheduled survey DAGs from the introductory post each do one thing: translate a natural language question into SQL, execute it against the CSV, and return the result. The LLM is involved once. The structure of the pipeline does not change based on what that LLM call returns. That is not a limitation to fix. It is the right design for that class of question. For a large fraction of production AI workflows, a single well-structured LLM call with good context is sufficient and preferable. The pattern becomes agentic when two things are true simultaneously: The question requires querying multiple independent dimensions The synthesis step, the thing that produces the final answer, depends on all of those results In an agent harness framework, this would be handled inside a reasoning loop: the LLM decides to call a to ## Ask Your Survey Anything: Building AI Analysis Pipelines with Airflow 3 DevFeed: [Ask Your Survey Anything: Building AI Analysis Pipelines with Airflow 3]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-04-15T00:00:00Z Content type: tutorial Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Artificial Intelligence](), [Large Language Model](), [SQL](), [CSV](), [data]() Tags: [ai](), [airflow](), [apache-airflow](), [automated](), [community](), [configuration](), [csv](), [executor](), [llm](), [pipelines](), [providers](), [scheduled](), [sql](), [tutorial]() ### AI overview A tutorial showing how to build interactive and scheduled AI analysis pipelines with Airflow 3. The pipelines use survey CSV data, generate and run SQL queries, validate schemas, and either request human approval or email results automatically. ### Source excerpt The 2025 Airflow Community Survey collected responses from nearly 6,000 practitioners across 168 questions. You can open a spreadsheet and filter, or write SQL by hand. But what if you could just ask a question and have Airflow figure out the query, run it, and bring the result back for your approval? This post builds two pipelines that do exactly that, using the apache-airflow-providers-common-ai provider for Airflow 3. The first pipeline is interactive: a human reviews the question before it reaches the LLM and approves the result before the DAG finishes. The second is scheduled: it downloads fresh survey data, validates the schema, runs the query unattended, and emails the result. If you haven't seen the common.ai provider overview yet, start there for a tour of all the operators. This post goes deep on a concrete end-to-end example. Two Pipelines, One Example File Both DAGs live in example_llm_survey_analysis.py and share the same schema context and datasource configuration. example_llm_survey_interactive: trigger manually, review at both ends: prompt_confirmation -> generate_sql -> run_query -> extract_data -> result_confirmation (HITLEntryOperator) (LLMSQLQuery) (Analytics) (@task) (ApprovalOperator) example_llm_survey_scheduled: runs @monthly, no human in the loop: download_survey -> prepare_csv -> check_schema -> generate_sql -> run_query -> extract_data -> send_result (HttpOperator) (@task) (LLMSchema (LLMSQLQuery) (Analytics) (@task) (@task / Email) Compare) The Data The Airflow Community Survey 2025 CSV has 5,856 rows and 168 columns covering everything from Airflow version and executor type to cloud provider, company size, and AI tool usage. A few highlights from the data: 3,320 respondents identify as Data Engineers 2,032 use AWS as their primary cloud provider for Airflow 1,445 are already running Airflow 3 1,351 say they often use AI tools to write Airflow code Those last two numbers together are part of why this example exists: the people most likely to use th ## Introducing the Common AI Provider: LLM and AI Agent Support for Apache Airflow DevFeed: [Introducing the Common AI Provider: LLM and AI Agent Support for Apache Airflow]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-04-14T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Large Language Model](), [AI Agent](), [Pydantic](), [Artificial Intelligence]() Tags: [2025](), [ai](), [ai-agent](), [airflow](), [apache](), [apache-airflow](), [community](), [llm](), [release]() ### AI overview Apache Airflow is releasing the Common AI Provider 0.1.0, a provider package that adds LLM and AI agent capabilities directly to Airflow. Built on Pydantic AI, it supports more than 20 model providers through a single install and requires Apache Airflow 3.0 or newer. ### Source excerpt At Airflow Summit 2025, we previewed what native AI integration in Apache Airflow could look like. Today we're shipping it. apache-airflow-providers-common-ai 0.1.0 adds LLM and agent capabilities directly to Airflow. Not a wrapper around another framework, but a provider package that plugs into the orchestrator you already run. It's built on Pydantic AI and supports 20+ model providers (OpenAI, Anthropic, Google, Azure, Bedrock, Ollama, and more) through a single install. pip install 'apache-airflow-providers-common-ai' Requires Apache Airflow 3.0+. Note: This is a 0.x release. We're actively looking for feedback and iterating fast, so breaking changes are possible between minor versions. Try it, tell us what works and what doesn't. Your input directly shapes the API. By the Numbers 6 Operators 6 TaskFlow decorators 5 Toolsets 4 Connection types 20+ Supported model providers via Pydantic AI The Decorator Suite Every operator has a matching TaskFlow decorator. @task.llm: Single LLM Call Send a prompt, get text or structured output back. from pydantic import BaseModel from airflow.providers.common.compat.sdk import dag, task @dag def my_pipeline(): class Entities(BaseModel): names: list[str] locations: list[str] @task.llm( llm_conn_id="my_openai_conn", system_prompt="Extract named entities.", output_type=Entities, ) def extract(text: str): return f"Extract entities from: {text}" extract("Alice visited Paris and met Bob in London.") my_pipeline() The LLM returns a typed Entities object, not a string you have to parse. Downstream tasks get structured data through XCom. @task.agent: Multi-Step Agent with Tools When the LLM needs to query databases, call APIs, or read files across multiple steps, use @task.agent. The agent picks which tools to call and loops until it has an answer. from airflow.providers.common.ai.toolsets.sql import SQLToolset from airflow.providers.common.compat.sdk import dag, task @dag def sql_analyst(): @task.agent( llm_conn_id="my_openai_conn", sys ## Apache Airflow 3.2.0: Data-Aware Workflows at Scale DevFeed: [Apache Airflow 3.2.0: Data-Aware Workflows at Scale]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-04-07T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [data](), [Orchestration](), [Release notes](), [SDK](), [Amazon S3](), [BigQuery]() Tags: [airflow](), [apache](), [apache-airflow](), [bigquery](), [data](), [docker-image](), [orchestration](), [pypi](), [release](), [release-notes](), [s3](), [sdk](), [workflows]() ### AI overview Apache Airflow 3.2.0 introduces asset partitioning for more granular data-aware scheduling, along with multi-team deployments, synchronous deadline alert callbacks, and continued Task SDK separation. Partition-aware workflows trigger downstream DAGs only when the relevant partition is updated. ### Source excerpt We're proud to announce the release of Apache Airflow 3.2.0! Airflow 3.1 puts humans at the center of automated workflows. 3.2 brings that same precision to data: Asset partitioning for granular pipeline orchestration, multi-team deployments for enterprise scale, synchronous deadline alert callbacks, and continued progress toward full Task SDK separation. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/3.2.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.2.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.2.0/release_notes.html 🐳 Docker Image: docker pull apache/airflow:3.2.0 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.2.0 🗂 Asset Partitioning (AIP-76): Only the Right Work Gets Triggered Asset partitioning has been one of the most requested additions to data-aware scheduling. If you work with date-partitioned S3 paths, Hive table partitions, BigQuery partitions, or really any partitioned data store, you've dealt with this: An upstream task updates one partition, and every downstream Dag fires regardless of which slice actually changed. It's wasteful, and for large deployments it creates real operational noise. Asset partitioning in 3.2 makes this granular. Downstream Dags trigger only when the specific partition they care about gets updated. It's the biggest change to data-aware scheduling since Assets were introduced, and it turns partition-driven orchestration into something Airflow handles natively rather than something you work around. Key Capabilities Partition-driven scheduling: Dags trigger on specific partition updates, not every asset change CronPartitionTimetable: Schedule Dags against partitions using cron expressions. Also available in the Task SDK Backfill for partitioned Dags: Backfill historical partitions without re-triggering everything downstream (#61464) Multi-asset partitions: A single Dag can listen for partitions across multiple assets, which matters when your downstream work depe ## Introducing the Apache Airflow Registry DevFeed: [Introducing the Apache Airflow Registry]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-03-19T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Data pipelines](), [API](), [JSON]() Tags: [airflow](), [apache](), [apache-airflow](), [community](), [data-pipelines](), [messaging](), [notifications](), [openai](), [registry]() ### AI overview Apache Airflow launches the Airflow Registry, a searchable catalog of official providers and modules. It includes provider and module search, installation and compatibility details, connection generation in URI, JSON, and environment-variable formats, ecosystem statistics, and a structured JSON API. ### Source excerpt Today we're launching the Apache Airflow Registry -- a searchable catalog of every official Airflow provider and its modules, live at airflow.apache.org/registry/. Need an S3 operator? A Snowflake hook? An OpenAI sensor? The Registry helps you find, compare, and configure the right components for your data pipelines -- without digging through docs or PyPI pages. By the Numbers 98 Official providers 1,602 Modules (operators, hooks, sensors, triggers, transfers, and more) 329M+ Monthly PyPI downloads across all providers 125+ Integrations with cloud platforms, databases, ML tools, and messaging services Search Everything Hit Cmd+K from any page and start typing. Results show up instantly, grouped by Providers and Modules, with type badges so you can tell a hook from an operator at a glance. Provider Pages Each provider gets a dedicated page with everything in one place: install command with copy-to-clipboard, version selector, extras dropdown, compatibility info, connection types, and the full module listing organized by type. The Amazon provider, for example, has 372 modules across operators, hooks, sensors, triggers, transfers, and more. Module type tabs let you filter to exactly what you're looking for, and a category sidebar groups modules by AWS service (S3, Lambda, Glue, Step Functions, etc.). Connection Builder Click any connection type badge on a provider page, fill in the fields, and the builder generates the connection in three formats -- URI, JSON, and Env Var -- ready to copy into your configuration. No more guessing URI encoding or JSON structure. Explore by Category Not sure which provider you need? The Explore page organizes providers into categories: Cloud Platforms, Databases, Data Warehouses, Messaging & Notifications, AI & Machine Learning, Data Processing, and more. Statistics The Stats page breaks down the ecosystem: 848 operators, 298 hooks, 164 triggers, 157 sensors, 83 transfers, and more -- plus top providers by downloads and module count. JSON API ## Data-to-Production: Bridging the Gap Between Iceberg and Live Microservices DevFeed: [Data-to-Production: Bridging the Gap Between Iceberg and Live Microservices]() Original publisher: [Read original article]() Author: Wix Engineering Published: 2026-02-17T11:04:08Z Content type: article Language: en Sources: [Wix Engineering]() Topics: [Apache Iceberg](), [data observability](), [Microservice](), [clickhouse](), [Amazon S3](), [Architecture & Design](), [Apache Spark](), [airflow](), [Back end](), [API]() Tags: [airflow](), [amazon-s3](), [apache-iceberg](), [api](), [architecture](), [clickhouse](), [data](), [data-engineering](), [json](), [microservices](), [spark]() ### AI overview Wix describes Data-to-Production, a platform that activates data from Amazon S3 and Apache Iceberg for backend microservices. The system ingests Iceberg data into ClickHouse and serves it through a type-safe JSON API, using metadata governance and an Airflow and Python ingestion engine. ### Source excerpt At Wix, our Data Warehouse (DWH) is a massive repository of insights. Built on Amazon S3 using Apache Iceberg table formats, and populated by Trino and Spark jobs, it houses petabytes of data--from user segmentation and logs to AI chat analytics. However, storage is only half the battle. The real challenge--and the "holy grail" for many data engineering teams--is Activation : taking that petabyte-scale data and exposing it to backend microservices with millisecond latency, high availability, and... ## The 2026 AI Data Engineer Roadmap DevFeed: [The 2026 AI Data Engineer Roadmap]() Original publisher: [Read original article]() Author: Zach Wilson Published: 2026-02-05T20:26:37Z Content type: article Language: en Sources: [DataExpert.io Newsletter]() Topics: [Artificial Intelligence](), [data-engineering](), [AI-assisted coding](), [Apache Spark](), [airflow](), [flink]() Tags: [ai](), [ai-coding-agents](), [best-practices](), [data](), [data-engineering]() ### AI overview This article presents a 2026 roadmap for data engineering as AI automates more pipeline, SQL, Spark, testing, migration, and operational work. It compares responsibilities that are becoming more automated with areas such as system design, deep technical debt, performance tradeoffs, and organizational constraints where human expertise remains important. ### Source excerpt And how to avoid getting replaced ## Airflow Survey 2025 DevFeed: [Airflow Survey 2025]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-01-22T00:00:00Z Content type: article Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [data](), [CSV]() Tags: [2025](), [airflow](), [analysis](), [apache-airflow](), [community](), [csv](), [download](), [form](), [graphics](), [interactive](), [report](), [responses](), [share](), [survey](), [users](), [view]() ### AI overview This page presents the Airflow Survey 2025 interactive report, hosted by Astronomer and sponsored and supported by the Apache Airflow community. The supplied evidence does not include the survey findings. ### Source excerpt The interactive report is hosted by Astronomer. The Apache Airflow community thanks Astronomer for running this survey, for sponsoring it and providing the report in this form, and for their effort in marketing, analysis, and preparing the graphics. View raw data Download survey responses (CSV) ## Wix's AirBot: An AI-Powered Slack Agent for Investigating Data Pipeline Failures DevFeed: [Wix's AirBot: An AI-Powered Slack Agent for Investigating Data Pipeline Failures]() Original publisher: [Read original article]() Author: Wix Engineering Published: 2026-01-15T07:30:31Z Content type: article Language: en Sources: [Wix Engineering]() Topics: [airflow](), [AI, ML & Data Engineering](), [data-engineering](), [Cloud](), [Slack]() Tags: [agent](), [ai](), [airflow](), [automation](), [bugs](), [data-engineering](), [engineering](), [http](), [process](), [processes](), [slack]() ### AI overview The article describes Wix's engineering environment, where thousands of Apache Airflow pipelines operate across a large cloud-based data platform. It presents AirBot, an AI-powered Slack agent intended to help engineers investigate pipeline failures across Airflow, Spark, Kubernetes, and related services. ### Source excerpt 1. Introduction: The Challenge Understanding the engineering need for AirBot starts with recognizing the scale of the ecosystem it operates within. Wix operates a massive cloud-based development platform serving 250 million users. This generates over 4 billion HTTP transactions daily, feeding a data lake that currently holds 7 petabytes of data. To manage this volume, Wix Data Engineering maintains over 3,500 Apache Airflow pipelines (DAGs) . These pipelines handle everything from ETL... ## How Vinted Standardizes Large-Scale Decentralized Data Pipelines DevFeed: [How Vinted Standardizes Large-Scale Decentralized Data Pipelines]() Original publisher: [Read original article]() Author: Oscar Ligthart Published: 2025-12-29T00:00:00Z Content type: article Language: en Sources: [Vinted]() Topics: [Data Infrastructure](), [Orchestration](), [airflow](), [Cloud]() Tags: [airflow](), [business-logic](), [cloud](), [data](), [dependencies](), [domain](), [gcp](), [infrastructure](), [orchestration](), [pipelines](), [scale]() ### AI overview Vinted describes its migration to cloud-based data infrastructure and the challenges of coordinating dependencies across decentralized teams and domains. The article explains why relying on teams to hand-craft Airflow DAGs would add unnecessary complexity, and discusses adapting the conventional dbt-with-Airflow approach to Vinted's scale and cost constraints. ### Source excerpt TL;DR: How Vinted standardizes large-scale decentralized data pipelines. ## The 2025 AI + Data Engineering Roadmap DevFeed: [The 2025 AI + Data Engineering Roadmap]() Original publisher: [Read original article]() Author: Zach Wilson Published: 2025-10-17T22:35:45Z Content type: tutorial Language: en Sources: [DataExpert.io Newsletter]() Topics: [data-engineering](), [Python](), [SQL](), [Artificial Intelligence](), [airflow](), [BigQuery](), [Orchestration](), [Apache Spark](), [data-modeling](), [Data Quality](), [Embeddings](), [Retrieval Augmented Generation (RAG)]() Tags: [2025](), [ai](), [airflow](), [bigquery](), [count](), [course](), [data-engineering](), [data-modeling](), [data-quality](), [embeddings](), [engineering](), [framer](), [newsletter](), [orchestration](), [python](), [rag](), [real-time](), [right-join](), [spark](), [sql]() ### AI overview A 2025 roadmap for entering data engineering, covering foundational SQL and Python skills, distributed computing, orchestration, data modeling, data quality, AI and data integrations, portfolio projects, and personal branding. ### Source excerpt Getting a data engineering job is complicated. ## Apache Airflow CTL aka airflowctl 0.1.0 DevFeed: [Apache Airflow CTL aka airflowctl 0.1.0]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2025-10-15T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Command-line interface](), [Release notes](), [API](), [REST API](), [releases](), [Authentication](), [audit]() Tags: [airflow](), [apache-airflow](), [audit-trail](), [authentication](), [cli](), [command-line](), [release](), [release-notes](), [rest-api]() ### AI overview Apache Airflow announces airflowctl 0.1.0, a command-line interface that interacts exclusively with the Airflow REST API. The release is intended to provide secure, auditable, remote management of Airflow deployments without direct metadata-database access, while remote commands gradually move from the existing Airflow CLI. ### Source excerpt We are thrilled to announce the first major release of airflowctl 0.1.0, the new secure, API-driven command-line interface (CLI) for Apache Airflow -- built under AIP-81. This release marks CLI to join the general posture on communicating through API. Airflow CLI joins the modern era of secure, auditable, and remote-first operations. Details: 📦 PyPI: https://pypi.org/project/apache-airflow-ctl/0.1.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow-ctl/stable/release_notes.html 🪶 Source Code: https://github.com/apache/airflow/tree/main/airflow-ctl 🎯 What is airflowctl? airflowctl is a new command-line interface for Apache Airflow that interacts exclusively with the Airflow REST API. It provides a secure, auditable, and consistent way to manage Airflow deployments -- without direct access to the metadata database. 🔄 Coexistence with Airflow CLI The Airflow CLI will continue as intended, primarily for admin tasks such as running Airflow components (airflow api-server, airflow scheduler) or managing the metadata database (airflow db init). airflowctl focuses on operational commands that interact with Airflow resources via the API (airflowctl dagrun trigger, airflowctl connection create, etc.). We defined the commands falls under two main categories: Remote Commands: Operations that can be provided via API (e.g., managing DAGs, connections, variables, triggering DAG runs) are now available in airflowctl and will be the recommended approach going forward. Local/Admin Commands: Operations that manage Airflow components or the metadata database will remain in the Airflow CLI. Of course, in the current state they will both have the remote commands. We are planning a zero-disruption migration path where Remote Commands will be gradually deprecated from the Airflow CLI as they achieve parity in airflowctl. 🔒 Why airflowctl? Until now, Airflow CLI connected directly to the metadata database, bypassing RBAC, authentication, and API logs. While convenient, this app ## Apache Airflow 3.1.0: Human-Centered Workflows DevFeed: [Apache Airflow 3.1.0: Human-Centered Workflows]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2025-09-25T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Automation](), [human review](), [Data Quality](), [Forms](), [Developer experience](), [Inference](), [React]() Tags: [apache-airflow](), [automation](), [data-quality](), [developer-experience](), [forms](), [human-review](), [inference](), [release](), [release-notes](), [ui]() ### AI overview Apache Airflow 3.1.0 introduces human-in-the-loop tasks for pausing automated workflows and collecting reviews through web forms. The release also adds internationalization support, developer experience improvements, restored Calendar and Gantt views, UI filtering, DAG pinning, and accessibility-focused theme updates. ### Source excerpt We are thrilled to announce the release of Apache Airflow 3.1.0, an update that puts humans at the center of data workflows. This release introduces powerful new capabilities for human decision-making in automated processes, comprehensive internationalization support, and significant developer experience improvements. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/3.1.0/ 📚 Core Airflow Docs: https://airflow.apache.org/docs/apache-airflow/3.1.0/ 📚 Task SDK Docs: https://airflow.apache.org/docs/task-sdk/1.1.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.1.0/release_notes.html 🪶 Sources: https://airflow.apache.org/docs/apache-airflow/3.1.0/installation/installing-from-sources.html 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.1.0 🤝 Human-in-the-Loop (HITL): When Automation Meets Human Judgment This powerful capability bridges the gap between automated processes and human expertise, making Airflow invaluable for: AI/ML Model Validation: Pause inference pipelines for human review of model outputs Content Moderation: Route content through human reviewers before publication Approval Workflows: Require manager approval for sensitive operations Data Quality Gates: Allow data stewards to validate critical datasets HITL tasks pause in a deferred state while presenting intuitive web forms in the Airflow UI. Users with appropriate roles can review context data, DAG parameters, and XCom values before making informed decisions. Example Code: from airflow.sdk import DAG from airflow.providers.standard.operators.hitl import HITLOperator with DAG("content_moderation", schedule="@daily") as dag: moderate_content = HITLOperator( task_id="review_content", message="Please review this content for publication", data_key="content_to_review" ) 📊 UI Enhancements & Performance Calendar and Gantt Views Make Their Comeback Remember those beloved Calendar and Gantt chart views from Airflow 2.x? They're back, completely rebuilt for the modern ## Apache Airflow® 3 is Generally Available! DevFeed: [Apache Airflow® 3 is Generally Available!]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2025-04-22T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [event driven](), [Architecture & Design](), [React](), [Messaging](), [Cloud]() Tags: [airflow](), [apache-airflow](), [architecture](), [event-driven](), [hybrid-cloud](), [messaging](), [react](), [release](), [releases]() ### AI overview Apache Airflow 3.0 has been released as a major version. The release introduces a React-based UI, DAG versioning, improved backfill support, event-driven scheduling, and a Task Execution Interface with Task SDKs supporting secure execution across multi-cloud, hybrid-cloud, and local data center deployments. ### Source excerpt We announced our intent to focus on Apache Airflow 3.0® as the next big milestone for the Airflow project at the Airflow Summit in September 2024. We are delighted to announce that Airflow 3.0 is now released! A Major Release, Four Years in the Making Airflow 3.0 is the biggest release in Airflow's history--2.0 was released in 2020, and the last 4 years have seen incremental updates and releases every quarter with version 2.10 released in Q4 2024. With over 30 million monthly downloads (up over 30x since 2020) and 80,000 organizations (up from 25,000 in 2020) now using Airflow, we've seen an incredible growth in popularity since 2.0. Over the last four years, Airflow has grown to power business critical data workflows within organizations of all sizes. We have seen an exponential increase in the use cases for Airflow from its beginnings with ETL, ELT, and Reverse ETL, with over 30% of Airflow users using it for MLOps, and 10% using it for GenAI workflows. Airflow 3 is a response to this use case expansion and is the standard for data application development across the enterprise. Here are some highlights: Airflow 3 is significantly easier to use for data practitioners and incorporates their key requests for critical changes to Airflow. Early user reactions to features such as the new React based UI, DAG Versioning, and improved Backfill support have been incredibly positive. I was ecstatic to see the reaction from data engineers when I demonstrated this at a recent Airflow meetup. The seamless UI transition of navigating between Asset-oriented workflows and Task-oriented workflows is beautiful. Once again, Airflow lets the developer choose how you want to develop and navigate without imposing any restrictions. Introduction of Event Driven Scheduling enables Airflow to seamlessly integrate with messaging providers and react to events happening and data assets being updated outside of Airflow. The big architecture change with the introduction of the Task Execution Inte ## April 2025 Newsletter DevFeed: [April 2025 Newsletter]() Original publisher: [Read original article]() Author: Mark Needham Published: 2025-04-16T08:29:42Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [real-time](), [Data Infrastructure](), [data observability](), [Artificial Intelligence](), [airflow]() Tags: [2025](), [ai](), [airflow](), [aws](), [clickhouse](), [data](), [data-infrastructure](), [data-observability](), [newsletter](), [openai](), [real-time]() ### AI overview The April 2025 ClickHouse newsletter covers CloudQuery's experience with ClickHouse, the new query condition cache in version 25.3, ClickHouse's Rust development, the acquisition of HyperDX, community news, upcoming events, training, and ClickHouse's role in data observability and AI applications. ### Source excerpt Welcome to the April ClickHouse newsletter, which will round up what's happened in real-time data warehouses over the last month. ## Trading Airflow + EMR for Temporal + Bauplan: The Mediaset tale DevFeed: [Trading Airflow + EMR for Temporal + Bauplan: The Mediaset tale]() Original publisher: [Read original article]() Author: Stu Kendall Published: 2025-04-10T00:00:00Z Content type: article Language: en Sources: [Temporal Blog]() Topics: [airflow](), [Orchestration](), [Data Management](), [Amazon Web Services](), [Python](), [Amazon S3](), [data](), [Apache Iceberg]() Tags: [airflow](), [aws](), [community](), [data-management](), [formats](), [orchestration](), [python](), [reliability](), [s3]() ### AI overview This article describes how Mediaset replaced an Airflow-based AWS data stack with Temporal and Bauplan. The change produced a near-real-time news dashboard, with refresh time reduced from one hour to five minutes according to the supplied summary. ### Source excerpt Mediaset replaced Airflow and six AWS services with Temporal and Bauplan, cutting dashboard refresh time from 1 hour to 5 minutes in just 6 weeks. ## Deploying Airflow Data Pipelines with a DAG Synchronization Service DevFeed: [Deploying Airflow Data Pipelines with a DAG Synchronization Service]() Original publisher: [Read original article]() Author: Doximity Published: 2025-03-12T14:28:00Z Content type: article Language: en Sources: [Doximity]() Topics: [airflow](), [data](), [Amazon S3](), [CI/CD]() Tags: [airflow](), [amazon-s3](), [ci-cd](), [container](), [deployment](), [pipelines](), [s3]() ### AI overview Doximity describes replacing Airflow container image rebuilds and scheduled service restarts with a DAG synchronization service that continuously updates DAGs from Amazon S3. The approach reduced deployment times from about 30 minutes to under one minute and reduced UI disruption and downtime. ### Source excerpt At Doximity, we have nearly 20 data teams responsible for the development of data pipelines to support product and business intelligence needs. These teams rely on Apache Airflow to orchestrate over 900 active DAGs (fancy word for data pipelines), with dozens of updates deployed daily. However, with the growth of our data platform team, the bottlenecks in our deployment process for data pipelines could no longer be ignored. Deploying new pipelines or updating existing ones required building, publishing, and deploying a new Airflow container image--a process that could take up to 20 minutes. To make matters worse, pulling in the latest updates meant restarting Airflow services, causing temporary UI disruptions and delaying workflows--the salt in the wound, if you will. In this article, we will discuss in further details the bottlenecks of our legacy deployment system and how we overcame them by developing a DAG synchronization service that continuously updates DAGs directly from Amazon S3 object storage. This new approach not only accelerated deployment times--reducing them from 30 minutes to under one--but also improved system reliability, reduced downtime, and enhanced operational flexibility. Problem Overview Prior to the sync service, DAGs were deployed by building them into the Airflow container image using a CI/CD pipeline. This was both time-consuming and resource-intensive, with each build and delivery process taking nearly 20 minutes to complete. With dozens of teams pushing updates daily, each triggering a full image rebuild, these delays were a clear bottleneck to our productivity. Pulling in these changes required scheduled restarts of the Airflow Scheduler and Web Server at regular intervals. These restarts, while necessary, caused temporary (and dare I say, annoying) disruptions to the Airflow Web UI. Refreshing your screen only to be greeted by a 503 error was the virtual equivalent of stubbing your toe--frustrating and painfully avoidable. Factoring in the [Next page]()