# apache-airflow Published articles for apache-airflow. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Quasi-Agentic Pipelines with Databricks and Apache Airflow DevFeed: [Quasi-Agentic Pipelines with Databricks and Apache Airflow]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-08-10T21:23:57Z Content type: tutorial Language: en Sources: [Data Engineering Central]() Topics: [databricks](), [airflow](), [Data pipelines](), [Large Language Model](), [data-engineering]() Tags: [airflow](), [apache-airflow](), [data-engineering](), [databricks](), [llms](), [pipelines]() ### AI overview A practical developer discussion of incorporating LLMs and agents into existing data workflows using Databricks and Apache Airflow. It also examines determinism in data pipelines and the gap between business requirements and engineering implementation. ### Source excerpt the strange space in between ## Apache Airflow 3.3.0: Stateful Tasks and Multi-Language Support DevFeed: [Apache Airflow 3.3.0: Stateful Tasks and Multi-Language Support]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-07-06T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [SDKs](), [Go Language](), [Java]() Tags: [airflow](), [apache-airflow](), [go](), [java](), [release](), [release-notes](), [sdk](), [state]() ### AI overview Apache Airflow 3.3.0 introduces durable state stores for tasks and assets, a Language Task SDK for Java and Go, expanded asset partitioning, and pluggable retry policies. ### Source excerpt We're proud to announce the release of Apache Airflow 3.3.0! Where 3.2 brought precision to data with asset partitioning, 3.3 gives your tasks memory and multi-language support: a first-class state store for tasks and assets, a Language Task SDK for writing task logic in Java and Go, a major expansion of asset partitioning, and pluggable retry policies. 🎯 Release Highlights 📦 PyPI: https://pypi.org/project/apache-airflow/3.3.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.3.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.3.0/release_notes.html 🐳 Docker Image: docker pull apache/airflow:3.3.0 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.3.0 🗃 Task & Asset State Store (AIP-103): Tasks That Remember Until now, if a task needed to remember something across retries or runs -- a cursor, a checkpoint, a high-water mark -- you reached for XComs, an external store, or a clever Variable hack. Airflow 3.3 makes durable task state a first-class concept. Tasks can persist arbitrary key-value state that survives across retries and runs via a new task_state_store accessor, and assets can carry their own state via asset_state_store -- both available directly from the Task SDK. State lives in the metadata database by default, or in a custom worker-side backend ([workers] state_store_backend), supports per-key retention with periodic garbage collection and an optional clear_on_success, and is fully manageable through the Core API and Execution API. Task State Store Task state is scoped to a specific task instance and persists across retries. Use it to track coordination state like remote job IDs, cursors, or progress checkpoints: @task def extract_data(**context): task_state = context["task_state_store"] # Resume from where we left off on retry cursor = task_state.get("last_cursor", default=0) records = fetch_records(since=cursor) new_cursor = records[-1]["id"] task_state.set("last_cursor", new_cursor) return records Persisted st ## Migrating from a Monolithic Orchestrator to Apache Airflow DevFeed: [Migrating from a Monolithic Orchestrator to Apache Airflow]() Original publisher: [Read original article]() Author: Poorva Patil Published: 2026-06-03T09:20:17Z Content type: article Language: en Sources: [Helpshift]() Topics: [airflow](), [Orchestration](), [Clojure](), [Python](), [Monitoring]() Tags: [apache-airflow](), [architecture](), [aws](), [clojure](), [data-engineering](), [etl](), [monitoring](), [monolithic-architecture](), [orchestration](), [python](), [software-architecture]() ### AI overview The article describes a migration from a monolithic Clojure-based scheduler to Apache Airflow. The legacy system coupled event pipelines, dependencies, and an EMR step, making workflows difficult to understand, maintain, monitor, and debug. The authors report that Airflow's Python-based DAGs made dependencies, retries, and scheduling more readable and helped improve onboarding and maintainability. ### Source excerpt Photo by Corinne Kutz on UnsplashBefore we knew better Our orchestration system started as a simple internal solution to manage event pipelines and trigger downstream jobs. Over time, as more workflows and dependencies were added, it gradually evolved into a tightly coupled monolithic scheduler that became increasingly difficult to understand and maintain. Understanding how a workflow executed often meant looking through multiple files, configurations and database tables. For newer team members, onboarding into the system took time because much of the workflow context was distributed across different parts of the codebase. Even relatively small changes required careful coordination to ensure existing pipelines continued to work as expected. Similarly, debugging typically involved manually tracing logs and rerunning jobs to better understand execution behavior. Limitations of our legacy design We had a monolithic architecture written in Clojure that bundled all our event pipelines together, added dependencies between them and triggered a Lambda function. Legacy Workflow This Lambda function added a single monolithic step to the EMR cluster. If there was an issue in any one of the pipelines, the entire flow would fail due to the single step on the cluster. We did not have step-wise monitoring in the old design, so during on-call situations it became very difficult to identify which part of the pipeline was causing the issue. Photo by Tim Gouw on Unsplash There was no single place to answer basic questions like: What runs first? What happens if this step fails? How do I re-run just one part safely? The scheduler worked, but it was hard to understand, hard to maintain and even harder to explain. That's when we realized we needed a better way. What we actually needed Our aim was less about fancy scheduling features and more about making our daily work easier and more reliable. Simpler onboarding, less mental overhead Our existing step scheduler was built in Clojure and c ## Ask Your Survey Anything: Building AI Analysis Pipelines with Airflow 3 DevFeed: [Ask Your Survey Anything: Building AI Analysis Pipelines with Airflow 3]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-04-15T00:00:00Z Content type: tutorial Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Artificial Intelligence](), [Large Language Model](), [SQL](), [CSV](), [data]() Tags: [ai](), [airflow](), [apache-airflow](), [automated](), [community](), [configuration](), [csv](), [executor](), [llm](), [pipelines](), [providers](), [scheduled](), [sql](), [tutorial]() ### AI overview A tutorial showing how to build interactive and scheduled AI analysis pipelines with Airflow 3. The pipelines use survey CSV data, generate and run SQL queries, validate schemas, and either request human approval or email results automatically. ### Source excerpt The 2025 Airflow Community Survey collected responses from nearly 6,000 practitioners across 168 questions. You can open a spreadsheet and filter, or write SQL by hand. But what if you could just ask a question and have Airflow figure out the query, run it, and bring the result back for your approval? This post builds two pipelines that do exactly that, using the apache-airflow-providers-common-ai provider for Airflow 3. The first pipeline is interactive: a human reviews the question before it reaches the LLM and approves the result before the DAG finishes. The second is scheduled: it downloads fresh survey data, validates the schema, runs the query unattended, and emails the result. If you haven't seen the common.ai provider overview yet, start there for a tour of all the operators. This post goes deep on a concrete end-to-end example. Two Pipelines, One Example File Both DAGs live in example_llm_survey_analysis.py and share the same schema context and datasource configuration. example_llm_survey_interactive: trigger manually, review at both ends: prompt_confirmation -> generate_sql -> run_query -> extract_data -> result_confirmation (HITLEntryOperator) (LLMSQLQuery) (Analytics) (@task) (ApprovalOperator) example_llm_survey_scheduled: runs @monthly, no human in the loop: download_survey -> prepare_csv -> check_schema -> generate_sql -> run_query -> extract_data -> send_result (HttpOperator) (@task) (LLMSchema (LLMSQLQuery) (Analytics) (@task) (@task / Email) Compare) The Data The Airflow Community Survey 2025 CSV has 5,856 rows and 168 columns covering everything from Airflow version and executor type to cloud provider, company size, and AI tool usage. A few highlights from the data: 3,320 respondents identify as Data Engineers 2,032 use AWS as their primary cloud provider for Airflow 1,445 are already running Airflow 3 1,351 say they often use AI tools to write Airflow code Those last two numbers together are part of why this example exists: the people most likely to use th ## Introducing the Common AI Provider: LLM and AI Agent Support for Apache Airflow DevFeed: [Introducing the Common AI Provider: LLM and AI Agent Support for Apache Airflow]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-04-14T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Large Language Model](), [AI Agent](), [Pydantic](), [Artificial Intelligence]() Tags: [2025](), [ai](), [ai-agent](), [airflow](), [apache](), [apache-airflow](), [community](), [llm](), [release]() ### AI overview Apache Airflow is releasing the Common AI Provider 0.1.0, a provider package that adds LLM and AI agent capabilities directly to Airflow. Built on Pydantic AI, it supports more than 20 model providers through a single install and requires Apache Airflow 3.0 or newer. ### Source excerpt At Airflow Summit 2025, we previewed what native AI integration in Apache Airflow could look like. Today we're shipping it. apache-airflow-providers-common-ai 0.1.0 adds LLM and agent capabilities directly to Airflow. Not a wrapper around another framework, but a provider package that plugs into the orchestrator you already run. It's built on Pydantic AI and supports 20+ model providers (OpenAI, Anthropic, Google, Azure, Bedrock, Ollama, and more) through a single install. pip install 'apache-airflow-providers-common-ai' Requires Apache Airflow 3.0+. Note: This is a 0.x release. We're actively looking for feedback and iterating fast, so breaking changes are possible between minor versions. Try it, tell us what works and what doesn't. Your input directly shapes the API. By the Numbers 6 Operators 6 TaskFlow decorators 5 Toolsets 4 Connection types 20+ Supported model providers via Pydantic AI The Decorator Suite Every operator has a matching TaskFlow decorator. @task.llm: Single LLM Call Send a prompt, get text or structured output back. from pydantic import BaseModel from airflow.providers.common.compat.sdk import dag, task @dag def my_pipeline(): class Entities(BaseModel): names: list[str] locations: list[str] @task.llm( llm_conn_id="my_openai_conn", system_prompt="Extract named entities.", output_type=Entities, ) def extract(text: str): return f"Extract entities from: {text}" extract("Alice visited Paris and met Bob in London.") my_pipeline() The LLM returns a typed Entities object, not a string you have to parse. Downstream tasks get structured data through XCom. @task.agent: Multi-Step Agent with Tools When the LLM needs to query databases, call APIs, or read files across multiple steps, use @task.agent. The agent picks which tools to call and loops until it has an answer. from airflow.providers.common.ai.toolsets.sql import SQLToolset from airflow.providers.common.compat.sdk import dag, task @dag def sql_analyst(): @task.agent( llm_conn_id="my_openai_conn", sys ## Apache Airflow 3.2.0: Data-Aware Workflows at Scale DevFeed: [Apache Airflow 3.2.0: Data-Aware Workflows at Scale]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-04-07T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [data](), [Orchestration](), [Release notes](), [SDK](), [Amazon S3](), [BigQuery]() Tags: [airflow](), [apache](), [apache-airflow](), [bigquery](), [data](), [docker-image](), [orchestration](), [pypi](), [release](), [release-notes](), [s3](), [sdk](), [workflows]() ### AI overview Apache Airflow 3.2.0 introduces asset partitioning for more granular data-aware scheduling, along with multi-team deployments, synchronous deadline alert callbacks, and continued Task SDK separation. Partition-aware workflows trigger downstream DAGs only when the relevant partition is updated. ### Source excerpt We're proud to announce the release of Apache Airflow 3.2.0! Airflow 3.1 puts humans at the center of automated workflows. 3.2 brings that same precision to data: Asset partitioning for granular pipeline orchestration, multi-team deployments for enterprise scale, synchronous deadline alert callbacks, and continued progress toward full Task SDK separation. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/3.2.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.2.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.2.0/release_notes.html 🐳 Docker Image: docker pull apache/airflow:3.2.0 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.2.0 🗂 Asset Partitioning (AIP-76): Only the Right Work Gets Triggered Asset partitioning has been one of the most requested additions to data-aware scheduling. If you work with date-partitioned S3 paths, Hive table partitions, BigQuery partitions, or really any partitioned data store, you've dealt with this: An upstream task updates one partition, and every downstream Dag fires regardless of which slice actually changed. It's wasteful, and for large deployments it creates real operational noise. Asset partitioning in 3.2 makes this granular. Downstream Dags trigger only when the specific partition they care about gets updated. It's the biggest change to data-aware scheduling since Assets were introduced, and it turns partition-driven orchestration into something Airflow handles natively rather than something you work around. Key Capabilities Partition-driven scheduling: Dags trigger on specific partition updates, not every asset change CronPartitionTimetable: Schedule Dags against partitions using cron expressions. Also available in the Task SDK Backfill for partitioned Dags: Backfill historical partitions without re-triggering everything downstream (#61464) Multi-asset partitions: A single Dag can listen for partitions across multiple assets, which matters when your downstream work depe ## Introducing the Apache Airflow Registry DevFeed: [Introducing the Apache Airflow Registry]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-03-19T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Data pipelines](), [API](), [JSON]() Tags: [airflow](), [apache](), [apache-airflow](), [community](), [data-pipelines](), [messaging](), [notifications](), [openai](), [registry]() ### AI overview Apache Airflow launches the Airflow Registry, a searchable catalog of official providers and modules. It includes provider and module search, installation and compatibility details, connection generation in URI, JSON, and environment-variable formats, ecosystem statistics, and a structured JSON API. ### Source excerpt Today we're launching the Apache Airflow Registry -- a searchable catalog of every official Airflow provider and its modules, live at airflow.apache.org/registry/. Need an S3 operator? A Snowflake hook? An OpenAI sensor? The Registry helps you find, compare, and configure the right components for your data pipelines -- without digging through docs or PyPI pages. By the Numbers 98 Official providers 1,602 Modules (operators, hooks, sensors, triggers, transfers, and more) 329M+ Monthly PyPI downloads across all providers 125+ Integrations with cloud platforms, databases, ML tools, and messaging services Search Everything Hit Cmd+K from any page and start typing. Results show up instantly, grouped by Providers and Modules, with type badges so you can tell a hook from an operator at a glance. Provider Pages Each provider gets a dedicated page with everything in one place: install command with copy-to-clipboard, version selector, extras dropdown, compatibility info, connection types, and the full module listing organized by type. The Amazon provider, for example, has 372 modules across operators, hooks, sensors, triggers, transfers, and more. Module type tabs let you filter to exactly what you're looking for, and a category sidebar groups modules by AWS service (S3, Lambda, Glue, Step Functions, etc.). Connection Builder Click any connection type badge on a provider page, fill in the fields, and the builder generates the connection in three formats -- URI, JSON, and Env Var -- ready to copy into your configuration. No more guessing URI encoding or JSON structure. Explore by Category Not sure which provider you need? The Explore page organizes providers into categories: Cloud Platforms, Databases, Data Warehouses, Messaging & Notifications, AI & Machine Learning, Data Processing, and more. Statistics The Stats page breaks down the ecosystem: 848 operators, 298 hooks, 164 triggers, 157 sensors, 83 transfers, and more -- plus top providers by downloads and module count. JSON API ## Airflow Survey 2025 DevFeed: [Airflow Survey 2025]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-01-22T00:00:00Z Content type: article Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [data](), [CSV]() Tags: [2025](), [airflow](), [analysis](), [apache-airflow](), [community](), [csv](), [download](), [form](), [graphics](), [interactive](), [report](), [responses](), [share](), [survey](), [users](), [view]() ### AI overview This page presents the Airflow Survey 2025 interactive report, hosted by Astronomer and sponsored and supported by the Apache Airflow community. The supplied evidence does not include the survey findings. ### Source excerpt The interactive report is hosted by Astronomer. The Apache Airflow community thanks Astronomer for running this survey, for sponsoring it and providing the report in this form, and for their effort in marketing, analysis, and preparing the graphics. View raw data Download survey responses (CSV) ## Apache Airflow CTL aka airflowctl 0.1.0 DevFeed: [Apache Airflow CTL aka airflowctl 0.1.0]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2025-10-15T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Command-line interface](), [Release notes](), [API](), [REST API](), [releases](), [Authentication](), [audit]() Tags: [airflow](), [apache-airflow](), [audit-trail](), [authentication](), [cli](), [command-line](), [release](), [release-notes](), [rest-api]() ### AI overview Apache Airflow announces airflowctl 0.1.0, a command-line interface that interacts exclusively with the Airflow REST API. The release is intended to provide secure, auditable, remote management of Airflow deployments without direct metadata-database access, while remote commands gradually move from the existing Airflow CLI. ### Source excerpt We are thrilled to announce the first major release of airflowctl 0.1.0, the new secure, API-driven command-line interface (CLI) for Apache Airflow -- built under AIP-81. This release marks CLI to join the general posture on communicating through API. Airflow CLI joins the modern era of secure, auditable, and remote-first operations. Details: 📦 PyPI: https://pypi.org/project/apache-airflow-ctl/0.1.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow-ctl/stable/release_notes.html 🪶 Source Code: https://github.com/apache/airflow/tree/main/airflow-ctl 🎯 What is airflowctl? airflowctl is a new command-line interface for Apache Airflow that interacts exclusively with the Airflow REST API. It provides a secure, auditable, and consistent way to manage Airflow deployments -- without direct access to the metadata database. 🔄 Coexistence with Airflow CLI The Airflow CLI will continue as intended, primarily for admin tasks such as running Airflow components (airflow api-server, airflow scheduler) or managing the metadata database (airflow db init). airflowctl focuses on operational commands that interact with Airflow resources via the API (airflowctl dagrun trigger, airflowctl connection create, etc.). We defined the commands falls under two main categories: Remote Commands: Operations that can be provided via API (e.g., managing DAGs, connections, variables, triggering DAG runs) are now available in airflowctl and will be the recommended approach going forward. Local/Admin Commands: Operations that manage Airflow components or the metadata database will remain in the Airflow CLI. Of course, in the current state they will both have the remote commands. We are planning a zero-disruption migration path where Remote Commands will be gradually deprecated from the Airflow CLI as they achieve parity in airflowctl. 🔒 Why airflowctl? Until now, Airflow CLI connected directly to the metadata database, bypassing RBAC, authentication, and API logs. While convenient, this app ## Apache Airflow 3.1.0: Human-Centered Workflows DevFeed: [Apache Airflow 3.1.0: Human-Centered Workflows]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2025-09-25T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Automation](), [human review](), [Data Quality](), [Forms](), [Developer experience](), [Inference](), [React]() Tags: [apache-airflow](), [automation](), [data-quality](), [developer-experience](), [forms](), [human-review](), [inference](), [release](), [release-notes](), [ui]() ### AI overview Apache Airflow 3.1.0 introduces human-in-the-loop tasks for pausing automated workflows and collecting reviews through web forms. The release also adds internationalization support, developer experience improvements, restored Calendar and Gantt views, UI filtering, DAG pinning, and accessibility-focused theme updates. ### Source excerpt We are thrilled to announce the release of Apache Airflow 3.1.0, an update that puts humans at the center of data workflows. This release introduces powerful new capabilities for human decision-making in automated processes, comprehensive internationalization support, and significant developer experience improvements. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/3.1.0/ 📚 Core Airflow Docs: https://airflow.apache.org/docs/apache-airflow/3.1.0/ 📚 Task SDK Docs: https://airflow.apache.org/docs/task-sdk/1.1.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.1.0/release_notes.html 🪶 Sources: https://airflow.apache.org/docs/apache-airflow/3.1.0/installation/installing-from-sources.html 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.1.0 🤝 Human-in-the-Loop (HITL): When Automation Meets Human Judgment This powerful capability bridges the gap between automated processes and human expertise, making Airflow invaluable for: AI/ML Model Validation: Pause inference pipelines for human review of model outputs Content Moderation: Route content through human reviewers before publication Approval Workflows: Require manager approval for sensitive operations Data Quality Gates: Allow data stewards to validate critical datasets HITL tasks pause in a deferred state while presenting intuitive web forms in the Airflow UI. Users with appropriate roles can review context data, DAG parameters, and XCom values before making informed decisions. Example Code: from airflow.sdk import DAG from airflow.providers.standard.operators.hitl import HITLOperator with DAG("content_moderation", schedule="@daily") as dag: moderate_content = HITLOperator( task_id="review_content", message="Please review this content for publication", data_key="content_to_review" ) 📊 UI Enhancements & Performance Calendar and Gantt Views Make Their Comeback Remember those beloved Calendar and Gantt chart views from Airflow 2.x? They're back, completely rebuilt for the modern ## Apache Airflow® 3 is Generally Available! DevFeed: [Apache Airflow® 3 is Generally Available!]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2025-04-22T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [event driven](), [Architecture & Design](), [React](), [Messaging](), [Cloud]() Tags: [airflow](), [apache-airflow](), [architecture](), [event-driven](), [hybrid-cloud](), [messaging](), [react](), [release](), [releases]() ### AI overview Apache Airflow 3.0 has been released as a major version. The release introduces a React-based UI, DAG versioning, improved backfill support, event-driven scheduling, and a Task Execution Interface with Task SDKs supporting secure execution across multi-cloud, hybrid-cloud, and local data center deployments. ### Source excerpt We announced our intent to focus on Apache Airflow 3.0® as the next big milestone for the Airflow project at the Airflow Summit in September 2024. We are delighted to announce that Airflow 3.0 is now released! A Major Release, Four Years in the Making Airflow 3.0 is the biggest release in Airflow's history--2.0 was released in 2020, and the last 4 years have seen incremental updates and releases every quarter with version 2.10 released in Q4 2024. With over 30 million monthly downloads (up over 30x since 2020) and 80,000 organizations (up from 25,000 in 2020) now using Airflow, we've seen an incredible growth in popularity since 2.0. Over the last four years, Airflow has grown to power business critical data workflows within organizations of all sizes. We have seen an exponential increase in the use cases for Airflow from its beginnings with ETL, ELT, and Reverse ETL, with over 30% of Airflow users using it for MLOps, and 10% using it for GenAI workflows. Airflow 3 is a response to this use case expansion and is the standard for data application development across the enterprise. Here are some highlights: Airflow 3 is significantly easier to use for data practitioners and incorporates their key requests for critical changes to Airflow. Early user reactions to features such as the new React based UI, DAG Versioning, and improved Backfill support have been incredibly positive. I was ecstatic to see the reaction from data engineers when I demonstrated this at a recent Airflow meetup. The seamless UI transition of navigating between Asset-oriented workflows and Task-oriented workflows is beautiful. Once again, Airflow lets the developer choose how you want to develop and navigate without imposing any restrictions. Introduction of Event Driven Scheduling enables Airflow to seamlessly integrate with messaging providers and react to events happening and data assets being updated outside of Airflow. The big architecture change with the introduction of the Task Execution Inte ## Apache Airflow 2.10.0 is here DevFeed: [Apache Airflow 2.10.0 is here]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2024-08-08T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [telemetry](), [configuration](), [data](), [PyPI](), [Docker Image]() Tags: [airflow](), [apache-airflow](), [configuration](), [docker-image](), [pypi](), [release](), [release-notes](), [releases](), [telemetry]() ### AI overview Apache Airflow 2.10.0 is available with default basic telemetry collection, support for multiple executors in one environment, and dynamic dataset scheduling through DatasetAlias. Deployments can opt out of telemetry collection. ### Source excerpt I'm happy to announce that Apache Airflow 2.10.0 is now available, bringing an array of noteworthy enhancements and new features that will greatly serve our community. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.10.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.10.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.10.0/release_notes.html 🐳 Docker Image: "docker pull apache/airflow:2.10.0" 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.10.0 Airflow now collects Telemetry data by default With the release of Airflow 2.10.0, we've introduced the collection of basic telemetry data, as outlined here. This data will play a crucial role in helping Airflow maintainers gain a deeper understanding of how Airflow is utilized across various deployments. The insights derived from this information are invaluable in guiding the prioritization of patches, minor releases, and security fixes. Moreover, this data will inform key decisions regarding the development roadmap, ensuring that Airflow continues to evolve in line with community needs. For those who prefer not to participate in data collection, deployments can easily opt out by setting the [usage_data_collection] enabled option to False or by using the SCARF_ANALYTICS=false environment variable. Multiple Executor Configuration (formerly "Hybrid Execution") Each executor comes with its unique set of strengths and weaknesses, typically balancing latency, isolation, and compute efficiency. Traditionally, an Airflow environment is limited to a single executor, requiring users to make trade-offs, as no single executor is perfectly suited for all types of tasks. We are introducing a new feature that allows for the concurrent use of multiple executors within a single Airflow environment. This flexibility enables users to take advantage of the specific strengths of different executors for various tasks, improving overall efficiency and mitigating weaknesses. Users can s ## Apache Airflow 2.9.0: Dataset and UI Improvements DevFeed: [Apache Airflow 2.9.0: Dataset and UI Improvements]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2024-04-08T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [dataset](), [datasets](), [releases](), [Release notes](), [ui](), [Python]() Tags: [2-9-0](), [airflow](), [apache-airflow](), [dataset](), [datasets](), [python](), [release](), [release-notes](), [scheduling](), [ui]() ### AI overview Apache Airflow 2.9.0 introduces expanded data-aware scheduling with logical OR and arbitrary AND/OR combinations, plus a timetable combining dataset events with time-based schedules. It also adds dataset event REST API endpoints, dataset UI enhancements, Python 3.12 support with Pendulum 3, and custom names for dynamically mapped tasks. ### Source excerpt I'm happy to announce that Apache Airflow 2.9.0 has been released! This time around we have new features for data-aware scheduling and a bunch of UI-related improvements. Apache Airflow 2.9.0 contains over 550 commits, which include 38 new features, 70 improvements, 31 bug fixes, and 18 documentation changes. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.9.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.9.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.9.0/release_notes.html 🐳 Docker Image: "docker pull apache/airflow:2.9.0" 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.9.0 Airflow 2.9.0 is also the first release that supports Python 3.12. However, Pendulum 2 does not support Python 3.12, so you'll need to use Pendulum 3 if you upgrade to Python 3.12. New data-aware scheduling options Logical operators and conditional expressions for DAG scheduling When Datasets were added in Airflow 2.4, DAGs only had scheduling support for logical AND combinations of Datasets. Simply, you could schedule against more than one Dataset, but a DAG run would only be created once all the Datasets were updated after the last run. Now in Airflow 2.9, we support logical OR and even arbitrary combinations of AND and OR. As an example, you can schedule a DAG whenever dataset_1 or dataset_2 are updated : with DAG(schedule=(dataset_1 | dataset_2), ...): ... You can have arbitrary combinations: with DAG(schedule=((dataset_1 | dataset_2) & dataset_3), ...): ... You can read more about this new functionality in the data-aware scheduling docs. Combining Dataset and Time-Based Schedules Airflow 2.9 comes with a new timetable, DatasetOrTimeSchedule, that allows you to schedule DAGs based on both dataset events and a timetable. Now you have the best of both worlds. For example, to run whenever dataset_1 updates and at midnight UTC: with DAG( schedule=DatasetOrTimeSchedule( timetable=CronTriggerTimetable("0 0 * * *", timezone="UTC"), ## Building a Data Pipeline to Track Strava's Bad Events DevFeed: [Building a Data Pipeline to Track Strava's Bad Events]() Original publisher: [Read original article]() Author: Bisman Sodhi Published: 2024-01-08T20:19:46Z Content type: opinion Language: en Sources: [Strava Engineering]() Topics: [data](), [airflow](), [Amazon S3](), [Python](), [Amazon Web Services](), [JSON](), [dashboards](), [Computer science]() Tags: [airflow](), [apache-airflow](), [aws](), [data](), [data-engineering](), [data-pipeline](), [data-platforms](), [integrity](), [json](), [python](), [s3](), [software-engineering](), [strava](), [tableau](), [warehouse]() ### AI overview A software engineering intern describes building a daily Apache Airflow pipeline that extracts schema-invalid user behavior events from S3, decompresses them into JSON, and loads them into Snowflake. Staging tables protect production data from partial loads, while materialized SQL views and a Tableau dashboard improve querying and monitoring. ### Source excerpt Hi my name is Bisman and I studied Computer Science at University of California, Santa Barbara. During summer of 2022, I had the most amazing experience working as a Software Engineer Intern on Strava's Data Platform Team. In the first fews weeks, I learned the tools my team uses and then spent the rest of the time working on my project. TRACKING BAD EVENTS For my major summer project, I created a data pipeline that pulls user behavior data out of external storage and persists it in our data warehouse. Strava uses a service called Snowplow to collect this user behavior data, like loading a club page or uploading a profile photo. Sometimes, this data fails to match the schema that we've set, and a piece of data that fails this schema validation is called a bad event. Previously, these bad events were temporarily stored in an Elastic Search. Persisting this data in Snowflake, our data warehouse, makes it accessible to a wider audience. It also makes it easier to incorporate the bad events data with other services used at Strava. To start my project, I created a directed acyclic graph in Apache Airflow, a scheduling framework, using python that extracts bad events data from the S3, AWS's storage service, buckets on a daily cadence. This data was stored as gzip files on S3 which I decompressed and stored the data as JSON blobs. As I was working with billions of rows of data, it was important to maintain data integrity and take measures in case data failed to load from S3. Therefore, I loaded data into a staging table in Snowflake. The staging table ensured that if loading from S3 failed, the production table would remain untouched. This data was then loaded into the production table free of any partial data. After all the data was loaded into the production table, I created six view tables because there were six different types of bad events stored in the production table. I collaborated with our stakeholders -- data analysts -- throughout this process to craft tables bas ## Apache Airflow 2.8.0 is here DevFeed: [Apache Airflow 2.8.0 is here]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2023-12-15T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Python](), [Amazon S3](), [Azure]() Tags: [apache-airflow](), [azure](), [configure](), [docker](), [logs](), [messages](), [object-storage](), [pypi](), [python](), [release](), [release-notes](), [s3]() ### AI overview Apache Airflow 2.8.0 is a software release that adds experimental object storage abstractions for systems such as S3, GCS, and Azure Blob Storage, integrates messages from other components into task logs, and introduces dataset listener hooks. It also adds extra index URL support for PythonVirtualEnvOperator and its decorator, including caching for additional Python package repositories. ### Source excerpt I am thrilled to announce the release of Apache Airflow 2.8.0, featuring a host of significant enhancements and new features that will greatly benefit our community. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.8.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.8.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.8.0/release_notes.html 🐳 Docker Image: "docker pull apache/airflow:2.8.0" 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.8.0 Airflow Object Storage (AIP-58) This feature is experimental and subject to change. Airflow now offers a generic abstraction layer over various object stores like S3, GCS, and Azure Blob Storage, enabling the use of different storage systems in DAGs without code modification. In addition, it allows you to use most of the standard Python modules, like shutil, that can work with file-like objects. Here is an example of how to use the new feature to open a file: from airflow.io.path import ObjectStoragePath base = ObjectStoragePath("s3://my-bucket/", conn_id="aws_default") # conn_id is optional @task def read_file(path: ObjectStoragePath) -> str: with path.open() as f: return f.read() The above example is just the tip of the iceberg. The new feature allows you to configure an alternative backend for a scheme or protocol. Here is an example of how to configure a custom backend for the dbfs scheme: from airflow.io.path import ObjectStoragePath from airflow.io.store import attach from fsspec.implementations.dbfs import DBFSFileSystem attach(protocol="dbfs", fs=DBFSFileSystem(instance="myinstance", token="mytoken")) base = ObjectStoragePath("dbfs://my-location/") For more information: Airflow Object Storage The support for a specific object storage system depends on the installed providers, with out-of-the-box support for the file scheme. Ship logs from other components to Task logs This feature seamlessly integrates task-related messages from various Airflow components, includi ## Airflow Survey 2023 DevFeed: [Airflow Survey 2023]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2023-09-21T00:00:00Z Content type: article Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [data-engineering]() Tags: [airflow](), [apache-airflow](), [community](), [data](), [data-engineering](), [survey](), [users]() ### AI overview The article presents the Apache Airflow Survey 2023, an annual survey intended to provide insights into Airflow usage and guide future efforts. The supplied text reports conflicting figures for participation: more than 5,818 responses from 122 countries and more than 5,250 responses from 116 countries. ### Source excerpt View Raw Data ## Apache Airflow 2.7.0 is here DevFeed: [Apache Airflow 2.7.0 is here]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2023-08-18T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [Security](), [Release notes](), [Python 3.8](), [Docker Image]() Tags: [apache-airflow](), [docker](), [python](), [python-3-8](), [release](), [release-notes](), [security]() ### AI overview Apache Airflow 2.7.0 has been released with new features, improvements, bug fixes, and documentation changes. The release focuses on security, drops support for end-of-life Python 3.7, and adds capabilities including setup and teardown tasks, a Cluster Activity UI, configurable deferrable tasks, and built-in OpenLineage integration. ### Source excerpt I'm happy to announce that Apache Airflow 2.7.0 has been released! Some notable features have been added that we are excited for the community to use. Apache Airflow 2.7.0 contains over 500 commits, which include 40 new features, 49 improvements, 53 bug fixes, and 15 documentation changes. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.7.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.7.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.7.0/release_notes.html 🐳 Docker Image: "docker pull apache/airflow:2.7.0" 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.7.0 Airflow 2.7.0 is a release that focuses on security. The Airflow security team, working together with security researchers, identified a number of areas that required strengthening of security. This resulted in, among others things, an improved description of the Airflow security model, a better explanation of our security policy and the disabling of certain, potentially dangerous, features by default - like, for example, connection testing (#32052). Airflow 2.7.0 is also the first release that drops support for end-of-life Python 3.7. This allows Airflow users and maintainers to make use of features and improvements in Python 3.8, and unlocks newer versions of our dependencies. Setup and Teardown (AIP-52) Airflow now has first class support for the concept of setup and teardown tasks. These tasks have special behavior in that: Teardown tasks will still run, no matter what state the upstream tasks end up in Teardown tasks failing won't, by default, cause the DAG run to fail Automatically clear setup/teardown tasks when clearing a dependent task You can read more about setup and teardown in the Introducing Setup and Teardown tasks blog post, or in the setup and teardown docs. Cluster Activity UI There is a new top level page in Airflow, the Cluster Activity page. This gives an overview of the cluster, including component health, dag and task state co ## what's new in Apache Airflow 2.6.0 DevFeed: [what's new in Apache Airflow 2.6.0]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2023-04-30T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [Release notes](), [Logging](), [ui](), [notifications](), [configuration]() Tags: [airflow](), [apache-airflow](), [bug](), [bug-fixes](), [docker-image](), [feature](), [features](), [logs](), [notifications](), [pypi](), [release](), [release-notes](), [ui]() ### AI overview Apache Airflow 2.6.0 has been released with 42 new features, 58 improvements, 38 bug fixes, and 17 documentation changes. Notable updates include trigger logs in the webserver, Grid view enhancements, a DAG-parameter trigger form, improved handling of stuck queued tasks, plugin-based cluster policy hooks, and notifications support for Slack. ### Source excerpt I am excited to announce that Apache Airflow 2.6.0 has been released, bringing many minor features and improvements to the community. Apache Airflow 2.6.0 contains over 500 commits, which include 42 new features, 58 improvements, 38 bug fixes, and 17 documentation changes. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.6.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.6.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.6.0/release_notes.html 🐳 Docker Image: "docker pull apache/airflow:2.6.0" 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.6.0 As the changelog is quite large, the following are some notable new features that shipped in this release. Trigger logs can now be viewed in webserver Trigger logs have now been added to task logs. They appear right alongside the rest of the logs from your task. Adding this feature required changes across the entire Airflow logging stack, so be sure to update your providers if you are using remote logging. Grid view improvements The grid view has received a number of minor improvements in this release. Most notably, there is now a graph tab in the grid view. This offers a more integrated graph representation of the DAG, where choosing a task in either the grid or graph will highlight the same task in both views. You can also filter upstream and downstream from a single task. For example, in the screenshot above, describe_integrity is the selected task. If you choose to filter downstream, this is the result: Trigger UI based on DAG level params A user-friendly form is now shown to users triggering runs for DAGs with DAG level params. See the Params docs for more details. Consolidation of handling stuck queued tasks Airflow now has a single configuration, [scheduler] task_queued_timeout, to handle tasks that get stuck in queued for too long. With a simpler implementation than the outgoing code handling these tasks, tasks stuck in queued will no longer slip through the ## Apache Airflow 2.5.0 Released with Dataset UI, DAG Testing, and Task Log Improvements DevFeed: [Apache Airflow 2.5.0 Released with Dataset UI, DAG Testing, and Task Log Improvements]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2022-12-02T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [dataset](), [debug](), [Docker Image]() Tags: [airflow](), [apache-airflow](), [console](), [dataset](), [debug](), [docker-image](), [release](), [release-notes](), [releases]() ### AI overview Apache Airflow 2.5.0 introduces usability improvements to the Datasets UI, a faster and easier-to-debug airflow dags test command, automatic task-log tailing in Grid view, and further improvements to Dynamic Task Mapping. The release also marks a shorter release cadence than previously used. ### Source excerpt Apache Airflow 2.5 has just been released, barely two and a half months after 2.4! Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.5.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.5.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.5.0/release_notes.html 🐳 Docker Image: docker pull apache/airflow:2.5.0 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.5.0 This quicker release cadence is a departure from our previous habit of releasing every five-to-seven months and was a deliberate effort to listen to you, our users, and get the changes and improvements into your workflows earlier. Usability improvements to the Datasets UI When we released Dataset aware scheduling in September we knew that the tools we gave to manage the Datasets were very much a Minimum Viable Product, and in the last two months the committers and contributors have been hard at work at making the UI much more usable when it comes to Datasets. But we aren't done yet - keep an eye out for more improvements coming over the next couple of releases too. Greatly improved airflow dags test command This airflow subcommand has been rethought and re-optimized to make it much easier to test your DAGs locally - the major changes are: a. Task logs are visible right there in the console, instead of hidden away inside the task log files b. It is about an order of magnitude quicker to run the tasks than before (i.e. it gets to running the task code so much quicker) c. Everything runs in one process, so you can put a breakpoint in your IDE, and configure it to run airflow dags test then debug code! Auto tailing task logs in the Grid view Hopefully the headline says enough. It's lovely, go check it out. More improvements to Dynamic-Task mapping In a similar vein to the improvements to the Dataset (UI), we have continued to iterate on and improve the feature we first added in Airflow 2.3, Dynamic Task Mapping, and 2.5 includes dozens of improvement ## Apache Airflow 2.4.0: That Data Aware Release DevFeed: [Apache Airflow 2.4.0: That Data Aware Release]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2022-09-19T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [datasets](), [releases]() Tags: [apache-airflow](), [data](), [datasets](), [new-features](), [release]() ### AI overview Apache Airflow 2.4.0 is a release with more than 650 user-facing commits, including new features, improvements, bug fixes, and documentation changes. Its main feature is data-aware scheduling, which allows DAGs to be scheduled when tasks update datasets and supports smaller, connected workflows. The release also introduces ExternalPythonOperator for running tasks in preconfigured virtual environments to manage conflicting Python dependencies. ### Source excerpt Apache Airflow 2.4.0 contains over 650 "user-facing" commits (excluding commits to providers or chart) and over 870 total. That includes 46 new features, 39 improvements, 52 bug fixes, and several documentation changes. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html 🐳 Docker Image: docker pull apache/airflow:2.4.0 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0 Data-aware scheduling (AIP-48) This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets. What does this mean, exactly? This is a great new feature that lets DAG authors create smaller, more self-contained DAGs, which chain together into a larger data-based workflow. If you are currently using ExternalTaskSensor or TriggerDagRunOperator you should take a look at datasets - in most cases you can replace them with something that will speed up the scheduling! But enough talking, lets have a short example. First lets write a simple DAG with a task called my_task that produces a dataset called my-dataset: from airflow import Dataset dataset = Dataset(uri='my-dataset') with DAG(dag_id='producer', ...) @task(outlets=[dataset]) def my_task(): ... Datasets are defined by a URI. Now, we can create a second DAG (consumer) that gets scheduled whenever this dataset changes: from airflow import Dataset dataset = Dataset(uri='my-dataset') with DAG(dag_id='dataset-consumer', schedule=[dataset]): ... With these two DAGs, the instant my_task finishes, Airflow will create the DAG run for the dataset-consumer workflow. We know that what exists right now won't fit all use cases that people might wish for datasets, and in the coming minor releases (2.5, 2.6, etc.) we will expand and improve upon this foundation. Datasets represent the abstract concept of a dataset, and (for now) do not have any dire ## Airflow Survey 2022 DevFeed: [Airflow Survey 2022]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2022-06-17T00:00:00Z Content type: article Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [data](), [Documentation](), [migration](), [Monitoring](), [Stack Overflow]() Tags: [airflow](), [apache-airflow](), [community](), [contributions](), [data](), [documentation](), [migration](), [monitoring](), [results](), [stack-overflow](), [survey](), [users]() ### AI overview The 2022 Airflow User Survey collected 210 responses over two weeks. Respondents were primarily data engineers, and Airflow was widely used in larger companies. Most participants were willing to recommend Apache Airflow, documentation and Stack Overflow were important information sources, and most deployments used Airflow 2. Migration from Airflow 1 was mainly constrained by resources and capacity. ### Source excerpt Airflow User Survey 2022 This year's survey has come and gone, and with it we've got a new batch of data for everyone! We collected 210 responses over two weeks. We continue to see growth in both contributions and downloads over the last two years, and expect that trend will continue through 2022. The raw response data will be made available here soon, in the meantime, feel free to email john.thomas@astronomer.io for a copy. TL;DR Overview of the user Like previous years, more than half of the Airflow users are Data Engineers (54%). Solutions Architects (13%), Developers (12%), DevOps (6%) and Data Scientists (4%) are also active Airflow users! There was a slight increase in the representation of Solutions Architect roles compared to results from 2020 and 2019 . Airflow is used and popular in bigger companies, 64% of Airflow users work for companies with 200+ employees which is an 11 percent increase compared to 2020. 62% of the survey participants have more than 6 Airflow users in their company. More Airflow users (65.9%) are willing to recommend Apache Airflow compared to the survey results in 2020 and 2019. There is a general positive trend in a willingness to recommend Airflow, 93% of surveyed Airflow users are willing to recommend Airflow ( 85.7% in 2019 and 92% in 2020 ), only 1% of users are not likely to recommend (3.6% in 2019 and 3.5% in 2020). Airflow documentation is a critical source of information, with more than 90% (15% increase compared to results from 2020) of survey participants using the documentation. Airflow documentation is also one of the top areas to improve! What's interesting, also Stack Overflow usage is critical, with about 60% users declaring to use it as a source of information (24% increase compared to results from 2020). Deployments 85% of the Airflow users have between 1 and 7 active Airflow instances. 62.5% of the Airflow users have between 11 and 250 DAGs in their largest Airflow instance. 75% of the surveyed Airflow users have be ## Airflow Summit 2022 DevFeed: [Airflow Summit 2022]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2022-05-16T00:00:00Z Content type: news Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Data pipelines](), [data-governance](), [Open Source](), [AI, ML & Data Engineering]() Tags: [airflow-summit](), [apache-airflow](), [community](), [data-governance](), [data-pipelines](), [events](), [machine-learning](), [network](), [open-source](), [summit]() ### AI overview Airflow Summit 2022 was scheduled for May 23-27 as a free conference for Apache Airflow practitioners and data leaders. The program covered Airflow practices, data pipelines, data governance, machine learning, the project's future, and non-code open-source contributions. ### Source excerpt The biggest Airflow Event of the Year returns May 23-27! Airflow Summit 2022 will bring together the global community of Apache Airflow practitioners and data leaders. What's on the Agenda During the free conference, you will hear about Apache Airflow best practices, trends in building data pipelines, data governance, Airflow and machine learning, and the future of Airflow. There will also be a series of presentations on non-code contributions driving the open-source project. How to Attend This year's edition will include a variety of online sessions across different time zones. Additionally, you can take part in local in-person events organized worldwide for data communities to watch the event and network. Interested? 🪶 Register for Airflow Summit 2022 today 🤝 Check out the in-person events planned for Airflow Summit 2022. ## Apache Airflow 2.3.0 is here DevFeed: [Apache Airflow 2.3.0 is here]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2022-04-30T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [Release notes](), [Command-line interface](), [Databases](), [Code](), [JSON](), [Docker]() Tags: [airflow](), [apache-airflow](), [bug-fixes](), [cli](), [database](), [docker](), [json](), [new-features](), [release](), [release-notes]() ### AI overview Apache Airflow 2.3.0 is a release containing over 700 commits since version 2.2.0, with 50 new features, 99 improvements, 85 bug fixes, and documentation changes. Highlighted features include Dynamic Task Mapping, a Grid View, metadata database cleanup, LocalKubernetesExecutor, standalone DAG processing, and JSON connection serialization. ### Source excerpt Apache Airflow 2.3.0 contains over 700 commits since 2.2.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.3.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.3.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.3.0/release_notes.html 🐳 Docker Image: docker pull apache/airflow:2.3.0 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.3.0 As the changelog is quite large, the following are some notable new features that shipped in this release. Dynamic Task Mapping(AIP-42) There's now first-class support for dynamic tasks in Airflow. What this means is that you can generate tasks dynamically at runtime. Much like using a for loop to create a list of tasks, here you can create the same tasks without having to know the exact number of tasks ahead of time. You can have a task generate the list to iterate over, which is not possible with a for loop. Here is an example: @task def make_list(): # This can also be from an API call, checking a database, -- almost anything you like, as long as the # resulting list/dictionary can be stored in the current XCom backend. return [1, 2, {"a": "b"}, "str"] @task def consumer(arg): print(list(arg)) with DAG(dag_id="dynamic-map", start_date=datetime(2022, 4, 2)) as dag: consumer.expand(arg=make_list()) More information can be found here: Dynamic Task Mapping Grid View replaces Tree View Grid view replaces tree view in Airflow 2.3.0. Screenshots: Purge history from metadata database Airflow 2.3.0 introduces a new airflow db clean command that can be used to purge old data from the metadata database. You would want to use this command if you want to reduce the size of the metadata database. More information can be found here: Purge history from metadata database LocalKubernetesExecutor There is a new executor named LocalKubernetesExecutor. This executor helps you run some tasks using LocalExecutor a ## What's new in Apache Airflow 2.2.0 DevFeed: [What's new in Apache Airflow 2.2.0]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2021-10-11T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [async](), [data]() Tags: [airflow](), [apache-airflow](), [async](), [bug-fixes](), [new-features](), [release](), [scheduling](), [sensors]() ### AI overview Apache Airflow 2.2.0 is presented as a release with more than 600 commits since version 2.1.4, including new features, improvements, bug fixes, and documentation changes. Highlighted features include pluggable custom timetables for DAG scheduling and deferrable tasks that can return worker resources while waiting for asynchronous checks. ### Source excerpt I'm proud to announce that Apache Airflow 2.2.0 has been released. It contains over 600 commits since 2.1.4 and includes 30 new features, 84 improvements, 85 bug fixes, and many internal and doc changes. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/2.2.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.2.0/ 🛠 Changelog: https://airflow.apache.org/docs/apache-airflow/2.2.0/changelog.html 🐳 Docker Image: docker pull apache/airflow:2.2.0 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.2.0 As the changelog is quite large, the following are some notable new features that shipped in this release. Custom Timetables (AIP-39) Airflow has historically used cron expressions and timedeltas to represent when a DAG should run. This worked for a lot of use cases, but not all. For example, running daily on Monday-Friday, but not on weekends wasn't possible. To provide more scheduling flexibility, determining when a DAG should run is now done with Timetables. Of course, backwards compatibility has been maintained - cron expressions and timedeltas are still fully supported, however, timetables are pluggable so you can add your own custom timetable to fit your needs! For example, you could write a timetable to schedule a DagRun execution_date has long been confusing to new Airflowers, so as part of this change a new concept has been added to Airflow to replace it named data_interval, which is the period of data that a task should operate on. The following are now available: logical_date (aka execution_date) data_interval_start (same value as execution_date for cron) data_interval_end (aka next_execution_date) If you write your own timetables, keep in mind they should be idempotent and fast as they are used in the scheduler to create DagRuns. More information can be found at: Customizing DAG Scheduling with Timetables Deferrable Tasks (AIP-40) Deferrable tasks allows operators or sensors to defer themselves until a light-weight async check pass [Next page]()