# Data pipelines Data pipelines are series of processing steps that systematically collect, transform, and store data from various sources, including for analysis. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## AI Changed How Spotify Builds. What We Learned (and Fixed) About Quality at Higher Velocity DevFeed: [AI Changed How Spotify Builds. What We Learned (and Fixed) About Quality at Higher Velocity]() Original publisher: [Read original article]() Author: Spotify Engineering Published: 2026-09-16T19:13:53Z Content type: article Language: en Sources: [Spotify Engineering](), [Spotify Engineering Blog]() Topics: [Monitoring](), [reliability](), [Microservices](), [Data pipelines](), [Artificial Intelligence](), [Job]() Tags: [ai](), [bug](), [data-pipelines](), [jobs](), [microservices](), [monitoring](), [reliability]() ### AI overview Spotify describes how rapid change, content-processing weaknesses, capacity limits, and a scheduling bug contributed to delays in publishing episodes. It reports adding end-to-end monitoring, fixing the scheduler, lowering batch-job priority, and increasing capacity. ### Source excerpt Quality and reliability have always been a point of pride for Spotify. We run an extraordinarily complex... The post AI Changed How Spotify Builds. What We Learned (and Fixed) About Quality at Higher Velocity appeared first on Spotify Engineering. ## Quasi-Agentic Pipelines with Databricks and Apache Airflow DevFeed: [Quasi-Agentic Pipelines with Databricks and Apache Airflow]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-08-10T21:23:57Z Content type: tutorial Language: en Sources: [Data Engineering Central]() Topics: [databricks](), [airflow](), [Data pipelines](), [Large Language Model](), [data-engineering]() Tags: [airflow](), [apache-airflow](), [data-engineering](), [databricks](), [llms](), [pipelines]() ### AI overview A practical developer discussion of incorporating LLMs and agents into existing data workflows using Databricks and Apache Airflow. It also examines determinism in data pipelines and the gap between business requirements and engineering implementation. ### Source excerpt the strange space in between ## The 5 Silent Failures in Data Pipelines DevFeed: [The 5 Silent Failures in Data Pipelines]() Original publisher: [Read original article]() Author: SeattleDataGuy Published: 2026-04-24T19:06:02Z Content type: article Language: en Sources: [SeattleDataGuy's Newsletter]() Topics: [Data pipelines](), [data](), [CSV]() Tags: [csv](), [dashboard](), [data-pipeline](), [data-pipelines](), [pipelines](), [reports](), [schema]() ### AI overview The article explains how data pipelines can fail silently without triggering errors or obvious warnings, causing stale or incorrect data to reach dashboards and reports. It introduces schema drift as one failure mode, including unexpected changes to CSV or XML files loaded from SFTP. ### Source excerpt How Your Pipelines Lie to You Without Throwing a Single Error ## Data Pipeline Foundations - Everything You Need To Know About Data Pipelines DevFeed: [Data Pipeline Foundations - Everything You Need To Know About Data Pipelines]() Original publisher: [Read original article]() Author: SeattleDataGuy Published: 2026-04-18T15:50:56Z Content type: article Language: en Sources: [SeattleDataGuy's Newsletter]() Topics: [Data pipelines](), [data](), [API](), [Database](), [Amazon S3]() Tags: [api](), [data](), [data-pipelines](), [database](), [pipeline](), [s3](), [sftp]() ### AI overview This article serves as a central collection of introductory resources about data pipelines. It explains that pipelines move data from sources to destinations and highlights sources such as S3 buckets, SFTP files, APIs, and databases. ### Source excerpt Hi, fellow future and current Data Leaders; Ben here 👋 ## Daily Tasks With Data Pipelines - Data Quality Checks And The Problem With Noisy Checks DevFeed: [Daily Tasks With Data Pipelines - Data Quality Checks And The Problem With Noisy Checks]() Original publisher: [Read original article]() Author: SeattleDataGuy Published: 2026-04-07T22:24:38Z Content type: article Language: en Sources: [SeattleDataGuy's Newsletter]() Topics: [Data Quality](), [Data pipelines](), [data]() Tags: [data-pipelines](), [data-quality](), [pipelines](), [quality]() ### AI overview The article discusses data quality checks in data pipelines and notes that teams may receive 137 data quality alerts every morning. ### Source excerpt Every morning, your team wakes up to 137 data quality alerts. ## Introducing the Apache Airflow Registry DevFeed: [Introducing the Apache Airflow Registry]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-03-19T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Data pipelines](), [API](), [JSON]() Tags: [airflow](), [apache](), [apache-airflow](), [community](), [data-pipelines](), [messaging](), [notifications](), [openai](), [registry]() ### AI overview Apache Airflow launches the Airflow Registry, a searchable catalog of official providers and modules. It includes provider and module search, installation and compatibility details, connection generation in URI, JSON, and environment-variable formats, ecosystem statistics, and a structured JSON API. ### Source excerpt Today we're launching the Apache Airflow Registry -- a searchable catalog of every official Airflow provider and its modules, live at airflow.apache.org/registry/. Need an S3 operator? A Snowflake hook? An OpenAI sensor? The Registry helps you find, compare, and configure the right components for your data pipelines -- without digging through docs or PyPI pages. By the Numbers 98 Official providers 1,602 Modules (operators, hooks, sensors, triggers, transfers, and more) 329M+ Monthly PyPI downloads across all providers 125+ Integrations with cloud platforms, databases, ML tools, and messaging services Search Everything Hit Cmd+K from any page and start typing. Results show up instantly, grouped by Providers and Modules, with type badges so you can tell a hook from an operator at a glance. Provider Pages Each provider gets a dedicated page with everything in one place: install command with copy-to-clipboard, version selector, extras dropdown, compatibility info, connection types, and the full module listing organized by type. The Amazon provider, for example, has 372 modules across operators, hooks, sensors, triggers, transfers, and more. Module type tabs let you filter to exactly what you're looking for, and a category sidebar groups modules by AWS service (S3, Lambda, Glue, Step Functions, etc.). Connection Builder Click any connection type badge on a provider page, fill in the fields, and the builder generates the connection in three formats -- URI, JSON, and Env Var -- ready to copy into your configuration. No more guessing URI encoding or JSON structure. Explore by Category Not sure which provider you need? The Explore page organizes providers into categories: Cloud Platforms, Databases, Data Warehouses, Messaging & Notifications, AI & Machine Learning, Data Processing, and more. Statistics The Stats page breaks down the ecosystem: 848 operators, 298 hooks, 164 triggers, 157 sensors, 83 transfers, and more -- plus top providers by downloads and module count. JSON API ## Backfills - The Necessary Evil of Data Engineering DevFeed: [Backfills - The Necessary Evil of Data Engineering]() Original publisher: [Read original article]() Author: SeattleDataGuy Published: 2026-02-23T23:37:52Z Content type: tutorial Language: en Sources: [SeattleDataGuy's Newsletter]() Topics: [data-engineering](), [Data pipelines](), [data](), [datasets](), [systems]() Tags: [bug](), [data-engineering](), [data-pipelines](), [data-type](), [databases](), [engineering](), [pipelines](), [schema]() ### AI overview This article explains why data teams perform backfills and why data engineers often dislike them. It describes backfills as rerunning or rebuilding tables and pipelines to account for corrected source data, pipeline bugs, schema changes, logic changes, or required data-type conversions. ### Source excerpt Why backfills happen, why we hate them, and how to handle them without breaking trust ## Common Data Pipeline Patterns You'll See in the Real World DevFeed: [Common Data Pipeline Patterns You'll See in the Real World]() Original publisher: [Read original article]() Author: SeattleDataGuy Published: 2026-01-05T19:58:06Z Content type: article Language: en Sources: [SeattleDataGuy's Newsletter]() Topics: [Data pipelines]() Tags: [data](), [data-pipeline](), [patterns](), [practical](), [real-world]() ### AI overview The article provides a practical overview of the different ways data pipelines appear within real companies. ### Source excerpt A practical look at the many ways data pipelines show up inside real companies ## How Can I Be An AI Engineer? DevFeed: [How Can I Be An AI Engineer?]() Original publisher: [Read original article]() Published: 2024-12-09T00:00:00Z Content type: tutorial Language: en Sources: [Tim Kellogg]() Topics: [AI Engineering](), [AI Development](), [Large Language Model](), [Retrieval-Augmented Generation](), [Data pipelines](), [Embeddings](), [Front end](), [React]() Tags: [ai-engineer](), [ai-engineering](), [data-pipelines](), [embeddings](), [frontend](), [genai](), [llms]() ### AI overview The article explains what AI engineers do, emphasizing that the role involves integrating generative AI models into applications and may include building user interfaces, APIs, and data pipelines. It describes several archetypes, including data-pipeline and UX-focused AI engineers, and discusses skills and experience that may be useful. ### Source excerpt You want to be an AI Engineer? Do you even have the right skills? What do they do? All great questions. I've had this same conversation several times, so I figured it would be best to write it down. Here I answer all those, and break down the job into archetypes that should help you understand how you'll contribute. ## Office Hours with Engineering Managing Director Mae Santos DevFeed: [Office Hours with Engineering Managing Director Mae Santos]() Original publisher: [Read original article]() Author: Emily Majewski Published: 2024-06-26T17:34:49Z Content type: article Language: en Sources: [Two Sigma Engineering]() Topics: [reliability](), [systems](), [Data pipelines](), [Resilience](), [Automation]() Tags: [automation](), [best-practices](), [collaboration](), [data-pipelines](), [engineering](), [integrity](), [reliability-engineering]() ### AI overview In this Office Hours interview, Two Sigma engineering leader Mae Santos discusses her career, leadership values, and responsibilities overseeing critical applications and data reliability engineering. She emphasizes integrity, collaboration, resilience, and designing reliability into applications and data systems from the beginning. ### Source excerpt The post Office Hours with Engineering Managing Director Mae Santos appeared first on Two Sigma. ## Insights from Workflow History Export on Temporal Cloud DevFeed: [Insights from Workflow History Export on Temporal Cloud]() Original publisher: [Read original article]() Author: Alice Yin Published: 2024-04-24T06:00:00Z Content type: tutorial Language: en Sources: [Temporal Blog]() Topics: [export](), [Amazon S3](), [Data pipelines](), [Data analysis](), [parquet](), [airflow](), [AWS Glue]() Tags: [airflow](), [amazon-s3](), [api](), [aws-glue](), [case-study](), [data](), [data-analysis](), [data-pipelines](), [export](), [parquet](), [product-news]() ### AI overview This tutorial explains how to export closed workflow histories from Temporal Cloud to Amazon S3, convert Protocol Buffer files to Parquet, and build data pipelines for analyzing execution metadata, operational efficiency, and bottlenecks. ### Source excerpt Explore how exporting workflow histories in Temporal Cloud gives you deep insights into execution metadata, events, and performance over time. ## Build AI, ML and Data Engineering Workflows in 2026 | Temporal DevFeed: [Build AI, ML and Data Engineering Workflows in 2026 | Temporal]() Original publisher: [Read original article]() Author: Joshua Smith Published: 2024-04-22T04:00:00Z Content type: article Language: en Sources: [Temporal Blog]() Topics: [AI, ML & Data Engineering](), [Orchestration](), [data-engineering](), [Data pipelines](), [Cloud]() Tags: [ai](), [data-engineering](), [gpu](), [long-running](), [ml](), [orchestration](), [pipelines](), [recovery](), [temporal-concepts](), [workflows]() ### AI overview This article explains how Temporal is used to orchestrate AI and machine learning workflows, including end-to-end processes and complex data pipelines. It describes job coordination across GPU resources, failure recovery, retries, and management of long-running tasks, with Descript as an example. ### Source excerpt Build reliable AI and ML pipelines faster. See how Temporal handles job coordination, failure recovery, and GPU orchestration so your team can do more. ## Leveraging Spark 3 and NVIDIA's GPUs to Reduce Cloud Cost by up to 70% for Big Data Pipelines DevFeed: [Leveraging Spark 3 and NVIDIA's GPUs to Reduce Cloud Cost by up to 70% for Big Data Pipelines]() Original publisher: [Read original article]() Author: Ilay Chen Published: 2024-02-21T16:42:14Z Content type: tutorial Language: en Sources: [PayPal Technology]() Topics: [Apache Spark](), [GPU](), [RAPIDS](), [Cloud](), [Data pipelines](), [migration](), [upgrade](), [Open Source]() Tags: [apache-spark](), [big-data](), [cloud](), [cloud-computing](), [data-pipelines](), [data-processing](), [gpu](), [machine-learning](), [migration](), [rapids](), [upgrade]() ### AI overview A PayPal engineering blog explains how upgrading from Apache Spark 2 to Spark 3 and migrating workloads to GPU clusters with NVIDIA Spark RAPIDS can accelerate selected big-data processing tasks and potentially reduce cloud costs by up to 70%. It covers the migration, parameter tuning, challenges, and reported benefits. ### Source excerpt By Ilay Chen and Tomer Akirav At PayPal, hundreds of thousands of Apache Spark jobs run on an hourly basis, processing petabytes of data and requiring a high volume of resources. To handle the growth of machine learning solutions, PayPal requires scalable environments, cost awareness and constant innovation. This blog explains how Apache Spark 3 and GPUs can help enterprises potentially reduce Apache Spark's jobs cloud costs by up to 70% for big data processing and AI applications. Our journey will begin with a brief introduction of Spark RAPIDS -- Apache Spark's accelerator that leverages GPUs to accelerate processing via the RAPIDS libraries. We will then review PayPal's CPU-based Spark 2 application, our upgrade to Spark 3 and its new capabilities, explore the migration of our Apache Spark application to a GPU cluster, and how we tuned Spark RAPIDS parameters. We will then discuss some challenges we encountered and the benefits of the updates. Libra scales in the cloud, generated by AIBackground GPUs are everywhere, and their parallelism characteristics are perfect for processing AI and graphics applications, among other things. For those unfamiliar: what makes GPUs different from CPUs, computation-wise, is that CPUs have a limited amount of very strong cores, whereas GPUs have thousands, or even tens of thousands or more, relatively weak cores that work together very well. PayPal has been leveraging GPUs to train models for some time now, and so we decided to evaluate if the parallelism of the GPU can be helpful with processing big data applications based on Apache Spark. In our research, we encountered NVIDIA's Spark RAPIDS open-source project. It has many purposes, however we focused on Spark RAPIDS's cost reduction potential, because enterprises like PayPal spend lots of money on running Spark jobs in the cloud. Using Spark with GPUs isn't common in the industry yet, but according to our findings as described in this blog, the potential benefits could be enorm ## Introducing Setup and Teardown tasks DevFeed: [Introducing Setup and Teardown tasks]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2023-08-18T00:00:00Z Content type: article Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Data pipelines](), [GPU]() Tags: [airflow](), [blog-post](), [cleanup](), [data-pipelines](), [dependencies](), [setup](), [tasks]() ### AI overview This article introduces setup and teardown tasks in Airflow 2.7 for managing infrastructure around work in data pipelines. It explains their dependency semantics, cleanup behavior, DAG run state handling, and behavior within task groups. ### Source excerpt In data pipelines, commonly we need to create infrastructure resources, like a cluster or GPU nodes in an existing cluster, before doing the actual "work" and delete them after the work is done. Airflow 2.7 adds "setup" and "teardown" tasks to better support this type of pipeline. This blog post aims to highlight the key features so you know what's possible. For full documentation on how to use setup and teardown tasks, see the setup and teardown docs. Why setup and teardown? Before we dig into examples, let me state at high level what setup and teardown bring to the table. More expressive dependencies Before setup and teardown, upstream and downstream relationships could only mean one thing: "this comes before that". With setup and teardown, in effect we can say "this requires that". And what it means in practice is, if you clear your task, and it requires a setup, that setup will be cleared too. And if that setup has a teardown, that will run again as well. Separating the work from the infra Sometimes the part of the dag you care about is not, say, the cleanup task. For example, suppose you have a dag that loads some data and then deletes temp files. As long as the data loads, you want your dag to be marked successful. By default, this is how teardown tasks work; that is, they are ignored when determining dag run state. Simple case A simple example is one setup / teardown pair, and one normal or "work" task. Setups and teardowns are indicated by the up and down arrows, respectively. From that we can see that .create_cluster is a setup task and delete_cluster is a teardown. The link between a setup and a teardown is always dotted to highlight the special relationship. Some things to observe: If create_cluster fails, neither run_query nor delete_cluster will run. If create_cluster succeeds and run_query fails, then delete_cluster will still run. If create_cluster is skipped, run_query and delete_cluster will be skipped By default, if run_query succeeds, and delete_c ## Airflow Summit 2022 DevFeed: [Airflow Summit 2022]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2022-05-16T00:00:00Z Content type: news Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Data pipelines](), [data-governance](), [Open Source](), [AI, ML & Data Engineering]() Tags: [airflow-summit](), [apache-airflow](), [community](), [data-governance](), [data-pipelines](), [events](), [machine-learning](), [network](), [open-source](), [summit]() ### AI overview Airflow Summit 2022 was scheduled for May 23-27 as a free conference for Apache Airflow practitioners and data leaders. The program covered Airflow practices, data pipelines, data governance, machine learning, the project's future, and non-code open-source contributions. ### Source excerpt The biggest Airflow Event of the Year returns May 23-27! Airflow Summit 2022 will bring together the global community of Apache Airflow practitioners and data leaders. What's on the Agenda During the free conference, you will hear about Apache Airflow best practices, trends in building data pipelines, data governance, Airflow and machine learning, and the future of Airflow. There will also be a series of presentations on non-code contributions driving the open-source project. How to Attend This year's edition will include a variety of online sessions across different time zones. Additionally, you can take part in local in-person events organized worldwide for data communities to watch the event and network. Interested? đŸĒļ Register for Airflow Summit 2022 today 🤝 Check out the in-person events planned for Airflow Summit 2022.