# etl ETL (extract, transform, and load) is a process that combines data from multiple sources, transforms it, and loads it into a data warehouse or other target store. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Windmill for Supporting Indigenous Communities - Conservation Metrics Case Study DevFeed: [Windmill for Supporting Indigenous Communities - Conservation Metrics Case Study]() Original publisher: [Read original article]() Author: Rudo Kemper Published: 2025-06-03T00:00:00Z Content type: article Language: en Sources: [Windmill Blog]() Topics: [data-processing](), [etl](), [Open Source](), [Monitoring](), [data]() Tags: [brazil](), [case-study](), [case-study-testimonial-tech-for-good](), [dagster](), [dashboards](), [data](), [data-processing](), [etl](), [open-source](), [open-source-software](), [tech-for-good](), [testimonial](), [whatsapp](), [windmill](), [work](), [workflow]() ### AI overview This case study describes how Conservation Metrics uses Windmill to support environmental monitoring and data sovereignty initiatives with indigenous communities. The workflows collect data from field applications and wildlife sensors, process and load it into databases, generate reports and alerts, notify communities through WhatsApp, and update monitoring dashboards. ### Source excerpt This is a testimonial from Rudo Kemper from Conservation Metrics about how Windmill has helped them build open-source software for indigenous communities, enabling automated data processing and critical alerts for environmental protection. ## Partitioning a large table in PostgreSQL with Rails DevFeed: [Partitioning a large table in PostgreSQL with Rails]() Original publisher: [Read original article]() Published: 2025-04-28T00:00:00Z Content type: tutorial Language: en Sources: [Aha! Engineering Blog]() Topics: [PostgreSQL](), [Rails](), [Database](), [Amazon RDS](), [backups](), [Amazon S3](), [etl](), [Disaster Recovery](), [datadog]() Tags: [active-record](), [amazon-rds](), [amazon-s3](), [backups](), [datadog](), [disaster-recovery](), [etl](), [latency](), [migration](), [partitioning](), [performance](), [postgresql](), [rails]() ### AI overview This article explains how Aha! partitioned a very large PostgreSQL audit table used with Rails. It covers the migration strategy, operational and performance problems caused by billions of rows, and an archival process that moves audits older than 12 months to Amazon S3. ### Source excerpt Keeping a reliable history of changes is essential for our users. They need to know when a record was updated, who made the change, and why. They could be anxiously awaiting a notification about a dependency that is blocking their work. Maybe they w ## Think About SQL MERGE in Terms of a RIGHT JOIN DevFeed: [Think About SQL MERGE in Terms of a RIGHT JOIN]() Original publisher: [Read original article]() Author: lukaseder Published: 2025-03-13T14:45:53Z Content type: tutorial Language: en Sources: [jOOQ]() Topics: [SQL](), [etl](), [rdbms]() Tags: [databricks](), [etl](), [firebird](), [merge](), [outer-join](), [postgresql](), [right-join](), [schema](), [semantics](), [source](), [sql](), [sql-server](), [statement](), [syntax](), [table](), [target-table](), [when-matched](), [when-not-matched](), [when-not-matched-by-source]() ### AI overview The article explains SQL MERGE by relating its target-and-source behavior to a RIGHT JOIN. It covers updating matching rows, inserting nonmatching rows, deletion behavior, and the restriction against duplicate matches. ### Source excerpt RIGHT JOIN is an esoteric feature in the SQL language, and hardly ever seen in the real world, because almost every RIGHT JOIN can just be expressed as an equivalent LEFT JOIN. The following two statements are equivalent: It's not unreasonable to expect these two statements to produce the same execution plan on most RDBMS, ... Continue reading Think About SQL MERGE in Terms of a RIGHT JOIN -> ## Windmill for AI Workflows - Investing.com Case Study DevFeed: [Windmill for AI Workflows - Investing.com Case Study]() Original publisher: [Read original article]() Author: Yonathan Adest Published: 2025-02-24T00:00:00Z Content type: article Language: en Sources: [Windmill Blog]() Topics: [Automation](), [data-processing](), [Artificial Intelligence](), [etl](), [tracing](), [Logging](), [PostgreSQL](), [airflow]() Tags: [ai](), [airflow](), [automation](), [case-study](), [case-study-testimonial-fintech-investing](), [data-processing](), [docker-compose](), [embeddings](), [etl](), [fintech](), [human-review](), [investing](), [pdf](), [postgresql](), [push-notifications](), [recommendations](), [testimonial](), [windmill](), [workflows]() ### AI overview This Investing.com case study describes how its AI team uses Windmill to orchestrate content processing and distribution, stock-analysis report generation, and ETL workflows. The workflows use webhooks, AI models, vector embeddings, human review, and PostgreSQL, while Windmill provides tracing, logging, and Docker Compose deployment. ### Source excerpt This is a testimonial from Yonathan Adest, CTO at Investing.com, about how Windmill has helped them to automate their workflows and improve their data processing capabilities. ## The unnecessary hype strategy behind Microsoft Fabric DevFeed: [The unnecessary hype strategy behind Microsoft Fabric]() Original publisher: [Read original article]() Published: 2023-05-29T00:00:00Z Content type: opinion Language: en Sources: [Mutt0-ds Notes]() Topics: [Microsoft](), [data-engineering](), [Azure](), [Cloud](), [data lake](), [data-processing](), [etl](), [AI, ML & Data Engineering]() Tags: [azure](), [cloud](), [copilot](), [data](), [data-engineering](), [data-lake](), [data-processing](), [etl](), [microsoft](), [microsoft-azure](), [technical]() ### AI overview An opinion article examines Microsoft Fabric, a unified data platform announced at Microsoft Build. It describes Fabric's integration of data storage, processing, ETL, analytics, and business intelligence tools, while criticizing Microsoft's secrecy and hype-oriented launch strategy and noting that the platform was still in beta. ### Source excerpt If you are into Data Engineering in Microsoft Azure Cloud Environment, you probaly heard about Microsoft Fabric being announced last week... ## My first weeks at Acer Europe DevFeed: [My first weeks at Acer Europe]() Original publisher: [Read original article]() Published: 2022-10-16T00:00:00Z Content type: opinion Language: en Sources: [Mutt0-ds Notes]() Topics: [Azure](), [dashboards](), [Databases](), [etl](), [DevOps](), [Purview](), [active directory](), [data-governance]() Tags: [active-directory](), [azure](), [dashboards](), [data-governance](), [databases](), [developer](), [devops](), [etl](), [purview]() ### AI overview A Business Intelligence Developer reflects on their first weeks at Acer Europe, describing the transition from a small company to a large IT organization. The post covers the scale of its data, reporting, ETL, Azure services, and cross-team work, along with the information overload and learning challenges of the first week. ### Source excerpt This is a post for my future self when I will be asking myself: "How were my first days at Acer like?". Note: I'm working at Acer EMEA HQ,... ## Lesser Known PostgreSQL Features DevFeed: [Lesser Known PostgreSQL Features]() Original publisher: [Read original article]() Author: Haki Benita Published: 2021-11-07T22:00:00Z Content type: tutorial Language: en Sources: [Haki Benita]() Topics: [PostgreSQL](), [etl]() Tags: [article](), [articles](), [etl](), [features](), [postgresql]() ### AI overview This article presents lesser-known PostgreSQL features, including a command for synchronizing table data and a technique for distinguishing inserted rows from updated rows using a system column. It also discusses logging ETL processes. ### Source excerpt A list of useful features you already have, but may not know about! In this article I share lesser known features of PostgreSQL. ## How we build the Image Gallery on trivago DevFeed: [How we build the Image Gallery on trivago]() Original publisher: [Read original article]() Author: Praneeth Peiris I want Published: 2021-07-07T00:00:00Z Content type: article Language: en Sources: [Trivago]() Topics: [Google Cloud Platform (GCP)](), [etl](), [data lake](), [Kafka](), [Front end](), [Amazon Web Services]() Tags: [amazon-web-services-aws](), [architecture](), [backend](), [cloud](), [data-lake](), [engineering](), [etl](), [frontend](), [gcp](), [kafka](), [platforms]() ### AI overview trivago describes migrating its hotel image-gallery ETL pipeline from Amazon Web Services to Google Cloud Platform. The redesigned architecture uses Dataflow jobs to snapshot images and tags, fetch changed accommodations, build and validate sorted galleries, and stream approved changes to frontend teams through Kafka. ### Source excerpt When was the last time you booked accommodation without checking its photos? Most probably never! Because having imagery information makes our decision-making process much easier and faster. How... ## Accommodation Consolidation: How we created an ETL pipeline on cloud DevFeed: [Accommodation Consolidation: How we created an ETL pipeline on cloud]() Original publisher: [Read original article]() Author: Praneeth Peiris I want Published: 2020-03-26T00:00:00Z Content type: article Language: en Sources: [Trivago]() Topics: [AWS Glue](), [AWS Step Functions](), [etl](), [data](), [Cloud]() Tags: [aws](), [aws-glue](), [aws-step-functions](), [backend](), [batch](), [cloud](), [data](), [engineering-culture](), [etl](), [overhead](), [pipeline](), [pipelines]() ### AI overview trivago describes a hybrid AWS architecture using AWS Glue and AWS Step Functions to build ETL pipelines for consolidating frequently changing hotel information from hundreds of partners. The approach batches updates to reduce computational overhead and supports separately tested consolidation models and sandbox environments. ### Source excerpt Imagine you go to your hotel for check-in and they say that your dog is not allowed even though the website clearly states that it is!trivago gets information about millions of accommodat... ## Airflow Survey 2019 DevFeed: [Airflow Survey 2019]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2019-12-11T00:00:00Z Content type: article Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [data-processing](), [etl](), [Machine learning](), [DevOps](), [Command-line interface](), [ui]() Tags: [apache-airflow](), [cli](), [community](), [data](), [devops](), [etl](), [overview](), [pipelines](), [survey](), [testing](), [ui](), [users](), [workflows]() ### AI overview This article reports findings from the 2019 Apache Airflow user survey, which received 308 responses. Respondents primarily used Airflow for data processing and ETL, with additional use for machine learning pipelines and DevOps operations. The survey also describes the scale of users' DAGs and tasks, onboarding difficulties, testing and UI concerns, and interface usage. ### Source excerpt Apache Airflow Survey 2019 Apache Airflow is growing faster than ever. Thus, receiving and adjusting to our users' feedback is a must. We created survey and we got 308 responses. Let's see who Airflow users are, how they play with it, and what they miss. Overview of the user What best describes your current occupation? No. % Data Engineer 194 62.99% Developer 34 11.04% Architect 23 7.47% Data Scientist 19 6.17% Data Analyst 13 4.22% DevOps 13 4.22% IT Administrator 2 0.65% Machine Learning Engineer 2 0.65% Manager 2 0.65% Operations 2 0.65% Chief Data Officer 1 0.32% Engineering Manager 1 0.32% Intern 1 0.32% Product owner 1 0.32% Quant 1 0.32% In your day to day job, what do you use Airflow for? No. % Data processing (ETL) 298 96.75% Artificial Intelligence and Machine Learning Pipelines 90 29.22% Automating DevOps operations 64 20.78% According to the survey, most of the Airflow users are the "data" people. Moreover, 28.57% uses Airflow to both ETL and ML pipelines meaning that those two fields are somehow connected. Only five respondents use Airflow for DevOps operations only, That means that other 59 people who use Airflow for DevOps stuff use it also for ETL / ML purposes. How many active DAGs do you have in your largest Airflow instance? No. % 0-20 115 37.34% 21-40 65 21.10% 41-60 44 14.29% 61-100 28 9.09% 101-200 28 9.09% 201-300 7 2.27% 301-999 8 2.60% 1000+ 13 4.22% The majority of users do not exceed 100 active DAGs per Airflow instance. However, as we can see there are users who exceed thousands of DAGs with a maximum number 5000. What is the maximum number of tasks that you have used in one DAG? No. % 0-10 61 19.81% 11-20 60 19.48% 21-30 31 10.06% 31-40 21 6.82% 41-50 26 8.44% 51-100 36 11.69% 101-200 28 9.09% 201-500 21 6.82% 501+ 24 11.54% The given maximum number of tasks in a single DAG was 10 000 (!). The number of tasks depends on the purposes of a DAG, so it's rather hard to say if users have "simple" or "complicated" workflows. When onboarding ne ## How We Solved a Storage Problem in PostgreSQL Without Adding a Single Byte of Storage DevFeed: [How We Solved a Storage Problem in PostgreSQL Without Adding a Single Byte of Storage]() Original publisher: [Read original article]() Author: Haki Benita Published: 2018-12-21T22:00:00Z Content type: tutorial Language: en Sources: [Haki Benita]() Topics: [PostgreSQL](), [etl](), [Database](), [hash](), [Sorting]() Tags: [articles](), [dataset](), [disk-space](), [dump](), [etl](), [hash](), [performance](), [postgresql](), [query](), [sorting](), [sql](), [storage]() ### AI overview This article explains how an ETL task for removing duplicate binary records in PostgreSQL caused increasing temporary disk-space usage. The author reduced the sorting key by hashing the binary value with md5, lowering additional disk usage from about 30 MB to about 4 MB without adding storage. ### Source excerpt A while back we started getting alerts in the middle of the night on low disk space. A quick investigation led us to one of our ETL tasks. Every night the task was fired to eliminate duplicate dumps, and free up some space. This is a short story about how we found our silver bullet and solved the issue without adding a single byte of storage. ## Addepar's Migration from Mongo 2.4 to Mongo 3.4 DevFeed: [Addepar's Migration from Mongo 2.4 to Mongo 3.4]() Original publisher: [Read original article]() Author: Elan Kugelmass Published: 2017-10-24T13:11:15Z Content type: article Language: en Sources: [Addepar]() Topics: [Databases](), [upgrade](), [etl](), [Dependency management](), [Replication]() Tags: [data-pipelines](), [database](), [databases](), [dependency-management](), [migration](), [mongodb](), [pipeline](), [replication](), [upgrade]() ### AI overview Addepar describes upgrading its database from Mongo 2.4 (TokuMX 2.0) to Mongo 3.4 as its dataset and stability and performance requirements grew. The article explains Mongo's role in the company's data ingestion and ETL pipelines and why database upgrades are risky because database guarantees and behavior can change between versions. ### Source excerpt At Addepar, we're building the world's most versatile financial analytics engine. To feed the calculations that give our clients an unprecedented view into their portfolios, we need data -- from as many sources, vendors, and intermediaries as possible. Our market and portfolio data pipelines ingest benchmarks, security terms, accounting, and performance data from hundreds of integration partners. Behind this data pipeline is a database. And like every database, ours requires maintenance and care. Maintaining an obsolete database instance is challenging due to lack of support, inferior performance, and a dwindling developer community. As our dataset grew and we faced increased stability and performance requirements, the engineering group at Addepar decided it was time to upgrade our venerable Mongo 2.4 (TokuMX 2.0) database to the latest and greatest Mongo 3.4. Every organization has at least one database saga, and we're excited to share one of ours. Dependency management extends to databases Upgrading a database is a tricky business. Like other dependencies that support a product, databases have a tendency to fall out of date. Upgrades are deferred until that imagined future where everything is stable, clients have exhausted their feature request lists, and there's not much to do in the office other than play foosball and exchange memes. It's an understandable decision! Databases are complicated, leaky abstractions that inevitably form an implicit extension of our application logic. Their data types, atomicity guarantees, and transactionality semantics define the constraints we place on data and drive how we store application state. And because these guarantees (or lack thereof) tend to change (in ways that are sometimes undocumented!) between database versions, moving to the latest release is a risky proposition. Motivated by our need for an extremely reliable datastore that could handle complex and evolving schemas, we chose Mongo as the sole database for Addepar's ## A tour of Postgres' Foreign Data Wrappers DevFeed: [A tour of Postgres' Foreign Data Wrappers]() Original publisher: [Read original article]() Author: Map Published: 2016-09-11T20:55:56Z Content type: tutorial Language: en Sources: [Craig Kerstiens]() Topics: [SQL](), [Databases](), [etl](), [MySQL]() Tags: [database](), [mysql](), [pipeline](), [postgres](), [sql](), [sql-database]() ### AI overview A practical overview of PostgreSQL foreign data wrappers (FDWs), which connect PostgreSQL to remote systems and let users query or join that data from PostgreSQL. The article explains setup, capabilities, tradeoffs versus ETL, and briefly discusses PostgreSQL and MySQL FDWs. ### Source excerpt SQL can be a powerful language for reporting. Whether you're just exploring some data, or generating reports that show month over month revenue growth it's the lingua franca for data analysis. But, your data isn't always in a SQL database, even then if you're using Postgres you can still likely use SQL to analyze, query, even joing with that data. Foreign data wrappers have been around for years in Postgres, but are continuing to mature and be a great option for joining disparate systems. Overview of foreign data wrappers If you're unfamiliar, foreign data wrappers, or FDW, allow you to connect from within Postgres to a remote system. Then you can query them from directly within Postgres. While there is an official Postgres FDW that ships with Postgres itself, that allows you to connect from one Postgres DB to another, there's also a broad community of others. At the core of it Postgres provides certain APIs under the covers which each FDW extension can implement. This can include the ability to map SQL to whatever makes sense for a given system, push down various operators like where clauses, and as of Postgres 9.3 can even write data. To setup a FDW you first would install the extension, then provide the connection to the remote system, setup your schema/tables, and then you're off to the races-or well ready to query. If you've got more than 2-3 databases or systems in your infrastructure, you'll often benefit from FDWs as opposed to introducing a heavyweight ETL pipeline. Don't mistake FDWs as the most performant method for joining data, but they are often the developer time efficient means of joining these data sets. Let's look at just a few of the more popular and interesting ones. Postgres FDW The Postgres one is the easiest to get started with. First you'll just enable it with CREATE EXTENSION, then you'll setup your remote server: CREATE EXTENSION postgres_fdw; CREATE SERVER core_db FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host 'foo', dbname 'core_db', por ## Postgres Tips: CTEs, .psqlrc Configuration, Query Statistics, and ETL DevFeed: [Postgres Tips: CTEs, .psqlrc Configuration, Query Statistics, and ETL]() Original publisher: [Read original article]() Author: Map Published: 2015-12-29T20:55:56Z Content type: tutorial Language: en Sources: [Craig Kerstiens]() Topics: [Databases](), [SQL](), [etl]() Tags: [cte](), [pg-stat-statements](), [postgres](), [postgresql](), [setup](), [sql](), [tips]() ### AI overview A practical collection of Postgres tips covering Common Table Expressions for readable queries, .psqlrc customization, pg_stat_statements for identifying indexing and performance opportunities, and caution around aggregating data through ETL when working across microservices or applications. ### Source excerpt I find during the holiday season many pick up new books, learn a new language, or brush up on some other skill in general. Here's my contribution to hopefully giving you a few new things to learn about Postgres and ideally utilize in the new year. It's not in a top 10 list as much as 10 tips and tricks you should be aware of as when you need them they become incredibly handy. But, first a shameless plug if you find any of the following helpful, consider subscribing to Postgres weekly a weekly newsletter with interesting Postgres content. 1. CTEs - Common Table Expressions CTEs allow you to do crazy awesome things like recursive queries but even the most simple form of them I don't go a day without using. Think of a CTE or commonly known as with clause as a view inside the time that query is running. This lets you more easily create readable query. Any query that's constructed that's even 100 lines long, but with 4-5 CTEs is undoubtedly going to be easier for someone new to come in and understand than a 20 line query that does the same thing. A few people like writing SQL, but no one likes reading someone else's so do them a favor and read up on CTEs. 2. Setup a .psqlrc You setup a bashrc, vimrc, etc. Why not do the same for Postgres. Some of the great things you can do: Setup pretty formatting by default with \x auto Set nulls to actually look like something \pset null ¤ Turn timing on by default \timing on Customize your prompt \set PROMPT1 '%[%033[33;1m%]%x%[%033[0m%]%[%033[1m%]%/%[%033[0m%]%R%# ' Save commonly run queries that you can run by name Here's an example of my own psqlrc: \set QUIET 1 \pset null '¤' -- Customize prompts \set PROMPT1 '%[%033[1m%][%/] # ' \set PROMPT2 '... # ' -- Show how long each query takes to execute \timing -- Use best available output format \x auto \set VERBOSITY verbose \set HISTFILE ~/.psql_history- :DBNAME \set HISTCONTROL ignoredups \set COMP_KEYWORD_CASE upper \unset QUIET 3. pg_stat_statements for where to index pg_stat_state ## Reformater avec pgloader DevFeed: [Reformater avec pgloader]() Original publisher: [Read original article]() Author: Dimitri Fontaine PostgreSQL Major Contributor; Author Published: 2011-08-05T09:26:00Z Content type: tutorial Language: fr Sources: [Dimitri Fontaine]() Topics: [etl]() Tags: [articles](), [etl](), [simple]() ### AI overview This article explains how to use pgloader's reformatting function. It presents reformatting as the Transform phase of an ETL process and describes pgloader as a simple solution for ETL needs. ### Source excerpt Dans la série de nos articles sur pgloader, le dernier venu détaille comment utiliser la fonction de reformatage de cet outil. Dans le cadre d'utilisation d'un ETL, cela est assimilé à la phase Transform, ce qui fait de pgloader une solution simple pour vos besoins d'ETL.