# etl Published articles for etl. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## A serverless, data-driven Git metrics dashboard using Amazon Quick Sight DevFeed: [A serverless, data-driven Git metrics dashboard using Amazon Quick Sight]() Original publisher: [Read original article]() Author: Saurabh Singhal Published: 2026-09-17T15:42:31Z Content type: tutorial Language: en Sources: [Artificial Intelligence]() Topics: [dashboards](), [GitHub](), [Amazon Web Services](), [GitLab](), [Amazon S3](), [AWS Step Functions](), [event driven](), [Orchestration](), [API](), [parallel](), [ai-coding]() Tags: [advanced-300](), [ai](), [ai-coding-tools](), [amazon-quick-sight](), [amazon-s3](), [automated](), [aws](), [aws-step-functions](), [concurrently](), [dashboard](), [dashboards](), [delivery](), [etl](), [event-driven](), [execution](), [github](), [gitlab](), [infrastructure](), [metrics](), [pipeline](), [real-time](), [technical-how-to]() ### AI overview This tutorial presents a serverless pipeline that collects Git metrics from GitHub and GitLab, processes repository activity through an event-driven workflow, stores results in Amazon S3, and visualizes them in interactive Amazon Quick Sight dashboards. It also describes change detection, incremental loads, and parallel processing for larger organizations. ### Source excerpt Learn how to build a fully serverless pipeline that automatically collects Git metrics from GitHub and GitLab and visualizes them in interactive Amazon Quick Sight dashboards, giving engineering teams near-real-time delivery analytics at low cost. ## Three principles for building a vector platform at Thumbtack DevFeed: [Three principles for building a vector platform at Thumbtack]() Original publisher: [Read original article]() Author: John Zhu Published: 2026-09-10T15:45:00Z Content type: article Language: en Sources: [Thumbtack Engineering - Medium]() Topics: [Embeddings](), [AI search](), [Database](), [data](), [Machine learning](), [API]() Tags: [api](), [data](), [embeddings](), [engineering](), [etl](), [infrastructure](), [machine-learning](), [ml-platform](), [pgvector](), [retrieval](), [search](), [vector](), [vector-search]() ### AI overview This article explains how Thumbtack built a vector platform that lets ML engineers deploy production vector search without managing database access, custom ETL, or query services. It describes three guiding principles: reuse existing infrastructure, treat embeddings as data, and reduce adoption costs for future teams. ### Source excerpt Reusing what we already had, treating embeddings as data, and lowering the next team's cost Today, an ML engineer at Thumbtack can stand up production vector search without negotiating database access, building a custom ETL, or writing a query service. The team brings their choice of embedding model, the data, and the query; the platform handles what connects them. It took several iterations to get to this point. In this post we'll walk through how we got there and the three principles that shaped what we built. A vector database stores high-dimensional numeric arrays (embeddings) and serves nearest-neighbor queries against them. It's how an ML system asks "what's most similar to this?" instead of "what matches this exact key?" The shift from exact lookup to semantic retrieval is what makes vectors useful: a search can return results that mean the same thing, not just results that spell the same. At Thumbtack, embeddings sit between the models that produce them and the services that consume them: language models for text, multimodal models for images, retrieval models for ranking. The platform we describe here is where those embeddings live and how teams reach for them when they need to. Three principles shaped what we built. Reuse what we have: extend the infrastructure we already run rather than stand up a new system. Treat embeddings as data: flow them through the same pipelines that move every other dataset at the company. Lower the next team's cost: make the platform easier to adopt than to work around. Each principle shaped one layer of the system, and together they took vector search from a one-off project to a platform that any team can build on. Architecture at a glance The platform has four moving parts: where embeddings come from, how they reach the database, where they live, and how consumers query them. Each is a layer, and together they form a pipeline that produces vectors and serves similarity searches as a typed API call. The diagram below traces a ## Migrating from a Monolithic Orchestrator to Apache Airflow DevFeed: [Migrating from a Monolithic Orchestrator to Apache Airflow]() Original publisher: [Read original article]() Author: Poorva Patil Published: 2026-06-03T09:20:17Z Content type: article Language: en Sources: [Helpshift]() Topics: [airflow](), [Orchestration](), [Clojure](), [Python](), [Monitoring]() Tags: [apache-airflow](), [architecture](), [aws](), [clojure](), [data-engineering](), [etl](), [monitoring](), [monolithic-architecture](), [orchestration](), [python](), [software-architecture]() ### AI overview The article describes a migration from a monolithic Clojure-based scheduler to Apache Airflow. The legacy system coupled event pipelines, dependencies, and an EMR step, making workflows difficult to understand, maintain, monitor, and debug. The authors report that Airflow's Python-based DAGs made dependencies, retries, and scheduling more readable and helped improve onboarding and maintainability. ### Source excerpt Photo by Corinne Kutz on UnsplashBefore we knew better Our orchestration system started as a simple internal solution to manage event pipelines and trigger downstream jobs. Over time, as more workflows and dependencies were added, it gradually evolved into a tightly coupled monolithic scheduler that became increasingly difficult to understand and maintain. Understanding how a workflow executed often meant looking through multiple files, configurations and database tables. For newer team members, onboarding into the system took time because much of the workflow context was distributed across different parts of the codebase. Even relatively small changes required careful coordination to ensure existing pipelines continued to work as expected. Similarly, debugging typically involved manually tracing logs and rerunning jobs to better understand execution behavior. Limitations of our legacy design We had a monolithic architecture written in Clojure that bundled all our event pipelines together, added dependencies between them and triggered a Lambda function. Legacy Workflow This Lambda function added a single monolithic step to the EMR cluster. If there was an issue in any one of the pipelines, the entire flow would fail due to the single step on the cluster. We did not have step-wise monitoring in the old design, so during on-call situations it became very difficult to identify which part of the pipeline was causing the issue. Photo by Tim Gouw on Unsplash There was no single place to answer basic questions like: What runs first? What happens if this step fails? How do I re-run just one part safely? The scheduler worked, but it was hard to understand, hard to maintain and even harder to explain. That's when we realized we needed a better way. What we actually needed Our aim was less about fancy scheduling features and more about making our daily work easier and more reliable. Simpler onboarding, less mental overhead Our existing step scheduler was built in Clojure and c ## Building an Enterprise Data Warehouse on Heroku: From Complex ETL to Seamless Salesforce Integration DevFeed: [Building an Enterprise Data Warehouse on Heroku: From Complex ETL to Seamless Salesforce Integration]() Original publisher: [Read original article]() Author: Sudarshan Hiray Published: 2025-11-05T20:05:38Z Content type: article Language: en Sources: [Heroku]() Topics: [Heroku](), [data-architecture](), [Databases](), [Monitoring]() Tags: [analytics](), [applications](), [article](), [building](), [data](), [ecosystems](), [engineering](), [enterprise](), [etl](), [heroku](), [heroku-postgres](), [integration](), [platforms](), [salesforce](), [services](), [systems](), [tools](), [warehouse]() ### AI overview Heroku describes an enterprise data warehouse architecture that unifies Salesforce and data from multiple application databases for real-time analytics. The approach uses Heroku Connect, Heroku Postgres follower databases, and a unified analytics platform to address API limits, ETL complexity, production database load, and fragmented monitoring. ### Source excerpt Modern businesses don't just run on Salesforce--they run on entire ecosystems of applications. At Heroku, we operate dozens of services alongside our Salesforce instance such as billing systems, user management platforms, analytics engines, and support tools. Traditional approaches to unifying this data create more problems than they solve. In this article, we'll see how we [...] The post Building an Enterprise Data Warehouse on Heroku: From Complex ETL to Seamless Salesforce Integration appeared first on Heroku. ## Windmill for Supporting Indigenous Communities - Conservation Metrics Case Study DevFeed: [Windmill for Supporting Indigenous Communities - Conservation Metrics Case Study]() Original publisher: [Read original article]() Author: Rudo Kemper Published: 2025-06-03T00:00:00Z Content type: article Language: en Sources: [Windmill Blog]() Topics: [data-processing](), [etl](), [Open Source](), [Monitoring](), [data]() Tags: [brazil](), [case-study](), [case-study-testimonial-tech-for-good](), [dagster](), [dashboards](), [data](), [data-processing](), [etl](), [open-source](), [open-source-software](), [tech-for-good](), [testimonial](), [whatsapp](), [windmill](), [work](), [workflow]() ### AI overview This case study describes how Conservation Metrics uses Windmill to support environmental monitoring and data sovereignty initiatives with indigenous communities. The workflows collect data from field applications and wildlife sensors, process and load it into databases, generate reports and alerts, notify communities through WhatsApp, and update monitoring dashboards. ### Source excerpt This is a testimonial from Rudo Kemper from Conservation Metrics about how Windmill has helped them build open-source software for indigenous communities, enabling automated data processing and critical alerts for environmental protection. ## Partitioning a large table in PostgreSQL with Rails DevFeed: [Partitioning a large table in PostgreSQL with Rails]() Original publisher: [Read original article]() Published: 2025-04-28T00:00:00Z Content type: tutorial Language: en Sources: [Aha! Engineering Blog]() Topics: [PostgreSQL](), [Rails](), [Database](), [Amazon RDS](), [backups](), [Amazon S3](), [etl](), [Disaster Recovery](), [datadog]() Tags: [active-record](), [amazon-rds](), [amazon-s3](), [backups](), [datadog](), [disaster-recovery](), [etl](), [latency](), [migration](), [partitioning](), [performance](), [postgresql](), [rails]() ### AI overview This article explains how Aha! partitioned a very large PostgreSQL audit table used with Rails. It covers the migration strategy, operational and performance problems caused by billions of rows, and an archival process that moves audits older than 12 months to Amazon S3. ### Source excerpt Keeping a reliable history of changes is essential for our users. They need to know when a record was updated, who made the change, and why. They could be anxiously awaiting a notification about a dependency that is blocking their work. Maybe they w ## Think About SQL MERGE in Terms of a RIGHT JOIN DevFeed: [Think About SQL MERGE in Terms of a RIGHT JOIN]() Original publisher: [Read original article]() Author: lukaseder Published: 2025-03-13T14:45:53Z Content type: tutorial Language: en Sources: [jOOQ]() Topics: [SQL](), [etl](), [rdbms]() Tags: [databricks](), [etl](), [firebird](), [merge](), [outer-join](), [postgresql](), [right-join](), [schema](), [semantics](), [source](), [sql](), [sql-server](), [statement](), [syntax](), [table](), [target-table](), [when-matched](), [when-not-matched](), [when-not-matched-by-source]() ### AI overview The article explains SQL MERGE by relating its target-and-source behavior to a RIGHT JOIN. It covers updating matching rows, inserting nonmatching rows, deletion behavior, and the restriction against duplicate matches. ### Source excerpt RIGHT JOIN is an esoteric feature in the SQL language, and hardly ever seen in the real world, because almost every RIGHT JOIN can just be expressed as an equivalent LEFT JOIN. The following two statements are equivalent: It's not unreasonable to expect these two statements to produce the same execution plan on most RDBMS, ... Continue reading Think About SQL MERGE in Terms of a RIGHT JOIN -> ## Windmill for AI Workflows - Investing.com Case Study DevFeed: [Windmill for AI Workflows - Investing.com Case Study]() Original publisher: [Read original article]() Author: Yonathan Adest Published: 2025-02-24T00:00:00Z Content type: article Language: en Sources: [Windmill Blog]() Topics: [Automation](), [data-processing](), [Artificial Intelligence](), [etl](), [tracing](), [Logging](), [PostgreSQL](), [airflow]() Tags: [ai](), [airflow](), [automation](), [case-study](), [case-study-testimonial-fintech-investing](), [data-processing](), [docker-compose](), [embeddings](), [etl](), [fintech](), [human-review](), [investing](), [pdf](), [postgresql](), [push-notifications](), [recommendations](), [testimonial](), [windmill](), [workflows]() ### AI overview This Investing.com case study describes how its AI team uses Windmill to orchestrate content processing and distribution, stock-analysis report generation, and ETL workflows. The workflows use webhooks, AI models, vector embeddings, human review, and PostgreSQL, while Windmill provides tracing, logging, and Docker Compose deployment. ### Source excerpt This is a testimonial from Yonathan Adest, CTO at Investing.com, about how Windmill has helped them to automate their workflows and improve their data processing capabilities. ## Analytics for Per-User Database Architecture DevFeed: [Analytics for Per-User Database Architecture]() Original publisher: [Read original article]() Author: Jamie Barton Published: 2024-11-18T00:00:00Z Content type: article Language: en Sources: [Turso Blog]() Topics: [Per-user Database](), [Multitenancy](), [Turso](), [SQLite](), [Platform API](), [Data Management](), [Architecture & Design](), [Software as a service](), [data-architecture](), [Databases]() Tags: [architecture](), [data-management](), [database](), [databases](), [etl](), [metrics](), [monitor](), [per-user-database](), [platform-api](), [reporting](), [saas](), [sqlite](), [turso]() ### AI overview This article explains how to aggregate analytics from a per-user or per-tenant Turso database architecture. It describes using an ETL script to query individual SQLite databases, collect metrics such as orders, revenue, shopping carts, and products, and store the results in a central database for reporting and platform monitoring. ### Source excerpt Aggregating Multi-Tenant Databases for Analytics and Reporting ## Enhancing Data Quality Using Better Designed ETLs DevFeed: [Enhancing Data Quality Using Better Designed ETLs]() Original publisher: [Read original article]() Author: Kasia Rachuta Published: 2024-09-30T16:00:00Z Content type: tutorial Language: en Sources: [Square Corner Blog](), [Square Corner Blog RSS Feed]() Topics: [Template](), [Data Quality](), [Data Science](), [Data Infrastructure]() Tags: [data-quality](), [data-science](), [documentation](), [etl](), [guide](), [metrics]() ### AI overview This article explains how an ETL design document template can help data science teams create more consistent, usable, and maintainable ETLs. It describes documenting technical decisions, goals, analytical questions, consumers, and monitored metrics to support data quality and shared understanding. ### Source excerpt Sharing an ETL design doc template ## Import Postgres tables into Tinybird with PostgreSQL Table Function DevFeed: [Import Postgres tables into Tinybird with PostgreSQL Table Function]() Original publisher: [Read original article]() Author: Cameron Archer Published: 2024-07-29T00:00:00Z Content type: release Language: en Sources: [Tinybird]() Topics: [PostgreSQL](), [data](), [Query (disambiguation)](), [systems]() Tags: [etl](), [import](), [pipelines](), [postgres](), [postgresql](), [product-updates]() ### AI overview Tinybird's PostgreSQL table function connects Postgres data directly to Tinybird and enables querying across systems without complex ETL pipelines. ### Source excerpt The PostgreSQL table function connects your Postgres data to Tinybird directly. Query across systems without complex ETL pipelines. ## A Taxonomy Of Data Change Events DevFeed: [A Taxonomy Of Data Change Events]() Original publisher: [Read original article]() Published: 2024-03-13T00:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [data](), [Databases](), [real-time](), [Microservice]() Tags: [comparison](), [data](), [database](), [debezium](), [etl](), [event](), [logging](), [microservices](), [real-time]() ### AI overview This article develops a taxonomy of data change events used in Change Data Capture systems such as Debezium. It explains full events, delta events, and ID-only events, along with their contents and use cases including real-time ETL, microservices data exchange, and audit logging. ### Source excerpt Table of Contents Full Events Delta Events Id-only Events Change Event Metadata Comparison This post originally appeared on the Decodable blog. All rights reserved. Data change events are at the core of Change Data Capture (CDC) solutions such as Debezium. They describe the changes made to a specific record in a database and allow event consumers to take action based on this information, enabling a wide range of use cases, such as real-time ETL (by propagating the updated data into downstream data stores such as data warehouses, analytics databases, or fulltext search indexes), microservices data exchange, or audit logging. ## Run analytics on Amazon S3 data with Tinybird S3 connector DevFeed: [Run analytics on Amazon S3 data with Tinybird S3 connector]() Original publisher: [Read original article]() Author: Tinybird Published: 2023-07-20T00:00:00Z Content type: article Language: en Sources: [Tinybird]() Topics: [Amazon S3](), [data](), [Query (disambiguation)]() Tags: [amazon-s3](), [analytics](), [data](), [data-lake](), [etl](), [pipelines](), [product-updates](), [s3]() ### AI overview The article presents the Tinybird S3 Connector as a way to run analytics on Amazon S3 data lake files without complex ETL pipelines. ### Source excerpt S3 analytics the easy way with the Tinybird Connector. Query your data lake files without complex ETL pipelines. ## The unnecessary hype strategy behind Microsoft Fabric DevFeed: [The unnecessary hype strategy behind Microsoft Fabric]() Original publisher: [Read original article]() Published: 2023-05-29T00:00:00Z Content type: opinion Language: en Sources: [Mutt0-ds Notes]() Topics: [Microsoft](), [data-engineering](), [Azure](), [Cloud](), [data lake](), [data-processing](), [etl](), [AI, ML & Data Engineering]() Tags: [azure](), [cloud](), [copilot](), [data](), [data-engineering](), [data-lake](), [data-processing](), [etl](), [microsoft](), [microsoft-azure](), [technical]() ### AI overview An opinion article examines Microsoft Fabric, a unified data platform announced at Microsoft Build. It describes Fabric's integration of data storage, processing, ETL, analytics, and business intelligence tools, while criticizing Microsoft's secrecy and hype-oriented launch strategy and noting that the platform was still in beta. ### Source excerpt If you are into Data Engineering in Microsoft Azure Cloud Environment, you probaly heard about Microsoft Fabric being announced last week... ## My first weeks at Acer Europe DevFeed: [My first weeks at Acer Europe]() Original publisher: [Read original article]() Published: 2022-10-16T00:00:00Z Content type: opinion Language: en Sources: [Mutt0-ds Notes]() Topics: [Azure](), [dashboards](), [Databases](), [etl](), [DevOps](), [Purview](), [active directory](), [data-governance]() Tags: [active-directory](), [azure](), [dashboards](), [data-governance](), [databases](), [developer](), [devops](), [etl](), [purview]() ### AI overview A Business Intelligence Developer reflects on their first weeks at Acer Europe, describing the transition from a small company to a large IT organization. The post covers the scale of its data, reporting, ETL, Azure services, and cross-team work, along with the information overload and learning challenges of the first week. ### Source excerpt This is a post for my future self when I will be asking myself: "How were my first days at Acer like?". Note: I'm working at Acer EMEA HQ,... ## Cinco de Trino recap: Learn how to build an efficient data lake DevFeed: [Cinco de Trino recap: Learn how to build an efficient data lake]() Original publisher: [Read original article]() Author: Brian Olsen, Brian Zhan Published: 2022-05-17T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [data](), [data-engineering](), [big-data](), [Query (disambiguation)](), [datasets](), [Apache Iceberg](), [Stack Overflow](), [X (Twitter)]() Tags: [analytics](), [architecture](), [big-data](), [build](), [clusters](), [community](), [conference](), [cost-savings](), [data](), [data-engineering](), [data-lake](), [etl](), [learn](), [pipelines](), [recap](), [recovery](), [scale](), [tools]() ### AI overview A recap of the Cinco de Trino conference covering Trino's role in data lakehouse architectures, interactive and federated querying, adaptive query planning, and fault-tolerant execution. It highlights Project Tardigrade for autoscaling, spot-instance use, cost savings, and failure recovery, plus a Starburst Galaxy lab for ingesting, cleaning, and analyzing Twitter and Stack Overflow data. ### Source excerpt When Trino (formerly PrestoSQL) arrived on the scene almost 10 years ago, it immediately became known as the much faster alternative to the data warehouse of big data, Apache Hive. The use cases that you, as the community, have built had far exceeded anything we had imagined in complexity. Together we've made Trino not only the fastest way to interactively query large data sets, but also a convenient way to run federated queries across data sources to make moving all the data optional. At Cinco de Trino, we came full circle back to the next iteration of analytics architecture with the data lake. This conference offers advice from industry thought leaders about how to use best lakehouse tools with Trino to manage that data complexity. Hear from industry thought leaders like Martin Traverso (Trino), Dain Sundstrom (Trino), James Campbell (Great Expectations), Jeremy Cohen (DBT Labs), Ryan Blue (Iceberg), Denny Lee (Delta Lake), Vinoth Chandar (Hudi). You can watch the talks on-demand on the Cinco de Trino playlist. In this post, I'd like to cover the key items from each talk you won't want to miss. ## Project Tardigrade delivers ETL at Trino speeds to early users DevFeed: [Project Tardigrade delivers ETL at Trino speeds to early users]() Original publisher: [Read original article]() Author: Andrii Rosa, Brian Olsen, Brian Zhan, Lukasz Osipiuk, Martin Traverso, Zebing Lin Published: 2022-05-05T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [User Experience](), [Query (disambiguation)](), [Concurrency]() Tags: [architecture](), [batch](), [clusters](), [concurrency](), [distributed](), [engineering](), [etl](), [launch](), [memory](), [release]() ### AI overview Project Tardigrade adds fault-tolerant execution to Trino, improving the experience of running resource-intensive ETL and batch queries. Its architecture supports granular retries, resource-aware scheduling, and fair progress for concurrent workloads without requiring queries to restart from scratch after failures. ### Source excerpt After six months of challenging work on Project Tardigrade, we are ready to launch. With the project we improved the user experience of running resource intensive queries that are common in the Extract, Transform, Load (ETL) and batch processing space. It required some significant and fascinating engineering to get us to the current status. The latest Trino release includes all the work from Project Tardigrade. Read on to learn how it all works, and how to enable the fault-tolerant execution in Trino. ## Lesser Known PostgreSQL Features DevFeed: [Lesser Known PostgreSQL Features]() Original publisher: [Read original article]() Author: Haki Benita Published: 2021-11-07T22:00:00Z Content type: tutorial Language: en Sources: [Haki Benita]() Topics: [PostgreSQL](), [etl]() Tags: [article](), [articles](), [etl](), [features](), [postgresql]() ### AI overview This article presents lesser-known PostgreSQL features, including a command for synchronizing table data and a technique for distinguishing inserted rows from updated rows using a system column. It also discusses logging ETL processes. ### Source excerpt A list of useful features you already have, but may not know about! In this article I share lesser known features of PostgreSQL. ## Extract, Transform, and Load in Go DevFeed: [Extract, Transform, and Load in Go]() Original publisher: [Read original article]() Published: 2021-09-13T00:00:00Z Content type: tutorial Language: en Sources: [William Kennedy]() Topics: [Go Language](), [CSV](), [SQL](), [Databases](), [Grafana]() Tags: [ardan-labs](), [blog](), [data](), [database](), [dataset](), [download](), [etl](), [go](), [go-programming](), [golang](), [grafana](), [programming](), [sql]() ### AI overview This tutorial explains extract, transform, and load (ETL) in Go using a food-violations dataset. It covers loading CSV data into an SQL database, querying it, and preparing fields through parsing, renaming, type conversion, and enrichment. ### Source excerpt Introduction You are about to visit Boston, and would like to taste some good food. You ask your friend who lives there what are good places to eat. They reply with "Everything is good, you can't go wrong". Which makes you think, maybe I should check where not to eat. The data geek in you arises, and you find out that the city of Boson has a dataset of food violations. You download it and decide to have a look. ## How we build the Image Gallery on trivago DevFeed: [How we build the Image Gallery on trivago]() Original publisher: [Read original article]() Author: Praneeth Peiris I want Published: 2021-07-07T00:00:00Z Content type: article Language: en Sources: [Trivago]() Topics: [Google Cloud Platform (GCP)](), [etl](), [data lake](), [Kafka](), [Front end](), [Amazon Web Services]() Tags: [amazon-web-services-aws](), [architecture](), [backend](), [cloud](), [data-lake](), [engineering](), [etl](), [frontend](), [gcp](), [kafka](), [platforms]() ### AI overview trivago describes migrating its hotel image-gallery ETL pipeline from Amazon Web Services to Google Cloud Platform. The redesigned architecture uses Dataflow jobs to snapshot images and tags, fetch changed accommodations, build and validate sorted galleries, and stream approved changes to frontend teams through Kafka. ### Source excerpt When was the last time you booked accommodation without checking its photos? Most probably never! Because having imagery information makes our decision-making process much easier and faster. How... ## Continuous Deployment for AWS Glue DevFeed: [Continuous Deployment for AWS Glue]() Original publisher: [Read original article]() Author: Mohammed Published: 2020-10-15T15:51:25Z Content type: tutorial Language: en Sources: [Brave New Geek]() Topics: [AWS Glue](), [Continuous Deployment (CD)](), [GitHub Actions](), [Jupyter Notebook](), [Amazon S3]() Tags: [analytics](), [analytics-pipeline](), [aws](), [aws-glue](), [ci-cd](), [continuous-delivery](), [continuous-deployment](), [etl](), [github](), [github-actions](), [jupyter](), [jupyter-notebook](), [s3](), [serverless]() ### AI overview A tutorial for automating continuous deployment of AWS Glue ETL jobs. It uses GitHub Actions to generate a Python script from a Jupyter notebook, copy it to Amazon S3, and update the Glue job to use the new script. ### Source excerpt AWS Glue is a managed service for building ETL (Extract-Transform-Load) jobs. It's a useful tool for implementing analytics pipelines in AWS without having to manage server infrastructure. Jobs are implemented using Apache Spark and, with the help of Development Endpoints, can be built using Jupyter notebooks. This makes it reasonably easy to write ETL processes in an interactive, iterative fashion. Once finished, the Jupyter notebook is converted into a Python script, uploaded to S3, and then run as a Glue job. ## SQL Tips for Application DBAs DevFeed: [SQL Tips for Application DBAs]() Original publisher: [Read original article]() Author: Haki Benita Published: 2020-07-26T21:00:00Z Content type: tutorial Language: en Sources: [Haki Benita]() Topics: [SQL](), [Development](), [Databases](), [schema design](), [data migrations](), [etl]() Tags: [articles](), [data-migrations](), [databases](), [development](), [etl](), [performance](), [schema-design](), [sql]() ### AI overview An application DBA shares practical SQL and database-development advice, including limiting updates to rows that need changes and considering constraint costs when loading or updating many rows. ### Source excerpt Some tips and misconceptions about database development I gathered along the way. ## Implementing ETL on GCP DevFeed: [Implementing ETL on GCP]() Original publisher: [Read original article]() Author: Deepmala Published: 2020-07-15T20:53:17Z Content type: tutorial Language: en Sources: [Brave New Geek]() Topics: [DataOps](), [Cloud](), [BigQuery](), [data loss prevention](), [Low code](), [No-code](), [olap]() Tags: [analytics](), [analytics-pipeline](), [bi-tools](), [bigquery](), [cdap](), [cloud-data-loss-prevention](), [cloud-dataflow](), [cloud-dataprep](), [cloud-dataproc](), [cloud-pub-sub](), [cloud-storage](), [cloud-tasks](), [data-analytics](), [data-fusion](), [data-lake](), [data-loss-prevention](), [elt](), [etl](), [gcp](), [no-code](), [serverless](), [sql]() ### AI overview A practical guide to building ETL pipelines on Google Cloud Platform using Google-managed services. It explains a two-phase architecture with Cloud Storage as a data lake, Cloud Data Loss Prevention for sensitive-data detection or redaction, and BigQuery as the curated data warehouse, with attention to low-code and no-code approaches. ### Source excerpt ETL (Extract-Transform-Load) processes are an essential component of any data analytics program. This typically involves loading data from disparate sources, transforming or enriching it, and storing the curated data in a data warehouse for consumption by different users or systems. An example of this would be taking customer data from operational databases, joining it with data from Salesforce and Google Analytics, and writing it to an OLAP database or BI engine. ## A Data Engineering Perspective on Go vs. Python (Part 2 - Dataflow) DevFeed: [A Data Engineering Perspective on Go vs. Python (Part 2 - Dataflow)]() Original publisher: [Read original article]() Author: Christian Hollinger Published: 2020-07-06T00:00:00Z Content type: article Language: en Sources: [Christian Hollinger]() Topics: [data-engineering](), [Go Language](), [Python](), [SDKs](), [AI, ML & Data Engineering](), [datasets](), [Google Cloud Platform (GCP)](), [etl](), [apache-flink](), [Apache Spark]() Tags: [apache-flink](), [apache-spark](), [beam](), [benchmarking](), [big-data](), [cloud-dataflow](), [data-engineering](), [dataflow](), [distributed](), [etl](), [gcp](), [go](), [golang](), [google-cloud](), [open-source](), [performance](), [programming](), [python](), [spark]() ### AI overview A comparison of Go and Python for data engineering using Apache Beam and Google Dataflow. It examines their SDK differences, drawbacks, benchmark performance, and the feasibility of switching between them. ### Source excerpt In Part 2 of our comparison of Python and go from a Data Engineering perspective, we'll finally take a look at Apache Beam and Google Dataflow and how the go SDK and the Python SDK differ, what drawbacks we're dealing with, how fast it is by running extensive benchmarks, and how feasible it is to make the switch [Next page]()