# deltalake Published articles for deltalake. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Building a Scalable AWS Backup System for S3 Data Warehouses DevFeed: [Building a Scalable AWS Backup System for S3 Data Warehouses]() Original publisher: [Read original article]() Author: Oleh Motrunych Published: 2025-09-22T00:00:00Z Content type: article Language: en Sources: [Scribd Tech]() Topics: [Amazon S3](), [Amazon Web Services](), [AWS Lambda](), [parquet](), [cloud-infrastructure]() Tags: [aws](), [aws-lambda](), [backup](), [backups](), [cleanup](), [compute](), [cost](), [data](), [data-lake](), [databases](), [datasets](), [deltalake](), [efficiency](), [featured](), [incremental](), [lambda](), [lifecycle](), [parquet](), [pipeline](), [reduce](), [retention](), [s3](), [terraform]() ### AI overview This article describes a monthly backup system for S3 data warehouses that uses incremental copying, S3 Inventory validation, parallel processing, and Glacier retention. It routes smaller workloads to AWS Lambda and larger workloads to ECS Fargate, with lifecycle-based cleanup for eligible files. ### Source excerpt We designed and implemented a scalable, cost-optimized backup system for S3 data warehouses that runs automatically on a monthly schedule. The system handles petabytes of data across multiple databases and uses a hybrid approach: AWS Lambda for small workloads and ECS Fargate for larger ones. At its core, the pipeline performs incremental backups -- copying only new or changed parquet files while always preserving delta logs -- dramatically reducing costs and runtime compared to full backups. Data is validated through S3 Inventory manifests, processed in parallel, and stored in Glacier for long-term retention. To avoid data loss and reduce storage costs, we also implemented a safe deletion workflow. Files older than 90 days, successfully backed up, and no longer present in the source are tagged for lifecycle-based cleanup instead of being deleted immediately. This approach ensures reliability, efficiency, and safety: backups scale seamlessly from small to massive datasets, compute resources are right-sized, and storage is continuously optimized. ## Cloud-native data ingestion architecture using AWS, Databricks, and open-source tools DevFeed: [Cloud-native data ingestion architecture using AWS, Databricks, and open-source tools]() Original publisher: [Read original article]() Author: R Tyler Croy Published: 2025-08-01T00:00:00Z Content type: tutorial Language: en Sources: [Scribd Tech]() Topics: [data](), [Amazon Web Services](), [databricks](), [event driven](), [Amazon Simple Queue Service (SQS)](), [Kafka](), [Open Source](), [Azure](), [Google Cloud Platform (GCP)]() Tags: [architecture](), [aws](), [azure](), [catalog](), [cloud](), [cloud-native](), [data](), [databricks](), [deltalake](), [event-driven](), [featured](), [google-cloud](), [kafka](), [open-source](), [rust](), [sqs]() ### AI overview This article presents Scribd's cloud-native data-ingestion architecture for building large datasets for Delta Lake. It describes using AWS services and open-source tools such as kafka-delta-ingest, oxbow, and Airbyte in a more event-driven and reliable platform, with Databricks and Unity Catalog. The approach can also be adapted to Azure, Google Cloud Platform, or on-premises environments. ### Source excerpt Delta Lake is a fantastic technology for quickly querying massive data sets, but first you need those massive data sets! In this talk from Data and AI Summit 2025 I dive into the cloud-native architecture Scribd has adopted to ingest data from AWS Aurora, SQS, Kinesis Data Firehose and more! ## Terraform module to manage Oxbow Lambda and its components DevFeed: [Terraform module to manage Oxbow Lambda and its components]() Original publisher: [Read original article]() Author: Oleh Motrunych Published: 2025-03-14T00:00:00Z Content type: release Language: en Sources: [Scribd Tech]() Topics: [Terraform](), [AWS Lambda](), [infrastructure as code (IAC)](), [Amazon Web Services](), [AWS Glue](), [Amazon Simple Queue Service (SQS)](), [data-processing](), [event driven](), [DynamoDB](), [Amazon S3]() Tags: [aws](), [aws-glue](), [aws-lambda](), [aws-s3](), [deltalake](), [dynamodb](), [event-driven](), [iac](), [oxbow](), [rust](), [security](), [sqs](), [terraform]() ### AI overview The article introduces terraform-oxbow, an open-source Terraform module for deploying and managing an Oxbow AWS Lambda workflow and its supporting components. It describes configurable integrations including AWS Glue, Kinesis Data Firehose, SQS, DynamoDB, IAM policies, and S3 notifications, while noting AWS notification limits and least-privilege considerations. ### Source excerpt Oxbow is a project to take an existing storage location which contains Apache Parquet files into a Delta Lake table. It is intended to run both as an AWS Lambda or as a command line application. We are excited to introduce terraform-oxbow, an open-source Terraform module that simplifies the deployment and management of AWS Lambda and its supporting components. Whether you're working with AWS Glue, Kinesis Data Firehose, SQS, or DynamoDB, this module provides a streamlined approach to infrastructure as code (IaC) in AWS. ## Pinion -- The Load Framework Part-2 DevFeed: [Pinion -- The Load Framework Part-2]() Original publisher: [Read original article]() Author: Saurabh Jain Published: 2021-10-29T16:50:24Z Content type: article Language: en Sources: [Groupon Engineering -- Medium]() Topics: [data lake](), [Framework](), [Data Quality](), [Transactions](), [Streaming](), [Open Source]() Tags: [acid](), [audit](), [aws](), [cloud](), [data-validation](), [delta-lake](), [deltalake](), [hdfs](), [logging](), [optimization](), [reporting](), [s3](), [schema](), [science](), [spark](), [streaming](), [transactions]() ### AI overview This second post in the Pinion -- The Load Framework series explains how Pinion extends Apache Delta Lake APIs for slowly changing dimension operations. It covers data validation, compaction, auditing, streamlined logging, and chained APIs, and introduces Delta Lake capabilities such as ACID transactions, schema enforcement, batch and streaming interfaces, and time travel. ### Source excerpt Pinion -- The Load Framework Part-2 This post is the 2nd part of the "Pinion -- The Load Framework" series. In case you have not read the 1st post, you can read it here. In this post, we are going to cover the following topics. How does Pinion use Delta Lake for SCD operations? Small file problem with Delta Lake and its resolution. Before we dive into the topics of this post, let's look at the definition of DeltaLake to set the context right. Apache Delta Lake - Apache Delta Lake is an open-source framework that enables the addition of ACID transactions support to a new data lake or an existing data lake created on top of S3, GCS, and HDFS. In addition to this, it provides other features such as scalable metadata handling, unified interface for both batch and streaming application, schema enforcement, time travel, and a rich interface of APIs to enable complex use cases like change-data-capture (CDC) and slowly-changing-dimension (SCD) operations. To keep the post concise and to the point, I won't go into much detail here about Delta Lake, since there is already great documentation available about it, that you can read it here. How does Pinion use Delta Lake for SCD operations? - Apache Delta Lake provides a rich set of APIs to handle slowly-changing dimensions, however, those APIs were not enough alone to build the features that we want to have in The Pinion Framework. So, we decided to enrich the APIs provided by Delta Lake by adding the following features to it: Data Validation Compaction Audit Streamlined logging infrastructure to make the data engineer's life easier during debugging of a failed job Chained APIs Let's dive a little further into the features that we had listed above. Data Validation -- By default, schema enforcement is enabled in Pinion for all the APIs where we have a need of inserting rows from source data(LRFs) into the target table. In case of a schema mismatch, Pinion raises an error and stops processing of further stages. It ensures the data a