# Pinion -- The Load Framework Part-2

DevFeed: [Pinion -- The Load Framework Part-2](<https://devfeed.tech/articles/pinion-the-load-framework-part-2-26224.md>)

Original publisher: [Read original article](<https://medium.com/groupon-eng/pinion-the-load-framework-part-2-e6a47586e7be?source=rss----5c13a88f9872---4>)

Author: Saurabh Jain

Published: 2021-10-29T16:50:24Z

Content type: article

Language: en

Sources: [Groupon Engineering -- Medium](<https://devfeed.tech/sources/groupon-engineering-medium.md>)

Topics: [data lake](<https://devfeed.tech/topics/data-lake.md>), [Framework](<https://devfeed.tech/topics/framework.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [acid](<https://devfeed.tech/tags/acid.md>), [audit](<https://devfeed.tech/tags/audit.md>), [aws](<https://devfeed.tech/tags/aws.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [data-validation](<https://devfeed.tech/tags/data-validation.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>), [deltalake](<https://devfeed.tech/tags/deltalake.md>), [hdfs](<https://devfeed.tech/tags/hdfs.md>), [logging](<https://devfeed.tech/tags/logging.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [reporting](<https://devfeed.tech/tags/reporting.md>), [s3](<https://devfeed.tech/tags/s3.md>), [schema](<https://devfeed.tech/tags/schema.md>), [science](<https://devfeed.tech/tags/science.md>), [spark](<https://devfeed.tech/tags/spark.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [transactions](<https://devfeed.tech/tags/transactions.md>)

## AI overview

This second post in the Pinion -- The Load Framework series explains how Pinion extends Apache Delta Lake APIs for slowly changing dimension operations. It covers data validation, compaction, auditing, streamlined logging, and chained APIs, and introduces Delta Lake capabilities such as ACID transactions, schema enforcement, batch and streaming interfaces, and time travel.

## Source excerpt

Pinion -- The Load Framework Part-2 This post is the 2nd part of the "Pinion -- The Load Framework" series. In case you have not read the 1st post, you can read it here. In this post, we are going to cover the following topics. How does Pinion use Delta Lake for SCD operations? Small file problem with Delta Lake and its resolution. Before we dive into the topics of this post, let's look at the definition of DeltaLake to set the context right. Apache Delta Lake - Apache Delta Lake is an open-source framework that enables the addition of ACID transactions support to a new data lake or an existing data lake created on top of S3, GCS, and HDFS. In addition to this, it provides other features such as scalable metadata handling, unified interface for both batch and streaming application, schema enforcement, time travel, and a rich interface of APIs to enable complex use cases like change-data-capture (CDC) and slowly-changing-dimension (SCD) operations. To keep the post concise and to the point, I won't go into much detail here about Delta Lake, since there is already great documentation available about it, that you can read it here. How does Pinion use Delta Lake for SCD operations? - Apache Delta Lake provides a rich set of APIs to handle slowly-changing dimensions, however, those APIs were not enough alone to build the features that we want to have in The Pinion Framework. So, we decided to enrich the APIs provided by Delta Lake by adding the following features to it: Data Validation Compaction Audit Streamlined logging infrastructure to make the data engineer's life easier during debugging of a failed job Chained APIs Let's dive a little further into the features that we had listed above. Data Validation -- By default, schema enforcement is enabled in Pinion for all the APIs where we have a need of inserting rows from source data(LRFs) into the target table. In case of a schema mismatch, Pinion raises an error and stops processing of further stages. It ensures the data a