# Data Quality at Udemy -- Part 1

DevFeed: [Data Quality at Udemy -- Part 1](<https://devfeed.tech/articles/data-quality-at-udemy-part-1-26351.md>)

Original publisher: [Read original article](<https://medium.com/udemy-engineering/data-quality-at-udemy-part-1-63e3b099ff81?source=rss----19c6d3367ed4---4>)

Author: Murat Migdisoglu

Published: 2023-09-06T22:01:16Z

Content type: article

Language: en

Sources: [Udemy Engineering](<https://devfeed.tech/sources/udemy-engineering.md>)

Topics: [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [airflow](<https://devfeed.tech/topics/airflow.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [data-architecture](<https://devfeed.tech/tags/data-architecture.md>), [data-catalog](<https://devfeed.tech/tags/data-catalog.md>), [data-governance](<https://devfeed.tech/tags/data-governance.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-lineage](<https://devfeed.tech/tags/data-lineage.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [data-quality-management](<https://devfeed.tech/tags/data-quality-management.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [principal-engineer](<https://devfeed.tech/tags/principal-engineer.md>), [quality](<https://devfeed.tech/tags/quality.md>), [spark](<https://devfeed.tech/tags/spark.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

## AI overview

This article describes Udemy's efforts to improve data quality by establishing an end-to-end data lineage solution. It explains how distributed data ownership and self-service analytics make lineage important for impact analysis, change management, and identifying unused columns or orphan tables.

## Source excerpt

Data Quality at Udemy -- Part 1Data Lineage Demystified- Why it Matters and How to Leverage its Magic for Informed Business Success! In late 2020, upon joining Udemy as a principal engineer for the data platform team, my focus shifted toward enhancing data quality within the organization. My journey began with conducting a comprehensive poll across the data organization, aimed at identifying the key pain points of data users. The results of the poll were eye-opening, revealing that 78% of users considered the absence of data provenance/lineage as a data quality issue. Furthermore, it was obvious that for a vast majority of the users, the inability to track data lineage was an important problem in impact analysis and detecting unused columns or orphan tables in the system. Inspired by these insights, I took the initiative to propose and launch two transformative projects. The first one, which is the subject of this article, is an ambitious initiative to establish a comprehensive end-to-end data lineage solution that will revolutionize our data ecosystem. The second project centers around data monitoring, which will be explored in another post. Udemy's sophisticated data architecture revolves around a data lake fed by diverse pipelines: system logs, streaming data from services, CDC listeners for replicated service databases, and more. The backbone of data transformations lies in Hive and Spark, while Airflow takes charge of orchestrating thousands of these pipelines. Unraveling Data Flow Complexity: Data Lineage in Growing Data Driven Organizations In the early stages of an organization's data-driven journey, data lineage may not be deemed crucial. With just a few pipelines managed by a centralized team, the dependency tree of the workflow orchestration typically suffices to comprehend the relationships between data entities. However, as the business scales up, relying on a single centralized team for all data flows becomes a bottleneck. Consequently, data organizatio