# data-quality-management Published articles for data-quality-management. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Data Quality at Udemy -- Part 1 DevFeed: [Data Quality at Udemy -- Part 1]() Original publisher: [Read original article]() Author: Murat Migdisoglu Published: 2023-09-06T22:01:16Z Content type: article Language: en Sources: [Udemy Engineering]() Topics: [Data Quality](), [data-architecture](), [data lake](), [Apache Spark](), [airflow](), [Streaming]() Tags: [airflow](), [architecture](), [data-architecture](), [data-catalog](), [data-governance](), [data-lake](), [data-lineage](), [data-quality](), [data-quality-management](), [pipelines](), [principal-engineer](), [quality](), [spark](), [streaming]() ### AI overview This article describes Udemy's efforts to improve data quality by establishing an end-to-end data lineage solution. It explains how distributed data ownership and self-service analytics make lineage important for impact analysis, change management, and identifying unused columns or orphan tables. ### Source excerpt Data Quality at Udemy -- Part 1Data Lineage Demystified- Why it Matters and How to Leverage its Magic for Informed Business Success! In late 2020, upon joining Udemy as a principal engineer for the data platform team, my focus shifted toward enhancing data quality within the organization. My journey began with conducting a comprehensive poll across the data organization, aimed at identifying the key pain points of data users. The results of the poll were eye-opening, revealing that 78% of users considered the absence of data provenance/lineage as a data quality issue. Furthermore, it was obvious that for a vast majority of the users, the inability to track data lineage was an important problem in impact analysis and detecting unused columns or orphan tables in the system. Inspired by these insights, I took the initiative to propose and launch two transformative projects. The first one, which is the subject of this article, is an ambitious initiative to establish a comprehensive end-to-end data lineage solution that will revolutionize our data ecosystem. The second project centers around data monitoring, which will be explored in another post. Udemy's sophisticated data architecture revolves around a data lake fed by diverse pipelines: system logs, streaming data from services, CDC listeners for replicated service databases, and more. The backbone of data transformations lies in Hive and Spark, while Airflow takes charge of orchestrating thousands of these pipelines. Unraveling Data Flow Complexity: Data Lineage in Growing Data Driven Organizations In the early stages of an organization's data-driven journey, data lineage may not be deemed crucial. With just a few pipelines managed by a centralized team, the dependency tree of the workflow orchestration typically suffices to comprehend the relationships between data entities. However, as the business scales up, relying on a single centralized team for all data flows becomes a bottleneck. Consequently, data organizatio ## Selective data deletion: a new feature for data quality management DevFeed: [Selective data deletion: a new feature for data quality management]() Original publisher: [Read original article]() Author: Jorge Sancha Published: 2020-08-27T00:00:00Z Content type: article Language: en Sources: [Tinybird]() Topics: [clickhouse](), [Data Quality]() Tags: [clickhouse](), [data-quality](), [data-quality-management](), [new-feature](), [product-updates]() ### AI overview The article discusses selective data deletion in ClickHouse, noting that incorrect deletion methods can reduce performance and presenting an approach intended to avoid that problem. ### Source excerpt Selective data deletion in ClickHouse is tricky. Delete the wrong way and performance tanks. Here's the approach that actually works.