# data-lineage Published articles for data-lineage. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## How we automatically handle ClickHouse® schema migrations DevFeed: [How we automatically handle ClickHouse® schema migrations]() Original publisher: [Read original article]() Author: Raquel Barbadillo Published: 2025-06-27T00:00:00Z Content type: article Language: en Sources: [Tinybird]() Topics: [clickhouse](), [Structured-data](), [data]() Tags: [clickhouse](), [data](), [data-lineage](), [data-migrations](), [migrations](), [scalable-analytics-architecture](), [schema]() ### AI overview The article explains how data lineage is used to optimize ClickHouse table deployments and avoid unnecessary, expensive data migrations. ### Source excerpt How we use data lineage to optimize ClickHouse® table deployments and avoid unnecessary and expensive data migrations. ## Data Quality at Udemy -- Part 1 DevFeed: [Data Quality at Udemy -- Part 1]() Original publisher: [Read original article]() Author: Murat Migdisoglu Published: 2023-09-06T22:01:16Z Content type: article Language: en Sources: [Udemy Engineering]() Topics: [Data Quality](), [data-architecture](), [data lake](), [Apache Spark](), [airflow](), [Streaming]() Tags: [airflow](), [architecture](), [data-architecture](), [data-catalog](), [data-governance](), [data-lake](), [data-lineage](), [data-quality](), [data-quality-management](), [pipelines](), [principal-engineer](), [quality](), [spark](), [streaming]() ### AI overview This article describes Udemy's efforts to improve data quality by establishing an end-to-end data lineage solution. It explains how distributed data ownership and self-service analytics make lineage important for impact analysis, change management, and identifying unused columns or orphan tables. ### Source excerpt Data Quality at Udemy -- Part 1Data Lineage Demystified- Why it Matters and How to Leverage its Magic for Informed Business Success! In late 2020, upon joining Udemy as a principal engineer for the data platform team, my focus shifted toward enhancing data quality within the organization. My journey began with conducting a comprehensive poll across the data organization, aimed at identifying the key pain points of data users. The results of the poll were eye-opening, revealing that 78% of users considered the absence of data provenance/lineage as a data quality issue. Furthermore, it was obvious that for a vast majority of the users, the inability to track data lineage was an important problem in impact analysis and detecting unused columns or orphan tables in the system. Inspired by these insights, I took the initiative to propose and launch two transformative projects. The first one, which is the subject of this article, is an ambitious initiative to establish a comprehensive end-to-end data lineage solution that will revolutionize our data ecosystem. The second project centers around data monitoring, which will be explored in another post. Udemy's sophisticated data architecture revolves around a data lake fed by diverse pipelines: system logs, streaming data from services, CDC listeners for replicated service databases, and more. The backbone of data transformations lies in Hive and Spark, while Airflow takes charge of orchestrating thousands of these pipelines. Unraveling Data Flow Complexity: Data Lineage in Growing Data Driven Organizations In the early stages of an organization's data-driven journey, data lineage may not be deemed crucial. With just a few pipelines managed by a centralized team, the dependency tree of the workflow orchestration typically suffices to comprehend the relationships between data entities. However, as the business scales up, relying on a single centralized team for all data flows becomes a bottleneck. Consequently, data organizatio ## DependenTree, Graph Visualization Library DevFeed: [DependenTree, Graph Visualization Library]() Original publisher: [Read original article]() Author: Amogh Kambale Published: 2022-03-23T19:00:00Z Content type: article Language: en Sources: [Square Corner Blog RSS Feed]() Topics: [Graphs](), [Library](), [Reverse Dependencies](), [API](), [data](), [Front end](), [JSON](), [Code]() Tags: [api](), [d3-js](), [data-lineage](), [dependency](), [engineering](), [front-end](), [graph](), [graphs](), [json](), [library](), [mapping](), [scale](), [visualization]() ### AI overview DependenTree is a D3.js-based visualization library for representing large dependency graphs as collapsible trees. It supports upstream and downstream dependencies, cyclic graphs, and client-side dependency mapping to avoid oversized server-to-client payloads. ### Source excerpt Visualize dependency graphs at scale