# data-platform-engineering Published articles for data-platform-engineering. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Data Quality at Petabyte Scale: Building Trust in the Data Lifecycle DevFeed: [Data Quality at Petabyte Scale: Building Trust in the Data Lifecycle]() Original publisher: [Read original article]() Author: Zakariah Siyaji Published: 2025-02-14T15:52:43Z Content type: article Language: en Sources: [Glassdoor Engineering]() Topics: [Data Quality](), [data-engineering](), [data-processing](), [DataOps](), [Usability]() Tags: [data-engineering](), [data-patterns](), [data-platform-engineering](), [data-quality](), [decision-making](), [gable](), [pipeline](), [shift-left](), [software-engineering](), [trust](), [usability]() ### AI overview Glassdoor describes a shift from reactive data engineering to a proactive, trust-centered approach to data quality. The article connects organizational culture with technical checks across the data lifecycle. ### Source excerpt The data Lifecycle with Data Quality Checks at GlassdoorMotivation Glassdoor has transformed from an employee review site to a community for workplace conversations [1]. As our platform evolves to support content creators, facilitate discussions, and offer rich content, it has become more apparent than ever that adopting a data-driven culture is essential. Businesses rely on accurate, high-quality data to understand their operations and assess strategic outcomes. Flawed or incomplete data results in misguided decisions and undermines trust. Recognizing this risk, we made data quality a foundational principle of our data-driven transformation. Although every company defines data quality differently, there is a universal expectation that data used for decision-making must be trustworthy. Additionally, data quality challenges are not solely technical; a psychological component is closely linked to trust in data. Airbnb recognized this and sought to develop a scoring system that acknowledges the belief that data quality is a multivariate issue, encompassing accuracy, reliability, stewardship, and usability, along with more detailed dimensions within each of these categories [2]. On the other hand, Netflix employs a more technically centered approach to quality: data is initially written to a temporary staging area, audited, and then published to the production location upon passing quality checks [3]. Ultimately, Glassdoor drew inspiration from these lessons and aimed to reinforce trust through a cultural shift and a series of technical solutions. This article demonstrates how a proactive, trust-centered approach that connects data producers and consumers establishes a foundation for more rigorous data quality methods, ultimately bolstering a strong company-wide strategy. Figure 1. Enhancing quality guards at the application code layer.Culture Shift: Reactive to Proactive Historically, Glassdoor's data engineering teams have been reactive, learning about issues only aft ## How to Reliably Scale Your Data Platform for High Volumes DevFeed: [How to Reliably Scale Your Data Platform for High Volumes]() Original publisher: [Read original article]() Author: Arbab Ahmed Published: 2020-12-08T17:30:29Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [data](), [Platform Engineering](), [Kafka](), [MySQL](), [parquet](), [Apache Spark](), [systems]() Tags: [analytics](), [apache](), [apache-parquet](), [apache-spark](), [data](), [data-platform-engineering](), [infrastructure](), [insights](), [kafka](), [parquet](), [platform](), [scale](), [spark](), [streaming](), [systems]() ### AI overview Shopify's Data Platform Engineering team describes how it prepared the data platform to handle the high-volume Black Friday and Cyber Monday event. The platform experienced an average throughput increase of 150 percent and processes data through ingestion, batch or stream processing, and delivery to merchants, partners, and internal teams. The article covers the use of Apache Parquet, Apache Spark, dbt, MySQL, Kafka, and tiered services to prioritize reliability and infrastructure investment. ### Source excerpt In this post, we'll outline the approach we took to reliably scale our data platform in preparation for Black Friday and Cyber Monday. ## Behind the Code: Mohammed Ridwanul Islam's Career and Work on Shopify's Eventscale Team DevFeed: [Behind the Code: Mohammed Ridwanul Islam's Career and Work on Shopify's Eventscale Team]() Original publisher: [Read original article]() Author: Toni Akinwumi Published: 2018-10-03T18:55:00Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [Shopify](), [Development](), [Platform Engineering](), [Streaming](), [Code](), [data](), [C#](), [Game Development]() Tags: [c-sharp](), [data-platform-engineering](), [development](), [platform-engineering](), [shopify](), [software-engineer](), [streaming]() ### AI overview A Behind the Code feature about Shopify software engineer Mohammed Ridwanul Islam. It covers his background, career path, transition into software engineering, and work on the Eventscale team building tools, libraries, and infrastructure for event-oriented streaming data. ### Source excerpt Mohammed's feature is part of our series called Behind The Code, where we share the stories of our employees and how they're solving meaningful problems at Shopify and beyond. Mohammed Ridwanul is a software engineer on the Eventscale team and joined Shopify a year and a half ago. Mohammed grew up in Dubai but was born in Noakhali, a small village in Bangladesh before moving when he was five.