# DataOps DataOps is a set of principles for developing and delivering analytics, including the orchestration of data, tools, code, and environments. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## New data pipeline management platform at Khan Academy DevFeed: [New data pipeline management platform at Khan Academy]() Original publisher: [Read original article]() Author: Khan Academy Published: 2018-04-30T22:00:00Z Content type: article Language: en Sources: [Khan Academy]() Topics: [data](), [DataOps](), [systems]() Tags: [batch](), [bigquery](), [cloud-dataflow](), [data](), [data-pipelines](), [debugging](), [engineering](), [infrastructure](), [news](), [pipeline](), [scheduling]() ### AI overview Khan Academy developed Khanalytics to manage its growing collection of data pipelines. The platform provides a sandboxed environment for batch jobs, a web interface, automatic parallelization, centralized logs, and pipeline scheduling with dependencies. ### Source excerpt By Ragini Gupta Data is very crucial to Khan Academy and is itself an internal product for the ... Read more ## Linux Patched For Silent User-Space Data Loss Bug That's Existed Since 2023 DevFeed: [Linux Patched For Silent User-Space Data Loss Bug That's Existed Since 2023]() Original publisher: [Read original article]() Author: Michael Larabel Published: 2026-09-14T10:13:45Z Content type: news Language: en Sources: [Phoronix]() Topics: [Linux](), [bug](), [DataOps](), [Code]() Tags: [bug](), [c](), [code](), [data-analytics](), [desktop-linux](), [kernel](), [linux](), [linux-benchmarking](), [linux-hardware-benchmarks](), [linux-hardware-reviews](), [linux-how-to](), [linux-performance](), [linux-server-benchmarks](), [open-source-graphics](), [phoronix](), [phoronix-test-suite](), [production](), [release](), [ubuntu-benchmarks](), [ubuntu-hardware]() ### AI overview A Linux kernel bug introduced in July 2023 can silently discard user-space writes when transparent hugepages are enabled, cgroup limits apply, and memory reclaim pressure is high. The issue has caused production data loss, including for users of the Polars data analytics library. A one-line fix was merged into the x86/urgent branch and marked for backporting to supported stable kernels. ### Source excerpt Being merged after yesterday's Linux 7.3-rc3 release was an important fix for addressing a silent, user-space data loss bug that has existed in the kernel the past three years... ## A practical approach to end-to-end Solvency II reporting in Databricks DevFeed: [A practical approach to end-to-end Solvency II reporting in Databricks]() Original publisher: [Read original article]() Author: Laurence Ryszka; Jack Yallop Published: 2026-09-09T16:20:00Z Content type: article Language: en Sources: [Databricks]() Topics: [databricks](), [DataOps](), [Orchestration](), [data](), [systems]() Tags: [business](), [data](), [databricks](), [eu](), [financial](), [financial-services](), [governance](), [industries](), [monitor](), [orchestration](), [platform](), [regulatory](), [solutions](), [uk](), [workflow]() ### AI overview The article presents Databricks as a governed data, orchestration, and reporting layer for end-to-end Solvency II reporting. It covers ingestion, data quality, actuarial reserving, capital calculation, QRT production, ORSA drafting, governance, approvals, and disclosure while allowing insurers to retain established actuarial and capital modeling systems. ### Source excerpt Solvency II reporting is not only a regulatory submission. It is a business process... ## Using Data Contracts to Coordinate Data Evolution at Enterprise Scale DevFeed: [Using Data Contracts to Coordinate Data Evolution at Enterprise Scale]() Original publisher: [Read original article]() Author: Keerthipriyan Published: 2026-08-25T20:22:17Z Content type: article Language: en Sources: [Walmart Global Tech]() Topics: [data-platforms](), [Data Quality](), [DataOps](), [systems]() Tags: [architecture](), [data](), [data-engineering](), [data-governance](), [data-platforms](), [data-quality](), [data-science](), [enterprise](), [organizational](), [pipeline](), [schema](), [software-engineering](), [teams](), [technology]() ### AI overview The article explains how data contracts help large enterprises coordinate changes across independently evolving data teams and downstream consumers. It argues that schema validation alone cannot identify ownership, downstream impact, or migration responsibilities, and presents data contracts as machine-enforceable coordination agreements. ### Source excerpt Coauthored by Satyajeet Coordinating Data Evolution at Enterprise Scale When you operate data platforms on a global enterprise scale, hundreds of engineering teams ship improvements every week, each moving independently to deliver value at the pace of the business demands. This velocity is a competitive advantage. The challenge: How do you enable hundreds of teams to evolve their data products independently while maintaining reliability for thousands of downstream consumers? Traditional coordination methods (messages, wiki updates, shared spreadsheets) work at small scale but break at Walmart scale. A source team ships an enhancement, perfectly valid within their domain, but that change ripples through fifteen downstream pipelines owned by different teams with different release schedules. Without a formal coordination mechanism, you discover the impact after it reaches production. The gap isn't technical debt or fragile systems. It's the absence of machine-enforceable agreements that scale with organizational complexity. Data contracts solve this: enabling teams to move fast independently while maintaining coordinated reliability across organizational boundaries. Here's the architecture we built. Why Schema Validation Alone Isn't Enough When data quality issues surface in production, the first instinct is often added to more schema validation. If a field is missing or has the wrong type, the pipeline catches it. This works for many data quality problems, but not all of them. Consider a scenario where a source team enhances their data model by restructuring field names to support new business capabilities. The schema still validates perfectly: every field exists; every type is correct; the data is well formed. But downstream consumers who depend on the original field names now receive empty results. Schema validation checks whether data has the right shape. It tells you that a field is missing. It does not tell you who owns that field, which downstream teams will bre ## The CI/CD moment for data analytics DevFeed: [The CI/CD moment for data analytics]() Original publisher: [Read original article]() Author: Gaurav Nanda Published: 2026-07-23T05:40:01Z Content type: article Language: en Sources: [Platform Engineering Blog]() Topics: [data analytics](), [DataOps](), [real-time](), [data-engineering](), [CI/CD](), [Kafka](), [apache-flink]() Tags: [ai](), [analytics](), [apache-flink](), [architectures](), [bridging](), [cost](), [data-analytics](), [data-engineering](), [engineering](), [event](), [fraud](), [google](), [insights](), [kafka](), [latency](), [real-time](), [scale](), [streaming](), [systems](), [tools]() ### AI overview The article argues that data analytics is approaching a CI/CD-like transition toward continuous analytics. It describes how multi-step pipelines introduce latency, operational risk, and maintenance burden, and explains why real-time insight is becoming a baseline platform capability for applications such as personalization, fraud detection, reliability, and AI-driven features. ### Source excerpt How converging OLTP and OLAP architectures are driving 'Continuous Analytics', the CI/CD moment for data to deliver real-time, unified insights and operational simplicity. ## The continuous validation framework for data pipelines. DevFeed: [The continuous validation framework for data pipelines.]() Original publisher: [Read original article]() Author: Niruta Talwekar Published: 2026-07-23T05:40:01Z Content type: article Language: en Sources: [Platform Engineering Blog]() Topics: [data](), [DataOps](), [Data Quality](), [data observability](), [CI/CD](), [data-processing](), [DevOps](), [Platform Engineering]() Tags: [ci-cd](), [data](), [data-quality](), [devops](), [platform-engineering](), [provenance](), [software-testing](), [streaming]() ### AI overview The article introduces the Continuous Validation Framework (CVF), an end-to-end methodology for validating data pipelines through architectural isolation, configuration-driven data quality management, and continuous automation based on lineage-driven impact analysis. It reports production results including a 50% reduction in incidents and an 80% improvement in detecting data quality issues. ### Source excerpt A framework for automated, end-to-end data pipeline validation using isolation, declarative quality checks, and lineage-driven impact analysis. ## Building a Centralized Alerting Framework for Data Quality Monitoring and Incident Management DevFeed: [Building a Centralized Alerting Framework for Data Quality Monitoring and Incident Management]() Original publisher: [Read original article]() Author: Manav Mehta Published: 2026-06-18T07:11:45Z Content type: article Language: en Sources: [Helpshift]() Topics: [Data Quality](), [incident management](), [Monitoring](), [DataOps]() Tags: [data-engineering](), [data-quality](), [email](), [incident-management](), [monitoring](), [notifications](), [observability](), [pipelines](), [slack](), [snowflake]() ### AI overview This article describes the design of a centralized alerting and incident management framework for data quality and pipeline monitoring. The framework uses Snowflake's native Alerting capabilities with Email, Slack, and Splunk On-Call integrations to detect issues, notify the appropriate engineers, escalate critical incidents, and provide actionable context. ### Source excerpt Before We Knew Better As our data platform grew, so did the number of pipelines, scheduled tasks, and data quality checks running every day. While Snowflake provided a reliable platform for storing and processing data, operational monitoring was fragmented across multiple systems. Data quality failures were often discovered only after downstream reports showed inconsistencies. Pipeline issues sometimes required engineers to manually inspect logs, query tables, and trace execution paths before identifying the root cause. The challenge wasn't detecting failures -- we already had mechanisms to identify them. The real challenge was ensuring the right people were notified quickly, with enough context to take action. Questions during on-call incidents were often similar: Did the pipeline fail or was data simply delayed? Which validation check triggered the alert? Who should respond to the issue? How can we ensure critical failures don't get missed overnight? As the number of pipelines increased, manually monitoring these failures became increasingly difficult. We needed a centralized alerting framework. What We Actually Needed Our goal wasn't simply to send more notifications. We wanted a system that could: Detect data quality issues automatically Notify engineers through channels they already use Escalate critical incidents to on-call responders Provide actionable context instead of generic failure messages Scale across multiple pipelines and monitoring use cases Most importantly, we wanted to keep the solution as close to the data platform as possible. Since our monitoring logic already lived in Snowflake, it made sense for the alerting framework to live there as well. The Architecture We Chose To address these challenges, we designed a centralized notification and incident management framework using Snowflake's native Alerting capabilities, combined with Email, Slack, and Splunk On-Call integrations. Rather than introducing another monitoring platform, we chose to build ## From SSH to REST: A Security-Driven Modernization of Slack's EMR Data Pipelines DevFeed: [From SSH to REST: A Security-Driven Modernization of Slack's EMR Data Pipelines]() Original publisher: [Read original article]() Author: Mahendran Vasagam Published: 2026-05-05T14:00:01Z Content type: article Language: en Sources: [Engineering at Slack]() Topics: [Security](), [OpenSSH](), [Amazon Web Services](), [airflow](), [DataOps](), [Architecture & Design](), [Kubernetes]() Tags: [airflow](), [architecture](), [aws](), [big-data](), [data-analytics](), [data-engineering](), [data-infrastructure](), [kubernetes](), [modernization](), [security](), [ssh](), [uncategorized]() ### AI overview Slack describes migrating more than 700 SSH-based data pipeline jobs to a REST-based architecture across eight data regions, eliminating SSH access to production AWS EMR clusters without downtime. The article explains the security and operational problems that motivated the modernization, including attack-surface exposure, key-management overhead, resource contention, broken connections, zombie jobs, and unreliable job-status detection. ### Source excerpt Excerpt By 2024, Slack's data platform had accumulated 700+ SSH-based operators orchestrating critical data pipelines. We're talking daily search indexing that processed terabytes of data, analytics jobs powering business intelligence, the whole shebang. Every single one of these jobs required direct SSH access to production AWS Elastic MapReduce (EMR) clusters. We had a massive security... ## Risk-Based Data Quality Testing for Reliable Finance Pipelines DevFeed: [Risk-Based Data Quality Testing for Reliable Finance Pipelines]() Original publisher: [Read original article]() Author: Jeremy Chia Published: 2026-03-11T00:00:00Z Content type: tutorial Language: en Sources: [Vinted]() Topics: [Data Quality](), [DataOps](), [Finance](), [Testing]() Tags: [data](), [data-quality](), [finance](), [migration](), [parquet](), [pipeline](), [pipelines](), [quality](), [reporting](), [schema](), [testing]() ### AI overview This article explains how Vinted addressed frequent upstream schema and format changes affecting finance reporting pipelines. It describes shifting testing closer to the source, applying materiality-based checks, and balancing data quality with pipeline reliability and timely availability. ### Source excerpt Upstream schema changes were breaking our finance pipelines daily. With monthly reporting deadlines looming, we needed to balance data quality with pipeline reliability. Here's how we solved it without compromising either. ## Streaming IoT and event data into Snowflake and ClickHouse DevFeed: [Streaming IoT and event data into Snowflake and ClickHouse]() Original publisher: [Read original article]() Author: Mdu Sibisi Published: 2025-12-09T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [Internet of things](), [Streaming](), [Redpanda-Connect](), [clickhouse](), [data observability](), [Compression](), [data-governance](), [DataOps](), [Database]() Tags: [clickhouse](), [cloud-storage](), [compliance](), [data](), [data-governance](), [event](), [iot](), [pipelines](), [redpanda-connect](), [streaming]() ### AI overview A guide to streaming IoT and event data through Redpanda and Redpanda Connect into Snowflake and ClickHouse. It compares ClickHouse for real-time analysis with Snowflake for scalable cloud storage, historical reporting, and querying, while discussing governance, compression, data freshness, and robust pipeline design. ### Source excerpt Learn how to stream IoT and event data into Snowflake and ClickHouse using Redpanda ## Stopping Silent Failures for Meta's Fake Accounts Pipeline DevFeed: [Stopping Silent Failures for Meta's Fake Accounts Pipeline]() Original publisher: [Read original article]() Author: Zach Wilson Published: 2025-08-12T19:38:14Z Content type: article Language: en Sources: [DataExpert.io Newsletter]() Topics: [Meta](), [Orchestration](), [DataOps]() Tags: [fake-accounts](), [meta](), [orchestration](), [pipeline]() ### AI overview An article about stopping silent failures in Meta's fake accounts pipeline. ### Source excerpt Data Orchestration Challenges I Faced at Airbnb, Netflix & Facebook - Part IV ## How I got a 12x speed up in a 50 TB pipeline at Meta DevFeed: [How I got a 12x speed up in a 50 TB pipeline at Meta]() Original publisher: [Read original article]() Author: Zach Wilson Published: 2025-08-04T19:39:26Z Content type: article Language: en Sources: [DataExpert.io Newsletter]() Topics: [Meta](), [Orchestration](), [data](), [DataOps]() Tags: [data](), [meta](), [orchestration](), [pipeline](), [speed]() ### AI overview The article describes a 12x speed improvement in a 50 TB data pipeline at Meta and places the topic within data orchestration challenges discussed across Airbnb, Netflix, and Facebook. ### Source excerpt Data Orchestration Challenges I Faced at Airbnb, Netflix & Facebook - Part III ## From siloed DataOps, MLOps, and LLMOps to a unified data-intelligence platform DevFeed: [From siloed DataOps, MLOps, and LLMOps to a unified data-intelligence platform]() Original publisher: [Read original article]() Author: Rajit Saha Published: 2025-08-04T18:03:19Z Content type: opinion Language: en Sources: [Udemy Engineering]() Topics: [DataOps](), [MLOps](), [Generative AI](), [Apache Spark](), [Amazon Bedrock](), [Amazon S3](), [Amazon Redshift](), [Amazon SageMaker](), [apache-flink]() Tags: [ai](), [aiops](), [amazon-s3](), [analytics](), [apache-flink](), [apache-spark](), [bedrock](), [dataops](), [large-language-models](), [llmops](), [mlops]() ### AI overview The article describes how DataOps, MLOps, and AI/LLM Ops commonly rely on separate systems and teams for data processing, model deployment, inference, evaluation, orchestration, governance, and monitoring. It then introduces Databricks' Data Intelligence Platform as a unified environment intended to bring these domains together. ### Source excerpt Introduction In modern data-driven businesses, the pace of innovation in analytics and artificial intelligence has outstripped the capacity of many teams. Three distinct disciplines emerged to handle this expansion: Data platform (DataOps) teams built data lakes on cloud storage such as Amazon S3, processed them with Apache Spark and Hive on EMR, ingested streaming data with Spark Structured Streaming or Apache Flink, and loaded tabular copies into MPP warehouses like Redshift for interactive SQL and BI. Cataloguing and governance were offloaded to external tools such as DataHub, and fine-grained access controls required third-party services like Privacera. This architecture worked, but it required separate workflows for batch and streaming, extra systems for lineage and governance, and a mosaic of operational teams. MLOps teams provided an additional layer. Data scientists used notebook environments (for example, Amazon SageMaker) to preprocess data, train, and evaluate models. Deploying models meant writing integration code to move features into a serving layer, to register models in disparate registries and to build custom APIs for inference. Feature stores and model registries were bought from additional vendors. Updates and monitoring were often manual processes. AI/LLM Ops teams are a new addition because generative AI requires specialized components: LLM gateways (e.g., Amazon Bedrock) to proxy access to foundation models; evaluation tooling to compare large language models; orchestration frameworks for agents; vector databases for retrieval augmented generation; and of course another layer of security, access management and cost control. These tools seldom integrate seamlessly with existing data and ML pipelines. This fragmented state makes it difficult to react quickly when product requirements change. Each new capability requires another system, another integration, and another team. Meanwhile, budgets tighten and go-to-market timelines shrink. The questio ## Overclocking dbt: Discord's Custom Solution in Processing Petabytes of Data DevFeed: [Overclocking dbt: Discord's Custom Solution in Processing Petabytes of Data]() Original publisher: [Read original article]() Author: Chris Dong Published: 2025-04-09T00:00:00Z Content type: article Language: en Sources: [Discord Blog]() Topics: [data](), [developer-productivity](), [data observability](), [DataOps](), [Testing](), [SQL](), [Open Source](), [Software Engineering]() Tags: [analytics](), [blog-post](), [code](), [data](), [developer](), [developer-productivity](), [open-source](), [performance](), [productivity](), [quality](), [software-engineering](), [sql](), [testing](), [tool]() ### AI overview Discord describes scaling dbt to process petabytes of warehouse data while supporting more than 100 developers and over 2,500 models. It explains how custom extensions addressed long recompilation times, inefficient incremental processing, conflicting test tables, breaking changes, and complex calculations, improving performance, developer productivity, and data quality. ### Source excerpt Explore how Discord supercharged dbt with a tailored solution designed for performance, developer productivity, and data quality. ## Data Quality at Petabyte Scale: Building Trust in the Data Lifecycle DevFeed: [Data Quality at Petabyte Scale: Building Trust in the Data Lifecycle]() Original publisher: [Read original article]() Author: Zakariah Siyaji Published: 2025-02-14T15:52:43Z Content type: article Language: en Sources: [Glassdoor Engineering]() Topics: [Data Quality](), [data-engineering](), [data-processing](), [DataOps](), [Usability]() Tags: [data-engineering](), [data-patterns](), [data-platform-engineering](), [data-quality](), [decision-making](), [gable](), [pipeline](), [shift-left](), [software-engineering](), [trust](), [usability]() ### AI overview Glassdoor describes a shift from reactive data engineering to a proactive, trust-centered approach to data quality. The article connects organizational culture with technical checks across the data lifecycle. ### Source excerpt The data Lifecycle with Data Quality Checks at GlassdoorMotivation Glassdoor has transformed from an employee review site to a community for workplace conversations [1]. As our platform evolves to support content creators, facilitate discussions, and offer rich content, it has become more apparent than ever that adopting a data-driven culture is essential. Businesses rely on accurate, high-quality data to understand their operations and assess strategic outcomes. Flawed or incomplete data results in misguided decisions and undermines trust. Recognizing this risk, we made data quality a foundational principle of our data-driven transformation. Although every company defines data quality differently, there is a universal expectation that data used for decision-making must be trustworthy. Additionally, data quality challenges are not solely technical; a psychological component is closely linked to trust in data. Airbnb recognized this and sought to develop a scoring system that acknowledges the belief that data quality is a multivariate issue, encompassing accuracy, reliability, stewardship, and usability, along with more detailed dimensions within each of these categories [2]. On the other hand, Netflix employs a more technically centered approach to quality: data is initially written to a temporary staging area, audited, and then published to the production location upon passing quality checks [3]. Ultimately, Glassdoor drew inspiration from these lessons and aimed to reinforce trust through a cultural shift and a series of technical solutions. This article demonstrates how a proactive, trust-centered approach that connects data producers and consumers establishes a foundation for more rigorous data quality methods, ultimately bolstering a strong company-wide strategy. Figure 1. Enhancing quality guards at the application code layer.Culture Shift: Reactive to Proactive Historically, Glassdoor's data engineering teams have been reactive, learning about issues only aft ## How to break into data in 2024? With DataCamp's CEO, Jonathan Cornelissen. DevFeed: [How to break into data in 2024? With DataCamp's CEO, Jonathan Cornelissen.]() Original publisher: [Read original article]() Author: Meri Nova Published: 2024-10-09T17:31:09Z Content type: opinion Language: en Sources: [Meri Nova]() Topics: [Data Science](), [data analytics](), [genai](), [Artificial Intelligence](), [DataOps]() Tags: [ai](), [beginners](), [career-advice](), [data](), [data-analytics](), [data-science](), [education](), [future-of-work](), [genai](), [podcast](), [scaling](), [technical]() ### AI overview A Technical Founder podcast episode featuring DataCamp co-founder and CEO Jonathan Cornelissen. The discussion covers building DataCamp, entering data careers, data education, GenAI's impact on edtech, data literacy, and career advice for new graduates. ### Source excerpt Listen to our first episode of the "Technical Founder" podcast, where we invite AI and Data leaders to learn from their entrepreneurial and technical journey! ## Data quality testing DevFeed: [Data quality testing]() Original publisher: [Read original article]() Author: Lambert Le Manh Published: 2024-09-04T16:30:00Z Content type: article Language: en Sources: [The incident.io Blog]() Topics: [Data Quality](), [data observability](), [DataOps](), [ci](), [Transactions]() Tags: [ci](), [data-observability](), [data-quality](), [flaky](), [incident](), [incident-channel](), [incident-management](), [incident-response](), [observability](), [outage](), [post-mortem](), [slack-incident](), [testing](), [transactions](), [workflow]() ### AI overview The article explains how incident.io uses dbt's native testing features for data quality testing and data observability. It describes integrating validation into production and CI workflows, including custom tests and relationship tests, and addresses flaky failures caused by ingestion and transformation pipelines running at different times. ### Source excerpt Our data observability workflow uses data quality testing to ensure data meets accuracy, consistency, and reliability standards, enabling confident, data-driven decisions. See how we built it, the common challenges we encountered, and the solutions. ## Emerging Kafka Trends in Connectors, Self-Service Environments, and Stream Processing DevFeed: [Emerging Kafka Trends in Connectors, Self-Service Environments, and Stream Processing]() Original publisher: [Read original article]() Published: 2021-05-28T08:30:00Z Content type: opinion Language: en Sources: [Gunnar Morling]() Topics: [Kafka](), [streaming-data-processing](), [DataOps](), [Open Source]() Tags: [kafka](), [open-source](), [pipelines](), [review](), [stream-processing]() ### AI overview The article identifies recurring themes in Kafka community activity: the expanding connector ecosystem and its operational maturity, self-service Kafka environments, and growing adoption of stream processing. It also mentions related concerns including access control, schema management, data lineage, quality, compliance, privacy, and observability. ### Source excerpt Table of Contents Cambrian Explosion of Connectors Democratization of Data Pipelines Stream Processing for Everyone Honorable Mentions Over the course of the last few months, I've had the pleasure to serve on the Kafka Summit program committee and review several hundred session abstracts for the three Summits happening this year (Europe, APAC, Americas). That's not only a big honour, but also a unique opportunity to learn what excites people currently in the Kafka eco-system (and yes, it's a fair amount of work, too ;). While voting on the proposals, and also generally aspiring to stay informed of what's going on in the Kafka community at large, I noticed a few repeating themes and topics which I thought would be interesting to share (without touching on any specific talks of course). At first I meant to put this out via a Twitter thread, but then it became a bit too long for that, so I decided to write this quick blog post instead. Here it goes! ## DataOps: How to Develop and Scale Data Intensive Projects DevFeed: [DataOps: How to Develop and Scale Data Intensive Projects]() Original publisher: [Read original article]() Author: Alberto Romeu Published: 2021-02-27T00:00:00Z Content type: tutorial Language: en Sources: [Tinybird]() Topics: [DataOps](), [DevOps](), [data]() Tags: [dataops](), [develop](), [devops](), [pipelines](), [scalable-analytics-architecture](), [scale]() ### AI overview The article presents DataOps practices for developing and scaling data-intensive projects. It describes these workflows as applying DevOps rigor to data pipelines in 2026. ### Source excerpt DataOps practices separate teams that ship from teams that struggle. These workflows bring DevOps rigor to data pipelines in 2026. ## DataOps: 10 principles to develop data intensive projects DevFeed: [DataOps: 10 principles to develop data intensive projects]() Original publisher: [Read original article]() Author: Alberto Romeu Published: 2021-02-27T00:00:00Z Content type: article Language: en Sources: [Tinybird]() Topics: [DataOps](), [data]() Tags: [data](), [dataops](), [develop](), [projects](), [scalable-analytics-architecture](), [teams]() ### AI overview The article presents 10 DataOps principles for developing data-intensive projects and supporting data teams. ### Source excerpt 10 of the principles of DataOps that we make available to data teams. ## Implementing ETL on GCP DevFeed: [Implementing ETL on GCP]() Original publisher: [Read original article]() Author: Deepmala Published: 2020-07-15T20:53:17Z Content type: tutorial Language: en Sources: [Brave New Geek]() Topics: [DataOps](), [Cloud](), [BigQuery](), [data loss prevention](), [Low code](), [No-code](), [olap]() Tags: [analytics](), [analytics-pipeline](), [bi-tools](), [bigquery](), [cdap](), [cloud-data-loss-prevention](), [cloud-dataflow](), [cloud-dataprep](), [cloud-dataproc](), [cloud-pub-sub](), [cloud-storage](), [cloud-tasks](), [data-analytics](), [data-fusion](), [data-lake](), [data-loss-prevention](), [elt](), [etl](), [gcp](), [no-code](), [serverless](), [sql]() ### AI overview A practical guide to building ETL pipelines on Google Cloud Platform using Google-managed services. It explains a two-phase architecture with Cloud Storage as a data lake, Cloud Data Loss Prevention for sensitive-data detection or redaction, and BigQuery as the curated data warehouse, with attention to low-code and no-code approaches. ### Source excerpt ETL (Extract-Transform-Load) processes are an essential component of any data analytics program. This typically involves loading data from disparate sources, transforming or enriching it, and storing the curated data in a data warehouse for consumption by different users or systems. An example of this would be taking customer data from operational databases, joining it with data from Salesforce and Google Analytics, and writing it to an OLAP database or BI engine.