# Data Quality Data quality measures how well a dataset meets criteria such as accuracy, completeness, validity, consistency, uniqueness, timeliness, and fitness for purpose. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Automating EDA With fg-data-profiling DevFeed: [Automating EDA With fg-data-profiling]() Original publisher: [Read original article]() Author: Real Python Published: 2026-09-15T14:00:00Z Content type: tutorial Language: en Sources: [Real Python]() Topics: [Data analysis](), [pandas](), [Python](), [Data Quality](), [Statistics](), [HTML](), [JSON]() Tags: [data-analysis](), [data-quality](), [pandas](), [python](), [statistics](), [tutorial]() ### AI overview A video course on automating exploratory data analysis with fg-data-profiling. It covers generating interactive reports from DataFrames, exporting them to HTML or JSON, analyzing time series, and comparing datasets. ### Source excerpt Automate exploratory data analysis by transforming DataFrames into interactive reports with one command from fg-data-profiling. ## Data Engineering Weekly #285 DevFeed: [Data Engineering Weekly #285]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-08-31T02:51:19Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [AI, ML & Data Engineering](), [Data Quality](), [observability](), [Artificial Intelligence]() Tags: [agentic-ai](), [automated](), [data-engineering](), [newsletter](), [observability](), [privacy](), [python](), [quality](), [recovery](), [weekly]() ### AI overview Data Engineering Weekly #285 covers building data platforms, AI chip architectures, preparing data for agentic AI, post-AI data stacks, data modernization, automated data contract breach handling, and privacy-preserving measurement tools. ### Source excerpt The Weekly Data Engineering Newsletter ## Using Data Contracts to Coordinate Data Evolution at Enterprise Scale DevFeed: [Using Data Contracts to Coordinate Data Evolution at Enterprise Scale]() Original publisher: [Read original article]() Author: Keerthipriyan Published: 2026-08-25T20:22:17Z Content type: article Language: en Sources: [Walmart Global Tech]() Topics: [data-platforms](), [Data Quality](), [DataOps](), [systems]() Tags: [architecture](), [data](), [data-engineering](), [data-governance](), [data-platforms](), [data-quality](), [data-science](), [enterprise](), [organizational](), [pipeline](), [schema](), [software-engineering](), [teams](), [technology]() ### AI overview The article explains how data contracts help large enterprises coordinate changes across independently evolving data teams and downstream consumers. It argues that schema validation alone cannot identify ownership, downstream impact, or migration responsibilities, and presents data contracts as machine-enforceable coordination agreements. ### Source excerpt Coauthored by Satyajeet Coordinating Data Evolution at Enterprise Scale When you operate data platforms on a global enterprise scale, hundreds of engineering teams ship improvements every week, each moving independently to deliver value at the pace of the business demands. This velocity is a competitive advantage. The challenge: How do you enable hundreds of teams to evolve their data products independently while maintaining reliability for thousands of downstream consumers? Traditional coordination methods (messages, wiki updates, shared spreadsheets) work at small scale but break at Walmart scale. A source team ships an enhancement, perfectly valid within their domain, but that change ripples through fifteen downstream pipelines owned by different teams with different release schedules. Without a formal coordination mechanism, you discover the impact after it reaches production. The gap isn't technical debt or fragile systems. It's the absence of machine-enforceable agreements that scale with organizational complexity. Data contracts solve this: enabling teams to move fast independently while maintaining coordinated reliability across organizational boundaries. Here's the architecture we built. Why Schema Validation Alone Isn't Enough When data quality issues surface in production, the first instinct is often added to more schema validation. If a field is missing or has the wrong type, the pipeline catches it. This works for many data quality problems, but not all of them. Consider a scenario where a source team enhances their data model by restructuring field names to support new business capabilities. The schema still validates perfectly: every field exists; every type is correct; the data is well formed. But downstream consumers who depend on the original field names now receive empty results. Schema validation checks whether data has the right shape. It tells you that a field is missing. It does not tell you who owns that field, which downstream teams will bre ## Project Lighthouse -- Part 3: Introducing project-lighthouse-anonymize DevFeed: [Project Lighthouse -- Part 3: Introducing project-lighthouse-anonymize]() Original publisher: [Read original article]() Author: Adam Bloomston Published: 2026-08-25T17:01:02Z Content type: article Language: en Sources: [The Airbnb Tech Blog - Medium]() Topics: [Algorithms](), [Open Source](), [Python](), [Data Quality](), [Code](), [Programming]() Tags: [algorithms](), [architecture](), [code](), [data](), [data-quality](), [engineering](), [open-source](), [privacy](), [python](), [scale](), [technology](), [validation]() ### AI overview Project Lighthouse -- Part 3 introduces project-lighthouse-anonymize, an open-source Python library for privacy-preserving anonymization. The article describes the project's k-anonymity methodology, scalable Core Mondrian algorithm, parallel processing architecture, and data quality validation framework, alongside two new technical papers. ### Source excerpt Project Lighthouse -- Part 3: Introducing project-lighthouse-anonymizeThe data in Project Lighthouse is powered by privacy-preserving anonymization code. We've put this code into open source, and published two new technical papers detailing the scalable algorithms and data quality frameworks behind it. By: Adam Bloomston Introduction In 2020, we launched Project Lighthouse, which we developed in partnership with leading civil rights and privacy organizations. As our 2020 announcement details, Project Lighthouse enables us to measure potential disparities in user experiences. This work uses perceived race data that is never linked to individual accounts; we only use this data for measuring potential disparities, and users who want to opt-out can do so by turning off the data use settings in their account's Privacy page. Our results, shared in 2024, demonstrate how we use these analyses to measure our progress in mitigating those disparities. Earlier this year, we open-sourced project-lighthouse-anonymize, the Python library that powers Project Lighthouse's anonymization process. To provide the full technical foundation for this work, we also published two new papers on arXiv alongside the code release. Together with our original 2020 paper, these three papers form a complete story: the foundational methodology, the scalable implementation, and the quality validation framework. The foundational methodology (2020) Our original 2020 paper established the privacy-by-design approach for Project Lighthouse and provides the rationale for choosing k-anonymity as the technical privacy model to prevent sensitive attribute disclosure at scale. For an introduction to this paper, see our first blog post on p-sensitive k-anonymity and our second blog post on measurement with anonymized data. Core Mondrian: Scalable partition-based anonymization (2025) The first of our new papers, Core Mondrian: Basic Mondrian beyond k-anonymity, presents the k-anonymity algorithm at the heart of th ## Optimizing Redshift Write Patterns: Tackling Tombstones and Ghost Rows DevFeed: [Optimizing Redshift Write Patterns: Tackling Tombstones and Ghost Rows]() Original publisher: [Read original article]() Author: Harichandan Pulagam Published: 2026-08-24T20:18:12Z Content type: article Language: en Sources: [Wealthfront]() Topics: [Amazon Redshift](), [dashboards](), [Data Quality]() Tags: [amazon-redshift](), [batch](), [dashboards](), [data-quality](), [latency](), [load](), [performance](), [redshift](), [space](), [wealthfront-engineering]() ### AI overview This Wealthfront engineering post examines how Redshift tables grew to nearly 10 times the size of their useful data because deleted rows remained on disk as ghost rows. It describes the resulting read and write latency and the write strategies adopted to control table size. ### Source excerpt Amazon Redshift is a core part of our analytics platform, powering dashboards, data quality checks, ad-hoc analytical workloads, and downstream reporting on a shared cluster. Because everything runs on the same cluster, the size and health of our tables directly affects every workload. At Wealthfront, data drives every decision we make, which means any performance... Read more ## Data Engineering Weekly #283 DevFeed: [Data Engineering Weekly #283]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-08-17T02:59:40Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [AI, ML & Data Engineering](), [Data Quality](), [Microservice](), [Apache Spark](), [Kafka](), [Feature Engineering](), [audit trail](), [Protocol (disambiguation)]() Tags: [ai](), [anthropic](), [article](), [data-engineering](), [data-quality](), [engineering](), [feature-engineering](), [kafka](), [newsletter](), [pipelines](), [services](), [testing]() ### AI overview Data Engineering Weekly #283 is a newsletter covering data platform fundamentals, multiagent system coordination, payments platform data contracts, financial data quality, declarative data engineering, and cost-efficient export workloads. ### Source excerpt The Weekly Data Engineering Newsletter ## Agentic SDLC in Practice: Insights from Engineering Leaders DevFeed: [Agentic SDLC in Practice: Insights from Engineering Leaders]() Original publisher: [Read original article]() Author: Matar Peles Published: 2026-08-10T11:42:34Z Content type: article Language: en Sources: [Developer Experience & Platform Engineering Blog | Port]() Topics: [sdlc](), [Artificial Intelligence](), [AI Agent](), [Platform Engineering](), [Model Context Protocol](), [MCP Server](), [incident](), [Data Quality](), [Amazon Bedrock](), [Slack](), [cursor](), [Pull Request]() Tags: [agent](), [agentic](), [agents](), [ai](), [ai-agent](), [bedrock](), [coding](), [cursor](), [data-quality](), [engineering](), [incident](), [mcp](), [mcp-server](), [platform-engineering](), [python](), [review](), [sdlc](), [server](), [slack](), [workflow]() ### AI overview Engineering leaders share practical examples of an agentic SDLC, including autonomous Jira ticket resolution through pull request creation and AI-powered incident triage. The article highlights adoption challenges such as siloed agents, missing governance, and the need for a shared registry of agents, skills, and MCPs. ### Source excerpt Engineering leaders share how agentic SDLC works in practice: what scales adoption, where teams get stuck, and how to maximize AI ROI. ## Data Engineering Weekly #280 DevFeed: [Data Engineering Weekly #280]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-07-27T03:37:19Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [Data Infrastructure](), [Open Source](), [Data Quality](), [DuckDB](), [Database]() Tags: [data](), [data-engineering](), [data-infrastructure](), [data-quality](), [database](), [duckdb](), [engineering](), [open-source](), [python]() ### AI overview Data Engineering Weekly #280 is a newsletter roundup covering updates to leetdata.ai and aidataengineer.io, agent-oriented data infrastructure, open-source modern data stack tools, data-tool landscapes, metric certification, and data quality in the AI era. ### Source excerpt The Weekly Data Engineering Newsletter ## The continuous validation framework for data pipelines. DevFeed: [The continuous validation framework for data pipelines.]() Original publisher: [Read original article]() Author: Niruta Talwekar Published: 2026-07-23T05:40:01Z Content type: article Language: en Sources: [Platform Engineering Blog]() Topics: [data](), [DataOps](), [Data Quality](), [data observability](), [CI/CD](), [data-processing](), [DevOps](), [Platform Engineering]() Tags: [ci-cd](), [data](), [data-quality](), [devops](), [platform-engineering](), [provenance](), [software-testing](), [streaming]() ### AI overview The article introduces the Continuous Validation Framework (CVF), an end-to-end methodology for validating data pipelines through architectural isolation, configuration-driven data quality management, and continuous automation based on lineage-driven impact analysis. It reports production results including a 50% reduction in incidents and an 80% improvement in detecting data quality issues. ### Source excerpt A framework for automated, end-to-end data pipeline validation using isolation, declarative quality checks, and lineage-driven impact analysis. ## Data Engineering Weekly #279 DevFeed: [Data Engineering Weekly #279]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-07-20T04:07:20Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [Data Quality](), [semantic-layer](), [Kafka](), [observability](), [knowledge-engineering](), [Embeddings](), [Databases](), [Amazon EKS](), [Amazon EC2](), [Availability]() Tags: [availability](), [data-engineering](), [data-quality](), [database](), [embeddings](), [engineering](), [kafka](), [knowledge-engineering](), [observability](), [semantic-layer]() ### AI overview Data Engineering Weekly #279 is a newsletter covering data platform fundamentals, data management for generative AI, semantic-layer portability, knowledge-base construction with Postgres and embeddings, Kafka migration, and Apache Pinot high availability. The supplied text includes sponsored material and ends mid-item. ### Source excerpt The Weekly Data Engineering Newsletter ## Data Engineering Weekly #275 DevFeed: [Data Engineering Weekly #275]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-06-22T04:02:10Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [semantic-layer](), [Data Quality](), [observability](), [Model Context Protocol](), [Python](), [SQL](), [YAML]() Tags: [data-engineering](), [data-quality](), [mcp](), [metrics](), [observability](), [pipelines](), [python](), [semantic-layer](), [sql](), [yaml]() ### AI overview Data Engineering Weekly #275 is a newsletter issue covering data platform fundamentals, semantic layers, metric governance, idempotent pipeline design, and AI modernization. It highlights how shared business definitions, data quality, observability, and retry-safe writes support reliable analytics and AI workflows. ### Source excerpt The Weekly Data Engineering Newsletter ## Building a Centralized Alerting Framework for Data Quality Monitoring and Incident Management DevFeed: [Building a Centralized Alerting Framework for Data Quality Monitoring and Incident Management]() Original publisher: [Read original article]() Author: Manav Mehta Published: 2026-06-18T07:11:45Z Content type: article Language: en Sources: [Helpshift]() Topics: [Data Quality](), [incident management](), [Monitoring](), [DataOps]() Tags: [data-engineering](), [data-quality](), [email](), [incident-management](), [monitoring](), [notifications](), [observability](), [pipelines](), [slack](), [snowflake]() ### AI overview This article describes the design of a centralized alerting and incident management framework for data quality and pipeline monitoring. The framework uses Snowflake's native Alerting capabilities with Email, Slack, and Splunk On-Call integrations to detect issues, notify the appropriate engineers, escalate critical incidents, and provide actionable context. ### Source excerpt Before We Knew Better As our data platform grew, so did the number of pipelines, scheduled tasks, and data quality checks running every day. While Snowflake provided a reliable platform for storing and processing data, operational monitoring was fragmented across multiple systems. Data quality failures were often discovered only after downstream reports showed inconsistencies. Pipeline issues sometimes required engineers to manually inspect logs, query tables, and trace execution paths before identifying the root cause. The challenge wasn't detecting failures -- we already had mechanisms to identify them. The real challenge was ensuring the right people were notified quickly, with enough context to take action. Questions during on-call incidents were often similar: Did the pipeline fail or was data simply delayed? Which validation check triggered the alert? Who should respond to the issue? How can we ensure critical failures don't get missed overnight? As the number of pipelines increased, manually monitoring these failures became increasingly difficult. We needed a centralized alerting framework. What We Actually Needed Our goal wasn't simply to send more notifications. We wanted a system that could: Detect data quality issues automatically Notify engineers through channels they already use Escalate critical incidents to on-call responders Provide actionable context instead of generic failure messages Scale across multiple pipelines and monitoring use cases Most importantly, we wanted to keep the solution as close to the data platform as possible. Since our monitoring logic already lived in Snowflake, it made sense for the alerting framework to live there as well. The Architecture We Chose To address these challenges, we designed a centralized notification and incident management framework using Snowflake's native Alerting capabilities, combined with Email, Slack, and Splunk On-Call integrations. Rather than introducing another monitoring platform, we chose to build ## Data Engineering Weekly #274 DevFeed: [Data Engineering Weekly #274]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-06-15T05:29:03Z Content type: news Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [Data Quality](), [anthropic](), [data-architecture](), [AI, ML & Data Engineering](), [Claude](), [Messaging]() Tags: [anthropic](), [architecture](), [claude](), [data](), [data-architecture](), [data-engineering](), [data-quality](), [engineering](), [software]() ### AI overview Data Engineering Weekly #274 covers data platform fundamentals, Anthropic's use of Claude for self-service analytics, Airbnb's data architecture and ownership conventions, Uber's data abstraction layer, and semantic search for AI agents. ### Source excerpt The Weekly Data Engineering Newsletter ## Granite 4.1 LLMs: How They're Built DevFeed: [Granite 4.1 LLMs: How They're Built]() Original publisher: [Read original article]() Author: Yousaf Shah Published: 2026-04-29T15:01:48Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [LLMs](), [ibm](), [long-context](), [Fine-tuning](), [grpo](), [Training AI Models](), [Transformer architecture](), [Data Quality]() Tags: [apache](), [fine-tuning](), [grpo](), [ibm](), [learning](), [llms](), [long-context](), [tokens](), [training](), [transformer-architecture]() ### AI overview The article introduces Granite 4.1, IBM's family of dense decoder-only LLMs in 3B, 8B, and 30B sizes. It describes their five-stage training process, which uses about 15 trillion tokens, data-quality refinement, long-context extension up to 512K tokens, supervised fine-tuning, and reinforcement learning with on-policy GRPO and DAPO loss. The models use a dense transformer architecture and are released under the Apache 2.0 license. ### Source excerpt Authors: Granite Team, IBM TL;DR -- Granite 4.1 is a family of dense, decoder-only LLMs (3B, 8B, and 30B) trained on ~15T tokens using a multi-stage pre-training pipeline, including long-context extension of up to 512K tokens. The models are further refined with supervised fine-tuning on ~4.1M high-quality curated samples and reinforcement learning via on-policy GRPO with DAPO loss (Yu et al., 2025). ## Chainguard customers safe from elementary-data compromise DevFeed: [Chainguard customers safe from elementary-data compromise]() Original publisher: [Read original article]() Published: 2026-04-25T00:00:00Z Content type: article Language: en Sources: [Chainguard: Unchained]() Topics: [chainguard](), [Malware](), [data observability](), [Data Quality](), [chainguard containers](), [chainguard libraries]() Tags: [chainguard](), [chainguard-containers](), [chainguard-customers](), [chainguard-libraries](), [data-observability](), [data-quality](), [elementary-data-compromise](), [malware](), [pypi](), [pypi-malware]() ### AI overview Chainguard reports that customers using its Python Libraries and Container images were unaffected by the compromised elementary-data 0.23.3 package on PyPI. Chainguard detected malicious patterns before building the package, while the compromised release was quarantined and related GitHub and Docker artifacts were removed. ### Source excerpt Malicious elementary-data version hit PyPI. Chainguard customers stayed protected by detecting malware pre-build and serving only verified safe versions. ## Daily Tasks With Data Pipelines - Data Quality Checks And The Problem With Noisy Checks DevFeed: [Daily Tasks With Data Pipelines - Data Quality Checks And The Problem With Noisy Checks]() Original publisher: [Read original article]() Author: SeattleDataGuy Published: 2026-04-07T22:24:38Z Content type: article Language: en Sources: [SeattleDataGuy's Newsletter]() Topics: [Data Quality](), [Data pipelines](), [data]() Tags: [data-pipelines](), [data-quality](), [pipelines](), [quality]() ### AI overview The article discusses data quality checks in data pipelines and notes that teams may receive 137 data quality alerts every morning. ### Source excerpt Every morning, your team wakes up to 137 data quality alerts. ## Risk-Based Data Quality Testing for Reliable Finance Pipelines DevFeed: [Risk-Based Data Quality Testing for Reliable Finance Pipelines]() Original publisher: [Read original article]() Author: Jeremy Chia Published: 2026-03-11T00:00:00Z Content type: tutorial Language: en Sources: [Vinted]() Topics: [Data Quality](), [DataOps](), [Finance](), [Testing]() Tags: [data](), [data-quality](), [finance](), [migration](), [parquet](), [pipeline](), [pipelines](), [quality](), [reporting](), [schema](), [testing]() ### AI overview This article explains how Vinted addressed frequent upstream schema and format changes affecting finance reporting pipelines. It describes shifting testing closer to the source, applying materiality-based checks, and balancing data quality with pipeline reliability and timely availability. ### Source excerpt Upstream schema changes were breaking our finance pipelines daily. With monthly reporting deadlines looming, we needed to balance data quality with pipeline reliability. Here's how we solved it without compromising either. ## Wayfair boosts catalog accuracy and support speed with OpenAI DevFeed: [Wayfair boosts catalog accuracy and support speed with OpenAI]() Original publisher: [Read original article]() Published: 2026-03-11T00:00:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [OpenAI](), [AI Models](), [AI Development](), [Data Quality](), [ChatGPT](), [Generative AI]() Tags: [agent](), [ai](), [ai-models](), [chatgpt](), [data](), [data-quality](), [ecommerce](), [openai](), [production]() ### AI overview Wayfair integrates OpenAI models into supplier support and product catalog workflows, automating ticket handling and improving the accuracy of product attributes across millions of items. A single tag-agnostic system replaces costly bespoke models and supports production-scale operations. ### Source excerpt Wayfair uses OpenAI models to improve ecommerce support and product catalog accuracy, automating ticket triage and enhancing millions of product attributes at scale. ## What really matters for performance: lessons from a year of benchmarks DevFeed: [What really matters for performance: lessons from a year of benchmarks]() Original publisher: [Read original article]() Author: Tom Schreiber Published: 2025-12-22T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [Benchmark](), [clickhouse](), [Compression](), [data](), [Data Quality](), [dataset](), [MongoDB](), [dashboards](), [Bluesky](), [pull-requests]() Tags: [benchmark](), [benchmarks](), [clickhouse](), [compression](), [data-quality](), [dataset](), [performance](), [pull-requests](), [storage]() ### AI overview The article reviews benchmark-backed ClickHouse work and argues that practical performance depends on how efficiently a database engine moves data end to end. It highlights JSONBench, which compares ClickHouse with other systems using up to one billion Bluesky events across ingestion quality, storage size, and cold and hot query runtimes. The reported results show ClickHouse achieving substantially faster queries and lower storage usage than traditional JSON data stores such as MongoDB. ### Source excerpt A look back at some of my favorite benchmark-backed ClickHouse blog posts this year, and the lessons they revealed about performance in practice. ## Inside Mirakl's agentic commerce vision DevFeed: [Inside Mirakl's agentic commerce vision]() Original publisher: [Read original article]() Published: 2025-12-01T22:00:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [Artificial Intelligence](), [ChatGPT](), [Documentation](), [Support](), [Data Quality]() Tags: [agentic](), [ai](), [ai-agents](), [catalog](), [chatgpt](), [customer](), [data-quality](), [documentation](), [efficiency](), [support]() ### AI overview Mirakl describes an agentic commerce strategy in which AI becomes a company-wide capability and employees build agents for individual and team workflows. Using ChatGPT Enterprise and other AI systems, Mirakl reports faster technical documentation, improved customer-support efficiency with high satisfaction, faster catalog onboarding, fewer categorization errors, and multilingual 24/7 support. ### Source excerpt Mirakl is redefining commerce through AI agents and ChatGPT Enterprise--achieving faster documentation, smarter customer support, and building toward agent-native commerce with Mirakl Nexus. ## The 2025 AI + Data Engineering Roadmap DevFeed: [The 2025 AI + Data Engineering Roadmap]() Original publisher: [Read original article]() Author: Zach Wilson Published: 2025-10-17T22:35:45Z Content type: tutorial Language: en Sources: [DataExpert.io Newsletter]() Topics: [data-engineering](), [Python](), [SQL](), [Artificial Intelligence](), [airflow](), [BigQuery](), [Orchestration](), [Apache Spark](), [data-modeling](), [Data Quality](), [Embeddings](), [Retrieval Augmented Generation (RAG)]() Tags: [2025](), [ai](), [airflow](), [bigquery](), [count](), [course](), [data-engineering](), [data-modeling](), [data-quality](), [embeddings](), [engineering](), [framer](), [newsletter](), [orchestration](), [python](), [rag](), [real-time](), [right-join](), [spark](), [sql]() ### AI overview A 2025 roadmap for entering data engineering, covering foundational SQL and Python skills, distributed computing, orchestration, data modeling, data quality, AI and data integrations, portfolio projects, and personal branding. ### Source excerpt Getting a data engineering job is complicated. ## 18x faster, 15x cheaper: How Datavations rebuilt its pipeline with ClickHouse DevFeed: [18x faster, 15x cheaper: How Datavations rebuilt its pipeline with ClickHouse]() Original publisher: [Read original article]() Author: ClickHouse Published: 2025-10-06T12:00:47Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [data-architecture](), [data](), [Data Quality](), [datasets](), [Amazon S3](), [databricks](), [AWS Glue](), [Amazon EC2](), [Amazon Web Services](), [Algorithms](), [synthetic-data]() Tags: [architecture](), [aws](), [aws-glue](), [clickhouse](), [cloud](), [data](), [data-architecture](), [data-quality](), [databricks](), [latency](), [machine-learning](), [optimization](), [performance](), [real-time](), [s3]() ### AI overview Datavations rebuilt its data architecture around ClickHouse after its S3, Databricks, AWS Glue, Athena, and Tableau pipeline became costly and slow at larger scale. The migration aimed to reduce costs, improve performance, and deliver faster, more actionable data-quality insights for its home-improvement analytics platform. ### Source excerpt Learn how Datavations rebuilt its data architecture around ClickHouse, cutting costs, improving performance, and giving clients faster, more actionable insights ## Apache Airflow 3.1.0: Human-Centered Workflows DevFeed: [Apache Airflow 3.1.0: Human-Centered Workflows]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2025-09-25T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Automation](), [human review](), [Data Quality](), [Forms](), [Developer experience](), [Inference](), [React]() Tags: [apache-airflow](), [automation](), [data-quality](), [developer-experience](), [forms](), [human-review](), [inference](), [release](), [release-notes](), [ui]() ### AI overview Apache Airflow 3.1.0 introduces human-in-the-loop tasks for pausing automated workflows and collecting reviews through web forms. The release also adds internationalization support, developer experience improvements, restored Calendar and Gantt views, UI filtering, DAG pinning, and accessibility-focused theme updates. ### Source excerpt We are thrilled to announce the release of Apache Airflow 3.1.0, an update that puts humans at the center of data workflows. This release introduces powerful new capabilities for human decision-making in automated processes, comprehensive internationalization support, and significant developer experience improvements. Details: ðŸ“Ķ PyPI: https://pypi.org/project/apache-airflow/3.1.0/ 📚 Core Airflow Docs: https://airflow.apache.org/docs/apache-airflow/3.1.0/ 📚 Task SDK Docs: https://airflow.apache.org/docs/task-sdk/1.1.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.1.0/release_notes.html ðŸŠķ Sources: https://airflow.apache.org/docs/apache-airflow/3.1.0/installation/installing-from-sources.html 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.1.0 ðŸĪ Human-in-the-Loop (HITL): When Automation Meets Human Judgment This powerful capability bridges the gap between automated processes and human expertise, making Airflow invaluable for: AI/ML Model Validation: Pause inference pipelines for human review of model outputs Content Moderation: Route content through human reviewers before publication Approval Workflows: Require manager approval for sensitive operations Data Quality Gates: Allow data stewards to validate critical datasets HITL tasks pause in a deferred state while presenting intuitive web forms in the Airflow UI. Users with appropriate roles can review context data, DAG parameters, and XCom values before making informed decisions. Example Code: from airflow.sdk import DAG from airflow.providers.standard.operators.hitl import HITLOperator with DAG("content_moderation", schedule="@daily") as dag: moderate_content = HITLOperator( task_id="review_content", message="Please review this content for publication", data_key="content_to_review" ) 📊 UI Enhancements & Performance Calendar and Gantt Views Make Their Comeback Remember those beloved Calendar and Gantt chart views from Airflow 2.x? They're back, completely rebuilt for the modern ## mmBERT: ModernBERT goes Multilingual DevFeed: [mmBERT: ModernBERT goes Multilingual]() Original publisher: [Read original article]() Author: Marc Marone; Orion Weller; William Fleshman; Eugene Yang; Dawn Lawrie; Ben Van Durme Published: 2025-09-09T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Artificial Intelligence](), [Training AI Models](), [Machine Learning & Artificial Intelligence](), [datasets](), [Open Source Models & Datasets](), [Data Quality](), [Boilerplate]() Tags: [artificial-intelligence](), [blog-post](), [community](), [data-quality](), [datasets](), [llm](), [model](), [models](), [nlp](), [open-source](), [open-source-collab](), [performance](), [research](), [speed](), [training]() ### AI overview This blog post introduces mmBERT, a massively multilingual encoder model trained on more than 3 trillion tokens across over 1,800 languages. It describes performance and speed improvements over earlier multilingual models, the model's ModernBERT-based architecture, and a progressive strategy for adding languages and balancing training data. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. [Next page]()