# data-engineering Published articles for data-engineering. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## The Future of Data Engineering in the Age of AI | Erfan Hesami DevFeed: [The Future of Data Engineering in the Age of AI | Erfan Hesami]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-09-16T13:21:19Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [data-engineering](), [AI Engineering](), [agent observability](), [coding](), [Security]() Tags: [agents](), [ai](), [ai-agents](), [ai-engineering](), [coding](), [data-engineering](), [fundamentals](), [governance](), [security]() ### AI overview An interview with Erfan Hesami examines how AI and agents may change data engineering, including the evolving role of data engineers, the overlap with AI engineering, the continuing importance of fundamentals, and the need to manage governance, security, costs, technical debt, and human judgment. ### Source excerpt AI Agents, Coding & Fundamentals ## Announcing On-Demand State Repartitioning for Apache Spark™ Structured Streaming on Databricks DevFeed: [Announcing On-Demand State Repartitioning for Apache Spark™ Structured Streaming on Databricks]() Original publisher: [Read original article]() Author: Thangam Vaiyapuri; Jay Palaniappan; B. Micheal Okutubo; Zifei Feng Published: 2026-09-14T21:04:30Z Content type: release Language: en Sources: [Databricks]() Topics: [Streaming](), [databricks](), [real-time](), [Amazon S3](), [API]() Tags: [amazon-s3](), [api](), [data-engineering](), [databricks](), [engineering](), [fraud-detection](), [infrastructure](), [monitoring](), [net-11-preview-7](), [production](), [real-time](), [streaming]() ### AI overview Databricks announces on-demand state repartitioning for Apache Spark Structured Streaming in Public Preview, available in Databricks Runtime 18 and later. The capability lets production stateful streaming queries resize their partition count while preserving checkpoint state, supporting workloads such as aggregations, stream-stream joins, deduplication, sessionization, and transformWithState. Coveo reports reducing related Amazon S3 API costs by 40%. ### Source excerpt Anyone running stateful Apache Spark™ Structured Streaming queries in production... ## Data Engineering Weekly #287 DevFeed: [Data Engineering Weekly #287]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-09-14T02:52:23Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [Kafka](), [data observability](), [Embeddings](), [Multi-tenancy](), [Event-Streaming](), [OpenAI](), [Library](), [Code]() Tags: [architecture](), [architectures](), [data-engineering](), [data-quality](), [embedding](), [engineering](), [kafka](), [multi-tenancy](), [object-storage](), [observability](), [openai](), [streaming](), [vector]() ### AI overview Data Engineering Weekly #287 covers building data platforms from scratch, including composable architectures, data quality, and observability. It also previews talks on governed machine-executable ontologies for marketing activation and fair, order-preserving Kafka consumption for many tenants. The issue links to OpenAI's storage platform scaling for ChatGPT and Pinterest's embedding retrieval platform. ### Source excerpt The Weekly Data Engineering Newsletter ## Improving Lakebase Postgres compute cache DevFeed: [Improving Lakebase Postgres compute cache]() Original publisher: [Read original article]() Author: David Wein; Sunil Kamath; Haoyu Huang Published: 2026-09-10T13:47:03Z Content type: article Language: en Sources: [Databricks]() Topics: [Caching](), [databricks](), [Databases](), [autoscaling](), [Amazon S3](), [Filesystems](), [cloud-infrastructure]() Tags: [autoscaling](), [cache](), [caching](), [data](), [data-engineering](), [databricks](), [engineering](), [filesystem](), [latency](), [low-latency](), [memory](), [s3](), [storage]() ### AI overview This Databricks article describes improvements to compute-side caching for Lakebase Postgres, a disaggregated storage system backed by object storage such as Amazon S3. It explains PostgreSQL shared buffers, the operating system page cache, and the planned use of dynamically autoscaling shared buffers consuming up to 75% of compute memory. It also introduces a local file cache as an incremental solution for fixed compute instances. ### Source excerpt The disaggregated storage model of Lakebase Postgres provides a feature rich, flexible... ## How we built Datadog Experiments DevFeed: [How we built Datadog Experiments]() Original publisher: [Read original article]() Author: Chas DeVeas; Aaron Silverman; Tyler Buffington; Jonathan Fulton; Taylor Overturf Published: 2026-09-10T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [experiments](), [real user monitoring]() Tags: [a-b-testing](), [acquisition](), [data-analytics](), [data-engineering](), [experimentation](), [experiments](), [metrics](), [observability](), [product-analytics](), [real-user-monitoring]() ### AI overview Datadog describes rebuilding its experimentation platform to speed up confident A/B-test decisions. The article explains a flexible CUPED approach that reduces metric variance and can be applied to segments. ### Source excerpt Datadog Experiments shortens the time from result to decision with CUPED on percentiles, verifiable warehouse results, and near real-time RUM metrics. ## We Cut Cloud Waste Before Touching Cluster Sizes: Lessons from Running a Data Platform DevFeed: [We Cut Cloud Waste Before Touching Cluster Sizes: Lessons from Running a Data Platform]() Original publisher: [Read original article]() Author: Deepika Saini Published: 2026-09-07T06:33:31Z Content type: article Language: en Sources: [Housing.com]() Topics: [BigQuery](), [databricks](), [Optimization](), [AWS Database Migration Service](), [data-platforms](), [Platform Engineering](), [migration](), [Cloud]() Tags: [bigquery](), [cloud-computing](), [cost](), [cost-optimization](), [data-engineering](), [data-platforms](), [databricks](), [delta-lake](), [finops](), [infrastructure](), [migration](), [optimization](), [platform-engineering]() ### AI overview This article explains how a data platform team reduced cloud costs by removing obsolete BigQuery data, adjusting Delta Lake retention, right-sizing DMS infrastructure, identifying unmonitored Databricks jobs, and standardizing pipeline onboarding and cost alerts. It reports that DMS costs were cut by over 50% and that retention was reduced from 90 days to 7 days for appropriate workloads after operational validation. ### Source excerpt How orphaned BigQuery storage, Delta retention, DMS right-sizing, and Databricks System Tables became our biggest cloud cost wins. The biggest cloud cost optimization we made wasn't shrinking clusters.It was deleting data we'd forgotten we were paying for.Like most teams, our first instinct was to tune infrastructure first. Instead, we discovered a treasure trove of hidden costs: orphaned BigQuery datasets, 90-day Delta retention, 24-hour jobs no one monitored, and DMS infrastructure that no longer matched business needs.We stopped treating cloud bills as a finance problem and started treating them as a platform engineering problem.30-second takeaway Why deleting forgotten data saved more than shrinking clusters. How we cut DMS costs by over 50%. How Databricks System Tables exposed hidden 24-hour jobs. How config.metadata standardized pipeline onboarding. How weekly Slack alerts turned cost optimization into a habit. Section 1: Storage Was Our Biggest Leak -- We Were Paying to Store Data Nobody Used This is the most overlooked cost on many data platforms. Storage duplication across platforms We had already migrated several workloads from BigQuery to Databricks. Large datasets were still sitting in BigQuery long after they had stopped serving production workloads - quietly generating storage costs month after month. Nothing failed. No alerts fired. Every month, we paid for storage that no longer served production workloads.A migration isn't complete until the old storage is decommissioned.The hidden cost of long retention The next surprise came from Delta Lake retention settings. Our workspace was configured to retain deleted table data and transaction history for 90 days to support time travel. Time travel is incredibly useful. But did every table need three months of historical recovery? Not really. We reduced retention to 7 days for appropriate workloads after validating operational needs. What changed immediately: Less storage tied up in deleted data. Faster clea ## Data Engineering Weekly #286 DevFeed: [Data Engineering Weekly #286]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-09-07T00:18:06Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [AI, ML & Data Engineering](), [agent observability](), [Large Language Model](), [data-modeling](), [Monitoring](), [Scalability](), [semantic-layer](), [parquet]() Tags: [ai](), [ai-agents](), [data-engineering](), [data-modeling](), [llm](), [monitoring](), [newsletter](), [observability](), [parquet](), [scalability](), [semantic-layer](), [weekly]() ### AI overview Data Engineering Weekly #286 is a curated newsletter covering data platform fundamentals, mathematics for machine learning, agentic machine learning at Instacart, Netflix's lifecycle for LLM-as-a-Judge systems, semantic layers and data modeling for AI analytics, and Apache Pinot scalability. ### Source excerpt The Weekly Data Engineering Newsletter ## Shehab Amin on Spark Compatibility, Rust, and LakeSail DevFeed: [Shehab Amin on Spark Compatibility, Rust, and LakeSail]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-09-02T12:22:23Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [Apache Spark](), [Rust](), [data-engineering](), [apache-arrow](), [agentic-coding]() Tags: [apache-arrow](), [apache-spark](), [data-engineering](), [rust]() ### AI overview A podcast conversation with LakeSail co-founder and CEO Shehab Amin about Spark compatibility, Rust, Apache Arrow, DataFusion, and data infrastructure. It also discusses data-stack choices, streaming and batch processing, agentic coding, and using existing data pipelines as a basis for AI pipelines. ### Source excerpt Data Engineering Central Podcast. ## Data Engineering Weekly #285 DevFeed: [Data Engineering Weekly #285]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-08-31T02:51:19Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [AI, ML & Data Engineering](), [Data Quality](), [observability](), [Artificial Intelligence]() Tags: [agentic-ai](), [automated](), [data-engineering](), [newsletter](), [observability](), [privacy](), [python](), [quality](), [recovery](), [weekly]() ### AI overview Data Engineering Weekly #285 covers building data platforms, AI chip architectures, preparing data for agentic AI, post-AI data stacks, data modernization, automated data contract breach handling, and privacy-preserving measurement tools. ### Source excerpt The Weekly Data Engineering Newsletter ## Planetary prediction engine: Automating global models via Earth AI DevFeed: [Planetary prediction engine: Automating global models via Earth AI]() Original publisher: [Read original article]() Published: 2026-08-27T17:37:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [Earth AI](), [Artificial Intelligence](), [data-engineering](), [Google](), [Machine learning](), [AI Development](), [Feature Engineering](), [Model Development]() Tags: [agents](), [ai](), [analytics](), [data-engineering](), [earth-ai](), [evaluation](), [feature-engineering](), [generative-ai](), [google](), [insights](), [machine-intelligence](), [mapping](), [research](), [training](), [validation](), [workflow](), [workflows]() ### AI overview Google Research introduces the Planetary Prediction Engine, an experimental Earth AI capability that autonomously performs geospatial data discovery, cleanup, feature engineering, model training, evaluation, and report generation from natural-language queries. The system targets applications including public health, food security, environmental risk, and socioeconomic analysis, reducing the stated workflow from weeks of manual data engineering to minutes. ### Source excerpt Earth AI ## Using Data Contracts to Coordinate Data Evolution at Enterprise Scale DevFeed: [Using Data Contracts to Coordinate Data Evolution at Enterprise Scale]() Original publisher: [Read original article]() Author: Keerthipriyan Published: 2026-08-25T20:22:17Z Content type: article Language: en Sources: [Walmart Global Tech]() Topics: [data-platforms](), [Data Quality](), [DataOps](), [systems]() Tags: [architecture](), [data](), [data-engineering](), [data-governance](), [data-platforms](), [data-quality](), [data-science](), [enterprise](), [organizational](), [pipeline](), [schema](), [software-engineering](), [teams](), [technology]() ### AI overview The article explains how data contracts help large enterprises coordinate changes across independently evolving data teams and downstream consumers. It argues that schema validation alone cannot identify ownership, downstream impact, or migration responsibilities, and presents data contracts as machine-enforceable coordination agreements. ### Source excerpt Coauthored by Satyajeet Coordinating Data Evolution at Enterprise Scale When you operate data platforms on a global enterprise scale, hundreds of engineering teams ship improvements every week, each moving independently to deliver value at the pace of the business demands. This velocity is a competitive advantage. The challenge: How do you enable hundreds of teams to evolve their data products independently while maintaining reliability for thousands of downstream consumers? Traditional coordination methods (messages, wiki updates, shared spreadsheets) work at small scale but break at Walmart scale. A source team ships an enhancement, perfectly valid within their domain, but that change ripples through fifteen downstream pipelines owned by different teams with different release schedules. Without a formal coordination mechanism, you discover the impact after it reaches production. The gap isn't technical debt or fragile systems. It's the absence of machine-enforceable agreements that scale with organizational complexity. Data contracts solve this: enabling teams to move fast independently while maintaining coordinated reliability across organizational boundaries. Here's the architecture we built. Why Schema Validation Alone Isn't Enough When data quality issues surface in production, the first instinct is often added to more schema validation. If a field is missing or has the wrong type, the pipeline catches it. This works for many data quality problems, but not all of them. Consider a scenario where a source team enhances their data model by restructuring field names to support new business capabilities. The schema still validates perfectly: every field exists; every type is correct; the data is well formed. But downstream consumers who depend on the original field names now receive empty results. Schema validation checks whether data has the right shape. It tells you that a field is missing. It does not tell you who owns that field, which downstream teams will bre ## How to take incremental steps towards data democratisation DevFeed: [How to take incremental steps towards data democratisation]() Original publisher: [Read original article]() Author: Andy Scotland Published: 2026-08-17T13:09:00Z Content type: article Language: en Sources: [Scott Logic]() Topics: [data](), [trust](), [decision-making](), [Artificial Intelligence]() Tags: [agent-factory](), [ai](), [ai-agents](), [artificial-intelligence](), [compliance](), [conway-s-law](), [data](), [data-democratisation](), [data-engineering](), [data-platform](), [data-products](), [governance](), [incremental](), [trust]() ### AI overview This article explains how organisations can pursue data democratisation incrementally while preserving control, risk management and governance. It describes potential benefits in financial services and discusses how trusted data, data products, platforms and AI agents could support innovation, decision-making and customer service. ### Source excerpt Organisations increasingly recognise the value of making data more accessible, but concerns around control, risk and governance often stand in the way. In this post, I explore why data democratisation doesn't require organisations to sacrifice oversight, and how data products, platforms and agent factories can unlock innovation while maintaining trust, compliance and accountability. ## Data Engineering Weekly #283 DevFeed: [Data Engineering Weekly #283]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-08-17T02:59:40Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [AI, ML & Data Engineering](), [Data Quality](), [Microservice](), [Apache Spark](), [Kafka](), [Feature Engineering](), [audit trail](), [Protocol (disambiguation)]() Tags: [ai](), [anthropic](), [article](), [data-engineering](), [data-quality](), [engineering](), [feature-engineering](), [kafka](), [newsletter](), [pipelines](), [services](), [testing]() ### AI overview Data Engineering Weekly #283 is a newsletter covering data platform fundamentals, multiagent system coordination, payments platform data contracts, financial data quality, declarative data engineering, and cost-efficient export workloads. ### Source excerpt The Weekly Data Engineering Newsletter ## AI success depends on data foundations DevFeed: [AI success depends on data foundations]() Original publisher: [Read original article]() Author: James Heward Published: 2026-08-14T14:19:00Z Content type: article Language: en Sources: [Scott Logic]() Topics: [data-architecture](), [AI Strategy](), [AI Development]() Tags: [agentic-ai](), [ai](), [ai-readiness](), [artificial-intelligence](), [data](), [data-engineering](), [governance](), [integration](), [legacy-it](), [legacy-modernisation](), [prototypes](), [quality](), [silos]() ### AI overview Strong data foundations--discoverable, high-quality, accessible, governed, and integrated--are presented as essential for reliable AI outcomes, especially as organisations adopt more autonomous agentic systems. ### Source excerpt As organisations invest in AI, many discover that their biggest challenges are not AI-related at all. In this post, I explore why strong data foundations, from quality and accessibility to governance and integration, are essential for turning AI ambition into reliable, production-ready outcomes. ## Data pipeline monitoring 101: Tracking health and performance across the data stack DevFeed: [Data pipeline monitoring 101: Tracking health and performance across the data stack]() Original publisher: [Read original article]() Author: Aaron Kaplan; Ryan Warrier Published: 2026-08-14T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [Monitoring](), [data observability](), [observability](), [data](), [data streams monitoring](), [Kafka](), [Apache Spark](), [event driven](), [Streaming]() Tags: [ai-ml](), [analytics](), [data](), [data-analytics](), [data-engineering](), [data-observability](), [data-streams-monitoring](), [event-driven](), [kafka](), [learn](), [monitoring](), [observability](), [performance](), [spark](), [streaming]() ### AI overview This article introduces end-to-end monitoring for modern data pipelines. It explains how to track pipeline health, performance, data quality, and availability across varied architectures and technology layers, with examples including Kafka, Flink, Apache Spark, data lakes, warehouses, and lakehouses. ### Source excerpt Learn about monitoring the end-to-end health and performance of modern data pipelines. ## Unlocking your data: the value is in collaboration DevFeed: [Unlocking your data: the value is in collaboration]() Original publisher: [Read original article]() Author: Sam Perridge Published: 2026-08-12T14:59:00Z Content type: opinion Language: en Sources: [Scott Logic]() Topics: [data](), [datasets](), [Artificial Intelligence](), [decision-making](), [systems]() Tags: [ai](), [business-intelligence](), [claude](), [collaboration](), [data](), [data-democratisation](), [data-engineering](), [data-maturity](), [data-platform](), [datasets](), [decision-making](), [governance](), [guardrails](), [reporting](), [self-service]() ### AI overview This opinion article argues that organisations unlock more value from data when datasets are connected and insights are accessible across teams. It describes a progression from paper records and siloed systems to connected and democratised data, including self-service analytics and AI, while emphasising governance and practical adoption. ### Source excerpt Organisations often focus on collecting data and connecting systems, but the greatest value comes from helping datasets work together and making insights accessible to the people who need them. In this post, I explore the journey from siloed data to democratised access, showing how self-service analytics and AI can unlock hidden value, while strong governance provides the guardrails for confident decision-making. ## Quasi-Agentic Pipelines with Databricks and Apache Airflow DevFeed: [Quasi-Agentic Pipelines with Databricks and Apache Airflow]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-08-10T21:23:57Z Content type: tutorial Language: en Sources: [Data Engineering Central]() Topics: [databricks](), [airflow](), [Data pipelines](), [Large Language Model](), [data-engineering]() Tags: [airflow](), [apache-airflow](), [data-engineering](), [databricks](), [llms](), [pipelines]() ### AI overview A practical developer discussion of incorporating LLMs and agents into existing data workflows using Databricks and Apache Airflow. It also examines determinism in data pipelines and the gap between business requirements and engineering implementation. ### Source excerpt the strange space in between ## Data Engineering Weekly #282 DevFeed: [Data Engineering Weekly #282]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-08-10T01:21:26Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [data-platforms](), [data observability](), [Graphs](), [semantic-layer](), [Large Language Model](), [Apache Iceberg](), [gRPC](), [Concurrency](), [Caching](), [Optimization](), [Streaming]() Tags: [caching](), [chaos](), [concurrency](), [data](), [data-engineering](), [data-platforms](), [data-quality](), [knowledge-graphs](), [llms](), [newsletter](), [observability](), [pipelines](), [semantic-layer](), [streaming]() ### AI overview Data Engineering Weekly #282 is a newsletter covering data platform fundamentals, semantic layers, ontology-backed knowledge graphs, converged databases, AI modernization, and Netflix's real-time distributed graph query architecture. It highlights composable architectures, data quality and observability, evolving schemas supported by LLM-assisted extraction, Iceberg full-text search, and optimization techniques including concurrency control, streaming filters, and caching. ### Source excerpt The Weekly Data Engineering Newsletter ## Data Engineering Weekly #281 DevFeed: [Data Engineering Weekly #281]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-08-03T12:34:40Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [AI, ML & Data Engineering](), [Loop Engineering](), [Artificial Intelligence](), [Large Language Model](), [post-training]() Tags: [agent-harness](), [ai](), [data-engineering](), [feature-engineering](), [genai](), [netflix](), [newsletter](), [post-training](), [weekly]() ### AI overview Data Engineering Weekly #281 covers building data platforms, emerging approaches to AI workflow architecture, data modernization, Netflix's GenRec recommendation system, AI infrastructure modernization, and evaluation practices for generative AI at scale. ### Source excerpt The Weekly Data Engineering Newsletter ## Modeling Device Capabilities for Analytics DevFeed: [Modeling Device Capabilities for Analytics]() Original publisher: [Read original article]() Author: Netflix Technology Blog Published: 2026-07-31T16:01:02Z Content type: article Language: en Sources: [Netflix](), [Netflix TechBlog - Medium]() Topics: [Netflix](), [data](), [data-processing](), [datasets](), [Hardware](), [cpu](), [Streaming](), [Software]() Tags: [analytics](), [cloud-gaming](), [cpu](), [data](), [data-engineering](), [data-modeling](), [datasets](), [devices](), [hardware](), [netflix](), [streaming]() ### AI overview Netflix describes a device-capability data model for analytics across a diverse ecosystem of streaming devices. Cumulative and histogram tables capture device capabilities, active device counts, software versions, and feature support, helping teams measure feature reach and make more granular enablement decisions. ### Source excerpt by Aarti Laddha, Richard Diaz-Cool, Rishika Idnani, Venkatesh Selveraj Netflix supports a vast and evolving set of features and content types, ranging from 4K streaming and immersive audio to live streaming and cloud gaming, across a diverse ecosystem of devices. However, not all devices are created equal. Hardware limitations such as available RAM, CPU cores, display capabilities, or platform support mean that some features cannot be supported on certain device models. To ensure the best possible user experience, we rely on a deep understanding of device capabilities. We have invested in building a comprehensive device capability data model and integrating feature flags from internal systems, paving the way for smarter, more granular feature management across our global device landscape. This approach helps us identify bottlenecks in feature penetration and accelerates the pace of innovation. We have designed our data storage and modeling strategies to efficiently support analytics at scale. We use a cumulative table to process information about the device's capabilities. This table is structured to efficiently capture the latest state of each device and its associated capabilities (like Screen resolutions, Video Profiles Supported, Surround Sound, RAM size etc) making it ideal for analytics and reporting use cases. { "Screen Height": ["720"], "Screen Width": ["1280"], "Video Profiles": [ "playready", "hevc", ], } For aggregate analytics, we leverage a histogram table that captures active device counts over the past 28 days, broken down by device model and software version. This table also records the number of devices supporting specific capabilities, enabling detailed distribution analysis. One use case for this histogram data is to analyze the distribution of external display capabilities attached to streaming sticks. For example, the histogram below shows that out of total X number of devices, all supported the HD profile (playready), while only 20% devices sup ## Data Engineering Weekly #280 DevFeed: [Data Engineering Weekly #280]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-07-27T03:37:19Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [Data Infrastructure](), [Open Source](), [Data Quality](), [DuckDB](), [Database]() Tags: [data](), [data-engineering](), [data-infrastructure](), [data-quality](), [database](), [duckdb](), [engineering](), [open-source](), [python]() ### AI overview Data Engineering Weekly #280 is a newsletter roundup covering updates to leetdata.ai and aidataengineer.io, agent-oriented data infrastructure, open-source modern data stack tools, data-tool landscapes, metric certification, and data quality in the AI era. ### Source excerpt The Weekly Data Engineering Newsletter ## The CI/CD moment for data analytics DevFeed: [The CI/CD moment for data analytics]() Original publisher: [Read original article]() Author: Gaurav Nanda Published: 2026-07-23T05:40:01Z Content type: article Language: en Sources: [Platform Engineering Blog]() Topics: [data analytics](), [DataOps](), [real-time](), [data-engineering](), [CI/CD](), [Kafka](), [apache-flink]() Tags: [ai](), [analytics](), [apache-flink](), [architectures](), [bridging](), [cost](), [data-analytics](), [data-engineering](), [engineering](), [event](), [fraud](), [google](), [insights](), [kafka](), [latency](), [real-time](), [scale](), [streaming](), [systems](), [tools]() ### AI overview The article argues that data analytics is approaching a CI/CD-like transition toward continuous analytics. It describes how multi-step pipelines introduce latency, operational risk, and maintenance burden, and explains why real-time insight is becoming a baseline platform capability for applications such as personalization, fraud detection, reliability, and AI-driven features. ### Source excerpt How converging OLTP and OLAP architectures are driving 'Continuous Analytics', the CI/CD moment for data to deliver real-time, unified insights and operational simplicity. ## Agentic Data Engineering Is Here -- But Can It Close the Loop? DevFeed: [Agentic Data Engineering Is Here -- But Can It Close the Loop?]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-22T14:05:27Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [data-engineering](), [Orchestration](), [data observability](), [Artificial Intelligence](), [AI-assisted coding](), [DuckDB](), [Apache Iceberg]() Tags: [agentic](), [ai-agents](), [data-engineering](), [duckdb](), [orchestration](), [pipelines]() ### AI overview A podcast conversation with Hugo Lu about agentic data engineering and the infrastructure needed for data platforms to execute work, observe outcomes, validate changes, and improve pipelines safely. It examines why production data systems remain difficult for AI, including schema changes, realistic testing, business semantics, and secure execution. ### Source excerpt a conversation with Hugo Lu ## Why Most Single Source of Truth Initiatives Fail (And What Successful Teams Do Differently) DevFeed: [Why Most Single Source of Truth Initiatives Fail (And What Successful Teams Do Differently)]() Original publisher: [Read original article]() Author: Deepika Saini Published: 2026-07-20T10:07:52Z Content type: article Language: en Sources: [Housing.com]() Topics: [data](), [product analytics]() Tags: [analytics](), [dashboards](), [data](), [data-architecture](), [data-engineering](), [data-strategy](), [databricks](), [governance](), [leadership](), [product-analytics](), [sql](), [technical]() ### AI overview This article argues that Single Source of Truth initiatives often fail because teams use different definitions for shared business metrics. It presents governance, business ownership of KPI definitions, and alignment between teams as more important than centralizing tables, pipelines, or dashboards. ### Source excerpt "We have multiple dashboards showing different numbers. Which one is correct?" If you've worked in data long enough, you've probably heard this question more times than you'd like. Sales reports one revenue figure. Finance reports another. Product Analytics has a third. Executives spend more time debating whose dashboard is correct than discussing what action to take. The natural response is often: "Let's build a Single Source of Truth." Sounds simple. Build a few centralized tables. Move everyone onto the same dashboards. Problem solved. Except...it rarely is. After leading an enterprise-wide Single Source of Truth (SSOT) initiative, I learned an important lesson: The hardest part wasn't building pipelines or writing SQL. It was aligning people. Technology was the easy part. Changing how the organization thought about data was the real challenge. The Biggest Myth About Single Source of Truth Many organizations believe an SSOT is simply a technical project. The thinking usually goes like this: Collect Data ↓ Transform Data ↓ Build Gold Tables ↓ Everyone Uses Them Unfortunately, reality looks more like this: Different Teams ↓ Different Definitions ↓ Different Dashboards ↓ Different Decisions ↓ Lost Trust The problem isn't that data lives in different places. The problem is that different teams define the same business metrics differently. Figure 1: Moving from fragmented metric definitions to a trusted Single Source of Truth is as much about standardization and governance as it is about technology. A table cannot solve that. Only governance can. Technology Doesn't Create Trust Imagine a metric as simple as Revenue. Ask five departments what "Revenue" means, and you might receive five different answers. Finance may recognize revenue after invoicing. Sales may count closed deals. Marketing may include projected pipeline. Product Analytics may track subscription purchases. Customer Success may exclude refunds. None of them are necessarily wrong. They're answering differen [Next page]()