# data-platforms A data platform is a technology environment that collects, stores, manages, processes, and analyzes structured and unstructured data. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## We Cut Cloud Waste Before Touching Cluster Sizes: Lessons from Running a Data Platform DevFeed: [We Cut Cloud Waste Before Touching Cluster Sizes: Lessons from Running a Data Platform]() Original publisher: [Read original article]() Author: Deepika Saini Published: 2026-09-07T06:33:31Z Content type: article Language: en Sources: [Housing.com]() Topics: [BigQuery](), [databricks](), [Optimization](), [AWS Database Migration Service](), [data-platforms](), [Platform Engineering](), [migration](), [Cloud]() Tags: [bigquery](), [cloud-computing](), [cost](), [cost-optimization](), [data-engineering](), [data-platforms](), [databricks](), [delta-lake](), [finops](), [infrastructure](), [migration](), [optimization](), [platform-engineering]() ### AI overview This article explains how a data platform team reduced cloud costs by removing obsolete BigQuery data, adjusting Delta Lake retention, right-sizing DMS infrastructure, identifying unmonitored Databricks jobs, and standardizing pipeline onboarding and cost alerts. It reports that DMS costs were cut by over 50% and that retention was reduced from 90 days to 7 days for appropriate workloads after operational validation. ### Source excerpt How orphaned BigQuery storage, Delta retention, DMS right-sizing, and Databricks System Tables became our biggest cloud cost wins. The biggest cloud cost optimization we made wasn't shrinking clusters.It was deleting data we'd forgotten we were paying for.Like most teams, our first instinct was to tune infrastructure first. Instead, we discovered a treasure trove of hidden costs: orphaned BigQuery datasets, 90-day Delta retention, 24-hour jobs no one monitored, and DMS infrastructure that no longer matched business needs.We stopped treating cloud bills as a finance problem and started treating them as a platform engineering problem.30-second takeaway Why deleting forgotten data saved more than shrinking clusters. How we cut DMS costs by over 50%. How Databricks System Tables exposed hidden 24-hour jobs. How config.metadata standardized pipeline onboarding. How weekly Slack alerts turned cost optimization into a habit. Section 1: Storage Was Our Biggest Leak -- We Were Paying to Store Data Nobody Used This is the most overlooked cost on many data platforms. Storage duplication across platforms We had already migrated several workloads from BigQuery to Databricks. Large datasets were still sitting in BigQuery long after they had stopped serving production workloads - quietly generating storage costs month after month. Nothing failed. No alerts fired. Every month, we paid for storage that no longer served production workloads.A migration isn't complete until the old storage is decommissioned.The hidden cost of long retention The next surprise came from Delta Lake retention settings. Our workspace was configured to retain deleted table data and transaction history for 90 days to support time travel. Time travel is incredibly useful. But did every table need three months of historical recovery? Not really. We reduced retention to 7 days for appropriate workloads after validating operational needs. What changed immediately: Less storage tied up in deleted data. Faster clea ## Postgres + ClickHouse Architectural Patterns DevFeed: [Postgres + ClickHouse Architectural Patterns]() Original publisher: [Read original article]() Author: Agus Syafaat Published: 2026-08-26T08:08:43Z Content type: article Language: en Sources: [SeveralNines]() Topics: [clickhouse](), [PostgreSQL](), [Databases](), [Architecture & Design](), [data-platforms](), [real-time](), [Transactions](), [Low Latency](), [dashboards]() Tags: [analytics](), [architecture](), [clickhouse](), [clustercontrol](), [database-general](), [hybrid-operations](), [latency](), [olap](), [postgres](), [postgresql](), [real-time](), [time](), [transactions]() ### AI overview This article explains architectural patterns that combine PostgreSQL and ClickHouse. PostgreSQL serves as the authoritative transactional system for OLTP workloads, while ClickHouse handles large-scale analytical queries and real-time analytics. Continuous Change Data Capture synchronization connects the systems and separates transactional and analytical workloads. ### Source excerpt The role of databases has shifted significantly as modern applications must deliver real-time analytics, dashboards, and machine learning alongside low-latency transaction processing. Handling these diverse demands with a single relational database has become unsustainable under growing data volumes. Consequently, organizations are adopting specialized database architectures where multiple engines work together based on their strengths, allowing [...] The post Postgres + ClickHouse Architectural Patterns appeared first on Severalnines. ## Using Data Contracts to Coordinate Data Evolution at Enterprise Scale DevFeed: [Using Data Contracts to Coordinate Data Evolution at Enterprise Scale]() Original publisher: [Read original article]() Author: Keerthipriyan Published: 2026-08-25T20:22:17Z Content type: article Language: en Sources: [Walmart Global Tech]() Topics: [data-platforms](), [Data Quality](), [DataOps](), [systems]() Tags: [architecture](), [data](), [data-engineering](), [data-governance](), [data-platforms](), [data-quality](), [data-science](), [enterprise](), [organizational](), [pipeline](), [schema](), [software-engineering](), [teams](), [technology]() ### AI overview The article explains how data contracts help large enterprises coordinate changes across independently evolving data teams and downstream consumers. It argues that schema validation alone cannot identify ownership, downstream impact, or migration responsibilities, and presents data contracts as machine-enforceable coordination agreements. ### Source excerpt Coauthored by Satyajeet Coordinating Data Evolution at Enterprise Scale When you operate data platforms on a global enterprise scale, hundreds of engineering teams ship improvements every week, each moving independently to deliver value at the pace of the business demands. This velocity is a competitive advantage. The challenge: How do you enable hundreds of teams to evolve their data products independently while maintaining reliability for thousands of downstream consumers? Traditional coordination methods (messages, wiki updates, shared spreadsheets) work at small scale but break at Walmart scale. A source team ships an enhancement, perfectly valid within their domain, but that change ripples through fifteen downstream pipelines owned by different teams with different release schedules. Without a formal coordination mechanism, you discover the impact after it reaches production. The gap isn't technical debt or fragile systems. It's the absence of machine-enforceable agreements that scale with organizational complexity. Data contracts solve this: enabling teams to move fast independently while maintaining coordinated reliability across organizational boundaries. Here's the architecture we built. Why Schema Validation Alone Isn't Enough When data quality issues surface in production, the first instinct is often added to more schema validation. If a field is missing or has the wrong type, the pipeline catches it. This works for many data quality problems, but not all of them. Consider a scenario where a source team enhances their data model by restructuring field names to support new business capabilities. The schema still validates perfectly: every field exists; every type is correct; the data is well formed. But downstream consumers who depend on the original field names now receive empty results. Schema validation checks whether data has the right shape. It tells you that a field is missing. It does not tell you who owns that field, which downstream teams will bre ## Data Engineering Weekly #282 DevFeed: [Data Engineering Weekly #282]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-08-10T01:21:26Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [data-platforms](), [data observability](), [Graphs](), [semantic-layer](), [Large Language Model](), [Apache Iceberg](), [gRPC](), [Concurrency](), [Caching](), [Optimization](), [Streaming]() Tags: [caching](), [chaos](), [concurrency](), [data](), [data-engineering](), [data-platforms](), [data-quality](), [knowledge-graphs](), [llms](), [newsletter](), [observability](), [pipelines](), [semantic-layer](), [streaming]() ### AI overview Data Engineering Weekly #282 is a newsletter covering data platform fundamentals, semantic layers, ontology-backed knowledge graphs, converged databases, AI modernization, and Netflix's real-time distributed graph query architecture. It highlights composable architectures, data quality and observability, evolving schemas supported by LLM-assisted extraction, Iceberg full-text search, and optimization techniques including concurrency control, streaming filters, and caching. ### Source excerpt The Weekly Data Engineering Newsletter ## Eric Schwartz on what it takes to run an AI SRE at petabyte scale DevFeed: [Eric Schwartz on what it takes to run an AI SRE at petabyte scale]() Original publisher: [Read original article]() Author: WorkOS Published: 2026-08-07T00:00:00Z Content type: article Language: en Sources: [WorkOS Blog]() Topics: [Artificial Intelligence](), [site-reliability-engineering](), [data-platforms](), [telemetry](), [observability ai agents]() Tags: [ai](), [data](), [data-platforms](), [on-call](), [production](), [root-cause-analysis](), [sre](), [telemetry]() ### AI overview An interview with Traversal product manager Eric Schwartz examines how the company operates an AI site reliability engineer for large enterprises. The article explains that petabyte-scale telemetry requires continuously analyzing, compressing, and indexing data ahead of runtime, with an SRE-focused tool and prompt harness. Traversal reports that deployments are generally running in production within a week with minimal tuning, supported by forward-deployed engineering for last-mile optimization. ### Source excerpt Traversal PM Eric Schwartz on data platforms, routing models by severity, and the permission ladder toward self-driving production, from AI Engineer 2026. ## The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering DevFeed: [The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-08T12:16:09Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [pandas](), [data-engineering](), [Artificial Intelligence](), [DuckDB](), [data-platforms](), [software-development](), [future of software](), [Apache Spark](), [Hadoop](), [Open Source](), [parquet]() Tags: [ai](), [apache-arrow](), [arrow](), [data](), [data-engineering](), [data-platforms](), [duckdb](), [hadoop](), [large-language-models](), [open-source](), [pandas](), [parquet](), [software-development]() ### AI overview An interview with Wes McKinney covers the origins of pandas and Apache Arrow, the evolution of modern data engineering from Hadoop to lakehouse architectures, and the roles of tools such as Parquet, DuckDB, DataFusion, and Spark. McKinney also discusses how AI affects software development, arguing that it can improve experienced engineers' productivity but does not replace software engineering, architecture, or judgment. ### Source excerpt interview with Wes McKinney ## Build a Governed Databricks Workspace with Pulumi DevFeed: [Build a Governed Databricks Workspace with Pulumi]() Original publisher: [Read original article]() Author: Pablo Seibelt Published: 2026-06-25T00:00:00Z Content type: tutorial Language: en Sources: [Pulumi]() Topics: [databricks](), [data-platforms](), [Tutorial](), [configuration](), [Security](), [Cloud]() Tags: [apis](), [cloud](), [configuration](), [data-engineering](), [databricks](), [governance](), [infrastructure-as-code](), [pulumi](), [security](), [tutorial](), [tutorials]() ### AI overview A tutorial on using Pulumi to establish a governed Databricks workspace baseline. It covers cluster policies, notebooks, secret scopes, permissions, and automated jobs, while distinguishing account-level from workspace-level resources and showing credential management with Pulumi ESC. ### Source excerpt Platform teams responsible for Databricks often find themselves manually configuring clusters and notebooks for every new data science team. This manual overhead leads to inconsistent cluster policies, runaway costs from oversized instances, and security gaps in workspace operations. Without a standardized way to provision workspace-local resources, data platforms become a fragmented collection of bespoke environments that are impossible to govern at scale. As Databricks usage grows across the enterprise, the lack of a governed workspace baseline becomes a major operational risk. Inconsistent policies lead to unpredictable billing and audit failures. Standardizing your Databricks environment with Pulumi ensures that every workspace starts with the correct cost controls, cluster policies, notebooks, and automated jobs, allowing your data teams to move faster without compromising governance. What you'll build In this post, you will learn how to provision a governed Databricks workspace baseline using Pulumi. You will build: Cluster policies to enforce instance types and cost controls. Workspace notebooks to standardize workload locations. Secret scopes and permissions to make access reviewable. Automated jobs that run on policy-constrained compute. By the end, you will have a reproducible workspace configuration that you can deploy to any new Databricks environment. The Databricks management boundary When managing Databricks with Pulumi, understand the boundary between account-level and workspace-level resources. Account-level: Creating cloud prerequisites such as VPCs, subnets, IAM roles, and Azure resource groups uses the cloud provider for your target platform. On AWS and GCP, Databricks workspace creation and account-level identity use the Databricks provider's account APIs, such as databricks.MwsWorkspaces; on Azure, the workspace itself is commonly managed with azure-native. Workspace-level: Managing clusters, jobs, notebooks, permissions, and secret scopes with ## Introducing CostBench: an open benchmark for data warehouse cost-performance DevFeed: [Introducing CostBench: an open benchmark for data warehouse cost-performance]() Original publisher: [Read original article]() Author: Tom Schreiber; Lionel Palacin Published: 2026-05-27T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [Benchmark](), [benchmarking](), [Cloud](), [data-platforms](), [real-time](), [SQL]() Tags: [benchmark](), [benchmarks](), [cloud](), [cost](), [data-platforms](), [performance](), [real-time](), [sql]() ### AI overview CostBench is an open benchmark for comparing cloud data warehouses by performance per dollar rather than speed alone. ### Source excerpt Introducing CostBench, an open benchmark that turns cloud data warehouse runtime and billing models into comparable performance-per-dollar results. ## Gala supercharges analytics performance with ClickHouse on AWS DevFeed: [Gala supercharges analytics performance with ClickHouse on AWS]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-05-04T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Amazon Web Services (AWS)](), [data analytics](), [Data Infrastructure](), [data-platforms](), [telemetry](), [databricks](), [Blockchain]() Tags: [aws](), [blockchain](), [clickhouse](), [data-analytics](), [data-infrastructure](), [data-platforms](), [databricks](), [telemetry]() ### AI overview Gala migrated from Databricks to ClickHouse on AWS to address growing data volumes and improve analytics performance. The article reports a threefold increase in analytics capacity, a 30% cost reduction, and query times reduced from minutes to sub-second. ### Source excerpt Learn how Gala migrated to the ClickHouse to improve analytics performance and cut costs ## AI is redrawing the database market DevFeed: [AI is redrawing the database market]() Original publisher: [Read original article]() Author: Tanya Bragin Published: 2026-03-17T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [Artificial Intelligence](), [clickhouse](), [Database](), [data-platforms](), [real-time](), [observability](), [dashboards]() Tags: [agentic](), [ai](), [analytics](), [clickhouse](), [dashboards](), [data](), [data-platforms](), [database](), [developers](), [observability](), [real-time]() ### AI overview AI is reshaping database platform requirements, driving demand for high-concurrency, real-time query processing, and full-fidelity data at scale. The article examines how real-time analytics, data warehousing, and observability are converging at the data layer, positioning ClickHouse as a unified platform for AI workloads. ### Source excerpt AI is reshaping database market requirements across real-time analytics, data warehousing, and observability -- and ClickHouse is uniquely positioned to meet them. ## 300-Node Clusters Now Supported in CockroachDB DevFeed: [300-Node Clusters Now Supported in CockroachDB]() Original publisher: [Read original article]() Author: David Bressler,Dipti Joshi Published: 2026-02-12T00:00:00Z Content type: release Language: en Sources: [Cockroach Labs]() Topics: [CockroachDB](), [Cockroach Labs](), [data-platforms](), [Benchmark](), [Cloud](), [Resilience]() Tags: [backups](), [benchmark](), [cloud](), [cockroach-labs](), [cockroachdb](), [data-platforms](), [distributed](), [performance](), [resilience](), [sql]() ### AI overview Cockroach Labs announces support for 300-node clusters in CockroachDB v25.4.4 and later, with up to 2.2M tpmC and 1.2PB of data. The article also describes testing involving operational stress and reports performance results from that test cycle. ### Source excerpt As AI-driven and agentic applications push data platforms into new territory, data architects are increasingly forced to choose between correctness, simplicity, and scale. Today, we're removing that tradeoff -- announcing support for 300-node clusters with 2.2M tpmC and 1.2PB of data in CockroachDB v25.4.4 and beyond. On CockroachDB Cloud, we're announcing support for 64 vCPU per node. Testing to this scale is important to ensure that we're testing ahead of customer deployments, and this milestone does just that. We believe these tests represent the largest node-scale testing completed by any distributed SQL vendor. And we're just getting started. ## Introducing OpenAI Frontier DevFeed: [Introducing OpenAI Frontier]() Original publisher: [Read original article]() Published: 2026-02-05T06:00:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [OpenAI](), [Artificial Intelligence](), [Authorization](), [data-platforms](), [systems]() Tags: [agents](), [ai](), [ai-agents](), [applications](), [building](), [customers](), [data](), [data-platforms](), [enterprise](), [governance](), [openai](), [platform](), [product](), [systems](), [technology](), [work]() ### AI overview OpenAI introduces Frontier, an enterprise platform for building, deploying, and managing AI agents. The platform is designed to help organizations move from isolated use cases to AI coworkers by providing shared context, onboarding, learning through feedback, and clear permissions and boundaries. ### Source excerpt OpenAI Frontier is an enterprise platform for building, deploying, and managing AI agents with shared context, onboarding, permissions, and governance. ## Migrating from NATS to Redpanda: Evolve from messaging to streaming DevFeed: [Migrating from NATS to Redpanda: Evolve from messaging to streaming]() Original publisher: [Read original article]() Author: Kavya Shivashankar Published: 2025-11-18T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Messaging](), [Event-Streaming](), [jetstream](), [event driven](), [data-platforms](), [Microservice](), [Persistence](), [real-time](), [Latency](), [personalization]() Tags: [database](), [event-driven](), [event-streaming](), [jetstream](), [latency](), [messaging](), [microservices](), [persistence](), [personalization](), [pipelines](), [real-time](), [streaming]() ### AI overview This article explains how organizations can migrate from NATS and JetStream messaging to Redpanda Streaming as their systems grow from low-latency message delivery toward durable event streaming. It compares messaging and streaming requirements and presents Redpanda Connect as a way to simplify the migration. ### Source excerpt Organizations are moving from message delivery to data streaming, unlocking new capabilities, insights, and improved business value. Here's how to do the same. ## Why streaming is the backbone for AI-native data platforms DevFeed: [Why streaming is the backbone for AI-native data platforms]() Original publisher: [Read original article]() Author: Tyler Rockwood Published: 2025-06-24T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Streaming](), [data-platforms](), [Artificial Intelligence](), [AI Development](), [Data analysis](), [Embeddings](), [SQL](), [dashboards](), [personalization](), [Development]() Tags: [ai](), [ai-agents](), [ai-and-data-warehouse-evolution](), [ai-automation-in-products](), [ai-data-analysis-automation](), [ai-enhanced-product-development](), [ai-native-data-platforms](), [ai-product-integration](), [ai-technology-integration](), [dashboards](), [data-analysis](), [data-platforms](), [data-platforms-for-ai-scalability](), [development](), [embeddings](), [modern-data-warehouse-ai](), [personalization](), [real-time-ai-personalization](), [sql](), [streaming](), [streaming-for-ai-data-platforms](), [structuring-data-lakes-with-ai](), [thought-leadership]() ### AI overview The article argues that streaming is the backbone of AI-native data platforms. It presents the modern data warehouse as a central source of context for AI and describes how AI can support analysis, dashboards, SQL queries, trend detection, embeddings, personalization, recommendations, automation, and faster product development. It also emphasizes guardrails and performance measurement for AI-driven automation. ### Source excerpt To power AI at scale, organizations must adapt to the evolving role of the modern data warehouse. Here's what you need to keep up in a rapidly evolving industry. ## ClickHouse raises $350 million Series C to power analytics for the AI era DevFeed: [ClickHouse raises $350 million Series C to power analytics for the AI era]() Original publisher: [Read original article]() Author: ClickHouse Published: 2025-05-29T14:22:53Z Content type: news Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Artificial Intelligence](), [Data Infrastructure](), [data-platforms](), [real-time](), [observability]() Tags: [ai-ml](), [announcement](), [clickhouse](), [data](), [funding](), [observability](), [real-time](), [series]() ### AI overview ClickHouse announced a $350 million Series C financing round, bringing its total funding to more than $650 million. The company plans to use the funding to scale product development, expand globally, and deepen partnerships supporting AI-native and real-time data applications. ### Source excerpt ClickHouse, Inc., a leader in real-time analytics, data warehousing, observability, and AI/ML, today announced it has raised $350 million in Series C financing. ## The foundations of Canva's continuous data platform with Snowpipe Streaming DevFeed: [The foundations of Canva's continuous data platform with Snowpipe Streaming]() Original publisher: [Read original article]() Author: Jack Caperon Published: 2025-01-06T00:00:01Z Content type: article Language: en Sources: [Canva Engineering]() Topics: [Streaming](), [product analytics](), [data-platforms](), [Amazon S3](), [Serverless]() Tags: [analytics](), [continuous](), [data](), [data-platform](), [partitioning](), [performance](), [s3](), [snowflake](), [streaming]() ### AI overview Canva describes how it is building a continuous data platform with Snowpipe Streaming to support product analytics as its user base, workforce, and data volume grow. The article discusses schema management, streaming changes, cost reduction, and the limitations of its previous AWS Data Firehose-based approach. ### Source excerpt Leveraging Snowpipe Streaming to build a continuous data platform. ## Failover Replication Slots with Postgres 17 DevFeed: [Failover Replication Slots with Postgres 17]() Original publisher: [Read original article]() Published: 2024-12-03T00:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Replication](), [Availability](), [Database](), [data-platforms]() Tags: [availability](), [data](), [data-platforms](), [database](), [debezium](), [failover](), [read-replica](), [replication]() ### AI overview This article explains how Postgres 17 introduces built-in failover slots. Replication slots created on a primary are automatically propagated to read replicas, allowing consumers such as Debezium to continue after failover without missing change events or creating a new initial snapshot. ### Source excerpt Table of Contents Hello, Failover Slots! Failover Slots in Decodable Wrapping Up This post originally appeared on the Decodable blog. All rights reserved. Postgres read replicas are commonly used not only to distribute query load amongst multiple nodes, but also to ensure high availability (HA) of the database. If the primary node of a Postgres cluster fails, a read replica can be promoted to be the new primary, processing write (and read) requests from thereon. ## MLOps seen by a CIO DevFeed: [MLOps seen by a CIO]() Original publisher: [Read original article]() Author: Raphaël Hoogvliets Published: 2024-02-07T09:41:29Z Content type: opinion Language: en Sources: [Raphaël Hoogvliets]() Topics: [MLOps](), [data-platforms](), [digital]() Tags: [article](), [insights](), [leadership](), [mlops]() ### AI overview An interview with Ahmed Fessi examines MLOps and broader technology transformation from a CIO's perspective. It discusses executive-level challenges, the differences between technology work in tech and non-tech companies, and the importance of evaluating technologies beyond hype, including their value, opportunities, risks, and implementation considerations. ### Source excerpt Beyond the hype to real-world applications ## Real-time data platforms explained: components & benefits DevFeed: [Real-time data platforms explained: components & benefits]() Original publisher: [Read original article]() Author: Cameron Archer Published: 2023-07-27T00:00:00Z Content type: article Language: en Sources: [Tinybird]() Topics: [data-platforms](), [real-time](), [data](), [Streams]() Tags: [architecture](), [components](), [data](), [data-platforms](), [real-time](), [scalable-analytics-architecture](), [streams]() ### AI overview An overview of real-time data platforms, focusing on how they process incoming streams and how their architecture can help readers choose a platform for their needs. ### Source excerpt Real-time data platforms process streams as they arrive. Understanding the architecture helps you pick the right one for your needs. ## Data mesh implementation using Hive views DevFeed: [Data mesh implementation using Hive views]() Original publisher: [Read original article]() Author: Alejandro Rojas, Cole Bowden Published: 2023-07-17T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [data-platforms](), [Authorization](), [Amazon S3](), [databricks](), [Apache Spark](), [Command-line interface]() Tags: [access-control](), [cli](), [data](), [data-platforms](), [databricks](), [s3](), [spark](), [storage]() ### AI overview The article recaps Comcast's data mesh implementation using Trino and Hive views. It describes a self-service data platform that federates storage systems and supports access through Trino, Databricks, and Spark, with centralized fine-grained access control. ### Source excerpt At Comcast, data is used in a data mesh ecosystem, with a vision where users can discover data and request data through a self-service platform. With federation, various tools, and the ability to create, read, and write data with different platforms, it's a full-blown data mesh. So how do you build that? With Trino, of course, and with the power of Hive views. Tune into the 10-minute lightning talk that Alejandro gave at Trino Fest to learn more about how Comcast pulled it off. ## Capturing Every Change From Shopify's Sharded Monolith DevFeed: [Capturing Every Change From Shopify's Sharded Monolith]() Original publisher: [Read original article]() Author: John Martin Published: 2021-03-12T18:51:39Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [data-platforms](), [Shopify](), [Kafka](), [data](), [big-data](), [Open Source](), [Apache Spark](), [Databases](), [Cloud](), [parquet](), [Amazon Redshift](), [BigQuery]() Tags: [analytics](), [big-data](), [bigquery](), [cloud](), [cloud-storage](), [data](), [data-platforms](), [data-science](), [databases](), [kafka](), [latency](), [low-latency](), [open-source](), [parquet](), [pipelines](), [platforms](), [shopify](), [spark](), [streaming]() ### AI overview Shopify describes how its data architecture evolved from an internal warehouse into a cloud-based data lake and separate platforms for internal workloads and merchant analytics. The article covers batch extraction, streaming change events, Apache Spark and dbt transformations, Kafka-based pipelines, and serving data through systems including BigQuery, Redshift, and Google Bigtable. ### Source excerpt Shopify is moving from existing query-based CDC (Change-Data Capture) tooling (Longboat) to an immutable, append-only, log-based mechanism. This has required a complete re-platforming, involving a shift to new, open source technology such as Kafka Connect and Debezium.