# data-platforms Published articles for data-platforms. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## We Cut Cloud Waste Before Touching Cluster Sizes: Lessons from Running a Data Platform DevFeed: [We Cut Cloud Waste Before Touching Cluster Sizes: Lessons from Running a Data Platform]() Original publisher: [Read original article]() Author: Deepika Saini Published: 2026-09-07T06:33:31Z Content type: article Language: en Sources: [Housing.com]() Topics: [BigQuery](), [databricks](), [Optimization](), [AWS Database Migration Service](), [data-platforms](), [Platform Engineering](), [migration](), [Cloud]() Tags: [bigquery](), [cloud-computing](), [cost](), [cost-optimization](), [data-engineering](), [data-platforms](), [databricks](), [delta-lake](), [finops](), [infrastructure](), [migration](), [optimization](), [platform-engineering]() ### AI overview This article explains how a data platform team reduced cloud costs by removing obsolete BigQuery data, adjusting Delta Lake retention, right-sizing DMS infrastructure, identifying unmonitored Databricks jobs, and standardizing pipeline onboarding and cost alerts. It reports that DMS costs were cut by over 50% and that retention was reduced from 90 days to 7 days for appropriate workloads after operational validation. ### Source excerpt How orphaned BigQuery storage, Delta retention, DMS right-sizing, and Databricks System Tables became our biggest cloud cost wins. The biggest cloud cost optimization we made wasn't shrinking clusters.It was deleting data we'd forgotten we were paying for.Like most teams, our first instinct was to tune infrastructure first. Instead, we discovered a treasure trove of hidden costs: orphaned BigQuery datasets, 90-day Delta retention, 24-hour jobs no one monitored, and DMS infrastructure that no longer matched business needs.We stopped treating cloud bills as a finance problem and started treating them as a platform engineering problem.30-second takeaway Why deleting forgotten data saved more than shrinking clusters. How we cut DMS costs by over 50%. How Databricks System Tables exposed hidden 24-hour jobs. How config.metadata standardized pipeline onboarding. How weekly Slack alerts turned cost optimization into a habit. Section 1: Storage Was Our Biggest Leak -- We Were Paying to Store Data Nobody Used This is the most overlooked cost on many data platforms. Storage duplication across platforms We had already migrated several workloads from BigQuery to Databricks. Large datasets were still sitting in BigQuery long after they had stopped serving production workloads - quietly generating storage costs month after month. Nothing failed. No alerts fired. Every month, we paid for storage that no longer served production workloads.A migration isn't complete until the old storage is decommissioned.The hidden cost of long retention The next surprise came from Delta Lake retention settings. Our workspace was configured to retain deleted table data and transaction history for 90 days to support time travel. Time travel is incredibly useful. But did every table need three months of historical recovery? Not really. We reduced retention to 7 days for appropriate workloads after validating operational needs. What changed immediately: Less storage tied up in deleted data. Faster clea ## PGConf India 2027 - Dates Announced and CFP Open DevFeed: [PGConf India 2027 - Dates Announced and CFP Open]() Original publisher: [Read original article]() Author: PGConf India Published: 2026-09-05T00:00:00Z Content type: news Language: en Sources: [PostgreSQL news]() Topics: [data](), [Replication]() Tags: [cloud](), [community](), [conference](), [data](), [data-platforms](), [disaster-recovery](), [monitoring](), [open-source](), [performance](), [postgres](), [security]() ### AI overview PGConf India 2027 will take place in Bengaluru from March 2-5, 2027. Its Call for Papers is open, with submissions due October 15, 2026, covering PostgreSQL topics including internals, performance, replication, cloud infrastructure, security, monitoring, production case studies, data platforms, extensions, and open source. ### Source excerpt Hey there, Mark your calendars: PGConf India 2027 is set for March 2-5, 2027 at the Sheraton Grand Hotel at Brigade Gateway, Bengaluru. The Call for Papers is open right now. Important dates CFP closes -- October 15, 2026 We tell you yes/no -- November 15, 2026 The actual conference -- March 2-5, 2027 What we're into Basically anything Postgres, including but not limited to: Core internals & features Performance & query tuning HA, replication & disaster recovery Cloud & infra Security & compliance Monitoring & ops Real production case studies ML, analytics & data platforms (pgvector fans, this means you) Extensions & ecosystem tools Community & open source To submit Give us a title, session length (25 or 40 min), your bio, and an abstract (≤500 words) on what you'll say and what people should walk away with. Submit your talk Ticket sales aren't open yet -- watch pgconf.in for that and everything else. See you in Bengaluru. The PGConf India Team https://pgconf.in Powered By ConfBase.io ## Rerouting the Stream: How Lyft Moved to the Apache Flink Operator DevFeed: [Rerouting the Stream: How Lyft Moved to the Apache Flink Operator]() Original publisher: [Read original article]() Author: Maheep Myneni Published: 2026-08-31T19:08:16Z Content type: article Language: en Sources: [Lyft Engineering - Medium]() Topics: [Apache Iceberg](), [Streaming](), [Kubernetes](), [Open Source](), [autoscaling](), [Tooling](), [Architecture & Design]() Tags: [apache](), [apache-flink](), [autoscaling](), [data-platforms](), [infrastructure](), [kubernetes](), [lyft](), [open-source](), [production](), [streaming](), [streaming-data-processing](), [tooling]() ### AI overview Lyft describes its migration from an internally developed Flink Kubernetes operator to the open-source Apache Flink Kubernetes Operator. The move addressed maintenance burden, technical debt, feature gaps, outdated dependencies, and growing real-time streaming demands, while providing capabilities such as autoscaling, memory tuning, safer upgrades, and automatic rollbacks. ### Source excerpt Written by Maheep Myneni, Arda Kuyumcu, and Prem Santosh Udaya Shankar at Lyft. Why We Migrated: Technical Debt Meets Modern Streaming Demands Over the past several quarters, Lyft's Streaming Compute team retired our internally developed Flink Kubernetes operator and moved our entire streaming fleet onto the open-source Apache Flink Kubernetes operator. This post is about why we made the switch, how we pulled it off incrementally without disrupting users, and the follow-on work it took to actually get the benefits we were after. Back in 2020, when we first architected the Lyft Flink Kubernetes Operator, it was exactly what we needed. At that time, the open-source community hadn't yet built a dedicated control plane, so we built our own to manage all streaming applications on Kubernetes. It worked well for our initial workloads, but as our real-time data needs increased and our engineers' scope of ownership grew in both breadth and complexity, the cracks started to show. First came the maintenance burden. Our operator had become a relic of Lyft's early Kubernetes days, kept alive by a growing pile of custom code. Every Flink version upgrade meant carefully picking through layers of accumulated technical debt and hoping nothing broke on the way through. Second came the feature gap. Streaming tooling kept moving, and our engineers kept asking for capabilities that had become table stakes elsewhere, such as autoscaling to right-size jobs, automatic rollbacks on failed deploys, and an end to hand-tuning CPU and memory. Each request left us with two options, neither of which was ideal. We could explain why we couldn't support it yet, or spend weeks rebuilding something the open-source community had already shipped. Third was the dependency problem. We were pinned to outdated libraries. That doesn't break anything today, but it almost always creates new issues down the line. Security patches lagged, modern Kubernetes features stayed out of reach, and every quarter we waite ## Using Data Contracts to Coordinate Data Evolution at Enterprise Scale DevFeed: [Using Data Contracts to Coordinate Data Evolution at Enterprise Scale]() Original publisher: [Read original article]() Author: Keerthipriyan Published: 2026-08-25T20:22:17Z Content type: article Language: en Sources: [Walmart Global Tech]() Topics: [data-platforms](), [Data Quality](), [DataOps](), [systems]() Tags: [architecture](), [data](), [data-engineering](), [data-governance](), [data-platforms](), [data-quality](), [data-science](), [enterprise](), [organizational](), [pipeline](), [schema](), [software-engineering](), [teams](), [technology]() ### AI overview The article explains how data contracts help large enterprises coordinate changes across independently evolving data teams and downstream consumers. It argues that schema validation alone cannot identify ownership, downstream impact, or migration responsibilities, and presents data contracts as machine-enforceable coordination agreements. ### Source excerpt Coauthored by Satyajeet Coordinating Data Evolution at Enterprise Scale When you operate data platforms on a global enterprise scale, hundreds of engineering teams ship improvements every week, each moving independently to deliver value at the pace of the business demands. This velocity is a competitive advantage. The challenge: How do you enable hundreds of teams to evolve their data products independently while maintaining reliability for thousands of downstream consumers? Traditional coordination methods (messages, wiki updates, shared spreadsheets) work at small scale but break at Walmart scale. A source team ships an enhancement, perfectly valid within their domain, but that change ripples through fifteen downstream pipelines owned by different teams with different release schedules. Without a formal coordination mechanism, you discover the impact after it reaches production. The gap isn't technical debt or fragile systems. It's the absence of machine-enforceable agreements that scale with organizational complexity. Data contracts solve this: enabling teams to move fast independently while maintaining coordinated reliability across organizational boundaries. Here's the architecture we built. Why Schema Validation Alone Isn't Enough When data quality issues surface in production, the first instinct is often added to more schema validation. If a field is missing or has the wrong type, the pipeline catches it. This works for many data quality problems, but not all of them. Consider a scenario where a source team enhances their data model by restructuring field names to support new business capabilities. The schema still validates perfectly: every field exists; every type is correct; the data is well formed. But downstream consumers who depend on the original field names now receive empty results. Schema validation checks whether data has the right shape. It tells you that a field is missing. It does not tell you who owns that field, which downstream teams will bre ## Data Engineering Weekly #282 DevFeed: [Data Engineering Weekly #282]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-08-10T01:21:26Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [data-platforms](), [data observability](), [Graphs](), [semantic-layer](), [Large Language Model](), [Apache Iceberg](), [gRPC](), [Concurrency](), [Caching](), [Optimization](), [Streaming]() Tags: [caching](), [chaos](), [concurrency](), [data](), [data-engineering](), [data-platforms](), [data-quality](), [knowledge-graphs](), [llms](), [newsletter](), [observability](), [pipelines](), [semantic-layer](), [streaming]() ### AI overview Data Engineering Weekly #282 is a newsletter covering data platform fundamentals, semantic layers, ontology-backed knowledge graphs, converged databases, AI modernization, and Netflix's real-time distributed graph query architecture. It highlights composable architectures, data quality and observability, evolving schemas supported by LLM-assisted extraction, Iceberg full-text search, and optimization techniques including concurrency control, streaming filters, and caching. ### Source excerpt The Weekly Data Engineering Newsletter ## Eric Schwartz on what it takes to run an AI SRE at petabyte scale DevFeed: [Eric Schwartz on what it takes to run an AI SRE at petabyte scale]() Original publisher: [Read original article]() Author: WorkOS Published: 2026-08-07T00:00:00Z Content type: article Language: en Sources: [WorkOS Blog]() Topics: [Artificial Intelligence](), [site-reliability-engineering](), [data-platforms](), [telemetry](), [observability ai agents]() Tags: [ai](), [data](), [data-platforms](), [on-call](), [production](), [root-cause-analysis](), [sre](), [telemetry]() ### AI overview An interview with Traversal product manager Eric Schwartz examines how the company operates an AI site reliability engineer for large enterprises. The article explains that petabyte-scale telemetry requires continuously analyzing, compressing, and indexing data ahead of runtime, with an SRE-focused tool and prompt harness. Traversal reports that deployments are generally running in production within a week with minimal tuning, supported by forward-deployed engineering for last-mile optimization. ### Source excerpt Traversal PM Eric Schwartz on data platforms, routing models by severity, and the permission ladder toward self-driving production, from AI Engineer 2026. ## The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering DevFeed: [The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-08T12:16:09Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [pandas](), [data-engineering](), [Artificial Intelligence](), [DuckDB](), [data-platforms](), [software-development](), [future of software](), [Apache Spark](), [Hadoop](), [Open Source](), [parquet]() Tags: [ai](), [apache-arrow](), [arrow](), [data](), [data-engineering](), [data-platforms](), [duckdb](), [hadoop](), [large-language-models](), [open-source](), [pandas](), [parquet](), [software-development]() ### AI overview An interview with Wes McKinney covers the origins of pandas and Apache Arrow, the evolution of modern data engineering from Hadoop to lakehouse architectures, and the roles of tools such as Parquet, DuckDB, DataFusion, and Spark. McKinney also discusses how AI affects software development, arguing that it can improve experienced engineers' productivity but does not replace software engineering, architecture, or judgment. ### Source excerpt interview with Wes McKinney ## Introducing CostBench: an open benchmark for data warehouse cost-performance DevFeed: [Introducing CostBench: an open benchmark for data warehouse cost-performance]() Original publisher: [Read original article]() Author: Tom Schreiber; Lionel Palacin Published: 2026-05-27T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [Benchmark](), [benchmarking](), [Cloud](), [data-platforms](), [real-time](), [SQL]() Tags: [benchmark](), [benchmarks](), [cloud](), [cost](), [data-platforms](), [performance](), [real-time](), [sql]() ### AI overview CostBench is an open benchmark for comparing cloud data warehouses by performance per dollar rather than speed alone. ### Source excerpt Introducing CostBench, an open benchmark that turns cloud data warehouse runtime and billing models into comparable performance-per-dollar results. ## Gala supercharges analytics performance with ClickHouse on AWS DevFeed: [Gala supercharges analytics performance with ClickHouse on AWS]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-05-04T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Amazon Web Services (AWS)](), [data analytics](), [Data Infrastructure](), [data-platforms](), [telemetry](), [databricks](), [Blockchain]() Tags: [aws](), [blockchain](), [clickhouse](), [data-analytics](), [data-infrastructure](), [data-platforms](), [databricks](), [telemetry]() ### AI overview Gala migrated from Databricks to ClickHouse on AWS to address growing data volumes and improve analytics performance. The article reports a threefold increase in analytics capacity, a 30% cost reduction, and query times reduced from minutes to sub-second. ### Source excerpt Learn how Gala migrated to the ClickHouse to improve analytics performance and cut costs ## AI is redrawing the database market DevFeed: [AI is redrawing the database market]() Original publisher: [Read original article]() Author: Tanya Bragin Published: 2026-03-17T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [Artificial Intelligence](), [clickhouse](), [Database](), [data-platforms](), [real-time](), [observability](), [dashboards]() Tags: [agentic](), [ai](), [analytics](), [clickhouse](), [dashboards](), [data](), [data-platforms](), [database](), [developers](), [observability](), [real-time]() ### AI overview AI is reshaping database platform requirements, driving demand for high-concurrency, real-time query processing, and full-fidelity data at scale. The article examines how real-time analytics, data warehousing, and observability are converging at the data layer, positioning ClickHouse as a unified platform for AI workloads. ### Source excerpt AI is reshaping database market requirements across real-time analytics, data warehousing, and observability -- and ClickHouse is uniquely positioned to meet them. ## 300-Node Clusters Now Supported in CockroachDB DevFeed: [300-Node Clusters Now Supported in CockroachDB]() Original publisher: [Read original article]() Author: David Bressler,Dipti Joshi Published: 2026-02-12T00:00:00Z Content type: release Language: en Sources: [Cockroach Labs]() Topics: [CockroachDB](), [Cockroach Labs](), [data-platforms](), [Benchmark](), [Cloud](), [Resilience]() Tags: [backups](), [benchmark](), [cloud](), [cockroach-labs](), [cockroachdb](), [data-platforms](), [distributed](), [performance](), [resilience](), [sql]() ### AI overview Cockroach Labs announces support for 300-node clusters in CockroachDB v25.4.4 and later, with up to 2.2M tpmC and 1.2PB of data. The article also describes testing involving operational stress and reports performance results from that test cycle. ### Source excerpt As AI-driven and agentic applications push data platforms into new territory, data architects are increasingly forced to choose between correctness, simplicity, and scale. Today, we're removing that tradeoff -- announcing support for 300-node clusters with 2.2M tpmC and 1.2PB of data in CockroachDB v25.4.4 and beyond. On CockroachDB Cloud, we're announcing support for 64 vCPU per node. Testing to this scale is important to ensure that we're testing ahead of customer deployments, and this milestone does just that. We believe these tests represent the largest node-scale testing completed by any distributed SQL vendor. And we're just getting started. ## Introducing OpenAI Frontier DevFeed: [Introducing OpenAI Frontier]() Original publisher: [Read original article]() Published: 2026-02-05T06:00:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [OpenAI](), [Artificial Intelligence](), [Authorization](), [data-platforms](), [systems]() Tags: [agents](), [ai](), [ai-agents](), [applications](), [building](), [customers](), [data](), [data-platforms](), [enterprise](), [governance](), [openai](), [platform](), [product](), [systems](), [technology](), [work]() ### AI overview OpenAI introduces Frontier, an enterprise platform for building, deploying, and managing AI agents. The platform is designed to help organizations move from isolated use cases to AI coworkers by providing shared context, onboarding, learning through feedback, and clear permissions and boundaries. ### Source excerpt OpenAI Frontier is an enterprise platform for building, deploying, and managing AI agents with shared context, onboarding, permissions, and governance. ## ClickHouse raises $400M Series D led by Dragoneer to accelerate expansion across analytics and AI infrastructure DevFeed: [ClickHouse raises $400M Series D led by Dragoneer to accelerate expansion across analytics and AI infrastructure]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-01-16T00:00:00Z Content type: news Language: en Sources: [ClickHouse Blog]() Topics: [ai observability]() Tags: [ai](), [ai-infrastructure](), [ai-ml](), [analytics](), [applications](), [clickhouse](), [cloud](), [cost](), [data](), [data-infrastructure](), [data-platforms](), [developers](), [evaluation](), [infrastructure](), [latency](), [llm](), [observability](), [performance](), [postgres](), [series]() ### AI overview ClickHouse announced a $400 million Series D, its acquisition of Langfuse for LLM observability, and a native Postgres service for unified transactional and analytical workloads. ### Source excerpt ClickHouse acquires Langfuse to enter LLM observability and introduces a native Postgres service to unify transactional and analytical workloads ## Why streaming is the backbone for AI-native data platforms DevFeed: [Why streaming is the backbone for AI-native data platforms]() Original publisher: [Read original article]() Author: Tyler Rockwood Published: 2025-06-24T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Streaming](), [data-platforms](), [Artificial Intelligence](), [AI Development](), [Data analysis](), [Embeddings](), [SQL](), [dashboards](), [personalization](), [Development]() Tags: [ai](), [ai-agents](), [ai-and-data-warehouse-evolution](), [ai-automation-in-products](), [ai-data-analysis-automation](), [ai-enhanced-product-development](), [ai-native-data-platforms](), [ai-product-integration](), [ai-technology-integration](), [dashboards](), [data-analysis](), [data-platforms](), [data-platforms-for-ai-scalability](), [development](), [embeddings](), [modern-data-warehouse-ai](), [personalization](), [real-time-ai-personalization](), [sql](), [streaming](), [streaming-for-ai-data-platforms](), [structuring-data-lakes-with-ai](), [thought-leadership]() ### AI overview The article argues that streaming is the backbone of AI-native data platforms. It presents the modern data warehouse as a central source of context for AI and describes how AI can support analysis, dashboards, SQL queries, trend detection, embeddings, personalization, recommendations, automation, and faster product development. It also emphasizes guardrails and performance measurement for AI-driven automation. ### Source excerpt To power AI at scale, organizations must adapt to the evolving role of the modern data warehouse. Here's what you need to keep up in a rapidly evolving industry. ## Building on open table formats DevFeed: [Building on open table formats]() Original publisher: [Read original article]() Author: Prashant Sridharan Published: 2025-05-29T07:00:00Z Content type: article Language: en Sources: [Supabase Blog]() Topics: [Apache Iceberg](), [data](), [datasets](), [schema-evolution](), [Amazon S3](), [cloud-infrastructure](), [Databases](), [parquet](), [systems](), [Filesystems](), [Streaming](), [Supabase]() Tags: [amazon-s3](), [analytics](), [apache-iceberg](), [compatibility](), [data](), [data-platforms](), [data-processing](), [databricks](), [datasets](), [object-storage](), [open](), [parquet](), [schema-evolution](), [spark](), [storage](), [streaming](), [systems](), [transactions]() ### AI overview The article explains how open table formats organize and manage large datasets on distributed storage. It compares Apache Iceberg, Delta Lake, and Apache Hudi, then focuses on Apache Iceberg's support for analytics, ACID transactions, schema evolution, partition evolution, hidden partitioning, time travel, and safe concurrent access across processing engines. Iceberg operates over Parquet and other data files in object storage such as Supabase Storage, Amazon S3, Google Cloud Storage, and Azure Blob Storage. ### Source excerpt Open table formats like Apache Iceberg, Delta Lake, and Apache Hudi are transforming how developers manage large-scale data on object storage systems. ## March 2025 Newsletter DevFeed: [March 2025 Newsletter]() Original publisher: [Read original article]() Author: Mark Needham Published: 2025-03-20T00:00:00Z Content type: news Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [real-time](), [Apache Iceberg](), [Data Infrastructure](), [data analytics](), [Large Language Model](), [Traces](), [Grafana](), [BigQuery]() Tags: [2025](), [apache-iceberg](), [aws](), [bigquery](), [clickhouse](), [conference](), [data](), [data-analytics](), [data-infrastructure](), [data-platforms](), [llms](), [newsletter](), [observability](), [real-time](), [traces](), [training]() ### AI overview The March 2025 ClickHouse newsletter covers recent product updates, including the Postgres CDC connector for ClickPipes entering Public Beta and general availability for Bring Your Own Cloud on AWS. It also highlights ClickHouse support for Apache Iceberg, data warehouse use cases, community work on an AI gateway using ClickHouse, and upcoming events, training, and workshops. ### Source excerpt Welcome to the March ClickHouse newsletter, which will round up what's happened in real-time data warehouses over the last month. ## Failover Replication Slots with Postgres 17 DevFeed: [Failover Replication Slots with Postgres 17]() Original publisher: [Read original article]() Published: 2024-12-03T00:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Replication](), [Availability](), [Database](), [data-platforms]() Tags: [availability](), [data](), [data-platforms](), [database](), [debezium](), [failover](), [read-replica](), [replication]() ### AI overview This article explains how Postgres 17 introduces built-in failover slots. Replication slots created on a primary are automatically propagated to read replicas, allowing consumers such as Debezium to continue after failover without missing change events or creating a new initial snapshot. ### Source excerpt Table of Contents Hello, Failover Slots! Failover Slots in Decodable Wrapping Up This post originally appeared on the Decodable blog. All rights reserved. Postgres read replicas are commonly used not only to distribute query load amongst multiple nodes, but also to ensure high availability (HA) of the database. If the primary node of a Postgres cluster fails, a read replica can be promoted to be the new primary, processing write (and read) requests from thereon. ## Building a Data Pipeline to Track Strava's Bad Events DevFeed: [Building a Data Pipeline to Track Strava's Bad Events]() Original publisher: [Read original article]() Author: Bisman Sodhi Published: 2024-01-08T20:19:46Z Content type: opinion Language: en Sources: [Strava Engineering]() Topics: [data](), [airflow](), [Amazon S3](), [Python](), [Amazon Web Services](), [JSON](), [dashboards](), [Computer science]() Tags: [airflow](), [apache-airflow](), [aws](), [data](), [data-engineering](), [data-pipeline](), [data-platforms](), [integrity](), [json](), [python](), [s3](), [software-engineering](), [strava](), [tableau](), [warehouse]() ### AI overview A software engineering intern describes building a daily Apache Airflow pipeline that extracts schema-invalid user behavior events from S3, decompresses them into JSON, and loads them into Snowflake. Staging tables protect production data from partial loads, while materialized SQL views and a Tableau dashboard improve querying and monitoring. ### Source excerpt Hi my name is Bisman and I studied Computer Science at University of California, Santa Barbara. During summer of 2022, I had the most amazing experience working as a Software Engineer Intern on Strava's Data Platform Team. In the first fews weeks, I learned the tools my team uses and then spent the rest of the time working on my project. TRACKING BAD EVENTS For my major summer project, I created a data pipeline that pulls user behavior data out of external storage and persists it in our data warehouse. Strava uses a service called Snowplow to collect this user behavior data, like loading a club page or uploading a profile photo. Sometimes, this data fails to match the schema that we've set, and a piece of data that fails this schema validation is called a bad event. Previously, these bad events were temporarily stored in an Elastic Search. Persisting this data in Snowflake, our data warehouse, makes it accessible to a wider audience. It also makes it easier to incorporate the bad events data with other services used at Strava. To start my project, I created a directed acyclic graph in Apache Airflow, a scheduling framework, using python that extracts bad events data from the S3, AWS's storage service, buckets on a daily cadence. This data was stored as gzip files on S3 which I decompressed and stored the data as JSON blobs. As I was working with billions of rows of data, it was important to maintain data integrity and take measures in case data failed to load from S3. Therefore, I loaded data into a staging table in Snowflake. The staging table ensured that if loading from S3 failed, the production table would remain untouched. This data was then loaded into the production table free of any partial data. After all the data was loaded into the production table, I created six view tables because there were six different types of bad events stored in the production table. I collaborated with our stakeholders -- data analysts -- throughout this process to craft tables bas ## Real-time data platforms explained: components & benefits DevFeed: [Real-time data platforms explained: components & benefits]() Original publisher: [Read original article]() Author: Cameron Archer Published: 2023-07-27T00:00:00Z Content type: article Language: en Sources: [Tinybird]() Topics: [data-platforms](), [real-time](), [data](), [Streams]() Tags: [architecture](), [components](), [data](), [data-platforms](), [real-time](), [scalable-analytics-architecture](), [streams]() ### AI overview An overview of real-time data platforms, focusing on how they process incoming streams and how their architecture can help readers choose a platform for their needs. ### Source excerpt Real-time data platforms process streams as they arrive. Understanding the architecture helps you pick the right one for your needs. ## Data mesh implementation using Hive views DevFeed: [Data mesh implementation using Hive views]() Original publisher: [Read original article]() Author: Alejandro Rojas, Cole Bowden Published: 2023-07-17T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [data-platforms](), [Authorization](), [Amazon S3](), [databricks](), [Apache Spark](), [Command-line interface]() Tags: [access-control](), [cli](), [data](), [data-platforms](), [databricks](), [s3](), [spark](), [storage]() ### AI overview The article recaps Comcast's data mesh implementation using Trino and Hive views. It describes a self-service data platform that federates storage systems and supports access through Trino, Databricks, and Spark, with centralized fine-grained access control. ### Source excerpt At Comcast, data is used in a data mesh ecosystem, with a vision where users can discover data and request data through a self-service platform. With federation, various tools, and the ability to create, read, and write data with different platforms, it's a full-blown data mesh. So how do you build that? With Trino, of course, and with the power of Hive views. Tune into the 10-minute lightning talk that Alejandro gave at Trino Fest to learn more about how Comcast pulled it off. ## Capturing Every Change From Shopify's Sharded Monolith DevFeed: [Capturing Every Change From Shopify's Sharded Monolith]() Original publisher: [Read original article]() Author: John Martin Published: 2021-03-12T18:51:39Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [data-platforms](), [Shopify](), [Kafka](), [data](), [big-data](), [Open Source](), [Apache Spark](), [Databases](), [Cloud](), [parquet](), [Amazon Redshift](), [BigQuery]() Tags: [analytics](), [big-data](), [bigquery](), [cloud](), [cloud-storage](), [data](), [data-platforms](), [data-science](), [databases](), [kafka](), [latency](), [low-latency](), [open-source](), [parquet](), [pipelines](), [platforms](), [shopify](), [spark](), [streaming]() ### AI overview Shopify describes how its data architecture evolved from an internal warehouse into a cloud-based data lake and separate platforms for internal workloads and merchant analytics. The article covers batch extraction, streaming change events, Apache Spark and dbt transformations, Kafka-based pipelines, and serving data through systems including BigQuery, Redshift, and Google Bigtable. ### Source excerpt Shopify is moving from existing query-based CDC (Change-Data Capture) tooling (Longboat) to an immutable, append-only, log-based mechanism. This has required a complete re-platforming, involving a shift to new, open source technology such as Kafka Connect and Debezium.