# spark Published articles for spark. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Unifying governance across engines and catalogs in the Open Lakehouse DevFeed: [Unifying governance across engines and catalogs in the Open Lakehouse]() Original publisher: [Read original article]() Author: Daniel Weeks; Ryan Blue; Andrei Tserakhau Published: 2026-09-10T15:05:10Z Content type: article Language: en Sources: [Databricks]() Topics: [Apache Iceberg](), [data](), [databricks](), [Apache Spark](), [DuckDB]() Tags: [apache](), [apis](), [catalog](), [data](), [databricks](), [duckdb](), [governance](), [open](), [product](), [spark]() ### AI overview The article examines two additions to the Apache Iceberg REST Catalog specification: read restrictions and catalog labels. It explains how they support governance across catalogs and engines, including centralized and delegated enforcement, and discusses trust requirements for engines such as Spark, DuckDB, and Trino. ### Source excerpt In our previous posts, we showed how open table formats, open APIs and unified governance... ## Delivering Real-Time Personalization with Databricks and Redis DevFeed: [Delivering Real-Time Personalization with Databricks and Redis]() Original publisher: [Read original article]() Author: Philip Laussermair, Anant Pingle Published: 2026-09-08T00:00:00Z Content type: article Language: en Sources: [Redis Blog]() Topics: [data]() Tags: [analytics](), [apache-flink](), [batch](), [data](), [databricks](), [e-commerce](), [performance](), [personalization](), [real-time](), [redis](), [spark](), [streaming](), [tech]() ### AI overview The article explains how Databricks Real-Time Mode and Redis support low-latency personalization by continuously processing event streams and serving fresh results quickly. ### Source excerpt Why real-time matters A customer is browsing an e-commerce site. They search for running shoes, open a product, read reviews, and add an item to the cart. Every one of those actions is a signal about what they want right now. If the homepage they lan... ## Muse Spark 1.3 now available on AI Gateway DevFeed: [Muse Spark 1.3 now available on AI Gateway]() Original publisher: [Read original article]() Author: Jerilyn Zheng Published: 2026-09-02T00:00:00Z Content type: release Language: en Sources: [Vercel News]() Topics: [AI Chat]() Tags: [agents](), [ai](), [ai-gateway](), [claude](), [claude-code](), [codex](), [coding](), [cursor](), [meta](), [models](), [pdf](), [playground](), [release](), [spark](), [vercel]() ### AI overview Muse Spark 1.3 from Meta is now available on AI Gateway in standard and contributor pricing tiers. The model adds a 1M-token context window and accepts text, image, and PDF input; the contributor tier uses the same model but allows Meta to use submitted inputs and outputs for model improvement in exchange for lower pricing. ### Source excerpt Muse Spark 1.3 from Meta is now available on AI Gateway, in both the standard and contributor pricing tiers. This model improves on prior Muse Spark models at agent work and coding, with a 1M token context window and text, image, and PDF input. On coding it takes fewer turns and writes less filler than the previous release. To use Muse Spark 1.3, set model to meta/muse-spark-1.3: Contributor tier Muse Spark 1.3 Contributor is a pricing tier on the same model rather than a separate one, with the same weights, capabilities, and context window. The difference is that Meta uses the inputs and outputs sent to this tier to train and improve its models, and pricing is lower in exchange. Model Input Output Cached input meta/muse-spark-1.3 $1.25 $4.25 $0.15 meta/muse-spark-1.3-contributor $0.10 $0.20 $0.002 Rates are per million tokens and unchanged from Muse Spark 1.2 on both tiers. To use it in a coding agent, see the coding agents guide, then run vercel ai-gateway coding-agents setup to connect Claude Code, Codex, Cursor, and more, then select meta/muse-spark-1.3 in the agent. Try Muse Spark 1.3 in the model playground. You can view all language models available on AI Gateway. Read more ## AWS Glue 6.0 now available with 30% lower price and full Apache Iceberg v3 support DevFeed: [AWS Glue 6.0 now available with 30% lower price and full Apache Iceberg v3 support]() Original publisher: [Read original article]() Author: Channy Yun (윤석찬) Published: 2026-08-21T18:53:26Z Content type: release Language: en Sources: [AWS News Blog]() Topics: [Apache Iceberg](), [Amazon Web Services](), [Apache Spark](), [Python](), [Scala](), [Geographic Information System](), [Internet of things]() Tags: [analytics](), [apache](), [apache-iceberg](), [aws](), [aws-glue](), [launch](), [news](), [performance](), [pricing](), [python-3-13](), [real-time](), [release](), [scala](), [spark](), [streaming]() ### AI overview AWS Glue 6.0 is generally available with 30% lower pricing, a modernized Spark 4.1 runtime, Python 3.13 and Scala 2.13 support, and full Apache Iceberg v3 support. The release adds improved handling of semi-structured data, declarative ETL pipelines, faster PySpark execution, and real-time streaming with single-digit millisecond latency. ### Source excerpt AWS Glue 6.0 is built on a fully modernized runtime, Apache Spark 4.1, Python 3.13, and Scala 2.13, delivering 30% lower pricing than previous AWS Glue versions. ## Lakesail: A Rust-based alternative to Spark without the JVM DevFeed: [Lakesail: A Rust-based alternative to Spark without the JVM]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-08-18T21:11:44Z Content type: opinion Language: en Sources: [Data Engineering Central]() Topics: [Apache Spark](), [Rust]() Tags: [apache-spark](), [rust](), [spark]() ### AI overview This commentary article introduces Lakesail as a Rust-based implementation or alternative to Spark that avoids the JVM overhead. It places Lakesail within the broader data-engineering ecosystem, where established tools such as Spark and Kafka continue to coexist with newer projects including Polars, DuckDB, and Daft. ### Source excerpt ya know ... black magic stuff ## Data pipeline monitoring 101: Tracking health and performance across the data stack DevFeed: [Data pipeline monitoring 101: Tracking health and performance across the data stack]() Original publisher: [Read original article]() Author: Aaron Kaplan; Ryan Warrier Published: 2026-08-14T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [Monitoring](), [data observability](), [observability](), [data](), [data streams monitoring](), [Kafka](), [Apache Spark](), [event driven](), [Streaming]() Tags: [ai-ml](), [analytics](), [data](), [data-analytics](), [data-engineering](), [data-observability](), [data-streams-monitoring](), [event-driven](), [kafka](), [learn](), [monitoring](), [observability](), [performance](), [spark](), [streaming]() ### AI overview This article introduces end-to-end monitoring for modern data pipelines. It explains how to track pipeline health, performance, data quality, and availability across varied architectures and technology layers, with examples including Kafka, Flink, Apache Spark, data lakes, warehouses, and lakehouses. ### Source excerpt Learn about monitoring the end-to-end health and performance of modern data pipelines. ## Muse Spark 1.2 is now available on Vercel AI Gateway DevFeed: [Muse Spark 1.2 is now available on Vercel AI Gateway]() Original publisher: [Read original article]() Author: Jerilyn Zheng Published: 2026-08-05T00:00:00Z Content type: release Language: en Sources: [Vercel News]() Topics: [AI-assisted coding](), [vercel ai sdk](), [debugging]() Tags: [agent](), [ai-gateway](), [api](), [code-generation](), [debugging](), [muse](), [spark](), [vercel]() ### AI overview Muse Spark 1.2 is available on Vercel AI Gateway as a coding-focused model update, with stated improvements in code generation, debugging, codebase understanding, and iterative developer workflows. ### Source excerpt Muse Spark 1.2 from Meta is now available on AI Gateway. It is a coding-focused update to the previous Muse Spark model. While keeping its general capabilities, 1.2 ships with improvements in code generation, complex debugging, codebase understanding, and end-to-end developer workflows. The model is built for long-horizon work like generating whole repositories, building out large projects end to end, and sustaining iterative loops where it writes, compiles, profiles, and improves code over many rounds. To use Muse Spark 1.2, set model to meta/muse-spark-1.2 in the AI SDK: To use Muse Spark in a coding agent, run vercel ai-gateway coding-agents setup to connect Claude Code, Codex, OpenCode, or Pi, then select meta/muse-spark-1.2 inside the agent. AI Gateway provides a unified API for calling models, tracking usage and cost, and configuring retries, failover, and performance optimizations for higher-than-provider uptime. It includes built-in custom reporting, Zero Data Retention support, budgets for API keys, routing rules, and more. AI Gateway reflects provider pricing with no markup and does not charge a platform fee on inference, including on Bring Your Own Key (BYOK) requests. Try Muse Spark 1.2 in the model playground. Read more ## Batch Jobs for SparkClient: Submitting and Managing Spark Workloads from Python DevFeed: [Batch Jobs for SparkClient: Submitting and Managing Spark Workloads from Python]() Original publisher: [Read original article]() Author: Sameer Yadav Published: 2026-07-25T05:00:00Z Content type: tutorial Language: en Sources: [Kubeflow]() Topics: [Apache Spark](), [SDKs](), [Python](), [Kubernetes]() Tags: [batch](), [cleanup](), [gsoc](), [jobs](), [kubernetes](), [logs](), [pipelines](), [python](), [scheduled](), [sdk](), [spark]() ### AI overview This tutorial explains how the Kubeflow SDK's SparkClient supports submitting and managing batch Spark workloads on Kubernetes from Python. It covers script- and function-based jobs, lifecycle operations, log retrieval, cleanup, and the implementation's current boundaries. ### Source excerpt How the SparkClient SDK's new batch job APIs work under the hood -- submit_job(), FileJob/FuncJob, the lifecycle APIs, and log retrieval. ## How We Refresh Razorpay's Data Warehouse 10x Faster with Graphs and Indexes DevFeed: [How We Refresh Razorpay's Data Warehouse 10x Faster with Graphs and Indexes]() Original publisher: [Read original article]() Author: Amit Prabhu Published: 2026-07-14T14:06:16Z Content type: article Language: en Sources: [Razorpay Engineering - Medium]() Topics: [data](), [airflow](), [Apache Spark](), [Amazon S3](), [Apache Iceberg](), [microservices architecture](), [parquet](), [Architecture & Design]() Tags: [airflow](), [analytics](), [apache-iceberg](), [architecture](), [batch](), [data](), [graphs](), [infrastructure](), [parquet](), [pipelines](), [razorpay](), [spark](), [trino](), [warehouse]() ### AI overview Razorpay describes its data warehouse refresh pipeline, which builds wide denormalized Facts by joining data from multiple microservices. The article covers the original Airflow- and Spark-based full-refresh process, the underlying lake formats and query layer, and the scaling challenges that led the team to reconsider refresh strategy, data layout, and high-cardinality dimensions. ### Source excerpt Contributors: Utkarsh Koppikar Rohan Background Razorpay provides the payment infrastructure for millions of merchants globally. Behind every payment, settlement, and refund is a microservices architecture where each service owns its own database. While this keeps services independent and scalable, it creates a challenge for stakeholders who need to see across those boundaries. The Data Platform team manages the infrastructure that bridges this gap. Transactional data flows into the lake via CDC pipelines, ingested onto S3 in Delta Lake, Apache Iceberg, or plain Parquet formats. On top of the lake, we build domain-specific warehouse tables -- wide, pre-joined tables that co-locate all the data a consumer needs, queryable via Trino. These power two use cases: Analytics (internal dashboards on Tableau and Superset) and Reporting (merchants and regulated entities who download structured data exports; Razorpay generates nearly a million such reports per month). The warehouse tables that power both use cases are called Facts. A Fact is a flat denormalised table on S3, produced by joining 10 to 30 microservice tables and materialising the result once. A settlement Fact, for example, merges payments, refunds, adjustments, and card details into a single wide row so that a dashboard or report reads from a single table instead of joining across services in real time. It is closer to a domain-specific materialised view than a classical data warehouse fact table. We maintain over 50 such Facts, and approximately 40% of all merchant reports are served directly from them. As data volumes and the number of entities per fact grew, the batch generation pipeline began to show its limits, prompting us to rethink the refresh strategy, the data layout, and how to handle high-cardinality dimensions. The rest of this post covers that journey. The Full Refresh Pipeline: Our Baseline and the Pain The original full-refresh pipeline was straightforward. Schedule: Airflow schedules Spark jobs o ## Training Orchestrator: Unifying Model Training at Yelp DevFeed: [Training Orchestrator: Unifying Model Training at Yelp]() Original publisher: [Read original article]() Author: Ying Wang and Nathan Sponberg, Software Engineer Published: 2026-07-14T00:00:00Z Content type: article Language: en Sources: [Yelp]() Topics: [Machine Learning & Artificial Intelligence](), [Orchestration](), [Apache Spark](), [Training AI Models](), [Model Development](), [configuration](), [configuration-management](), [Monitoring]() Tags: [configuration](), [machine-learning](), [maintenance](), [mlflow](), [model-training](), [orchestration](), [reproducibility](), [spark](), [tooling]() ### AI overview Yelp's Core Machine Learning Team developed Training Orchestrator to standardize how machine learning teams define and run Spark-based model training. The configuration-driven system addresses duplicated code, inconsistent configurations, limited local testing, scattered validation and monitoring, and poor reproducibility across environments. ### Source excerpt At Yelp, we train many machine learning models on different schedules. Applied machine learning teams all have their own set of Spark-based training batches, scripts, and configurations. Over time, these diverged, leading to duplicated code, subtle inconsistencies, and a growing maintenance burden. Yelp's Core Machine Learning Team has developed excellent tooling across our ML ecosystem over the years: feature stores for reproducible data, a unified training library for neural networks and gradient-boosted trees, seamless Spark integration, and MLflow services for model tracking and deployment. But there was still one key piece missing right in the middle: a standardized way to... ## Inside Cilium CNI: solving mysterious Kubernetes pod setup timeouts DevFeed: [Inside Cilium CNI: solving mysterious Kubernetes pod setup timeouts]() Original publisher: [Read original article]() Author: Adyen Published: 2026-07-13T09:17:40Z Content type: article Language: en Sources: [Adyen Tech]() Topics: [Cilium](), [Kubernetes](), [Kubernetes networking](), [networking](), [eBPF](), [Latency](), [hdfs](), [Apache Spark](), [big-data](), [cpu]() Tags: [big-data](), [cilium](), [cni](), [cpu](), [data](), [devops](), [ebpf](), [hdfs](), [kubernetes](), [kubernetes-networking](), [latency](), [spark](), [speed](), [tracing]() ### AI overview An Adyen engineer investigates Cilium CNI timeouts that prevented new Kubernetes pods from starting. The article attributes the issue to sequential connection-tracking-table garbage collection, which became a bottleneck under workloads involving HDFS, Trino, and short-lived Spark jobs. ### Source excerpt By Jorrick Sleijster - Senior Data Platform Engineer, Adyen I was fully aware a year ago that a single configuration line could break the Kubernetes networking stack. But if they told me that leftovers from Kubernetes pods which terminated hours prior could block new ones from starting, I would have thought they were joking. In high-performance networking, 35 seconds is a lifetime. This was the latency required to iterate through our connection tracking table of 7 million entries at a maximum speed of 200,000 entries per second. At our 16-million-entry peak, this sequential lookup could take up to 80 seconds, leading to Cilium CNI timeouts preventing new pods from starting on affected nodes. We uncovered this linear-time behavior at Adyen by tracing syscalls, inspecting codebases, and analyzing eBPF internals. This investigation revealed how our varied workloads turned the connection tracking table's garbage collection algorithm into a critical bottleneck. Our setup: why we're different At Adyen, we run Cilium CNI across all our 100+ Kubernetes clusters. When we switched from Calico to Cilium, we knew we'd face challenges adapting it to our production workloads. Our production big data Kubernetes clusters have a unique usage pattern compared to the other Kubernetes environments within Adyen: Data extraction from HDFS. Our infrastructure relies on more than 500 datanodes. Trino represents one of our most demanding HDFS workloads, processing analytical queries against data stored on HDFS. Due to the distributed nature of HDFS, each file you download requires a new connection to any of these 500 nodes. Therefore, during peak hours, a single pod can produce approximately 50,000 connections every minute. Pod churn. Many pods we spawn on the Kubernetes cluster run batch jobs, such as Spark jobs. They stay around for anywhere from a second to a couple of hours. Wide variety of workloads. Some workloads are very CPU-intensive, like Spark pods executing complex joins and tra ## Data Engineering Weekly #278 DevFeed: [Data Engineering Weekly #278]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-07-13T02:44:42Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [AI, ML & Data Engineering](), [Apache Spark](), [Kafka](), [Data Quality]() Tags: [agents](), [ai](), [data-engineering](), [data-pipeline](), [data-quality](), [engineering](), [kafka](), [releases](), [spark]() ### AI overview Data Engineering Weekly #278 is a curated newsletter covering AI-assisted engineering workflows, new leetdata.ai features, agent-oriented data systems, multilingual AI concerns, visualization with Flint, and data platform fundamentals. ### Source excerpt The Weekly Data Engineering Newsletter ## Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption DevFeed: [Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption]() Original publisher: [Read original article]() Author: Rahul Penti Published: 2026-07-10T00:00:00Z Content type: article Language: en Sources: [Grab Tech]() Topics: [Apache Iceberg](), [data](), [Amazon S3](), [parquet](), [Apache Spark](), [Latency](), [Concurrency]() Tags: [apache-iceberg](), [compute](), [concurrency](), [data](), [database](), [engineering](), [latency](), [parquet](), [performance](), [s3](), [scale](), [spark]() ### AI overview Grab describes its transition from Hive Parquet and a directory-based Hive Metastore architecture toward Apache Iceberg and a table-centric lakehouse. The article outlines scalability problems involving metadata latency, small files, S3 request overhead, and operational management, and mentions a unified Spark catalog built to hide table-format differences. ### Source excerpt Introduction: The evolution of Grab's Data Lake At Grab's scale, managing petabytes of data across billions of S3 objects demands more than a storage layer. It demands a robust architectural primitive that supports the high-concurrency needs of a modern "Lakehouse." Our goal is full storage-compute separation, leveraging S3 as an elastic foundation for both near-real-time metrics and large-scale batch transformations. For years, the vast majority of our tables were Hive Parquet, managed through the Hive Metastore with a directory-based layout. This model served us well, but as data volume grew, the directory-and-metastore approach became the limiting factor. We are now transitioning to a table-centric architecture built on modern table formats, treating data as a first-class primitive to ensure consistency and performance across our internal data transformation platforms: Slide, which powers batch transformations, and Hugo, which handles online-to-data-lake ingestion. Along the way, we also built the UnifiedSparkCatalog, a unified Spark catalog that hides table-format differences from users entirely, which we are open-sourcing alongside this post. The catalyst for change: Challenges with Hive Parquet For years, Hive Parquet was the backbone of our Data Lake, representing the vast majority of our tables. However, as data volume scaled, the architectural limitations of directory-based storage became apparent. We identified four primary bottlenecks: Catalog latency: The Hive Metastore (HMS) became a centralized failure point. High concurrency during metadata access led to O(n) listing overhead, where query planning time scaled linearly with partition count, crippling throughput. The small file problem: The directory layout left us with severe file fragmentation. Certain Machine Learning (ML) datasets had an average file size under 1 MB, with thousands of files in each partition. At this scale, the overhead of S3 object listing and metadata request latency drove up Appl ## Muse Spark 1.1 is now available on AI Gateway DevFeed: [Muse Spark 1.1 is now available on AI Gateway]() Original publisher: [Read original article]() Author: Jerilyn Zheng Published: 2026-07-09T00:00:00Z Content type: release Language: en Sources: [Vercel News]() Topics: [AI Models](), [multimodal](), [long-context](), [Model Context Protocol](), [vercel ai sdk](), [API keys](), [Routing (disambiguation)]() Tags: [agent](), [agentic](), [ai](), [audio](), [context-window](), [image](), [leaderboard](), [model](), [multimodal](), [muse](), [pricing](), [ranking](), [routing](), [sdk](), [spark](), [tokens](), [tool](), [tools]() ### AI overview Muse Spark 1.1 from Meta is now available through Vercel's AI Gateway. It is a multimodal reasoning model with a 1M-token context window for agentic tasks, supporting multiple input types, tool orchestration, MCP servers, custom skills, parallel tool calls, structured output, and search with citations. ### Source excerpt Muse Spark 1.1 from Meta is now available on AI Gateway. It is a multimodal reasoning model with a 1M token context window built for agentic tasks, accepting text, image, video, PDF, and audio inputs. Muse Spark 1.1 plans and orchestrates work across tools and services, operating as a main agent or as a subagent, and it works with new tools, MCP servers, and custom skills without examples. The model supports parallel tool calling, structured output, and built-in search with citations. To use Muse Spark 1.1, set model to meta/muse-spark-1.1 in the AI SDK: AI Gateway provides a unified API for calling models, tracking usage and cost, and configuring retries, failover, and performance optimizations for higher-than-provider uptime. It includes built-in custom reporting, Zero Data Retention support, budgets for API keys, routing rules, and more. AI Gateway reflects provider pricing with no markup and does not charge a platform fee on inference, including on Bring Your Own Key (BYOK) requests. Try Muse Spark 1.1 in the model playground. Read more ## Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs DevFeed: [Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs]() Original publisher: [Read original article]() Author: Sagibhuvana Published: 2026-06-30T11:01:01Z Content type: article Language: en Sources: [Expedia]() Topics: [Apache Spark](), [Large Language Model](), [debugging](), [Optimization](), [MCP]() Tags: [apache-spark](), [big-data](), [data-engineering](), [debugging](), [innovation](), [llm](), [llms](), [mcp](), [mcp-server](), [optimization](), [spark]() ### AI overview This article describes an automated workflow that uses large language models to analyze Spark SQL physical plans, identify performance anti-patterns, and suggest fixes for long-running jobs. It uses an open-source Spark MCP server to provide SQL execution metadata, plans, stage and task metrics, shuffle, spill, and I/O statistics. ### Source excerpt Expedia Group Technology -- InnovationUsing large language models to reveal bottlenecks in Spark SQL execution plansPhoto by Luis del Río If you've ever stared at a 300-plus-node physical plan at 2 a.m. trying to spot a missing broadcast or one cursed skewed partition, this is for you. Spark makes it deceptively easy to write complex SQL that looks correct but quietly turns into a performance and cost problem at scale. A query that runs fine on day one can slow to a crawl as data grows, joins get wider, and aggregations become more nested. Suddenly, jobs take hours instead of minutes, clusters need to be over-provisioned, and costs climb with no clear explanation. When things go wrong, the situation gets worse. Debugging Spark SQL often means wading through deeply nested physical plans, fragmented UI metrics, and stage-level statistics that don't clearly point to the root cause. Engineers are left guessing whether the issue is a bad join strategy, skewed data, excessive shuffling, or memory pressure -- turning optimization into a time-consuming, trial-and-error exercise. So we built an automated, LLM-powered workflow that reads Spark SQL plans, flags performance anti-patterns, and suggests concrete fixes. Here's why we built it, how it works, and what we learned after running it on real workloads. Why we even needed this Across teams we kept bumping into the same pain points: Slow or failing jobs that needed deep Spark expertise to debug SQL that didn't behave the way people assumed Skew, spills, or partition issues hiding beneath the surface Tuning efforts that were reactive and inconsistent At some point, the obvious question came up: Can an LLM look at a Spark SQL plan and tell you what's likely to go wrong -- where cost will spike, where execution will drag, and where seemingly harmless SQL hides real operational risk? Turns out: yes How we approached it1. Bring in real context with the Spark MCP server We used an open-source Spark MCP server that exposes: SQL exec ## Automated Schema Evolution in Pinterest's Next-Generation DB Ingestion Framework DevFeed: [Automated Schema Evolution in Pinterest's Next-Generation DB Ingestion Framework]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-06-24T18:01:01Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [schema-evolution](), [Apache Iceberg](), [Automation](), [Databases](), [Apache Spark](), [Kafka](), [Pull Request](), [Code generation](), [datasets](), [monitor](), [Provisioning]() Tags: [automation](), [code-generation](), [datasets](), [engineering](), [icebergs](), [kafka](), [monitor](), [pinterest](), [process](), [schema-evolution](), [software](), [spark](), [storage]() ### AI overview Pinterest describes an automated schema-evolution framework for its CDC-based database ingestion platform, which uses Kafka, Flink, Spark, and Iceberg. The framework propagates supported schema changes across the stack, uses pull-request-based versioning and auditing, targets SLA-based consistency between online and offline schemas, and provides recovery paths for unsupported or ambiguous changes. ### Source excerpt Yisheng Zhou | Software Engineer II Liang Mou | Sr Staff Software Engineer Gabriel Raphael Garcia Montoya | Staff Software Engineer Istvan Podor | Staff Software Engineer Introduction In the first post of this series, we introduced Pinterest's next-generation CDC-based ingestion platform built on Kafka, Flink, Spark, and Iceberg. In production, upstream schemas are constantly evolving, and in a distributed CDC pipeline, schema is not just metadata -- it is a cross-system contract spanning ingestion, transformation, storage, and historical backfill. A schema change that is not handled carefully can break Flink jobs, block Spark upserts, or create inconsistencies between online and offline representations. This post walks through how we make schema evolution safe in practice: the onboarding model it builds on, the changes we support and the tradeoffs we accept, how updates propagate across the stack, and how we monitor and recover when things go wrong. We end with where we're heading next -- zero-gap schema evolution. Background & Motivation Schema evolution was particularly challenging in our environment because our ingestion pipeline is composed of multiple tightly related stages, each of which depends on schema in a different way. A single table onboarding flow spans CDC source configuration, Kafka provisioning, Flink and Spark code generation, Iceberg CDC and base table creation, and historical bootstrap -- all driven by the same schema. Once a source schema changes, that change must be reflected across code, metadata, and storage in a controlled and auditable way. Without a unified automation framework, manual updates across these layers increase the risk of drift, failed deployments, and inconsistent datasets. We therefore needed a way to make schema evolution safe, repeatable, and scalable. Our Solution: SLA-Based Automated Schema Evolution To address this problem, we built an automated schema evolution framework on top of our CDC ingestion platform, providing: au ## ClickHouse integrates with Lakehouse Runtime Catalog DevFeed: [ClickHouse integrates with Lakehouse Runtime Catalog]() Original publisher: [Read original article]() Author: Melvyn Peignon Published: 2026-04-22T13:27:18Z Content type: release Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Apache Iceberg](), [Google](), [BigQuery](), [Apache Spark](), [Cloud](), [API](), [data](), [Authentication]() Tags: [analytics](), [apache-iceberg](), [api](), [authentication](), [bigquery](), [clickhouse](), [cloud](), [data](), [google](), [google-cloud](), [integration](), [spark]() ### AI overview ClickHouse announces a beta integration with Google's Lakehouse Runtime Catalog, enabling direct querying of Google Cloud Lakehouse Apache Iceberg tables through the Iceberg REST Catalog. The integration supports ClickHouse 26.2 and later, ClickHouse Cloud, Google Application Default Credentials, and querying or loading data without data movement or metadata synchronization. ### Source excerpt We're excited to announce a new integration between ClickHouse and Google's Lakehouse Runtime CatalogCatalog ## Databricks is abstracting away physical data engineering controls DevFeed: [Databricks is abstracting away physical data engineering controls]() Original publisher: [Read original article]() Author: Zach Wilson Published: 2026-02-24T01:03:11Z Content type: opinion Language: en Sources: [DataExpert.io Newsletter]() Topics: [databricks](), [data-engineering](), [Apache Iceberg](), [Open Source]() Tags: [architecture](), [data-engineering](), [databricks](), [open-source](), [partition](), [partitioning](), [sorting](), [spark]() ### AI overview This opinion article argues that Databricks is shifting away from hands-on data engineering by abstracting physical data modeling through features such as liquid clustering and predictive optimization. It also criticizes Databricks' support for managed Apache Iceberg tables after acquiring Tabular. ### Source excerpt Databricks abstracts away almost all of the data engineering skills. Liquid clustering is the first place where things will get messy! ## Data-to-Production: Bridging the Gap Between Iceberg and Live Microservices DevFeed: [Data-to-Production: Bridging the Gap Between Iceberg and Live Microservices]() Original publisher: [Read original article]() Author: Wix Engineering Published: 2026-02-17T11:04:08Z Content type: article Language: en Sources: [Wix Engineering]() Topics: [Apache Iceberg](), [data observability](), [Microservice](), [clickhouse](), [Amazon S3](), [Architecture & Design](), [Apache Spark](), [airflow](), [Back end](), [API]() Tags: [airflow](), [amazon-s3](), [apache-iceberg](), [api](), [architecture](), [clickhouse](), [data](), [data-engineering](), [json](), [microservices](), [spark]() ### AI overview Wix describes Data-to-Production, a platform that activates data from Amazon S3 and Apache Iceberg for backend microservices. The system ingests Iceberg data into ClickHouse and serves it through a type-safe JSON API, using metadata governance and an Airflow and Python ingestion engine. ### Source excerpt At Wix, our Data Warehouse (DWH) is a massive repository of insights. Built on Amazon S3 using Apache Iceberg table formats, and populated by Trino and Spark jobs, it houses petabytes of data--from user segmentation and logs to AI chat analytics. However, storage is only half the battle. The real challenge--and the "holy grail" for many data engineering teams--is Activation : taking that petabyte-scale data and exposing it to backend microservices with millisecond latency, high availability, and... ## The 2025 AI + Data Engineering Roadmap DevFeed: [The 2025 AI + Data Engineering Roadmap]() Original publisher: [Read original article]() Author: Zach Wilson Published: 2025-10-17T22:35:45Z Content type: tutorial Language: en Sources: [DataExpert.io Newsletter]() Topics: [data-engineering](), [Python](), [SQL](), [Artificial Intelligence](), [airflow](), [BigQuery](), [Orchestration](), [Apache Spark](), [data-modeling](), [Data Quality](), [Embeddings](), [Retrieval Augmented Generation (RAG)]() Tags: [2025](), [ai](), [airflow](), [bigquery](), [count](), [course](), [data-engineering](), [data-modeling](), [data-quality](), [embeddings](), [engineering](), [framer](), [newsletter](), [orchestration](), [python](), [rag](), [real-time](), [right-join](), [spark](), [sql]() ### AI overview A 2025 roadmap for entering data engineering, covering foundational SQL and Python skills, distributed computing, orchestration, data modeling, data quality, AI and data integrations, portfolio projects, and personal branding. ### Source excerpt Getting a data engineering job is complicated. ## How we scaled raw GROUP BY to 100 B+ rows in under a second DevFeed: [How we scaled raw GROUP BY to 100 B+ rows in under a second]() Original publisher: [Read original article]() Author: Tom Schreiber Published: 2025-09-29T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [AI, ML & Data Engineering]() Tags: [analytics](), [benchmark](), [bigquery](), [clickhouse](), [cloud](), [internals](), [performance](), [scale](), [spark]() ### AI overview ClickHouse Cloud introduces parallel replicas to distribute a single analytical query across cluster cores, aiming to scale GROUP BY workloads horizontally without data reshuffling. ### Source excerpt ClickHouse Cloud now scales analytical queries with parallel replicas, fanning a single query across thousands of cores for terabyte-per-second throughput. This post dives into the internals and lets you see and feel the speed. ## DuckDB benchmarked against Spark DevFeed: [DuckDB benchmarked against Spark]() Original publisher: [Read original article]() Author: Matt Martin Published: 2025-09-22T20:13:34Z Content type: comparison Language: en Sources: [DataExpert.io Newsletter]() Topics: [DuckDB](), [Apache Spark]() Tags: [duckdb](), [spark]() ### AI overview A comparison article about benchmarking DuckDB against Apache Spark. ### Source excerpt You Don't Always Need A Sledgehammer ## How Changan Ford cut costs by 40% and powered precision marketing with ClickHouse Enterprise Edition DevFeed: [How Changan Ford cut costs by 40% and powered precision marketing with ClickHouse Enterprise Edition]() Original publisher: [Read original article]() Author: ClickHouse Published: 2025-09-22T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [data](), [dashboards]() Tags: [ads](), [analytics](), [apache-flink](), [architecture](), [clickhouse](), [cloud](), [customer](), [dashboards](), [data](), [enterprise](), [errors](), [open-source](), [performance](), [platform](), [real-time](), [serverless](), [spark](), [storage]() ### AI overview Changan Ford migrated its customer data platform from ClickHouse Community Edition to ClickHouse Enterprise Edition on Alibaba Cloud. The company reports a 40% cost reduction, lower operational workload, and improved stability for high-demand marketing campaigns. ### Source excerpt Changan Ford migrated their Customer Data Platform to ClickHouse Enterprise Edition, cut costs by 40%, eased the operational workload, and gave the team stability and confidence even during their most demanding campaigns. ## Announcing the ClickHouse Connector for AWS Glue DevFeed: [Announcing the ClickHouse Connector for AWS Glue]() Original publisher: [Read original article]() Author: Luke Gannon Published: 2025-08-21T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [AWS Glue](), [clickhouse](), [Apache Spark](), [aws-marketplace](), [Serverless]() Tags: [apache](), [aws](), [aws-glue](), [aws-marketplace](), [blog](), [clickhouse](), [launch](), [python](), [scala](), [spark]() ### AI overview This article announces the official ClickHouse Connector for AWS Glue, a serverless Apache Spark-based ETL integration available through AWS Marketplace. It explains how the connector supports PySpark and Scala, simplifies setup, and enables production-ready Spark jobs that connect AWS Glue with ClickHouse. ### Source excerpt Today, we're announcing the launch of the official ClickHouse Connector for AWS Glue, which utilizes their Apache Spark-based serverless ETL engine. [Next page]()