# Apache Spark Apache Spark is a unified engine for large-scale data analytics. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Shehab Amin on Spark Compatibility, Rust, and LakeSail DevFeed: [Shehab Amin on Spark Compatibility, Rust, and LakeSail]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-09-02T12:22:23Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [Apache Spark](), [Rust](), [data-engineering](), [apache-arrow](), [agentic-coding]() Tags: [apache-arrow](), [apache-spark](), [data-engineering](), [rust]() ### AI overview A podcast conversation with LakeSail co-founder and CEO Shehab Amin about Spark compatibility, Rust, Apache Arrow, DataFusion, and data infrastructure. It also discusses data-stack choices, streaming and batch processing, agentic coding, and using existing data pipelines as a basis for AI pipelines. ### Source excerpt Data Engineering Central Podcast. ## AWS Glue 6.0 now available with 30% lower price and full Apache Iceberg v3 support DevFeed: [AWS Glue 6.0 now available with 30% lower price and full Apache Iceberg v3 support]() Original publisher: [Read original article]() Author: Channy Yun (윤석찬) Published: 2026-08-21T18:53:26Z Content type: release Language: en Sources: [AWS News Blog]() Topics: [Apache Iceberg](), [Amazon Web Services](), [Apache Spark](), [Python](), [Scala](), [Geographic Information System](), [Internet of things]() Tags: [analytics](), [apache](), [apache-iceberg](), [aws](), [aws-glue](), [launch](), [news](), [performance](), [pricing](), [python-3-13](), [real-time](), [release](), [scala](), [spark](), [streaming]() ### AI overview AWS Glue 6.0 is generally available with 30% lower pricing, a modernized Spark 4.1 runtime, Python 3.13 and Scala 2.13 support, and full Apache Iceberg v3 support. The release adds improved handling of semi-structured data, declarative ETL pipelines, faster PySpark execution, and real-time streaming with single-digit millisecond latency. ### Source excerpt AWS Glue 6.0 is built on a fully modernized runtime, Apache Spark 4.1, Python 3.13, and Scala 2.13, delivering 30% lower pricing than previous AWS Glue versions. ## Lakesail: A Rust-based alternative to Spark without the JVM DevFeed: [Lakesail: A Rust-based alternative to Spark without the JVM]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-08-18T21:11:44Z Content type: opinion Language: en Sources: [Data Engineering Central]() Topics: [Apache Spark](), [Rust]() Tags: [apache-spark](), [rust](), [spark]() ### AI overview This commentary article introduces Lakesail as a Rust-based implementation or alternative to Spark that avoids the JVM overhead. It places Lakesail within the broader data-engineering ecosystem, where established tools such as Spark and Kafka continue to coexist with newer projects including Polars, DuckDB, and Daft. ### Source excerpt ya know ... black magic stuff ## Data Engineering Weekly #283 DevFeed: [Data Engineering Weekly #283]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-08-17T02:59:40Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [AI, ML & Data Engineering](), [Data Quality](), [Microservice](), [Apache Spark](), [Kafka](), [Feature Engineering](), [audit trail](), [Protocol (disambiguation)]() Tags: [ai](), [anthropic](), [article](), [data-engineering](), [data-quality](), [engineering](), [feature-engineering](), [kafka](), [newsletter](), [pipelines](), [services](), [testing]() ### AI overview Data Engineering Weekly #283 is a newsletter covering data platform fundamentals, multiagent system coordination, payments platform data contracts, financial data quality, declarative data engineering, and cost-efficient export workloads. ### Source excerpt The Weekly Data Engineering Newsletter ## Batch Jobs for SparkClient: Submitting and Managing Spark Workloads from Python DevFeed: [Batch Jobs for SparkClient: Submitting and Managing Spark Workloads from Python]() Original publisher: [Read original article]() Author: Sameer Yadav Published: 2026-07-25T05:00:00Z Content type: tutorial Language: en Sources: [Kubeflow]() Topics: [Apache Spark](), [SDKs](), [Python](), [Kubernetes]() Tags: [batch](), [cleanup](), [gsoc](), [jobs](), [kubernetes](), [logs](), [pipelines](), [python](), [scheduled](), [sdk](), [spark]() ### AI overview This tutorial explains how the Kubeflow SDK's SparkClient supports submitting and managing batch Spark workloads on Kubernetes from Python. It covers script- and function-based jobs, lifecycle operations, log retrieval, cleanup, and the implementation's current boundaries. ### Source excerpt How the SparkClient SDK's new batch job APIs work under the hood -- submit_job(), FileJob/FuncJob, the lifecycle APIs, and log retrieval. ## How We Refresh Razorpay's Data Warehouse 10x Faster with Graphs and Indexes DevFeed: [How We Refresh Razorpay's Data Warehouse 10x Faster with Graphs and Indexes]() Original publisher: [Read original article]() Author: Amit Prabhu Published: 2026-07-14T14:06:16Z Content type: article Language: en Sources: [Razorpay Engineering - Medium]() Topics: [data](), [airflow](), [Apache Spark](), [Amazon S3](), [Apache Iceberg](), [microservices architecture](), [parquet](), [Architecture & Design]() Tags: [airflow](), [analytics](), [apache-iceberg](), [architecture](), [batch](), [data](), [graphs](), [infrastructure](), [parquet](), [pipelines](), [razorpay](), [spark](), [trino](), [warehouse]() ### AI overview Razorpay describes its data warehouse refresh pipeline, which builds wide denormalized Facts by joining data from multiple microservices. The article covers the original Airflow- and Spark-based full-refresh process, the underlying lake formats and query layer, and the scaling challenges that led the team to reconsider refresh strategy, data layout, and high-cardinality dimensions. ### Source excerpt Contributors: Utkarsh Koppikar Rohan Background Razorpay provides the payment infrastructure for millions of merchants globally. Behind every payment, settlement, and refund is a microservices architecture where each service owns its own database. While this keeps services independent and scalable, it creates a challenge for stakeholders who need to see across those boundaries. The Data Platform team manages the infrastructure that bridges this gap. Transactional data flows into the lake via CDC pipelines, ingested onto S3 in Delta Lake, Apache Iceberg, or plain Parquet formats. On top of the lake, we build domain-specific warehouse tables -- wide, pre-joined tables that co-locate all the data a consumer needs, queryable via Trino. These power two use cases: Analytics (internal dashboards on Tableau and Superset) and Reporting (merchants and regulated entities who download structured data exports; Razorpay generates nearly a million such reports per month). The warehouse tables that power both use cases are called Facts. A Fact is a flat denormalised table on S3, produced by joining 10 to 30 microservice tables and materialising the result once. A settlement Fact, for example, merges payments, refunds, adjustments, and card details into a single wide row so that a dashboard or report reads from a single table instead of joining across services in real time. It is closer to a domain-specific materialised view than a classical data warehouse fact table. We maintain over 50 such Facts, and approximately 40% of all merchant reports are served directly from them. As data volumes and the number of entities per fact grew, the batch generation pipeline began to show its limits, prompting us to rethink the refresh strategy, the data layout, and how to handle high-cardinality dimensions. The rest of this post covers that journey. The Full Refresh Pipeline: Our Baseline and the Pain The original full-refresh pipeline was straightforward. Schedule: Airflow schedules Spark jobs o ## Training Orchestrator: Unifying Model Training at Yelp DevFeed: [Training Orchestrator: Unifying Model Training at Yelp]() Original publisher: [Read original article]() Author: Ying Wang and Nathan Sponberg, Software Engineer Published: 2026-07-14T00:00:00Z Content type: article Language: en Sources: [Yelp]() Topics: [Machine Learning & Artificial Intelligence](), [Orchestration](), [Apache Spark](), [Training AI Models](), [Model Development](), [configuration](), [configuration-management](), [Monitoring]() Tags: [configuration](), [machine-learning](), [maintenance](), [mlflow](), [model-training](), [orchestration](), [reproducibility](), [spark](), [tooling]() ### AI overview Yelp's Core Machine Learning Team developed Training Orchestrator to standardize how machine learning teams define and run Spark-based model training. The configuration-driven system addresses duplicated code, inconsistent configurations, limited local testing, scattered validation and monitoring, and poor reproducibility across environments. ### Source excerpt At Yelp, we train many machine learning models on different schedules. Applied machine learning teams all have their own set of Spark-based training batches, scripts, and configurations. Over time, these diverged, leading to duplicated code, subtle inconsistencies, and a growing maintenance burden. Yelp's Core Machine Learning Team has developed excellent tooling across our ML ecosystem over the years: feature stores for reproducible data, a unified training library for neural networks and gradient-boosted trees, seamless Spark integration, and MLflow services for model tracking and deployment. But there was still one key piece missing right in the middle: a standardized way to... ## Inside Cilium CNI: solving mysterious Kubernetes pod setup timeouts DevFeed: [Inside Cilium CNI: solving mysterious Kubernetes pod setup timeouts]() Original publisher: [Read original article]() Author: Adyen Published: 2026-07-13T09:17:40Z Content type: article Language: en Sources: [Adyen Tech]() Topics: [Cilium](), [Kubernetes](), [Kubernetes networking](), [networking](), [eBPF](), [Latency](), [hdfs](), [Apache Spark](), [big-data](), [cpu]() Tags: [big-data](), [cilium](), [cni](), [cpu](), [data](), [devops](), [ebpf](), [hdfs](), [kubernetes](), [kubernetes-networking](), [latency](), [spark](), [speed](), [tracing]() ### AI overview An Adyen engineer investigates Cilium CNI timeouts that prevented new Kubernetes pods from starting. The article attributes the issue to sequential connection-tracking-table garbage collection, which became a bottleneck under workloads involving HDFS, Trino, and short-lived Spark jobs. ### Source excerpt By Jorrick Sleijster - Senior Data Platform Engineer, Adyen I was fully aware a year ago that a single configuration line could break the Kubernetes networking stack. But if they told me that leftovers from Kubernetes pods which terminated hours prior could block new ones from starting, I would have thought they were joking. In high-performance networking, 35 seconds is a lifetime. This was the latency required to iterate through our connection tracking table of 7 million entries at a maximum speed of 200,000 entries per second. At our 16-million-entry peak, this sequential lookup could take up to 80 seconds, leading to Cilium CNI timeouts preventing new pods from starting on affected nodes. We uncovered this linear-time behavior at Adyen by tracing syscalls, inspecting codebases, and analyzing eBPF internals. This investigation revealed how our varied workloads turned the connection tracking table's garbage collection algorithm into a critical bottleneck. Our setup: why we're different At Adyen, we run Cilium CNI across all our 100+ Kubernetes clusters. When we switched from Calico to Cilium, we knew we'd face challenges adapting it to our production workloads. Our production big data Kubernetes clusters have a unique usage pattern compared to the other Kubernetes environments within Adyen: Data extraction from HDFS. Our infrastructure relies on more than 500 datanodes. Trino represents one of our most demanding HDFS workloads, processing analytical queries against data stored on HDFS. Due to the distributed nature of HDFS, each file you download requires a new connection to any of these 500 nodes. Therefore, during peak hours, a single pod can produce approximately 50,000 connections every minute. Pod churn. Many pods we spawn on the Kubernetes cluster run batch jobs, such as Spark jobs. They stay around for anywhere from a second to a couple of hours. Wide variety of workloads. Some workloads are very CPU-intensive, like Spark pods executing complex joins and tra ## Data Engineering Weekly #278 DevFeed: [Data Engineering Weekly #278]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-07-13T02:44:42Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [AI, ML & Data Engineering](), [Apache Spark](), [Kafka](), [Data Quality]() Tags: [agents](), [ai](), [data-engineering](), [data-pipeline](), [data-quality](), [engineering](), [kafka](), [releases](), [spark]() ### AI overview Data Engineering Weekly #278 is a curated newsletter covering AI-assisted engineering workflows, new leetdata.ai features, agent-oriented data systems, multilingual AI concerns, visualization with Flint, and data platform fundamentals. ### Source excerpt The Weekly Data Engineering Newsletter ## Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption DevFeed: [Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption]() Original publisher: [Read original article]() Author: Rahul Penti Published: 2026-07-10T00:00:00Z Content type: article Language: en Sources: [Grab Tech]() Topics: [Apache Iceberg](), [data](), [Amazon S3](), [parquet](), [Apache Spark](), [Latency](), [Concurrency]() Tags: [apache-iceberg](), [compute](), [concurrency](), [data](), [database](), [engineering](), [latency](), [parquet](), [performance](), [s3](), [scale](), [spark]() ### AI overview Grab describes its transition from Hive Parquet and a directory-based Hive Metastore architecture toward Apache Iceberg and a table-centric lakehouse. The article outlines scalability problems involving metadata latency, small files, S3 request overhead, and operational management, and mentions a unified Spark catalog built to hide table-format differences. ### Source excerpt Introduction: The evolution of Grab's Data Lake At Grab's scale, managing petabytes of data across billions of S3 objects demands more than a storage layer. It demands a robust architectural primitive that supports the high-concurrency needs of a modern "Lakehouse." Our goal is full storage-compute separation, leveraging S3 as an elastic foundation for both near-real-time metrics and large-scale batch transformations. For years, the vast majority of our tables were Hive Parquet, managed through the Hive Metastore with a directory-based layout. This model served us well, but as data volume grew, the directory-and-metastore approach became the limiting factor. We are now transitioning to a table-centric architecture built on modern table formats, treating data as a first-class primitive to ensure consistency and performance across our internal data transformation platforms: Slide, which powers batch transformations, and Hugo, which handles online-to-data-lake ingestion. Along the way, we also built the UnifiedSparkCatalog, a unified Spark catalog that hides table-format differences from users entirely, which we are open-sourcing alongside this post. The catalyst for change: Challenges with Hive Parquet For years, Hive Parquet was the backbone of our Data Lake, representing the vast majority of our tables. However, as data volume scaled, the architectural limitations of directory-based storage became apparent. We identified four primary bottlenecks: Catalog latency: The Hive Metastore (HMS) became a centralized failure point. High concurrency during metadata access led to O(n) listing overhead, where query planning time scaled linearly with partition count, crippling throughput. The small file problem: The directory layout left us with severe file fragmentation. Certain Machine Learning (ML) datasets had an average file size under 1 MB, with thousands of files in each partition. At this scale, the overhead of S3 object listing and metadata request latency drove up Appl ## The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering DevFeed: [The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-08T12:16:09Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [pandas](), [data-engineering](), [Artificial Intelligence](), [DuckDB](), [data-platforms](), [software-development](), [future of software](), [Apache Spark](), [Hadoop](), [Open Source](), [parquet]() Tags: [ai](), [apache-arrow](), [arrow](), [data](), [data-engineering](), [data-platforms](), [duckdb](), [hadoop](), [large-language-models](), [open-source](), [pandas](), [parquet](), [software-development]() ### AI overview An interview with Wes McKinney covers the origins of pandas and Apache Arrow, the evolution of modern data engineering from Hadoop to lakehouse architectures, and the roles of tools such as Parquet, DuckDB, DataFusion, and Spark. McKinney also discusses how AI affects software development, arguing that it can improve experienced engineers' productivity but does not replace software engineering, architecture, or judgment. ### Source excerpt interview with Wes McKinney ## Operating Kubernetes at scale: a few stories from running Amazon EKS DevFeed: [Operating Kubernetes at scale: a few stories from running Amazon EKS]() Original publisher: [Read original article]() Author: Neelendra Bhandari Published: 2026-06-30T13:00:00Z Content type: article Language: en Sources: [Kubernetes Overview, News and Trends | The New Stack]() Topics: [Amazon EKS](), [Kubernetes](), [Availability](), [systems](), [Apache Spark](), [AI Platform]() Tags: [agentic-ai](), [ai-infrastructure](), [amazon-eks](), [apache-spark](), [availability](), [aws-marketplace](), [cloud-services](), [data](), [gpu](), [kubernetes](), [post-contributed](), [real-time](), [sponsor-aws-marketplace](), [sponsored-post-contributed](), [systems]() ### AI overview This article describes how operating Amazon EKS at very large scale shaped approaches to Kubernetes control-plane resilience. It argues that availability problems often arise when components amplify faults, and explains how changing workload patterns--including foundation-model training, real-time inference, Apache Spark analytics, and agentic AI--create extremely rapid state transitions that challenge traditional Kubernetes operating assumptions. ### Source excerpt Amazon EKS runs hundreds of thousands of Kubernetes clusters across more than thirty AWS regions. Operating at that scale has The post Operating Kubernetes at scale: a few stories from running Amazon EKS appeared first on The New Stack. ## Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs DevFeed: [Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs]() Original publisher: [Read original article]() Author: Sagibhuvana Published: 2026-06-30T11:01:01Z Content type: article Language: en Sources: [Expedia]() Topics: [Apache Spark](), [Large Language Model](), [debugging](), [Optimization](), [MCP]() Tags: [apache-spark](), [big-data](), [data-engineering](), [debugging](), [innovation](), [llm](), [llms](), [mcp](), [mcp-server](), [optimization](), [spark]() ### AI overview This article describes an automated workflow that uses large language models to analyze Spark SQL physical plans, identify performance anti-patterns, and suggest fixes for long-running jobs. It uses an open-source Spark MCP server to provide SQL execution metadata, plans, stage and task metrics, shuffle, spill, and I/O statistics. ### Source excerpt Expedia Group Technology -- InnovationUsing large language models to reveal bottlenecks in Spark SQL execution plansPhoto by Luis del Río If you've ever stared at a 300-plus-node physical plan at 2 a.m. trying to spot a missing broadcast or one cursed skewed partition, this is for you. Spark makes it deceptively easy to write complex SQL that looks correct but quietly turns into a performance and cost problem at scale. A query that runs fine on day one can slow to a crawl as data grows, joins get wider, and aggregations become more nested. Suddenly, jobs take hours instead of minutes, clusters need to be over-provisioned, and costs climb with no clear explanation. When things go wrong, the situation gets worse. Debugging Spark SQL often means wading through deeply nested physical plans, fragmented UI metrics, and stage-level statistics that don't clearly point to the root cause. Engineers are left guessing whether the issue is a bad join strategy, skewed data, excessive shuffling, or memory pressure -- turning optimization into a time-consuming, trial-and-error exercise. So we built an automated, LLM-powered workflow that reads Spark SQL plans, flags performance anti-patterns, and suggests concrete fixes. Here's why we built it, how it works, and what we learned after running it on real workloads. Why we even needed this Across teams we kept bumping into the same pain points: Slow or failing jobs that needed deep Spark expertise to debug SQL that didn't behave the way people assumed Skew, spills, or partition issues hiding beneath the surface Tuning efforts that were reactive and inconsistent At some point, the obvious question came up: Can an LLM look at a Spark SQL plan and tell you what's likely to go wrong -- where cost will spike, where execution will drag, and where seemingly harmless SQL hides real operational risk? Turns out: yes How we approached it1. Bring in real context with the Spark MCP server We used an open-source Spark MCP server that exposes: SQL exec ## Automated Schema Evolution in Pinterest's Next-Generation DB Ingestion Framework DevFeed: [Automated Schema Evolution in Pinterest's Next-Generation DB Ingestion Framework]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-06-24T18:01:01Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [schema-evolution](), [Apache Iceberg](), [Automation](), [Databases](), [Apache Spark](), [Kafka](), [Pull Request](), [Code generation](), [datasets](), [monitor](), [Provisioning]() Tags: [automation](), [code-generation](), [datasets](), [engineering](), [icebergs](), [kafka](), [monitor](), [pinterest](), [process](), [schema-evolution](), [software](), [spark](), [storage]() ### AI overview Pinterest describes an automated schema-evolution framework for its CDC-based database ingestion platform, which uses Kafka, Flink, Spark, and Iceberg. The framework propagates supported schema changes across the stack, uses pull-request-based versioning and auditing, targets SLA-based consistency between online and offline schemas, and provides recovery paths for unsupported or ambiguous changes. ### Source excerpt Yisheng Zhou | Software Engineer II Liang Mou | Sr Staff Software Engineer Gabriel Raphael Garcia Montoya | Staff Software Engineer Istvan Podor | Staff Software Engineer Introduction In the first post of this series, we introduced Pinterest's next-generation CDC-based ingestion platform built on Kafka, Flink, Spark, and Iceberg. In production, upstream schemas are constantly evolving, and in a distributed CDC pipeline, schema is not just metadata -- it is a cross-system contract spanning ingestion, transformation, storage, and historical backfill. A schema change that is not handled carefully can break Flink jobs, block Spark upserts, or create inconsistencies between online and offline representations. This post walks through how we make schema evolution safe in practice: the onboarding model it builds on, the changes we support and the tradeoffs we accept, how updates propagate across the stack, and how we monitor and recover when things go wrong. We end with where we're heading next -- zero-gap schema evolution. Background & Motivation Schema evolution was particularly challenging in our environment because our ingestion pipeline is composed of multiple tightly related stages, each of which depends on schema in a different way. A single table onboarding flow spans CDC source configuration, Kafka provisioning, Flink and Spark code generation, Iceberg CDC and base table creation, and historical bootstrap -- all driven by the same schema. Once a source schema changes, that change must be reflected across code, metadata, and storage in a controlled and auditable way. Without a unified automation framework, manual updates across these layers increase the risk of drift, failed deployments, and inconsistent datasets. We therefore needed a way to make schema evolution safe, repeatable, and scalable. Our Solution: SLA-Based Automated Schema Evolution To address this problem, we built an automated schema evolution framework on top of our CDC ingestion platform, providing: au ## Announcing Apache Iceberg 1.11.0 DevFeed: [Announcing Apache Iceberg 1.11.0]() Original publisher: [Read original article]() Author: Google Open Source (noreply@blogger.com) Published: 2026-05-27T18:30:00Z Content type: release Language: en Sources: [Google Open Source Blog]() Topics: [Apache Iceberg](), [Apache Spark](), [apache-flink](), [schema-evolution](), [Streaming](), [Encryption](), [Cryptography]() Tags: [apache-flink](), [apache-iceberg](), [apache-spark](), [encryption](), [kms](), [lakehouse](), [releases](), [schema-evolution](), [streaming]() ### AI overview Apache Iceberg 1.11.0 adds support for Apache Spark 4.1 and Apache Flink 2.1, makes both default build targets, and introduces changes including server-side REST catalog scan planning, partition statistics APIs, built-in envelope encryption with Google KMS support, and Google Storage Analytics integration. ### Source excerpt by Alex Stephen & Talat Uyarer, Lakehouse Apache Iceberg project has just launched version 1.11.0! A lot has happened since the last version. Iceberg 1.11.0 adds support for Apache Spark 4.1 and Apache Flink 2.1, the latest releases of the two engines and makes both the default build targets The rest are more structural. The REST catalog learns to plan scans server-side, shifting metadata work off the query engine. A new partition statistics scan API gives optimizers a clean, supported way to read a table's shape. Built-in table encryption arrives with envelope encryption and Google KMS support. And Google Storage Analytics library integration makes your Iceberg workloads faster than before. Let's take a look at some of the biggest changes. Spark & Flink Updates As Spark and Flink are moving forward, the 1.11.0 release is pushing forward for new version support in both. Spark 4.1 & DSv2 Migration: Spark 4.1 unlocks is MERGE INTO with automatic schema evolution: Spark's newer MERGE syntax accepts a WITH SCHEMA EVOLUTION clause, so a MERGE whose source carries columns the target table lacks can add those columns to the table within the same statement, with no separate ALTER TABLE round trip. Beyond the version bump, the 1.11 Spark connector also modernizes against Spark's newer DataSource V2 APIs and adds an asynchronous micro-batch planner that speeds up Structured Streaming. Flink Ecosystem Updates: Initial work for Flink 2.1 support has landed in the core repository, continuing Iceberg's promise of providing first-class, low-latency streaming sink capabilities. The centerpiece of the Flink work is the DynamicIcebergSink, an experimental sink that breaks the old one-sink-per-table model: a single sink routes each record to a table chosen at runtime, creating tables on demand and evolving their schemas and partition specs on the fly as the input changes including dropping columns once you opt in with dropUnusedColumns. In addition to DynamicIcebergSInk work Flink sta ## ClickHouse integrates with Lakehouse Runtime Catalog DevFeed: [ClickHouse integrates with Lakehouse Runtime Catalog]() Original publisher: [Read original article]() Author: Melvyn Peignon Published: 2026-04-22T13:27:18Z Content type: release Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Apache Iceberg](), [Google](), [BigQuery](), [Apache Spark](), [Cloud](), [API](), [data](), [Authentication]() Tags: [analytics](), [apache-iceberg](), [api](), [authentication](), [bigquery](), [clickhouse](), [cloud](), [data](), [google](), [google-cloud](), [integration](), [spark]() ### AI overview ClickHouse announces a beta integration with Google's Lakehouse Runtime Catalog, enabling direct querying of Google Cloud Lakehouse Apache Iceberg tables through the Iceberg REST Catalog. The integration supports ClickHouse 26.2 and later, ClickHouse Cloud, Google Application Default Credentials, and querying or loading data without data movement or metadata synchronization. ### Source excerpt We're excited to announce a new integration between ClickHouse and Google's Lakehouse Runtime CatalogCatalog ## Kubeflow SDK v0.4.0: Model Registry, SparkConnect, and Enhanced Developer Experience DevFeed: [Kubeflow SDK v0.4.0: Model Registry, SparkConnect, and Enhanced Developer Experience]() Original publisher: [Read original article]() Author: Kubeflow SDK Team Published: 2026-03-19T05:00:00Z Content type: release Language: en Sources: [Kubeflow]() Topics: [SDKs](), [MLOps](), [Apache Spark](), [data-processing](), [Hyperparameter optimization](), [Machine learning](), [Kubernetes](), [Python]() Tags: [apache-spark](), [data-processing](), [hyperparameter-optimization](), [kubernetes](), [mlops](), [python](), [release](), [sdk]() ### AI overview Kubeflow SDK v0.4.0 introduces a Model Registry Client, SparkClient with SparkConnect support, namespaced TrainingRuntimes, dataset and model initializers, and new documentation. The release targets a unified Python interface for AI workloads on Kubernetes across data processing, model management, and ML pipelines. ### Source excerpt Explore the full documentation at sdk.kubeflow.org ## Data-to-Production: Bridging the Gap Between Iceberg and Live Microservices DevFeed: [Data-to-Production: Bridging the Gap Between Iceberg and Live Microservices]() Original publisher: [Read original article]() Author: Wix Engineering Published: 2026-02-17T11:04:08Z Content type: article Language: en Sources: [Wix Engineering]() Topics: [Apache Iceberg](), [data observability](), [Microservice](), [clickhouse](), [Amazon S3](), [Architecture & Design](), [Apache Spark](), [airflow](), [Back end](), [API]() Tags: [airflow](), [amazon-s3](), [apache-iceberg](), [api](), [architecture](), [clickhouse](), [data](), [data-engineering](), [json](), [microservices](), [spark]() ### AI overview Wix describes Data-to-Production, a platform that activates data from Amazon S3 and Apache Iceberg for backend microservices. The system ingests Iceberg data into ClickHouse and serves it through a type-safe JSON API, using metadata governance and an Airflow and Python ingestion engine. ### Source excerpt At Wix, our Data Warehouse (DWH) is a massive repository of insights. Built on Amazon S3 using Apache Iceberg table formats, and populated by Trino and Spark jobs, it houses petabytes of data--from user segmentation and logs to AI chat analytics. However, storage is only half the battle. The real challenge--and the "holy grail" for many data engineering teams--is Activation : taking that petabyte-scale data and exposing it to backend microservices with millisecond latency, high availability, and... ## The 2026 AI Data Engineer Roadmap DevFeed: [The 2026 AI Data Engineer Roadmap]() Original publisher: [Read original article]() Author: Zach Wilson Published: 2026-02-05T20:26:37Z Content type: article Language: en Sources: [DataExpert.io Newsletter]() Topics: [Artificial Intelligence](), [data-engineering](), [AI-assisted coding](), [Apache Spark](), [airflow](), [flink]() Tags: [ai](), [ai-coding-agents](), [best-practices](), [data](), [data-engineering]() ### AI overview This article presents a 2026 roadmap for data engineering as AI automates more pipeline, SQL, Spark, testing, migration, and operational work. It compares responsibilities that are becoming more automated with areas such as system design, deep technical debt, performance tradeoffs, and organizational constraints where human expertise remains important. ### Source excerpt And how to avoid getting replaced ## Booking.com 2026 GenAI and ML PhD Research Internship in Amsterdam DevFeed: [Booking.com 2026 GenAI and ML PhD Research Internship in Amsterdam]() Original publisher: [Read original article]() Author: Yang Yang Published: 2026-02-05T10:39:26Z Content type: article Language: en Sources: [Booking.com Data Science]() Topics: [Machine Learning & Artificial Intelligence](), [genai](), [AI Research](), [Python](), [Large Language Model](), [Apache Spark](), [SQL](), [Hadoop](), [Reinforcement learning](), [Synthetic Data Generation](), [Transformer]() Tags: [2026](), [ai](), [big-data](), [blog-posts](), [featured](), [genai](), [internship](), [machine-learning](), [ml](), [python](), [research]() ### AI overview Booking.com is recruiting current PhD students in quantitative fields for a three-month GenAI and machine learning research internship in Amsterdam in 2026. Projects include LLM alignment, transformer explainability, embeddings, context engineering, and synthetic data generation. ### Source excerpt At Booking.com, we don't just use Machine Learning -- we use it to solve some of the most complex travel challenges in the world. We're looking for the next generation of researchers to join our Machine Learning community in Amsterdam for a 3-month deep dive into cutting-edge AI. The Program As a Research Intern, you'll be embedded in our teams, working alongside world-class mentors. Your mission? To tackle real-world problems and push the boundaries of the state-of-the-art. Are You the One? We're looking for current PhD students in quantitative fields (CS, Math, AI, Physics) who can conduct independent research and have a solid grip on Python and Big Data tech (SQL, Spark, Hadoop). What's in it for you? You won't just be "an intern". You'll be a contributor to our Machine Learning community. You'll have the opportunity to contribute to the existing efforts of the Machine Learning teams, participate in internal knowledge-sharing sessions, and enjoy the collaborative, high-energy environment of our Amsterdam HQ. Projects Regularized Target Encoding for large real-world datasets Multi-Agent Collaboration Aligning LLMs with user feedback via reinforcement learning Multi-level treatments Interpretable Foundations: Explainability Methods for Transformer Models on Sequential Event Data Scalable and generalisable ID embedding learning Improving property embeddings with better handling of rich and long-context data Utility-aware retrieval for context engineering in travel planning Synthetic Data Generation in Images Requirements We are looking for independent researchers with strong understanding of Machine Learning topics (see requirements for each project in the Linkedin ad), have a track record of peer-reviewed publications and a passion for solving complex problems. Why Booking.com? You'll join a vibrant, diverse community of data scientists and researchers who love to experiment. Beyond the code, you'll experience the unique culture of our Amsterdam headquarters -- a hub ## The 2025 AI + Data Engineering Roadmap DevFeed: [The 2025 AI + Data Engineering Roadmap]() Original publisher: [Read original article]() Author: Zach Wilson Published: 2025-10-17T22:35:45Z Content type: tutorial Language: en Sources: [DataExpert.io Newsletter]() Topics: [data-engineering](), [Python](), [SQL](), [Artificial Intelligence](), [airflow](), [BigQuery](), [Orchestration](), [Apache Spark](), [data-modeling](), [Data Quality](), [Embeddings](), [Retrieval Augmented Generation (RAG)]() Tags: [2025](), [ai](), [airflow](), [bigquery](), [count](), [course](), [data-engineering](), [data-modeling](), [data-quality](), [embeddings](), [engineering](), [framer](), [newsletter](), [orchestration](), [python](), [rag](), [real-time](), [right-join](), [spark](), [sql]() ### AI overview A 2025 roadmap for entering data engineering, covering foundational SQL and Python skills, distributed computing, orchestration, data modeling, data quality, AI and data integrations, portfolio projects, and personal branding. ### Source excerpt Getting a data engineering job is complicated. ## Announcing the ClickHouse Connector for AWS Glue DevFeed: [Announcing the ClickHouse Connector for AWS Glue]() Original publisher: [Read original article]() Author: Luke Gannon Published: 2025-08-21T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [AWS Glue](), [clickhouse](), [Apache Spark](), [aws-marketplace](), [Serverless]() Tags: [apache](), [aws](), [aws-glue](), [aws-marketplace](), [blog](), [clickhouse](), [launch](), [python](), [scala](), [spark]() ### AI overview This article announces the official ClickHouse Connector for AWS Glue, a serverless Apache Spark-based ETL integration available through AWS Marketplace. It explains how the connector supports PySpark and Scala, simplifies setup, and enables production-ready Spark jobs that connect AWS Glue with ClickHouse. ### Source excerpt Today, we're announcing the launch of the official ClickHouse Connector for AWS Glue, which utilizes their Apache Spark-based serverless ETL engine. ## Streaming optimized data to S3 for analytics with Parquet DevFeed: [Streaming optimized data to S3 for analytics with Parquet]() Original publisher: [Read original article]() Author: Chandler Mayo Published: 2025-08-13T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [parquet](), [Redpanda-Connect](), [Amazon S3](), [data analytics](), [Streaming](), [Apache Spark](), [pandas](), [Serverless]() Tags: [analytics](), [analytics-systems-data-feed](), [apache-spark-and-athena-data-queries](), [athena](), [automating-data-pipelines-in-s3](), [building-dashboards-with-s3-data](), [compressing-data-with-parquet](), [data](), [data-pipeline](), [data-pipeline-for-analytics](), [event-driven-pipelines-with-s3](), [how-to](), [learn](), [pandas](), [parquet](), [parquet-files-in-amazon-s3](), [pipeline](), [real-time-data-streaming](), [redpanda-connect](), [redpanda-data-streaming](), [s3](), [serverless](), [setting-up-redpanda-observability](), [spark](), [streaming](), [streaming-data-to-s3-with-parquet](), [tutorial]() ### AI overview This tutorial explains how to use Redpanda Connect to continuously batch streaming data and write compressed Apache Parquet files to Amazon S3 for analytical workloads. It covers the benefits of Parquet and querying the resulting files with tools such as Pandas, Apache Spark, and Athena. ### Source excerpt Learn how to build a powerful data pipeline that feeds analytics systems from Redpanda using clean, compressed Parquet files in Amazon S3. ## From siloed DataOps, MLOps, and LLMOps to a unified data-intelligence platform DevFeed: [From siloed DataOps, MLOps, and LLMOps to a unified data-intelligence platform]() Original publisher: [Read original article]() Author: Rajit Saha Published: 2025-08-04T18:03:19Z Content type: opinion Language: en Sources: [Udemy Engineering]() Topics: [DataOps](), [MLOps](), [Generative AI](), [Apache Spark](), [Amazon Bedrock](), [Amazon S3](), [Amazon Redshift](), [Amazon SageMaker](), [apache-flink]() Tags: [ai](), [aiops](), [amazon-s3](), [analytics](), [apache-flink](), [apache-spark](), [bedrock](), [dataops](), [large-language-models](), [llmops](), [mlops]() ### AI overview The article describes how DataOps, MLOps, and AI/LLM Ops commonly rely on separate systems and teams for data processing, model deployment, inference, evaluation, orchestration, governance, and monitoring. It then introduces Databricks' Data Intelligence Platform as a unified environment intended to bring these domains together. ### Source excerpt Introduction In modern data-driven businesses, the pace of innovation in analytics and artificial intelligence has outstripped the capacity of many teams. Three distinct disciplines emerged to handle this expansion: Data platform (DataOps) teams built data lakes on cloud storage such as Amazon S3, processed them with Apache Spark and Hive on EMR, ingested streaming data with Spark Structured Streaming or Apache Flink, and loaded tabular copies into MPP warehouses like Redshift for interactive SQL and BI. Cataloguing and governance were offloaded to external tools such as DataHub, and fine-grained access controls required third-party services like Privacera. This architecture worked, but it required separate workflows for batch and streaming, extra systems for lineage and governance, and a mosaic of operational teams. MLOps teams provided an additional layer. Data scientists used notebook environments (for example, Amazon SageMaker) to preprocess data, train, and evaluate models. Deploying models meant writing integration code to move features into a serving layer, to register models in disparate registries and to build custom APIs for inference. Feature stores and model registries were bought from additional vendors. Updates and monitoring were often manual processes. AI/LLM Ops teams are a new addition because generative AI requires specialized components: LLM gateways (e.g., Amazon Bedrock) to proxy access to foundation models; evaluation tooling to compare large language models; orchestration frameworks for agents; vector databases for retrieval augmented generation; and of course another layer of security, access management and cost control. These tools seldom integrate seamlessly with existing data and ML pipelines. This fragmented state makes it difficult to react quickly when product requirements change. Each new capability requires another system, another integration, and another team. Meanwhile, budgets tighten and go-to-market timelines shrink. The questio [Next page]()